This blog post is based on a summary by Isla Kuhn of the study day run by HCLU earlier this year. The day was lead by Jo Hooper, with input from Kieran Lamb and Natalie King.
What is it?
The definition is ever evolving! The 2010 Prague International Conference on Grey Literature criteria was that this needs to be “of sufficient quality to be collected by libraries” but not created by commercial bodies/publishers.
Grey literature includes: reports, newsletters, conference abstracts, government papers, board papers, power point presentations, data sets, patient information leaflets, social media posts, film/video, audio, newspapers, email discussion lists, discussion forums and so on.
Grey literature is not: books, chapters or journal articles. It is not peer reviewed. It is not “Controlled” either by a commercial publisher, or in terms of the language it uses.
Why use it?
There are 2 striking examples of why including grey / “unpublished” data in a systematic review can significantly impact the conclusions of the work.
One is the Tamiflu expose (read here and here), and the other involves an antidepressant Reboxetine. Both demonstrate that simply using traditionally published data alone will only give you a partial view. They are also illustrations of some of the publication bias that is part of the current system.
Challenges to using Grey Literature
- It’s not in one place
- It’s not easily exportable to a reference management tool
- The language used varies
- It’s not necessarily going to be available tomorrow, even if you found it today (websites crash or are abandoned)
- Your web search may not be reproducible owing to algorithmic variations.
- Deciding which resources to search for in the first place can be a challenge
- Your search terms are likely to be simplified
- Screening criteria and processes might need to be adjusted from those used for journal articles
Where is it?
Grey literature can be found in databases, but not always.
Here are some places you might want to look, depending on what you are looking for you might need to select the best suited:
Databases of grey literature
OpenGrey Grey Literature in Europe.
GreyNet International Includes the various resources including Grey Source, The Grey Journal and many more.
Clinical Trials Registers
ICTRP – World Health Organisation register, which searches across multiple national trials registers.
ClinicalTrials.gov is a database of privately and publicly funded clinical studies conducted around the world, spanning 209 countries.
EU Clinical Trials Register contains information on interventional clinical trials on medicines conducted in the European Union (EU), or the European Economic Area (EEA) which started after 1 May 2004.
Also consider searching by pharmaceutical company – they often have their own trials registers (eg https://www.astrazeneca.com/our-science/pipeline.html and https://uk.gsk.com/en-gb/research/trials-in-people/clinical-study-register/ )
Institutional Repositories
CORE hosts the world’s largest collection of open access full texts, which are used and referenced by people globally.
OpenDOAR is a global directory of Open Access repositories and their policies.
OAIster includes more than 50 million records that represent digital resources from more than 2,000 contributors.
BASE is one of the world’s most voluminous search engines especially for academic web resources. BASE provides more than 100 million documents from more than 5,000 sources.
Prospero includes protocol details for systematic reviews relevant to health and social care, welfare, public health, education, crime, justice, and international development, where there is a health related outcome.
Guidelines
Dissertations
There are specific databases of theses and dissertations:
PQDT (ProQuest Dissertations & Theses Abstracting & Indexing Service) – requires a subscription
PQDT Open (ProQuest Dissertations and Theses Open)
Open Access Theses and Dissertations (OATD)
Conference abstracts
Web of Science Core Collection – conference proceedings citation index science, and conference proceedings citation index social science and humanities (or search by “conference”) – requires a subscription
Zetoc The British Library’s Electronic Table of Contents, Zetoc contains 16,000 conference proceedings as well as journal articles and other material.
Tweets
https://twitter.com/search-advanced?lang=en-gb
UK Non-Governmental Institutions
The King’s Fund’s Information and Knowledge Services provide a unique and free source of information on health and social care policy and management
Nuffield Trust is an independent think tank which provides evidence-based research and policy analysis.
The Health Foundation Publications are available to download. Hard copies of many publications can also be ordered free of charge.
Charity Choice is a charity directory and fundraising website with a list of all UK registered charities.
Use Google to help you search better
Use the advanced search tools in Google to:
- bring up only specific file formats such as PDF or ppt or xls
- search in specific domains – to avoid having to use the search functionality of individual websites: use Google advanced search, or include “site:domain.net” in your google search (changing domain.net as appropriate)
How do I assess quality?
The AACODS checklist is designed to enable evaluation and critical appraisal of grey literature. It prompts you to thoroughly question authority, accuracy, coverage, objectivity, date and significance. Find it here
How do I report my method?
The search process needs to be documented in enough detail throughout the process to ensure that it can be reported correctly in the review, to the extent that all the searches of all the databases are reproducible. (Current Cochrane Handbook, section 6.6.1)
If you’re doing a web search of any search engines you can try out more than one and consider including the following in your work: search engine name, dates searched, search terms, how results were selected (eg title only, or click through to each result), screen between 10 -15 pages on Google (100 – 150 results per search), # results retrieved and # actually included, website names, url, date searched, any specific sections searched, search terms you used.
Stansfield et al (2016) discusses best practice when using online sources in systematic reviews HERE
Other Useful information:
Searching for Grey Literature University of Exeter (Bethel A, Briscoe S, Rogers M.)
What is ORCID?