Accessing the grey literature
As knowledge transfer and exchange gains strength and visibility, there is growing awareness among primary health care (PHC) researchers and policy makers that the most representative and therefore strongest evidential base must include grey literature. This Guide provides explanations around, and avenues for, accessing the grey literature with a focus on resources relevant to PHC.
This topic guide was designed for researchers, policy makers and anyone interested in gaining access to 'hard to find' literature or non-peer-reviewed documents relevant to PHC. Due to the competitive nature of peer-reviewed publications, among other factors, only a limited amount of information about PHC will appear in academic journals. By accessing only the 'black' literature, researchers and policy makers might miss cutting edge programs presented at conferences, or key policy documents available on government websites. PHC is a broad field and incorporates a wide range of information sources including clinical skills guides, government reports and local initiatives - many of these cannot be accessed using traditional literature searching tools.
This Guide provides a greater understanding of how 'grey literature' is defined, the benefits of searching the grey literature, and links to a number of useful sites providing access to grey literature relevant to PHC.
What is Grey Literature
'Grey literature' refers to written material or information that is unpublished or not published commercially. Where traditionally researchers and policy makers use databases to explore peer-reviewed publications or 'black literature' to inform their practice, in recent times the value of 'grey literature' has been formally acknowledged by prestigious evidence-based research organisations, the Cochrane Collaboration, the Campbell Collaboration and the Joanna Briggs Institute, all of whom stipulate that, in addition to black literature, searches for grey and unpublished literature must be conducted by their systematic reviewers in order to avoid publication bias.
Of the many available definitions, the 'Luxembourg definition', developed and approved during the Fourth International Conference on Grey Literature in 1999, defines grey literature as "that which is produced on all levels of government, academics, business and industry, in print and electronic formats, but which is not controlled by commercial publishers".
In discussing these "fugitive, ephemeral [documents]" (Turner, Liddy, & Bradley, 2002), the National Library of Australia (2006) add that grey literature can be considered "information that is not searchable or accessible through conventional search engines or subject directories". This situation is improving with open access1 publishing, institutional repositories, increasingly sophisticated search engines and the growing use of metadata, so that the problem is now less about visibility and more about the proliferation of sources.
Grey literature should not be thought of as strictly 'literature', but rather as grey 'resources' as it can encompass many different formats depending on the discipline (Bichteler, 1991; Tyndall, 2008). Examples of types of grey literature are provided in Table 1.
Table 1: Examples of grey literature
Why is grey literature important?
Grey literature plays an important role in the rapid and timely distribution of in-depth, recent, scientific and technical information. Grey literature provides access to a broad range of information and often contains new ideas (Simkhada, 2004). Searching the grey literature also offers the potential to balance any tendencies for publication bias (i.e., an increased likelihood of reporting positive or significant results), in the published literature. Research that is not published in journals but available in other formats (such as reports, theses or conference proceedings) is often much more detailed, is more recent and can be more rapidly disseminated. Due to the competitive and time consuming nature of publishing in peer-reviewed academic journals, some research may never make it into journals and would therefore be inaccessible to interested parties without the availability of grey literature.
Identifying the grey literature
Despite the pros of grey literature, it can be challenging to find. A search for grey literature thus requires the searcher to be creative (Tyndall, 2012). It is common now to see 'grey literature searches' included in research methods sections. The difficulty is that when conducting a web search, many search engines are unable to 'access all areas' yet accessing the grey literature means identifying the part of the information iceberg that is below the surface. For example, search engines may be able to find a site on 'Primary Health Care' but may not be able to sufficiently search within that site. To resolve this, there are increasing numbers of sites being developed to promote direct access to the grey literature without relying on these surface web crawlers (Tyndall, 2012).
Grey literature can be obtained from various sources such as specific organisations, professional bodies, government, universities, libraries, discussion groups, communities of practice and individuals (Tyndall, 2012). Figure 1 illustrates one approach to accessing grey literature.
Evaluation and critical appraisal of grey literature
Web developments have meant that grey literature is much easier to access now than it was previously. Grey literature is now readily collected in digital formats and catalogued by libraries (Lawrence, 2012; Tyndall, 2008). However, this leads to an additional challenge of determining what is useful and relevant to the researcher seeking the information.
Ideally, unpublished studies (grey literature) should be appraised to the same standard, and using the same tools as those used to evaluate published literature (black literature). The AACODS (Authority, Accuracy, Coverage, Objectivity, Date, Significance) checklist provides five criteria for judging the quality of information sources. Tyndall (2008) provides a summary of the tool which can be used when evaluating a grey literature source. Details can be found at http://dspace.flinders.edu.au/jspui/handle/2328/3326. Swinburne University and Victoria University in partnership with the National Library of Australia are also conducting an ARC linkage research project to develop best practice guidelines for producing and managing grey literature in Australia. For further information visit http://greylitstrategies.info/about
Notes: Open Access
1. As discussed in the PHCRIS Getting Started Guide:How to Access Journal Articles, “if an article is 'Open Access' it means that it can be freely accessed by anyone in the world using an internet connection” (SHERPA, 2006) and many journals are choosing to make all or a selection of their articles freely available in this way. In addition, recently core research funding bodies such as the Australian Research Council and National Health and Medical Research Council have released open access policies with a condition of their grant funding that published outcomes of all research grants be made publicly available in an open access database/repository within 12 months of publication. While some of the documents made available in response to these policies may be peer-reviewed journal articles (i.e., black literature), the stipulated release of additional publication outputs such as book chapters and reports will provide another source of grey literature.
Resources: Grey Literature Databases
There are a range of established databases which refer specifically to grey literature or enable searches of the grey literature, beyond the capabilities of a traditional web search.
DATABASES FOR PRIMARY HEALTH CARE GREY LITERATURE
While numerous general databases provide access to grey literature (see 'other databases' section), there are also increasing numbers of topic-specific sites. Listed below are a range of sources with particular relevance to primary health care.
Designed to include information from all primary health care researchers and users in Australia, Research Profiles provide a platform for anyone involved in research to locate and/or publish information from projects, to presentations, posters, reports and articles.
This American-based database focuses on older adults and ageing with information for researchers, professionals and consumers. Indexes journals and grey literature sources such as book chapters and reports.
Australian Drug Information Network
Australia's leading alcohol and drug search directory (ADIN) provides links to services, research, guidelines, policies and many other resources relevant to alcohol, drugs and mental health. The site allows organisations to share information about their programs, projects and resources. A formal quality review is conducted for every website included in the directory.
Australian Indigenous HealthInfoNet
Useful website for anyone with an interest in Indigenous health. This is a fully searchable database providing access to government documents, theses, conference papers, reports etc. It aims to make research and other knowledge readily accessible to inform policy and practice around all aspects of Indigenous health.
CareSearch is an online resource of palliative care information and evidence. A range of search filters are available for palliative care, heart failure, lung cancer, residential aged care and dementia. In addition their comprehensive Grey Literature database provides access to conference abstracts, non-indexed journals, theses, government department and other organisation documents, national programs and research studies which address palliative care information and evidence.
Focusing on public health information, WONDER (Wide-ranging On-line Data for Epidemiologic Research) makes the information resources of the Centers for Disease Control and Prevention available to public health professionals and the public.
Health Systems Evidence
Health Systems Evidence is a repository of syntheses of evidence about implementing change, governance, financial and delivery arrangements within health systems. Syntheses are presented in a range of forms including evidence briefs for policy, systematic reviews and their protocols. The site also provides access to a repository of economic evaluations, descriptions of health systems and reforms. There is a subsite specific to PHC, 'PHC Evidence'.
Health Services Research Projects in Progress
The US National Library of Medicine supports the HSRProj page, a searchable database which provides information about ongoing health services research and projects, before they are published.
JBI COnNECT+, the Joanna Briggs Institute's 'Clinical Online Network of Evidence for Care and Therapeutics' database provides easy access to evidence-based resources to inform clinical decision-making.
This UK National Institute for Health and Clinical Excellence site provides reliable information on health and social care in one place, enabling access to clinical, non-clinical and best practice evidence.
New York Academy of Medicine Grey Literature Report
This NYAM site provides both a search facility and links to the bimonthly publication 'The Grey Literature Report', which alerts readers to new grey literature publications in public health and health services research.
PsycEXTRA is the American Psychological Association's grey literature database which accompanies the scholarly PsycINFO database. It provides bibliographic records and full-text professional literature including technical reports, conference presentations and popular magazines.
Social Care Online
Social Care Online is a UK database providing information and research on all aspects of social care and social work. Regularly updated resources include legislation, government reports, practice and guidance, systematic reviews, research briefings, journal articles and websites.
USEFUL (GENERAL) DATABASES
The databases listed below provide access to grey literature on a wide range of topics.
Australian Bureau of Statistics
The ABS, Australia's official national statistical agency, provides a high quality, objective and responsive statistical service with information and resources on a range of topics, including health.
Australian Institute of Health and Welfare
The AIHW is the national agency providing reliable, regular and relevant information and statistics on Australia's health and welfare.
APO specialises in grey literature and provides access to a range of research reports, working papers, discussion papers and statistics for effective public policy debate, decision making and implementation. The site monitors a range of sources including academic research centres, government departments, think tanks and non-government organisations. There is also a page dedicated to health.
Explore The British Library
The British Library allows resources to be shared using inter-library loan. The search options also allow specific types of grey literature to be selected e.g. conference literature, sounds.
Flinders Academic Commons
Administered by the Flinders University Library, the Academic Commons is an institutional repository which showcases the research outputs of university staff.
Global Health Observatory
The WHO's GHO is a gateway to comprehensive health-related data and statistics from all around the world.
Provides a selection of web-based grey literature resources covering a range of topics.
.idcommunity Demographic Resource Centre
This .id site develops online information products specifically for local government decision makers. It covers a range of information types including community profiles, population forecasts, economic profiles and social atlases.
Produced by the Australian government this site provides access to searchable patent, trademark and design databases.
This multidisciplinary database has material from a range of European sites with documents such as research reports, conference papers, dissertations and other types of grey literature covering science, biomedical science, social science and humanities.
Preserving and Accessing Networked Documentary Resources of Australia (PANDORA), was originally established by the National Library of Australia and is a growing collection of Australian online publications and websites. The content documents the cultural, social, political, intellectual and expressive activities of Australians.
Trove (National Library of Australia)
Trove provides a free search service/repository of Australian material including digital theses, digitised magazines and newspapers.
WorldCat is a network of library services built upon databases of bibliographic and institutional metadata, covering over 10,000 libraries worldwide. One key grey literature-related ability is the listing of materials that are 'hard to find'.
This Global Science Gateway includes a range of international scientific databases and portals. Real-time search results provide not only coverage of global science but multilingual scientific literature.
Clearinghouses receive, organise and disseminate valuable information in specific fields, aiming to make that information available to people working in the area. A type of knowledge broker, a clearinghouse is a central access point that serves the needs of users of a specific body of knowledge (Tyndall, 2012). One role of clearinghouses is to identify, describe and evaluate information relevant to the knowledge area, preventing duplication. They are “specialist resource and information collections” (Lawrence, 2012).
Agency for Clinical Innovation: Innovation Exchange
ACI is an information and networking service for Australian health professionals which aims to support and increase the implementation of effective innovations in clinical care across healthcare settings.
Closing the Gap Clearinghouse
The AIHW and Australian Institute of Family Studies clearinghouse provides evidence-based research on overcoming disadvantage for Indigenous Australians.
National Guideline Clearinghouse
Developed by the US Department of Health and Human Services, this clearinghouse offers a comprehensive, searchable collection of evidence-based clinical practice guidelines.
Some databases specifically index conference proceedings, a popular form of grey literature. It is also often possible to access conference papers not only via the conference's website or the convening organisation's website but through more traditional literature databases which include a conference search option.
When conducting a search through Scopus it is possible to select 'conference' as the document type.
This database, with an emphasis on sociology and related disciplines in the social and behavioural sciences, provides abstracts from a number of grey literature sources including book chapters, dissertations, working papers and conference papers. Using the advanced search it is possible to specify the source type as 'conference papers and proceedings'.
Web of Science
This Web of Knowledge database tool will search conference proceedings if the searcher selects 'conference proceedings citation index' in the current limits option.
THESES AND DISSERTATIONS
There are a number of sites that promote access to electronic theses and dissertations, another useful grey literature source. Australian theses may be found through searching Trove (National Library of Australia, see link above). Others may be found in more traditional black literature databases such as CINAHL, ERIC, PsycInfo or Sociological Abstracts.
The British Library's Electronic Thesis Online Service provides open access to UK theses in a range of disciplines. Where immediate download is not possible EThOS can arrange delivery of the full text.
The Networked Digital Library of Theses and Dissertations supports electronic publishing and open access to international theses and dissertations to enhance sharing of knowledge.
ProQuest Dissertations & Theses
Dissertation and Theses Dissemination and Ordering provides the full text of open access dissertations and theses. While an international source, many of the available materials are from the US.
Theses Canada aims to acquire and preserve a collection of Canadian Theses to provide access to this research around the world.
SHARING DATA, PROJECTS AND MEASUREMENT TOOLS
A new trend in the field of grey literature is making data available for re-use or providing easy access to different measurement instruments for researchers.
Australian National Data Service
This site allows Australian researchers to reuse research data. It provides national services such as Research Data Australia and is building the Australian Research Data Commons; promoting partnerships and resources to enable researchers to share data collections.
BMC Research Notes
One aspect of BMC Research Notes' scope refers to the publication of software tools, databases and datasets in reusable formats.
Health and Psychosocial Instruments
Ovid's Health and Psychosocial Instruments database provides access to information on measurement tools relevant to disciplines including the health fields, psychosocial sciences, organisational behaviour, library and information science.
Health Services & Sciences Research Resources
The US National Library of Medicine is responsible for the HSRR site designed for health care researchers and health science librarians seeking data resources (including information about research datasets, instruments and software) used in research related to health services, behavioural and social sciences, and public health.
The SA-NT DataLink provides a data linkage service which supports health, social and economic research, policy development, education, service planning and evaluation in SA and the NT.
Further Reading and Resources
- McKimmie T, Szurmak J. (2002). Beyond grey literature: How grey questions can drive research. Journal of Agriculture and Food Information, 4, 71-79. http://www.tandfonline.com/doi/abs/10.1300/J108v04n02_06
- Olsan TH, Bianchi C, White P, Glessner T, Mapstone PL. (2011). Finding electronic information for health policy advocacy: A guide to improving search results. Journal of the American Academy of Nurse Practitioners, 23, 648-658. http://onlinelibrary.wiley.com/doi/10.1111/j.1745-7599.2011.00670.x/abstract
- Sweet M, Hagger C. (2012). Some tips re digging for useful health policy information on the web. Croakey blog. 27 April 2012. http://blogs.crikey.com.au/croakey/2012/04/27/some-tips-re-digging-for-useful-health-policy-information-on-the-web/
- Swinburne Institute for Social Research. (2012). Grey literature strategies. http://greylitstrategies.info/
- Bichteler J. (1991). Geologists and gray literature: Access, use and problems. Science and Technology Libraries, 11, 39-50.
- Duffield A, Reid G, Walker D, Shoham J. (2004). Review of the published literature for the impact and cost-effectiveness of six nutrition related emergency interventions. Retrieved 30 August 2012 from http://www.ennonline.net/reviewsixnutritioninterventions
- Lawrence A. (2012). Electronic documents in a print world: Grey literature and the internet. Media International Australia, 143, 122-131.
- National Library of Australia. (2006). Preserving access to digital information: Grey literature. Retrieved 27 November 2012 from http://pandora.nla.gov.au/pan/10691/20110824-1153/www.nla.gov.au/padi/topics/372.html
- SHERPA. (2006). Authors and open access: What is Open Access? Retrieved 1 February 2013 from http://www.sherpa.ac.uk/guidance/authors.html
- Simkhada P, Knight J, Reid G, Wyness L, Mandava L. (2004). Chasing the grey evidence: A standardised systematic critical literature review approach. Presented at Sixth International Conference on Grey Literature, 6-7 December 2004, New York. http://www.opengrey.eu/data/69/78/51/GL6_Simkhada_et_al_2005_ConferencePreprint.pdf
- Turner AM, Liddy ED, Bradley J. (2002). Breathing life into digital archives: Use of natural language processing to revitalize the grey literature of public health, Robert Wood Johnson Foundation. Retrieved 27 November 2012 from http://dl.acm.org/citation.cfm?doid=544220.544359.
- Tyndall J. (2008). How low can you go? Towards a hierarchy of grey literature. Presented at Dreaming08: Australian Library and Information Association Biennial Conference, 2-5 September 2008, Alice Springs. http://hdl.handle.net/2328/3326
- Tyndall J. (2012, May). Accessing the “Grey” literature. Research Higher Degree Professional Development Program Workshop, Flinders University, South Australia.
Related resources on the PHCRIS site
Was this guide useful? We welcome your comments and suggestions, please use the feedback form and let us know what you think.
Primary Health Care Research & Information Service (2016). Getting Started Guide: Grey Literature. From phcris.org.au/guides/grey_literature.php (Accessed 10 Dec 2016)