Primary Health Care Research Impact Project : Phase 1
The first phase of the PHC RIP project was undertaken by PHC RIS during 2006. The purpose of this project was to develop a way to assess the impact of primary health care research and as part of this aim we trialled the use of the Buxton and Hanney Payback Framework (Hanney, Gonzalez-Block et al. 2003; Hanney, Grant et al. 2004) and their methods of data collection. This included interviews with the Chief Investigators of four nationally funded research projects and end users, and a bibliometric study of the project publications. We were also interested in pathways to impact and how they can be enhanced.
We considered the relevance of our findings to the Research Quality Framework (RQF) for Australian Universities which will commence in 2007. In the RQF, assessment of research impact, defined as social, economic, environmental and/or cultural outcomes, will be considered as well as research quality, meaning academic impact, including publication measures.
The preferred RQF model, released in October 2006, (Australian Government Department of Education Science and Training 2006) will assess both the quality and the impact of university research, with the unit of analysis being the research group.
The quality of the research is defined in terms of its academic impact. Ways of measuring academic impact or quality will be:
- Citation data, such the as number of journal articles in high impact journals and citations to those articles (bibliometric measures).
- Disciplines are challenged also to develop rankings based on discipline specific outputs, such as conferences, journals or exhibition venues, with researchers being encouraged to publish in the most prestigious outlets for their discipline.
- Grant income data.
The impact of the research “ relates to the recognition by qualified end users that methodologically sound and rigorous research has been successfully applied to achieve social, economic, environmental and/or cultural outcomes.” (Australian Government Department of Education Science and Training 2006)
The basis of the impact assessment will be an impact statement which includes:
- an “evidence based statement of claims for the Group against generic and panel specific impact criteria including verifiable indicators in support of those claims”;
- up to four case studies that illustrate the groups claims of impact;
- and details of end users who can be contacted to verify claims of impact. (Australian Government Department of Education Science and Training 2006).
The analysis of citation data is used to assess the relative impact of one set of peer reviewed journal articles compared to another set as measured by the Journal Impact Factor of the journal in which articles are published and the number of citations to those articles. Journal Impact Factor (JIF) is a statistic calculated on the number of citations to articles in a particular journal over a defined period and is calculated using the ISI Web of Science databases.
The ISI Web of Science databases are compiled in the US and their coverage is problematic for Australian primary health care. In an analysis by PHCRIS of the publications derived from General Practice Evaluation Program (GPEP) projects (Lowcay, McIntyre et al. 2004) we found that of the 13 most used journals only 6 are indexed. Only 32% of articles are on the ISI database and in a journal with a recorded Journal Impact Factor.
Any assessment of the Journal Impact Factor as a measure of research impact in this field would therefore be misleading. Consideration of Generic and Specific Outcome Data as an alternative measure of research quality (academic impact) is in its infancy but will be a necessary development in this field.
We found that impact is facilitated by strong collaborative links, personal relationships, the involvement of people with links to decision making processes, and the congruence of the findings with users' priorities. New behaviours will be keeping records of conference and other presentations, media stories, reports and policy documents, and anything which quotes your work. Note individual requests for information and whether anything has followed from it and maintain contact with those who have used your research as their testimony will be needed to provide evidence of research impact.
While the discipline has work to do in preparing for the RQF, there are grounds for optimism. The Primary Health Care Research Impact Project has shown that PHC research can have a great deal of success in producing social, economic, environmental and/or cultural outcomes and that our structures and processes for ensuring good research transfer are working.
Our report gives guidance for researchers in compiling case studies and gathering Generic and Specific Outcome data for the RQF. The second stage of our project is in the planning stages and will be conducted during 2007.
Key documents on the Research Quality Framework
Australian Government Department of Education Science and Training (DEST):
Research Quality Framework (RQF) Web page
Some other relevant documents from DEST
Resources on the Assessment of Research Impact
Focus on…. Understanding and measuring research impact
Payback publications from the Health Economics Research Group, Brunel University
Publications from the Research Unit for Research Utilization, University of St Andrews
Describing the impact of health research: a Research Impact Framework
- Shyama Kuruvilla, Nicholas Mays, Andrew Pleasant and Gill Walt.
BMC Health Services Research 2006, 6:134, doi:10.1186/1472-6963-6-134
http://www.biomedcentral.com/1472-6963/6/134
- The Research Impact Framework was developed by identifying potential areas of health research impact from the research impact assessment literature and based on research assessment criteria, for example, as set out by the UK Research Assessment Exercise panels. A prototype of the framework was used to guide an analysis of the impact of selected research projects at the London School of Hygiene and Tropical Medicine.
Resources on research transfer
Focus on…Knowledge Brokering
Canadian Health Services Research Foundation
- The premier site for innovative models of knowledge transfer, notably the Linkage and Exchange Model and Knowledge Brokering
http://www.chsrf.ca/home_e.php
Sax Institute
Health Evidence network, World Health Organisation
From Transfer to Transformation: Rethinking the Relationship between Research and Policy
View: Bibliography of research impact and knowledge transfer publications and resources
|
|
print version
email this page
Publications arising from this research
