assessing the international use
DESCRIPTION
TRANSCRIPT
25/06/2012
Assessing the International Use of Health Technology Assessments
Dr David Wright, Prof Ruairidh Milne, Alison Price, Nicola Tose, Dr Nick HicksNIHR Evaluation, Trials and Studies Coordinating Centre (NETSCC)www.netscc.ac.uk
Why measure HTA use?
• HTA commissioning organisations need to demonstrate uptake of findings to:• justify existing levels of spend• meet requirements for accountability• inform and improve funding and commissioning
processes• Most studies explore use and impact within the
country in which the research was conducted• We know little about the use of research
internationally
Study Question and Design
Study aim:•To explore the merits of different methods for assessing the international use of UK funded research by the NIHR HTA programme
Study design•Phase I: Updated Literature Review •Phase II: Exploring Methods
• Bibliometrics• Webtrends• HTA CRD database search
Phase I: Literature review
• Hanney’s systematic review examined the question: ‘What models are available to assess the impact of health research programmes, and what are their strengths and weaknesses?
• The current study used the same search terms, inclusion / exclusion criteria for the period 2005 – September 2010
1 Hanney S, Buxton M, Green C, et al. An assessment of the impact of the NHS Health Technology Assessment Programme. Health Technol Assess. 2007; 11(53): 1 – 200.
Phase I: Literature review
• Most models move beyond assessing research outputs (i.e. publications) to assessing research outcomes (i.e. changes in behaviour or practice)
• The Payback framework is the most widely used approach
• The dominance of the Payback framework supports findings in the Hanney et al. review (2007)
1 Hanney S, Buxton M, Green C, et al. An assessment of the impact of the NHS Health Technology Assessment Programme. Health Technol Assess. 2007; 11(53): 1 – 200.
The PayBack model: (Hanney et al. 2007)
Knowledge
(e.g. published
output)
Benefits to future
research & research
use
(e.g. research
capacity)
Health sector
benefits (e.g. health
service delivery)
Political and
administrative
benefits (e.g.
policy
decisions)
Broader
economic benefits
(e.g. healthy
Workforce)
Hanney S, Buxton M, Green C, et al. An assessment of the impact of the NHS Health Technology Assessment Programme. Health Technol Assess. 2007; 11(53): 1 – 200.
Phase II: Exploring methods
Knowledge(Academic Use)
Methods:Bibliometric analysis of no. publications,
impact factor, citations and
international citations
Internet Use
Methods:Webtrend analysis of
no. UK and non-UK visits
HTA Use
Methods:Analysis of citations in international HTA
reports identified through the CRD
database.
Phase II: Bibliometric results
• Bibliometric analysis was undertaken on the top 10 most cited HTA reports
• The average number of published journal papers for the top 10 HTA reports was 2 publications
• The average journal impact factor for the publications was 5.22
• 41% of the 549 journals citing NIHR HTA reports had their editorial base in the United States. 36% were based in the UK.
Phase II: Webtrends results
International visits of top 5 downloaded HTA reports,Jan 1st 2004 – 30th Jun 2010
Author Title Research Type
No. downloads
UK visits (%)
Non-UK visits (%)
Avenell et al. (2004)
Systematic review of the long-term effects and economic consequences of treatments for obesity
Systematic Review / Primary Research
354,166 27.76 72.24
Murphy et al. (1998)
Qualitative research methods in health technology assessment: a review of the literature
Methodology 254,750 47.95 52.05
Wald et al. (2003)
First and second trimester antenatal screening for Down's syndrome
Primary Research
223,921 43.83 56.17
Chen et al. (2006)
A systematic review of the effectiveness of adalimumab, etanercept and infliximab for the treatment of rheumatoid arthritis in adults
Systematic Review
182,299 17.40 82.60
Dinnes et al. (2007)
A systematic review of rapid diagnostic tests for the detection of tuberculosis infection
Systematic Review
177,047 12.23 87.77
Proportion of website visits to Dinnes et al (2007) - Review of rapid diagnostic tests for tuberculosis by country, Jan 2004 – Jun 2010
Proportion of website visits to Avenell et al (2004)- Review of effects and consequences of obesity treatments by country, Jan 2004 – Jun 2010
Phase II: CRD HTA database
Phase II: CRD HTA database
• Five highly cited NIHR HTA reports were ‘tracked’ through to identify pick up by international HTA agencies
• Four HTA reports were collectively cited 28 times, 18 of which were by non-UK HTA agencies.
• Canada cited HTA Programme reports the most with 5 citations.
• Avenell et al. (2004) was the most internationally cited report with 10 non-UK HTA agencies citations.
To conclude…
• Methods used for research impact assessment are useful in generating data on international uptake of HTA findings.
• Webtrends revealed a high proportion of international visits, particularly for systematic reviews, although less so for methodology reports
• HTA report citations provided evidence of the transfer of HTA findings internationally.
Recommendations…
• A multi-dimensional model of HTA uptake is recommended
• Bibliometrics can identify international academic uptake, but more developed analysis of international citation is recommended
• Webtrends and HTA report citation can be used to provide indicators of international uptake
• Further research is recommended using a case-study approach to explore the nature of HTA use.
Any questions…?
Dr David Wright
Senior Research Fellow
t: +44 (0) 23 8059 7484
f: +44 (0) 23 8080 5639
NIHR Evaluation, Trials and Studies Coordinating Centre
(NETSCC) part of the National Institute of Health Research (NIHR)
Alpha House, University of Southampton Science Park,
Southampton SO16 7NS
w: www.netscc.ac.uk
Acknowledgement:This research was funded by the National Institute for Health Research (NIHR) and carried out by the Evaluation, Trials and Studies Coordinating Centre at the University of Southampton. The views expressedare those of the research team and not necessarily those of the NHS, the NIHR or the Department of Health.