division of library services | office of research services | national institutes of health telling...
TRANSCRIPT
DIVISION OF LIBRARY SERVICES | OFFICE OF RESEARCH SERVICES | NATIONAL INSTITUTES OF HEALTH
Telling the Research Story:
A Role for Librarians in Analyzing
Research Impact Based on Evidence
Ex Post Evaluation of the Ocular Hypertension Treatment Study: Finding Evidence of Meaningful Health Outcomes
Cathy C. Sarli, MLS, AHIPPresented by Kristi Holmes, PhDBernard Becker Medical LibraryWashington University in St. Louis School of Medicine
Meaningful Metrics to Identify Value
Terrie Wheeler, AMLSNational Institutes of Health (NIH) Library (DHHS/NIH/ORS/DLS)
Genesis
Citation Analysis: What is It?
Examination of an individual publication (or a group of publications) and counting how often it has been
cited, if ever, by subsequent publications.
Citation analysis is a traditional tool for measuring impact using publication data.
Inherent assumption is that significant publications will demonstrate a high citation count.
Citation Analysis for OHTS
A select group of six publications demonstrated high citation rates:
– Cited 919 times as of September 2007– Citation rates exceeded average citation rates and ranked high
in percentile rankings– Three of four Core Papers in ESI for the topic of “Glaucoma” as
of September 2007 were OHTS publications
Citation Analysis: So What?• Do numbers tell a story?
• Is citation analysis meaningful for non-academia audiences such as the public, policy-makers, clinical trial participants, healthcare providers, or grant funding agencies?
• Are citations indicative or predictive of translational or clinical applications?
• Do citations provide a full narrative of meaningful health outcomes?
How Do we measure what matters
“It is no longer enough to measure what we can – we
need to measure what matters.”
How do we measure what matters?
[Source: Wells R, Whitworth A. 2007. Assessing outcomes of health and medical research: do we measure what counts or count what we can measure? Australia and New Zealand Health Policy, 4:14. http://www.anzhealthpolicy.com/content/4/1/14]
Consensus Development Conferences Canadian Perspectives in Glaucoma Management: Setting Target
Intraocular Pressure Range Can J Ophthalmol. 2003 Apr;38(3):189-97. “Recent evidence supports the concept of aggressively lowering IOP to
help prevent glaucoma related blindness.” OHTS is referred to as one of the “gold standard” prospective randomized clinical trials
that support this concept.
Private Healthcare Benefit PlansMany private insurance companies allow for
pachymetry for measurement of corneal thickness, with variations in coverage.
Continuing Education Materials10th Annual Glaucoma Report: High Tech Glaucoma
Devices Hasten Diagnosis, 2004.“Perhaps the most important point: The Ocular
Hypertension Treatment Study (OHTS) identified corneal thickness not just as a variable that affects Goldmann IOP readings but as an independent risk factor for developing glaucoma. Specifically, OHTS uncovered a linear relationship between IOP and
corneal thickness in patients with "thicker" corneas.”
13th Annual Glaucoma Report: Calculate the Risk Factors For Glaucoma, 2007.
“. . . the Ocular Hypertension Treatment Study (OHTS) and other studies published during the last
decade have fundamentally changed how we diagnose and manage glaucoma, and have helped us
to identify such risks. In past years, we diagnosed open-angle glaucoma based on elevated intraocular
pressure (IOP), visible optic nerve damage and correlated visual field loss.
Cost-Effective InterventionManagement of ocular hypertension: a cost-
effectiveness approach from the Ocular Hypertension Treatment Study. Kymes S, et al. Am J Ophthal 2006Jun;141(6):997-1008.The Ocular Hypertension Treatment Study
(OHTS) demonstrated that medical treatment of people with intraocular
pressure (IOP) of > or =24 mm Hg reduces the risk of the development of primary open-angle glaucoma (POAG) by 60%.Although the treatment of individual patients is largely dependent on their
attitude toward the risk of disease progression and blindness, the treatment of those patients with IOP of > or =24 mm Hg
and a > or =2% annual risk of the development of glaucoma is likely to be cost-
effective.
Change in Delivery of Healthcare ServicesMurphy, John. National Panel: OHTS affects when optometrists treat
glaucoma. Review of Optometry, July 15, 2003;140(7).
“Three out of four panelists say that the Ocular Hypertension Treatment Study (OHTS) has affected the way they manage glaucoma
patients.”
“We now routinely perform ultrasound pachymetry on all glaucoma suspects and ocular hypertensives.”
“Three in 10 optometrists now perform baseline pachymetry on new glaucoma patients. Many say they have since ordered (or are about
to order) a pachymeter.
“I treat patients with a family history and thinner corneas earlier than I would have before.”
“Five years ago, only about one-third of optometrists typically treated glaucoma patients themselves. Now, two-thirds of
optometrists say they treat glaucoma patients largely on their own.”
Quality Measure GuidelinesThe National Committee for Quality Assurance (NCQA), Health Plan Employer Data and Information Set (HEDIS)
HEDIS Glaucoma Screening in Older Adults (GSO) Measure (first year optional Senior measure), 2005.HEDIS Glaucoma Screening in Older Adults (GSO) Measure (required Senior measure), 2006-2009.
American Medical Association Current Procedural Terminology (CPT) Codes
CPT Code Category III for pachymetry established in 2002. CPT Code Category I for pachymetry established in 2004.
“Corneal pachymetry (the measurement of corneal thickness) is an essential tool for glaucoma diagnosis and management because measurement of IOP is
affected by the thickness of the cornea.”
The determination of an accurate IOP improves the diagnosis, screening and management of patients with glaucoma and OHT.” CPT Assistant: Authoritative
Coding Information. The American Medical Association, July 2004.
Clinical/Practice GuidelinesAmerican Academy of Ophthalmology
Primary Open-Angle Glaucoma Suspect Preferred Practice Pattern™ Guideline, 2005. American Optometric Association
Care of the Patient with Open Angle Glaucoma: Reference Guide for Clinicians, 2002. International Council of Ophthalmology/International Fed. of Ophthalmological Societies
Primary Open-Angle Glaucoma (Initial Evaluation), 2007.Primary Open-Angle Glaucoma (Follow-up Evaluation), 2007.
Primary Open-Angle Glaucoma Suspect (Initial and Follow-up Evaluation), 2007.Royal College of Ophthalmologists
Guidelines for the Management of Open Angle Glaucoma and Ocular Hypertension, 2004.
New Research StudiesR01 Corneal Endothelial Cell DensityR01 Short Wavelength Automated Perimetry R01 Confocal Scanning Laser Ophthalmoscopy Pachymetry Ancillary StudyR01 Computer Analysis of Optic Disc Images in GlaucomaR01 Genetic Study of the OHTS Cohort to Identify Glaucoma GenesR03 EGPS/OHTS Collaborative AnalysisR03 Development of a Vision Specific Utilities Elicitation MethodR21 EGPS/OHTS Confocal Scanning Laser Ophthalmoscopy CollaborationR21 European Glaucoma Prevention Study (EGPS)/OHTS Collaborative AnalysisR21 Evaluation of the Frequency of Visual Field Testing in Ocular HypertensivesR21 Follow-up Intraocular Pressure and the Risk of Developing Primary Open-Angle GlaucomaR21 Peripapillary Atrophy Progression During the Ocular Hypertension Treatment StudyR21 Supplemental Grant for the Optic Disc Reading Center for the Ocular Hypertension Treatment Study (OHTS)U10 Quantitative Analysis of the Optic Disc/Ocular HypertensionPfizer COA Change in Mean Deviation Project: Value of IOP Model
ReviewsOhba N, Nakao K, Isashiki Y, Ohba A. The 100 most frequently cited articles in
ophthalmology journals. Arch Ophthalmol. 2007 Jul;125(7):952-60.Included as one of the top 100 most frequently cited articles is: Kass MA, Heuer DK, Higginbotham EJ;
et al. The Ocular Hypertension Treatment Study: a randomized trial determines that topical ocular hypotensive medication delays or prevents the onset of primary open angle glaucoma.
Measurement Instruments Devers Risk Calculator
Glaucoma Risk Calculator—The Scoring Tool for Assessing Risk Pfizer Hand-Held Glaucoma Five Year Estimator Risk Calculator
Glaucoma Five Year Estimator Risk Calculator
New Diagnostic Criteria: “Central Corneal Thickness”Thimons, J. James. Pachymetry: The New Standard of Care In Glaucoma.
Optometric Management. May 2006.
“Since the publication of the Ocular Hypertension Treatment Study (OHTS), central corneal thickness as a significant risk factor for glaucoma has been central
to the discussion of its management. Pachymetry is a remarkable tool and has rapidly become a mainstay of clinical practice in the area of glaucoma. The
question of its status as a standard-of care has, to a large degree, been answered by the clinical community. The American Academy of Ophthalmology deemed
CCT an integral part of the glaucoma evaluation. The American Optometric Association incorporated CCT into its recommendations for Preferred Practice Patterns. Medicare issued a unique reimbursement code more rapidly than for
any other technology in the history of ophthalmic science.
New Standard of CareHutcheson, Kelly A. Application of new ophthalmic technology in the pediatric
patient: Pediatrics and strabismus. Current Opinion in Ophthalmology, September 2007;18(5):384–391.
“In adult patients, measurement of the central corneal thickness (CCT) has become standard of care for patients with known or suspected glaucoma. In
the Ocular Hypertension Treatment Study [2], the CCT was found to be a strong predictor for the development of glaucoma, with patients having
corneas thinner than 555 µm having a three times greater risk than those with thicker corneas. . .it has become the standard of care to perform pachymetry
measurements on all children with or suspected of having glaucoma.”
Curriculum Guidelines International Council on Ophthalmology
Principles and Guidelines for the Curriculum for Education of the Ophthalmic Specialist Guidelines and Standards for Education of an Ophthalmologist: A Curricular Outline
Appendix: Glaucoma: Ocular Hypertension Treatment StudyRoyal College of Ophthalmologists
Guidelines for the Management of Open Angle Glaucoma and Ocular Hypertension
iPhone Application: “OHT Calc”The OHT Calc app is a clinical tool for ophthalmologists, optometrists
and patients to determine an individual’s risk of developing glaucoma. The app is based on the risk model developed by the Ocular Hypertension Treatment Study (OHTS) and the European
Glaucoma Prevention Study (EGPS), two large clinical trials.
Advanced Careers ofEarly Stage Investigators
OHTS data enabled six early stage investigators to be awarded eight NIH awards: R01, R21, and R03.
Increase in Usage of “Central Corneal Thickness”PubMed records with the term “central corneal thickness” noted in 1,350 records from 2003-2011
compared to 300 records from 1969 to 2002. ARVO abstracts with the term “corneal thickness” more than doubled from 54 in 2002 to more than 100 following the OHTS 2002 publications.
Reduction in Incidence of Disease Identification of risk factors by OHTS and use of
medication has reduced the incidence of glaucoma.
Quality of LifeEarly intervention for treatment of glaucoma has improved the
quality of life for individuals at high risk for glaucoma.
Point of Care Guideline: Johns Hopkins POC-IT Guides
“Incorporating the Results of the Ocular Hypertension Treatment Study into Clinical Practice.”
New Research DirectionOHTS led the way for research related to
biomechanical properties of the eye.
Measuring what matters
Finding the EvidencePEOPLE
InvestigatorsSupport staff for PIsConsultantsPolicy-makersProgram officersLibrariansColleaguesHealthcare providers
DATABASESPubMed SCOPUS Web of ScienceGoogleClinicalTrials.govRePORTER.govNGCNCBI
ORGANIZATIONSFederal agenciesProfessional organizationsStandard-issuing organizationsAcademic organizationsFunding agenciesAdministration
RESOURCESTrade publicationsGray literatureClinical guidelinesGovernment documentsPolicy statementsCurriculum materialsPatentsMTAsLicensesDissertationsAnecdotal evidence
General Issues
• Time lag between research discovery and translational applications.
• Optimal timeframe for starting assessment is unknown and process can vary by type of research and discipline.
• Supporting documentation may not be publicly available.
• It can be difficult to establish a direct correlation from a specific research output.
• Not a linear process nor is there a standard method for locating evidence of research impact.
Idea
Hybrid Framework
The Becker Model is a framework for tracking research outputs that have
been disseminated/diffused
to locate indicators that
demonstrate evidence of biomedical research impact.
Research Output
Dissemination
Diffusion
Indicators
Evidence of Research Impact
Project Website
https://becker.wustl.edu/impact-assessment
Quotes “I had anecdotal evidence of the OHTS study impact, but I
wondered if quantitative, independent indicators were available.
Without the thorough analysis recommended by the Becker Model for Assessment of Research Impact, we
would have never realized just how far reaching the impact of our research has been.”
Mae Gordon, PhD Department of Ophthalmology and Visual Sciences Principal Investigator, Ocular Hypertension Treatment Study (OHTS)
Establishing linkages: using the literature to identify value
Using the literature to assess science requires clear linkages from funding to papers to new science, and clean data
It further requires meaningful metrics
Concerns with standard publication metrics
Impact Factor average # of citations
(within one year) received by all papers in a journal published within the subsequent 2 years
h-index g-index tapered h-index a-index etc.
Source: http://en.wikipedia.org/wiki/File:H-index-en.svg
Percentile Ranking
Normalizes citation counts Is not as affected by a skewed
distribution Provides information about
publication impact Is calculated fairly cleanly Over time percentile ranking in
a discipline is anticipated
Percentiles for papers published by field, 1999 - 2009Multidisciplinary 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 All Years
0.01% 493 401 346 584 350 399 198 164 91 72 13 4930.10% 468 305 346 584 350 399 162 163 71 59 13 1961.00% 62 69 79 121 107 65 57 52 40 26 5 65
10.00% 5 5 12 16 14 14 16 16 14 7 2 1020.00% 2 2 5 5 4 5 5 8 7 3 1 450.00% 1 1 1 1 1 1 1 1 1 1 0 1
Neuroscience & Behavior 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 All Years0.01% 1254 1286 1306 954 680 557 345 295 122 69 12 8480.10% 596 564 532 440 310 325 206 140 78 28 7 4031.00% 260 230 219 182 145 116 94 68 36 13 3 151
10.00% 74 71 66 59 49 41 32 22 13 5 1 4220.00% 45 43 40 36 31 27 21 15 9 3 1 2550.00% 18 17 16 15 13 11 9 6 4 1 0 8
Psychiatry/Psychology 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 All Years0.01% 943 909 630 821 871 335 656 197 87 35 10 6240.10% 409 397 297 282 250 183 135 100 50 17 6 2451.00% 160 144 130 111 99 81 56 40 22 8 3 92
10.00% 47 44 41 35 31 26 19 13 7 3 1 2520.00% 27 26 24 21 19 16 12 8 5 2 0 1450.00% 9 8 8 8 7 6 5 3 2 1 0 4
Clinical Medicine 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 All Years0.01% 1388 1400 1541 1528 1013 817 742 390 244 88 16 10200.10% 573 519 471 434 402 316 252 175 99 34 6 3541.00% 179 172 156 145 127 105 83 56 33 12 3 112
10.00% 48 46 43 39 35 30 24 17 10 4 1 2820.00% 28 27 26 24 21 19 15 10 6 2 0 1650.00% 9 9 9 8 8 7 6 4 2 1 0 5
All Fields 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 All Years0.01% 1216 1210 1139 1030 816 638 499 316 197 71 12 8040.10% 480 452 411 361 308 248 193 129 76 28 6 2961.00% 161 152 139 124 105 88 67 46 27 11 3 97
10.00% 42 40 37 34 30 25 20 14 8 3 1 2420.00% 24 23 22 20 18 15 12 9 5 2 0 1350.00% 8 7 7 7 6 5 4 3 2 1 0 3
Percentile ranking chart from TR’s Essential Science Indicators, 2009
Percientile Ranking Chart
Balanced Scorecard Example
# of publications/year (productivity)
# of citations to publications (influence)
Percentile placement of WRAIR authors above 50%
Publication metrics reported to Walter Reed Army Institute of Research (WRAIR) Balanced Scorecard.
Using the literature to identify impact
NIAID – impact on standards of care
NIMH – impact of extramural funding
CSR – seeking which study sections have funded most drug/device discoveries
Can we predict future science with publication data?
Reliable metrics Clean data Establish linkages
funding
papers
new science
Evaluate trends
Questions
Terrie Wheeler
301.496.1157
Cathy C. Sarli
Kristi L. Holmes