searching grey literature: is the effort worth the yield? ahrq annual meeting track c, session 105...
TRANSCRIPT
Searching Grey Literature: Is the Effort Worth the Yield?AHRQ Annual Meeting Track C, Session 105 Methods for Synthesizing Evidence
Marian McDonagh, PharmDEvidence-based Practice Center
The Problem:
• Searching for and including grey literature adds substantial workload to a systematic review.
• The benefit is unclear in terms of value for time spent
• Information from grey literature may be limited and risk of bias cannot be determined.
2
Session Purpose and Goals
• This session will explore –The methodological argument in
favor of searching for grey literature–Different approaches to searching
for grey literature with highest yield –The usefulness of including grey
literature based on real-world examples
3
Discussion Leaders
• Dr. David Moher– Ottawa Hospital Research Institute – University of Ottawa– Ottawa Evidence-based Practice Center
• Rose Relevo, MLIS, MS– Oregon Evidence-based Practice Center– Oregon Health & Science University
• Marian McDonagh, PharmD– Oregon Evidence-based Practice Center– Oregon Health & Science University
Usefulness of Grey Literature
• What have we discussed so far today:• Evidence that failure to include grey literature
can lead to biased result (Dr. Moher)– Methods to identify grey literature for comparative
effectiveness reviews (Ms. Relevo)
• Interactive Session: What’s it like in practice? – Examples from the Drug Effectiveness Review
Project use of FDA materials as grey literature
Drug Effectiveness Review Project
• DERP is a collaboration of primarily state Medicaid agencies that have joined together to commission systematic reviews of the comparative effectiveness and harms of drugs in many widely used drug classes, and to apply the findings to inform public policy in local settings.
• Project began in 2003– OHSU Center for Evidence-based Policy coordinates– EPCs at Oregon and University of North Carolina
produce systematic reviews for the project.
6
DERP I (15)2003-2006
DERP II (17)2006-2009
DERP III (12)2009 - 2012
AlaskaArkansasCADTH*California HCFIdahoKansasMichiganMinnesotaMissouriMontanaNorth CarolinaOregonWashingtonWisconsinWyoming
AlaskaArkansasCaliforniaCalifornia HCFCADTH*IdahoKansasMichiganMinnesotaMissouriMontanaNew YorkNorth CarolinaOregonWashingtonWisconsinWyoming
ArkansasCADTH*ColoradoIdahoMarylandMissouriMontanaNew YorkOregonWashingtonWisconsinWyoming
*Canadian Agency for Drugs and Technologies in Health
DERP Organizations
114 Reports Completed to date
Proton Pump Inhibitors
Long-acting Opioids
Statins
Non-steroidal Anti-Inflammatory Drugs (NSAIDs)
Estrogens
Triptans
Skeletal Muscle Relaxants
Oral Hypoglycemics
Drugs for Overactive Bladder
ACE Inhibitors
Beta Blockers
Calcium Channel Blockers
Angiotensin II Receptor Antagonists
2nd Generation Antidepressants
Drugs for Constipation
Direct Renin Inhibitors/ACEI/ AIIRA
Fibromyalgia, Drugs to treat
Drugs for Neuropathic Pain
Antiepileptic Drugs in Bipolar Disorder/ Pain
2nd Generation Antihistamines
Atypical Antipsychotics
Inhaled Corticosteroids
ADHD and ADD, Drugs to treat
Alzheimer's, Drugs to treat
Antiplatelet Drugs
Thiazolidinediones (glitazones)
Drugs for Hepatitis C
Newer Drugs for Insomnia
Targeted Immune Modulators
Beta Agonists
Newer Anti-emetics
Drugs for Multiple Sclerosis
Combination Products
Controller Drugs for Asthma
Newer Drugs for Diabetes
Topical Calcineurin Inhibitors
DERP Methods in Using FDA Documents
• Systematic review methodology for DERP includes searching FDA web site– FDA Medical and Statistical reviews of NDAs
• Documents available on FDA web site > 1997• Approved drugs, newly approved for updates• Include studies if enough information to rate quality
• What we are looking for:– Unique unpublished studies– Unpublished data related to published studies
Workload?
• Number of documents reviewed varies• Range 0 to 9 FDA documents per report (mean 1.5)
• Document Size and Organization Issues• Several hundred pages each• Organization varies• Historically pdfs of scanned originals
• Matching published and unpublished studies• Review after electronic searching and selection
• Dual review: eligibility, quality, abstraction
DERP Studying Experience with FDA Documents
• We are analyzing DERP reports for• Frequency of information found in FDA documents
being included in a report• Value of information from FDA documents included• Value is being assessed by assigning an impact
category to each instance, e.g. fills a gap in evidence
• 175 FDA documents eligible for review in114 reports over 9 years– 48 instances of evidence from FDA documents
included (27%)
Example 1: Unique unpublished study
• Atypical Antipsychotic Drugs (2004)– Head to head comparisons of primary
relevance to review– No eligible published studies of aripiprazole
(Abilify®) identified through usual searches and request for manufacturer information
– Abilify® FDA document found• > 200 hundred pages, Examined 5 trials• Identified one trial head to head trial (N = 255)
Your thoughts?
If you came across this evidence what would do with it?
1.Nothing. Don’t use grey literature.
2.Include in report qualitatively only
3.Include in report qualitatively, but influences publication bias assessment
4.Fully include, incorporate into meta-analysis, adjust strength of evidence
Your Thoughts?
If you had included this information, how do you think it would have changed the SOE?
1.Yes
2.No
3.Unsure
Your Thoughts?
Do you believe the added information was valuable to the review’s conclusions?
1.Yes
2.No
3.Unsure
Your thoughts?
Do you think that the time and resources spent to obtain this information was worthwhile?
1.Yes
2.No
3.Unsure
Discussion
• Panelists Thoughts?• What we did in DERP
– Included study in report; changed conclusions only in a minor way• Secondary outcome measure used (cognitive
function)• Aripiprazole superior to olanzapine on some, but
not all measures
• Further discussion?
Example 2: Unpublished data
• Drugs for Insomnia, Update 2 (2008)• Zolpidem ER (Ambien CR®) new
– 3 published placebo-controlled trials identified– 5 placebo-controlled trials in FDA documents
• Matched 2 of the published trials
• FDA docs: Primary outcome at 8 hours – Ambien superior to placebo up to 6 hours, not
different at hours 7 and 8– Publication: reports only 6-hour results
Your thoughts?
If you came across this evidence what would do with it?
1.Nothing. Don’t use grey literature.
2.Include in report qualitatively only
3.Include in report qualitatively, but influences publication bias assessment
4.Fully include, incorporate into meta-analysis, adjust strength of evidence
Your Thoughts?
If you had included this information, how do you think it would have changed the SOE?
1.Yes
2.No
3.Unsure
Your Thoughts?
Do you believe the added information was valuable to the review’s conclusions?
1.Yes
2.No
3.Unsure
Your thoughts?
Do you think that the time and resources spent to obtain this information was worthwhile?
1.Yes
2.No
3.Unsure
Discussion
• Panelists Thoughts?• What we did in DERP
– Included results from FDA docs – changed conclusions of report on Ambien CR®
• Further discussion?
Example 3: Unique unpublished study
• Proton Pump Inhibitors, Update 2 (2004)• Esomeprazole (Nexium®) vs omeprazole
– FDA documents: 4 trials submitted prior to the 2001 approval of Nexium®
– 2 published studies identified • Both found esomeprazole superior to omeprazole
– 2 unpublished studies (in FDA docs) found no statistically significant difference between the drugs
Your thoughts?
If you came across this evidence what would do with it?
1.Nothing. Don’t use grey literature.
2.Include in report qualitatively only
3.Include in report qualitatively, but influences publication bias assessment
4.Fully include, incorporate into meta-analysis, adjust strength of evidence
Your Thoughts?
If you had included this information, how do you think it would have changed the SOE?
1.Yes
2.No
3.Unsure
Your Thoughts?
Do you believe the added information was valuable to the review’s conclusions?
1.Yes
2.No
3.Unsure
Your thoughts?
Do you think that the time and resources spent to obtain this information was worthwhile?
1.Yes
2.No
3.Unsure
Discussion
• Panelists Thoughts?• What we did in DERP
– Included study in report only qualitatively– Inadequate information in FDA docs to
evaluate quality– Subsequent update: obtained data from
manufacturer and incorporated into meta-analysis
• Further discussion?