introduction to systematic review and meta-analysis brennan spiegel, md, mshs va greater los angeles...
Post on 27-Dec-2015
222 Views
Preview:
TRANSCRIPT
Introduction to Systematic Review and Meta-Analysis
Brennan Spiegel, MD, MSHS
VA Greater Los Angeles Healthcare SystemDavid Geffen School of Medicine at UCLA
UCLA School of Public HealthCURE Digestive Diseases Research Center
UCLA/VA Center for Outcomes Research and Education (CORE)
Objectives• Define and discuss “systematic review”
– Contrast with “narrative review”
– Describe the 4 components of appropriate question
– Define steps for a successful search strategy
– Review construction of evidence tables
• Define and discuss “meta-analysis”– Describe calculations of summary estimates
– Review how to evaluate for heterogeneity
– Define fixed versus random effects models
– Describe “funnel plots” for publication bias
Purposes of Systematic Review and Meta-Analysis
• Combine data from multiple studies to arrive at summary conclusion
• Calculate summary estimate of effect size– May overcome Type II error
• Test for and explain heterogeneity
• Test for publication bias
• Inform decision models
Some Basic Premises
• All meta-analyses must begin with a systematic review
• Knowledge and application of statistical models cannot overcome inadequacies in qualitative systematic review
• Qualitative approach is primary – quantitative approach is secondary
Decision Analysis and Systematic Review
If decision analysis is the engine for making decisions under conditions of uncertainty, then systematic review provides the fuel to run the engine.
The Nature of Meta-Analysis
“Meta-analysis should not be used exclusively to arrive at an average or ‘typical’ value for effect size. It is not simply a statistical method but rather a multicomponent approach for making sense of information.”
• Diana Petitti, in Meta-Analysis, Decision Analysis, and Cost-Effectiveness Analysis, Oxford U Press 2000
Feature Narrative Review Systematic Review
Question Broad in Scope Focused
Sources and Search
Not usually specified, potentially biased
Comprehensive sources and explicit search strategy
Selection Not usually specified, potentially biased
Criterion-based selection, uniformly applied
Appraisal Variable Rigorous critical appraisal
Synthesis Often a qualitative summary
May include quantitative summary (meta-analysis)
Systematic versus Narrative Review
Adapted from Mulrow C, Cook D: Systematic Reviews; ACP Press 1998
Steps to Systematic Review
Step 1 Define focused question
Step 2 Define inclusion / exclusion criteria
Step 3 Develop search strategy
Step 4 Identify databases to search
Step 5 Run search and abstract data
Step 6 Compile data into evidence tables
Step 6 Pool data
Step 7 Interpret data
Four Elements of a Systematic Review Question
1. Type of person involved
2. Type of exposure experienced• Risk factor• Prognostic factor• Intervention• Diagnostic test
3. Type of control with which the exposure is being compared
4. Outcomes to be addressed
Adapted from Mulrow C, Cook D: Systematic Reviews; ACP Press 1998
Example of Inadequate Question
Does smoking cause lung cancer?
Exposure Outcome
Better Question
What is the relative risk of…
lung cancer…
in cigarette smokers…
compared to non cigarette smokers?
Control
Exposure and Type of Person
Outcome
Inadequate Question
Better
Do SSRI improve health related quality of life in patients with depression compared with Elavil?
Are SSRIs, like Prozac, effective for depression?
Feels Better
Does not Feel Better
Feels Better
Does not Feel Better
Chance Nodes
Decision Node
Depression
Developing Inclusion / Exclusion Criteria
• Think of each study as a patient in an RCT
– Must carefully specify inclusion and exclusion criteria to include in the study
• Criteria should mirror carefully formulated question
• Criteria should strike a balance in scope – avoid being too narrow or too broad
• Make sure you target clinically relevant outcomes
• Consider limiting to RCTs if possible
Considerations for Inclusion / Exclusion Criteria
• Definition of target disease/condition
• Stage or severity of condition
• Patient sub-groups (age, sex, symptoms)
• Population or setting (community, hospital)
• Intensity, timing, or duration of exposure
• Method of delivery (e.g. group therapy or individual therapy, oral or IV, etc)
• Type of outcome (survival, HRQOL, adverse events)
• Study design (experimental vs. observational; randomized vs. unrandomized)
Search Strategy Principles
• Balance sensitivity with specificity
– Highly sensitive search strategy may yield untenable number of titles by casting the net too widely
– Highly specific search may yield too few titles and miss key articles by failing to cast a wide enough net
• Said another way:
“The overall goal of any search strategy is to identify all of the relevant material and nothing else.”
• Diana Petitti, in Meta-Analysis, Decision Analysis, and Cost-Effectiveness Analysis, Oxford U Press 2000
Components of Search Strategy
• Select target databases– US National Library of Medicine (MEDLINE)– EMBASE– “Fugitive” or “gray” literature – Cochrane Database of Systematic Review
• Determine language restrictions
• Establish time horizon for search
• Operationalize targeted material with MeSH terms, text words (tw), and publication types (pt)
• Operationalize excluded material and set after “NOT” operator
Example of Defining the Search Strategy
Group Search Terms Significance of Grouping
1 RANDOMIZED-CONTROLLED-TRIAL OR CONTROLLED-CLINICAL-TRIAL OR RANDOMIZED-CONTROLLED-TRIALS OR RANDOM-ALLOCATION OR DOUBLE-BLIND-METHOD OR SINGLE-BLIND-METHOD OR CLINICAL-TRIAL OR CLINICAL-TRIALS OR (CLIN* NEAR TRIAL*) OR ((SINGL* OR DOUBL* OR TREBL* OR TRIPL*) NEAR (BLIND* OR MASK*))
Filter for Randomized
Controlled Trials
2 (ROFECOXIB OR CELECOXIB OR VALDECOXIB OR ETORICOXIB OR COXIB OR COX-2 OR CYCLOOXYGENASE-2) OR ((NAPROXEN OR DICLOFENAC OR IBUPROFEN OR KETOROLAC OR MELOXICAM OR INDOMETHACIN OR KETOPROFEN OR NABUMETONE OR ETODOLAC OR PIROXICAM OR SULINDAC OR ASPIRIN OR ASA OR SALSALATE OR NSAID) AND (LANSOPRAZOLE OR OMEPRAZOLE OR ESOMEPRAZOLE OR RABEPRAZOLE OR PANTOPRAZOLE OR PROTON PUMP INHIBITOR*))
Targeted Content Keywords
3 (TG=ANIMAL OR LETTER [pt] OR EDITORIAL [pt] OR REVIEW [pt] OR NEWS [pt] OR CANCER OR CARCINOMA OR MALIGNANCY OR NEOPLASM)
Excluded Study Types and Content
Spiegel et al. Am J Med 20061 AND 2 NOT 3
Another Example
Spiegel et al. Alim Pharm Ther 2007
Example Search Strategy
Spiegel et al. Arch Int Med 2001
Example Flow Diagram
Spiegel et al. Arch Int Med 2001
Other Best Practices for Systematic Review
• Identify titles, abstract, and manuscripts in 3 separate steps
• Two reviewers search in tandem– Test set for training
– Target high inter-rater reliability (>0.7)
• Develop standardized abstraction form for manuscript review
• Transfer data onto evidence tables
Example of Data Abstraction Using Evidence Tables
Spiegel et al. Am J Med 2006
Another Example
Spiegel et al. Arch Int Med 2001
Evaluating Study Quality
Quality Indicator Points Assessed
Was study described as “randomized?”
If yes, score +1If no, score 0
If study randomized, was there concealed allocation?
If yes, score +1If no, score -1
Was study described as “double-blind?”
If yes, score +1If no, score 0
If study blinded, was it appropriate?
If yes, score +1If no, score -1
Was there a description of withdrawals and dropouts?
If yes, score+1If no, score 0
Jadad et al. Control Clin Trials 1996
Abstracting Data: 2x2 Table
Exposed Unexposed
Event
No Event
NE NU
NE - nE NU - nU
nUnE
= nE NERiskExposed = nE NERiskUnexposed
Abstracting Data: 2x2 Table
Exposed Unexposed
Event
No Event
C D
BA
OR = AD / BC
Before you Combine Data
• Look at the studies you’ve collected. Ask yourself, are they qualitatively similar in terms of 4 key characteristics:
– Patient population
– Exposure
– Comparision group
– Outcome
Before you Combine Data
• Test for statistical evidence of heterogeneity
– Cochrane’s Q statistic
– I2 statistic
• Measure degree of between-study variance
– Wider the variance, higher the heterogeneity
• Tests to see if you are combining “apples” and “oranges”
Cochrane’s Q Statistic
• Tests the sum of the weighted difference between the summary effect measure and the measure of effect from each study
• Compared against 2 distribution with k-1 degrees of freedom, where k=N of studies
• Null hypothesis is that studies are homogeneous
• Test has low sensitivity for detecting heterogeneity, especially when small N of studies – most use p<0.1 for significance
Visual Evidence of Heterogeneity
Juni et al. Lancet 2004
I2 Statistic
• Improves upon Q statistics because less conditional on sample size of studies
• Describes the percentage of total variation across studies that is due to heterogeneity rather than chance.
• I2 calcuation based on Q as follows:
I2 = 100% x (Q-df) / QHiggins et al. BMJ 2003;327
Interpreting I2 Statistic
Range of 0-100%
0-25% = “Low” Heterogeneity
26-50% = “Moderate” Heterogeneity
>50% = “High” Heterogeneity
Higgins et al. BMJ 2003;327
What if there is Heterogeneity?
• More important to explain heterogeneity than to force a summary estimate
• Some turn to “random effects model” (more soon – not a good solution for heterogeneity)
• Can explain heterogeneity through various mechanisms:
– Perform sensitivity analyses stratified by key study characteristics
– Perform meta-regression if sample size permits
Example of Sub-Group Analyses
Watson et al. Curr Med Res Opin 2004
Fixed vs. Random Effects Models
• Two types of statistical procedures to combine data from multiple studies:
– Fixed effects models
• Mantel-Haenszel Method
• Peto Method
– Random effects models
• DerSimonian & Laird Method
Fixed Effects Models
• Inference is conditional on the studies actually done – i.e. the studies at hand
• Assumes there are no other studies outside of the group being evaluated
• Focuses on “within study variance,” which assumes a fixed effect in each study with a variance around the study
– Weight of each study is thus driven by sample size
Random Effects Models
• Inference is based on the assumption that studies in analysis are random sample of larger hypothetical population of studies
• Assumes there are other studies outside of the group being evaluated
• Focuses on both “within study variance” and “between study variance”
– Heterogeneity driven by 2 factors: random variation of each study around fixed effect, and random variation of each study compared to other studies
Within Study Variance
Between Study Variance
More on Fixed vs. Random Models
• Fixed effects model answers question:
“Did the treatment produce benefit on average in the studies at hand?”
• Random effect model answer question:
“Will the treatment produce benefit on average?”
More on Fixed vs. Random Models
• Random effects model usually more conservative than fixed effects model
– Random effects usually has narrower confidence intervals
• When between-study variance is large, within study variance becomes relatively less important, and large and small studies tend to be weighted equally
• Fixed effect is special case of random effect in which between-study variance is zero
• If there is no heterogeneity, then fixed and random effects models yield similar results
Random Effects Model as Solution for Heterogeneity
“The use of the random-effects model is not a defensible solution to the problem of heterogeneity… When there is lack of homogeneity, calculating a summary estimate of effect size is of dubious value… Random effects models should not be used to ‘adjust for’ or ‘explain away’ heterogeneity. The main focus should be on trying to understand sources of heterogeneity.”
- Diana Petitti
Weighted Mean OR = wi*ORi /
W
n
i=1
Where
W = wi
n
i=1
wi = 1 / variancei ORi = ai di/ bi ci
Mantel-Haenszel Method
Coxibs vs. NSAIDS: Dyspepsia Forest Plot
Spiegel et al. Am J Med 2006
Running Meta-Analysis in STATA
Spreadsheet set-up:
Study N_Group_A N_Group_B n_Event_Group_A n_Event_Group_B
Jones 10 10 5 5
James 20 18 3 8
Johnson 100 95 25 40
Marshall 300 280 59 88
Gen n_No_Event_Group_A=N_Group_A-n_Event_Group_A
Gen n_No_Event_Group_B=N_Group_B-n_Event_Group_B
Metan n_Event_Group_A n_No_Event_Group_A n_Event_Group_B n_No_Event_Group_B, rr fixed xlab (.8,1,2) texts(5) label(namevar=study)
Publication Bias
• Editors and journal readers like big, positive studies
• Small, negative studies are inherently less exciting or publishable
• When small negative studies are suppressed, there is an artificially inflated effect
Symmetric Funnel Plot
Effect Size
Sample Size
Asymmetric Funnel Plot
Effect Size
Sample Size
Asymmetric Funnel Plot
Effect Size
Sample Size
Begg's funnel plot with pseudo 95% confidence limits
lo
go
r_6
mo
s.e. of: logor_6mo0 .5 1 1.5
-2
0
2
4
Study Size (SE)
Stu
dy E
ffec
t (L
og O
dds)
Smaller Studies
Larger Studies
Larger Effect
Smaller Effect
Question
Does every probability estimate mandate a full
systematic review and/or meta-analysis?
Answer NO!
Considerations for Determining Rigor of Probability Development
• A priori hypotheses based on literature
• Physical location of variable in tree
• Impact of variable in sensitivity analysis
• Editor pet-peeves and targeted journal for submission
top related