Systematic Review Module 9: Systematic Review Module 9: Quantitative Synthesis IQuantitative Synthesis I

Joseph Lau, MDJoseph Lau, MDThomas Trikalinos, MD, PhDThomas Trikalinos, MD, PhD

Tufts EPCTufts EPC

CER Process OverviewCER Process Overview

Prepare topic:

· Refine key questions

· Develop analytic frameworks

Search for and select

studies:

· Identify eligibility criteria

· Search for relevant studies

· Select evidence for inclusion

Abstract data:

· Extract evidence from studies

· Construct evidence tables

Analyze and synthesize data:

· Assess quality of studies

· Assess applicability of studies

· Apply qualitative methods

· Apply quantitative methods (meta-analyses)

· Rate the strength of a body of evidence

Present findings

2

Learning ObjectivesLearning Objectives

Basic principles of combining data Common metrics for meta-analysis Basics of combining results across

studies and effects of weights Meaning of heterogeneity Fixed effect and random effects model

3

Synonyms for Meta-analysisSynonyms for Meta-analysis

Quantitative overview Research (evidence) synthesis Research integration Pooling (less precise—suggests data

from multiple sources are simply lumped together)

Combining (preferred by some—connotation of applying procedures to data)

4

Caveats of Meta-analysisCaveats of Meta-analysis

Few will criticize you for doing a systematic review. But as soon as you combine data (or draw conclusions based on “similarly grouped” studies), you will likely get disagreements.

Most meta-analyses are retrospective exercises, suffering from all the problems of being an observational design. We cannot make up missing information or fix badly collected, analyzed, or reported data. Therefore, care is needed when deciding on the type of data that should be included in a meta-analysis.

5

Why Is That?Why Is That?

Apples and oranges (heterogeneity) Garbage in, garbage out (quality) Selection of outcomes (soft or hard) Selection of studies Publication bias Many assumptions are used in

quantifying results (there are no “assumption-free” statistics)

6

Reasons for Meta-analysisReasons for Meta-analysis

Get an overall estimate of treatment effect

Appreciate the degree of uncertainty Appreciate heterogeneity Forces you to think rigorously about the

data

7

Types of Data that Could Be Combined Types of Data that Could Be Combined in Meta-analysis of Summary Datain Meta-analysis of Summary Data

Dichotomous (events, e.g., deaths) Measures (odds ratios, correlations) Continuous data (mmHg, pain scores,

proportions) Survival curves Diagnostic test performance (sensitivity,

specificity) “Effect size”

8

Basic Principles in Basic Principles in Combining DataCombining Data

For each analysis, one study should contribute only one effect.

The effect may be a single outcome or a composite of several independent outcomes.

Effect being combined should be the same across studies or similar.

Know your question. The question drives your systematic review, meta-analysis, and interpretation of the results.

9

Things to Know about the Things to Know about the Data before CombiningData before Combining

Biological plausibility Scale Fragility of small numbers

10

True Associations May Disappear When True Associations May Disappear When You Combine Data InappropriatelyYou Combine Data Inappropriately

Eff

ect

of

inte

rest

Variable of interest

MEN WOMEN

11

Apparent Association May Be Apparent Association May Be Seen When There is None Seen When There is None

Eff

ect

of

inte

rest

variable of interest

MEN

WOMEN

12

Same Changes in One Scale May Same Changes in One Scale May Have Different Meaning in AnotherHave Different Meaning in Another

Eff

ect

of

inte

rest

Variable of interest

A

B

C

D

Both A–B and C–D involve a change of one absolute unit.

A–B change (1 to 2) represents a 100% relative change.

C–D change (7 to 8) represents only a 14% relative change.

13

Same Change in One Scale May Have Same Change in One Scale May Have Different Meaning in Another ScaleDifferent Meaning in Another Scale

TreatmentTreatment ControlControl

StudyStudy EventsEvents TotalTotal RateRate EventsEvents TotalTotal RateRate

Relative Relative RiskRisk

Absolute Absolute RiskRisk

AA 100100 10001000 10%10% 200200 10001000 20%20% 0.50.5 10%10%

BB 11 10001000 0.1%0.1% 22 10001000 0.2%0.2% 0.50.5 0.1%0.1%

14

Effect of Small Changes on the EstimateEffect of Small Changes on the Estimate(small numbers give fragile estimates)(small numbers give fragile estimates)

Baseline Baseline casecase

Effect of Effect of decrease of decrease of

1 event1 event

Effect of Effect of increase of increase of

1 event1 event

Relative Relative change of change of estimateestimate

2/102/10

20%20%

1/101/10

10%10%

3/103/10

30%30%

± ± 50%50%

20/10020/100

20%20%

19/10019/100

19%19%

21/10021/100

21%21%

±± 5% 5%

200/1,000200/1,000

20%20%

199/1,000199/1,000

19.9%19.9%

201/1,000201/1,000

20.1%20.1%

±± 0.5% 0.5%

15

Dichotomous OutcomesDichotomous Outcomes

Binary outcomes, event or no event, yes or no It is the most common type of outcomes reported

in clinical trials Some examples are dead or alive, stroke or no

stroke, cure or failure 2x2 tables commonly used to report their results Sometimes continuous variables are converted

into dichotomous outcomes. For example, a threshold value may be used to report pain scores as improved or not improved

16

Example of 2x2 Table: ISIS-2 Example of 2x2 Table: ISIS-2

Streptokinase Streptokinase PlaceboPlacebo

Vascular Vascular deathsdeaths

791791 1,0291,029

SurviveSurvive 7,8017,801 7,5667,566

TOTALTOTAL 8,5928,592 8,5958,595

Randomized trial of intravenous streptokinase, oral aspirin, both, Randomized trial of intravenous streptokinase, oral aspirin, both, or neither among 17,817 cases of suspected acute myocardial or neither among 17,817 cases of suspected acute myocardial infarction Lancet 1988;ii:349-360infarction Lancet 1988;ii:349-360

17

a b

c d

EventsNo

Events

Treatment

Control

Group Rates

TR =

CR =

Treatment Effects

a

a + b

c

c + d

Risk Difference Odds Ratio Risk Ratio

RD = TR - CR OR = RR =TR / (1 - TR) TR

CRCR / (1 - CR)

OR = (a d) / (b c)

Definitions of Treatment Definitions of Treatment Effects from 2x2 Table Effects from 2x2 Table

18

Available Metrics for Combining Available Metrics for Combining Dichotomous Outcome DataDichotomous Outcome Data

Odds ratio (OR) Risk ratio (RR) Risk difference (RD) NNT (number needed to treat) can be

derived (inverse of the combined risk difference) = 1/RD

19

Effect SizeEffect Size

Dimensionless metric The basic idea is to combine standard

deviations of diverse types of related effects However, availability and selection of reported

effects may be biased, variable importance of different effects

Frequently used in education, social science literature

Infrequently used in medicine, difficulty in interpreting results

20

Properties of Odds RatioProperties of Odds Ratio

Desirable mathematical properties, unbiased estimator

Symmetrical outcome meaning (the odds of dying is equal to the opposite [inverse] of the odds of living). The 0.5 odds of dying is 2.0 odds of living.

Can approximate risk ratio at low event rates

Not easy to interpret

21

Properties of Risk DifferenceProperties of Risk Difference

Symmetrical meaning of outcome (5% more of dying is 5% less of living)

Magnitude of effect directly interpretable NNT can be calculated and clinically useful Risk difference across studies more likely to

be heterogeneous Combining heterogeneous RD in a meta-

analysis may not be meaningful Unbiased estimator

22

Properties of Risk RatioProperties of Risk Ratio

Easy to understand by clinicians Needs to be interpreted in view of baseline

rate Asymmetric meaning for outcome (the risk

ratio of dying is not the same as the opposite of the risk ratio of living)

Less desirable mathematical properties (not an unbiased estimator)

Unstable variance (usually not a big problem)

23

The Complementary Outcome of The Complementary Outcome of Risk Ratio Is Not Symmetrical Risk Ratio Is Not Symmetrical

DeadDead AliveAlive TotalTotal

TreatmentTreatment 2020 8080 100100ControlControl 4040 6060 100100

OR (dead) = 20x60 / 40x80 = 0.25

OR (alive) = 80x40 / 20x60 = 4.0

RR (dead) = 20/100 / 40/100 = 0.5

RR (alive) = 80/100 / 60/100 = 1.33

24

Calculation of Treatment Calculation of Treatment Effects of ISIS-2 Data Effects of ISIS-2 Data

Streptokinase Streptokinase PlaceboPlacebo

Vascular Vascular deathsdeaths

791791 1,0291,029

SurviveSurvive 7,8017,801 7,5667,566

TOTALTOTAL 8,5928,592 8,5958,595

RR = 0.0921 / 0.1197 = 0.77

OR = (791 x 7566) / (1029 x 7801) = 0.75

RD = 0.0921 – 0.1197 = -0.028

TR = 791/8592 = 0.0921 CR = 1029 / 8595 = 0.1197

25

ISIS-2 Streptokinase vs. Placebo Vascular ISIS-2 Streptokinase vs. Placebo Vascular Death Estimate with the 95% CIDeath Estimate with the 95% CI

EstimateEstimate 95% CI95% CI

Risk ratio (RR)Risk ratio (RR) 0.770.77 0.70–0.840.70–0.84

Odds ratio (OR)Odds ratio (OR) 0.750.75 0.68–0.820.68–0.82

Risk difference Risk difference (RD)(RD)

−−0.0280.028 −−0.037–0.037–−−0.0190.019

NNT (1/RD)NNT (1/RD) 3636 27–5427–54

26

Beta-Blockers after Myocardial Infarction - Secondary Prevention  Experiment Control Odds 95% CI N Study Year Obs Tot Obs Tot Ratio Low High === ============ ==== ====== ====== ====== ====== ===== ===== =====   1 Reynolds 1972 3 38 3 39 1.03 0.19 5.45 2 Wilhelmsson 1974 7 114 14 116 0.48 0.18 1.23 3 Ahlmark 1974 5 69 11 93 0.58 0.19 1.76 4 Multctr. Int 1977 102 1533 127 1520 0.78 0.60 1.03 5 Baber 1980 28 355 27 365 1.07 0.62 1.86 6 Rehnqvist 1980 4 59 6 52 0.56 0.15 2.10 7 Norweg.Multr 1981 98 945 152 939 0.60 0.46 0.79 8 Taylor 1982 60 632 48 471 0.92 0.62 1.38 9 BHAT 1982 138 1916 188 1921 0.72 0.57 0.90 10 Julian 1982 64 873 52 583 0.81 0.55 1.18 11 Hansteen 1982 25 278 37 282 0.65 0.38 1.12 12 Manger Cats 1983 9 291 16 293 0.55 0.24 1.27 13 Rehnqvist 1983 25 154 31 147 0.73 0.40 1.30 14 ASPS 1983 45 263 47 266 0.96 0.61 1.51 15 EIS 1984 57 858 45 883 1.33 0.89 1.98 16 LITRG 1987 86 1195 93 1200 0.92 0.68 1.25 17 Herlitz 1988 169 698 179 697 0.92 0.73 1.18

27

Simpson’s ParadoxSimpson’s Paradox(Rothman, Modern Epidemiology) (I)(Rothman, Modern Epidemiology) (I)

Suppose a man enters a shop to buy a hat and finds a table of 30 hats, 10 black and 20 gray. He discovers that 9 of 10 black hats fit, but only 17 of the 20 gray hats fit. Thus, he notes that the proportion of black hats that fit is 90% compared with 85% of the gray hats.

At another table in the same shop, he finds another 30 hats, 20 black and 10 gray. At this table, 3 (15%) of the black hats fit, but only 1 (10%) of the gray hats fits.

28

Simpson’s Paradox (II)Simpson’s Paradox (II)

Before he chooses a hat, the shop closes for the evening, so he returns on the following morning. Overnight, the clerk has piled all the hats on the same table: Now there are 30 hats of each color. The shopper remembers that yesterday the proportion of black hats that fit was greater at each of the two tables. Today he finds that, although all the same hats are displayed, when mixed together only 40% (12 of 30) of the black hats fit, whereas 60% (18 of 30) of the gray hats fit.

29

Simpson’s Paradox (III)Simpson’s Paradox (III)

FitFit Not fitNot fit

BlackBlack 99 11

GrayGray 1717 33

FitFit Not fitNot fit

BlackBlack 33 1717

GrayGray 11 99

FitFit Not fitNot fit

BlackBlack 1212 1818

GrayGray 1818 1212

Table 1 Table 2

Pooling Tables 1 and 2

black (90%) > gray (85%) black (15%) > gray (10%)

black (40%) < gray (60%)

30

What Is the “Average (Overall)” What Is the “Average (Overall)” TreatmentTreatment——Control Difference in DBP?Control Difference in DBP?

Study Sample Size

mmHg

95% Confidence Interval

ANBP 554 −6.2 −6.9 to −5.5

EWPHE 304 −7.7 −10.2 to −5.2

Kuramoto 39 −0.1 −6.5 to +6.3 4

31

Simple AverageSimple Average

(−6.2) + (−7.7) + (−0.1)(−6.2) + (−7.7) + (−0.1)

33== −−4.7 mmHg4.7 mmHg

X x

ii1

k

n

Study Sample Size

mmHg

95% Confidence Interval

ANBP 554 -6.2 -6.9 to -5.5

EWPHE 304 -7.7 -10.2 to -5.2

Kuramoto 39 -0.1 -6.5 to +6.3 4

____

32

Weighted AverageWeighted Average

(554 x (554 x −6.2) + (304 x 6.2) + (304 x −7.7) + (39 x 7.7) + (39 x −0.1)0.1)

554 + 304 + 39554 + 304 + 39== −6.4 mmHg6.4 mmHg

X w

ix

ii1

k

wi

i1

k

Study SampleSize

mmHg

95% ConfidenceInterval

ANBP 554 -6.2 -6.9 to -5.5

EWPHE 304 -7.7 -10.2 to -5.2

Kuramoto 39 -0.1 -6.5 to +6.34

33

General Formula: Weighted General Formula: Weighted Average Effect Size (d+)Average Effect Size (d+)

d w

id

ii1

k

wi

i1

k

where: di = effect size of the ith studywi = weight of the ith studyk = number of studies

34

Calculation of WeightsCalculation of Weights

Generally the inverse of the variance of treatment effect (that captures both study size and precision)

Different formula for odds ratio, risk ratio, risk difference

Readily available in books and software

35

Heterogeneity (Diversity)Heterogeneity (Diversity)

Is it reasonable (are studies and effects sufficiently similar) to estimate an average effect?

Types of heterogeneity– Conceptual (clinical) heterogeneity

– Statistical heterogeneity

36

Conceptual (Clinical) Conceptual (Clinical) HeterogeneityHeterogeneity

Are the studies of similar treatments, populations, settings, design, etc., such that an average effect would be clinically meaningful?

37

Endoscopic Hemostasis. An Effective Therapy for Endoscopic Hemostasis. An Effective Therapy for Bleeding Peptic Ulcers. Sacks HS, Chalmers TC, et al. Bleeding Peptic Ulcers. Sacks HS, Chalmers TC, et al.

JAMA 1990 264:494-499JAMA 1990 264:494-499

25 RCTs compared endoscopic hemostasis with standard therapy for bleeding peptic ulcer

5 different types of treatment (monopolar electrode, bipolar electrode, argon laser, neodymium-YAG laser, sclerosant injection)

4 different conditions (active bleeding, nonspurting blood vessel, no blood vessels seen, undesignated)

3 different outcomes (emergency surgery, overall mortality, recurrent bleeding)

38

Statistical HeterogeneityStatistical Heterogeneity

Is the observed variability of effects greater than that expected by chance alone?

39

A Container with a Fixed (Known) A Container with a Fixed (Known) Number of White and Black BallsNumber of White and Black Balls

(fixed effect model)(fixed effect model)

40

Random Sampling from a Container with a Random Sampling from a Container with a Fixed Number of White and Black Balls Fixed Number of White and Black Balls

(equal sample size)(equal sample size)

41

Random Sampling from a Container with Random Sampling from a Container with Fixed Number of White and Black Balls Fixed Number of White and Black Balls

(different sample size)(different sample size)

42

Different Containers with Different Different Containers with Different Proportions of White and Black BallsProportions of White and Black Balls

(random effects model)(random effects model)

43

Random Sampling from Containers to Random Sampling from Containers to Get an Overall Estimate of the Get an Overall Estimate of the

Percentage of White (or Black) BallsPercentage of White (or Black) Balls

44

Statistical Models of Pooling Statistical Models of Pooling 2x2 Tables2x2 Tables

Fixed Effect Model: weights studies by the inverse of the within-study (sampling) variance. Assumes a common treatment effect. The size of the study and the number of events are the main determinants of its importance.

Random Effect Model: weights studies by the inverse of the sum of the within-study variation and the among-study variation. Allows for different treatment effects. Tends to be more conservative (gives broader confidence interval) when heterogeneity is present.

45

Fixed Effect ExampleFixed Effect Example

I ask all of you to measure the height of the flag pole outside of this building. There will be some variations in the reported values, but all of you are measuring the same flag pole. Discounting potential errors (biases) from using different measuring instruments, the variation is due to “random errors” around the truth.

46

Fixed Effects ModelFixed Effects Model

TREATMENT EFFECTS (RD, OR, RR)

SINGLETRUETREATMENTEFFECT

47

Fixed Effects ModelFixed Effects Model

TREATMENT EFFECTS (RD, OR, RR)

SINGLETRUETREATMENTEFFECT

POOLED RESULTESTIMATED TREATMENTEFFECT

RESULTS OF MULTIPLE CLINICAL TRIALS RANDOMLY DISTRIBUTED AROUND THE TRUE TREATMENT EFFECT

48

49

Random Effects ExampleRandom Effects Example

Suppose that I am interested in knowing the average height of the flag poles in a city so that I can compare with another city’s average flag pole heights. I ask all of you to randomly measure the height of flag poles around the city. There will be a lot more variations in the reported values because of measurements of different flag poles and different measurements of the same flag pole. The greater variation is due to “random errors” around the true height of each flag pole and the distribution of the heights of different flag poles.

50

Random Effects ModelRandom Effects Model

TREATMENT EFFECTS (RD, OR, RR)

MULTIPLE TRUETREATMENT EFFECTS(distribution of treatment effects)

51

Random Effects ModelRandom Effects Model

TREATMENT EFFECTS (RD, OR, RR)

MULTIPLE TRUETREATMENT EFFECTS(distribution of treatment effects)POOLED RESULT

SINGLE ESTIMATED TREATMENT EFFECT

RESULTS OF MULTIPLE CLINICAL TRIALS RANDOMLY DISTRIBUTED AROUND EACH OF THE TRUE TREATMENT EFFECT

52

Influenza Vaccine Efficacy from Influenza Vaccine Efficacy from Observational StudiesObservational Studies

Gross et al. Gross et al. Ann Intern Med Ann Intern Med 19951995

53

Fixed Effect and Random Fixed Effect and Random Effects ModelsEffects Models

wv vii

*

*

1

wvii

1

Random Effects WeightFixed Effect Weight

where: vi = within study variance

v* = between study variance

54

HETEROGENEOUS TREATMENT EFFECTS

IGNORE INCORPORATEESTIMATE(insensitive)

EXPLAIN

FIXED EFFECTS MODEL

DO NOT COMBINE WHEN

HETEROGENEITY IS PRESENT

RANDOM EFFECTS MODEL

SUBGROUP ANALYSES

META-REGRESSION(control rate, covariates)

Dealing with HeterogeneityDealing with Heterogeneity

55

Chi-Square Homogeneity Chi-Square Homogeneity Test Mantel-HaenszelTest Mantel-Haenszel

Q w d dk df i ii

k

( )1

2 2

1

NOTE: d = ln(ORi d+ = ln(ORMH) wi = 1/variance (ORi)

Variance (ORi) = 1/ai + 1/bi + 1/ci + 1/di

56

Summary: Basic Statistical Summary: Basic Statistical Methods of Combining 2x2 TablesMethods of Combining 2x2 Tables

OddsOdds

RatioRatio

RiskRisk

RatioRatioRisk Risk

DifferenceDifference

Fixed Effect Fixed Effect ModelModel

Mantel-Mantel-HaenszelHaenszel

PetoPeto

ExactExact

Mantel-Mantel-HaenszelHaenszel

Inverse Inverse variance variance weightedweighted

Random Random Effects Effects ModelModel

DerSimonianDerSimonian& Laird& Laird

DerSimonianDerSimonian& Laird& Laird

DerSimonianDerSimonian& Laird& Laird

57

Summary: Statistical Models Summary: Statistical Models of Combining 2x2 Tablesof Combining 2x2 Tables

Most meta-analyses of clinical trials combine treatment effects (risk ratio, odds ratio, risk difference) across studies to produce a common estimate, using either a fixed effect or random effect model.

In practice, the results using these two models are often similar when there is little or no heterogeneity.

When heterogeneity is present, the random effect model generally produces a more conservative result (smaller Z-score) with a similar estimate but with a wider confidence interval. However, there are rare exceptions of extreme heterogeneity where random effects model may yield counterintuitive results.

58

SummarySummary

Decision to do a meta-analysis should be based on well-formulated question and an appreciation of how the results will be used.

Math is relatively simple. Can be easily programmed using statistical or

spreadsheet software. Many commercial or shareware meta-analysis software are readily available.

Decisions (e.g., fixed or random effects models, measure of effect) can be complex.

Results may vary with assumptions made.

59