1

Lecture 10:Meta-analysis of intervention

studies

• Introduction to meta-analysis

• Selection of studies

• Abstraction of information

• Quality scores

• Methods of analysis and presentation

• Sources of bias

2

Definitions

• Traditional (narrative) review:– Selective, biassed

• Systematic review (overview):– Synthesis of studies of a research question– Explicit methods for study selection, data

abstraction, and analysis (repeatable)

• Meta-analysis:– Quantitative pooling of study results

3 Source: l’Abbé et al, Ann Intren Med 1987, 107: 224-233

4

Protocol preparation

• Research question

• Study “population”:– search strategy– inclusion/exclusion criteria

5

Protocol: Search strategy

– computerized databases• E.g. Medline, CINAHL, Embase, Cochrane clinical trial database,

PubMed, Psychinfo

• test sensitivity and predictive value of search strategy

– hand-searches (reference list, relevant journals, colleagues)

– “grey” (unpublished) literature:

• pro: publication bias

• con: results less reliable

– Search strategy should be reliable

6

Identifying relevant studies for systematic reviews of RCTs in vision research (Dickerson, in Systematic Reviews, BMJ,1995)

• Sensitivity and “precision” of Medline searches• Gold standard:

– registry of RCTs in vision research• extensive computer and hand searches

• contacts with investigators to clarify design

• Sensitivity:– proportion of known RCTs identified by the search

• “Precision” (PV+):– proportion of publications identified by search that were

RCTs

7Source: Chalmers + Altman, Systematic Reviews, BMJ Publishing Group, 1995

8Source: Chalmers + Altman, Systematic Reviews, BMJ Publishing Group, 1995

9Source: Chalmers + Altman, Systematic Reviews, BMJ Publishing Group, 1995

10

Protocol preparation

• Study “population”:– inclusion/exclusion criteria:

• language

• study design

• outcome of interest

• etc.

Source: Data abstraction form for meta-analysis project

11

Protocol preparation

• Data collection:– standardized abstraction form– number of abstractors– blinding of abstractors– rules for resolving discrepancies (consensus,

other)– use of quality scores

12Source: l’Abbé et al, Ann Intren Med 1987, 107:224-233

13

Analysis

• Measure of effect:– odds ratio, risk/rate ratio– risk/rate difference – relative risk reduction

• Graphical methods:– conventional (individual studies)– cumulative– exploring heterogeneity

14

Analyses

• Pooling results:– is it appropriate?

– equivalent to pooling results from multi-centre trials

– fixed effect (e.g., Mantel-Haenzel) methods• assume that all trials have same underlying treatment effect

– random effect methods (e.g., DerSimonian & Laird):• allow for heterogeneity of treatment effects

15Source: Chalmers + Altman Systematic Reviews, BMJ Publishing Group, 1995

16Source: Chalmers + Altman Systematic Reviews, BMJ Publishing Group, 1995

17

Heterogeneity

• Sources of heterogeneity:– Study design

– Study population

– Intervention

– Methodological features

• Approaches:– Descriptive and graphical analyses

– Meta-regression• Effect size is outcome

18Source: l’Abbé et al, Ann Intren Med 1987, 107:224-233

19

Quality scores

• Rating scales and checklists to assess methodological quality of RCTs

• How should they be used?– Qualitative assessment– Exclusion of weaker studies– Weighting of estimates

20

Does quality of trials affect estimate of intervention efficacy? (Moher et al, 1998)

• Random sample of 11 meta-analyses of 127 RCTs • Replicated analysis • Used quality scales/measures• Results:

– masked abstraction provided higher quality score than unmasked

– low quality trials found stronger effects than high quality trials

– quality-weighted analysis resulted in lower statistical heterogeneity

21Source: Moher et al, Lancet 1998, 352: 609-13

22Source: Moher et al, Lancet 1998, 352; 609-13

23

Unresolved questions about meta-analysis

• Apples and oranges?– Between-study differences in study population,

design, outcome measures, etc.

• Inclusion of weak studies?

• Publication bias– methods to evaluate impact – - particularly with small studies

• Is it better to do good original studies?

24

Large trials vs meta-analyses of smaller trials (Cappelleri et al, 1996)

• Selected meta-analyses from Medline and Cochrane pregnancy and childbirth database with at least 1 “large” study and 2 smaller studies:– sample size approach (n=1000+) - 79 meta-analyses– statistical power approach (adequate size to detect treatment

effect from pooled analysis - 61 meta-analyses

• Results:– agreement between larger trials and meta-analysis 82-90%

using random effects models– greater disagreement using fixed effects models

25

Large trials vs meta-analyses of smaller trials (Cappelleri et al, 1996)

• Results:– agreement between larger trials and meta-analysis 82-

90% using random effects models

– greater disagreement using fixed effects models

• Conclusion:– large and small trial results generally agree

– each type of trial has advantages and disadvantages:• large trials provide more stable estimates of effect

• small trials may better effect heterogeneity of clinical populations

26

Risk ratios from large studies vs pooled smaller studies (Cappeleri et al,1996)

(Left- sample size approach; right - statistical power approach)

Source: Cappeleri et al, JAMA 1996, 276: 1332-1338

27

Source: Cappeleri et al, JAMA 1996, 276: 1332-1338

28

Discrepancies between meta-analyses and subsequent large RCTs (LeLorier et al, 1997)

• Compared results of 12 large (n=1000+) RCTs with results of 19 prior meta-analyses on same topics

29Source: Lelorier et al, NEJM 1997, 337: 536-42

30Source: Lelorier et al, NEJM 1997, 337: 536-42