Systematic Reviews &Meta-analysis

Clay BordleyDepartment of Pediatrics

Division of Hospital & Emergency Medicine

Jane P. GagliardiDepartment of Psychiatry and Behavioral Sciences

Department of Medicine

First a Case….

3 mo w/ sneezin’, wheezin’ and fever

• Congested for 4 days, fast breathing for past 24 hours– Breast and bottle fed, but having trouble sucking and breathing– Decreased wet diapers

• 3yr cousin with URI sx’s• PMHx: 39 wk, unremarkable P/L/D• SHx: 16yr mother (sucking her thumb), lives with MGM, mo and cousins• PE

– 38.0 186 64, 93% sat– Alert, obvious congestion and increased WOB’ing, nl TM’s, (+) retractions, defuse

wheezing, no murmur, nl CR and pulses, abd soft, no HSM

• Your impression: – Bronchiolitis, fever, borderline sats

What Clinical Questions Do You Have?

Disclosures

W. Clayton Bordley, MD, MPH

Today’s Clinical Question

Is hypertonic saline effective in infants with bronchiolitis?

Exercise #1

Search in groups of 2

Time: 3-5 minutes

Be prepared to report to group

1. How did you do your search2. What source did you choose?

3. Did you get an answer?

What Did You Find?

Is This a Systematic Review?

eMedicne

Did You Use PICOTT?

• Patient• Intervention• Control• Outcome• Type of question• Type of study

• Infants w/ bronchiolitis• Hypertonic saline• None• Effectiveness (?)• ?• ?

Exercise #2

• Connie searched and came up with 7 candidate studies

Exercise #2• In the next FIVE MINUTES:

– Critically appraise your article• Follow-up• Randomization• Intention to treat• Similar at baseline• Blinding• Equal treatment outside the intervention

• Summarize the methods and results• Be ready to share your answers out loud

• 5 minutes to CAT the articles• Follow-up• Randomization• Intention to treat• Similar at baseline• Blinding• Equal treatment outside the intervention

What overall grade would you assign your study?

1 2 3 4 5

0% 0% 0%0%0%

1. A (excellent, few sources of bias)2. B (not bad, despite some bias)3. C (mediocre, some obvious bias)4. D (barely makes the cut, lots of

bias)5. F (may not even be worth

reading)

Please remember the grade you assigned(will return to this a bit later)

Learning Objectives

• Understand the distinction between reviews, systematic reviews and meta-analysis

• Understand the basic methods & outcomes reported in meta-analysis

• Interpret a forest plot• Understand the critical appraisal elements that

apply to systematic reviews• Understand role systematic of reviews in

guidelines and other publication types• Use a systematic review to help guide clinical

decision-making

Components of SystematicReviews

• Specific objective(s)• Clearly described, reproducible methods

– Databases and search terms• A priori inclusion / exclusion criteria• Data extracted by more than one reviewer• Consideration of quality• Evidence table(s)• Results

– Qualitative and/or quantitative

Specific Objective(s)

• A focused clinical question• (May look suspiciously like a PICOTT)

Clearly Described Methods

• LITERATURE SEARCH:

– Search Terms– Database(s)– Time Frame

Inclusion/Exclusion Criteria

• This needs to be specified A PRIORI:– What types of studies will be included?– What characteristics of studies will be

accepted?– Is there any consideration of QUALITY of the

studies to be included?

Consideration of quality - Assessments of study quality should resemble:

1 2 3 4 5

0% 0% 0%0%0%

1. PICOTT2. Validity Criteria3. Cochrane review4. Performance

Improvement5. Systematic review

Data Abstraction

• How many reviewers?• Should the reviewer(s) be aware of the

study hypothesis?• What sort of information should the

reviewer(s) abstract?• How are conflicts resolved?

Results

• Flow Diagram for Study Selection• Evidence Tables• Forest Plots

Impact of Diet Soda on Impulsivity

Favors regular Favors diet

*adapted from article addressing different question

Summarizing the Evidence

BREAK!

Which of the following is NOT a necessary part of a systematic review?

a) b) c) d) e)

0% 0% 0%0%0%

a) Focused clinical questionb) Reproducible search

methodsc) Inclusion/exclusion criteriad) Statistical analysise) Consideration of study

quality

Meta-analysis

Meta-analysis

(definition): mathematical synthesis of the results of two or more primary studies that addressed the same hypothesis in the same way

The Basic Principles• Each study reports a measure of effect

– Rate, RR, OR, risk difference, others• Summary (pooled) effect measure is calculated

– weighted average of the individual studies• Weighting may be based on sample size (usually) and/or study quality

• Confidence intervals – Communicates the precision of the summary estimate, – Allows calculation of a P value

• Measure of variability among results (aka heterogeneity)– Is the variation compatible with random variation?– Is the variation sufficiently large to suggest the studies assessed

different “truths” in their results? – Note: estimate of whether or not methods are similar enough to

combine should have been done while compiling evidence table

Review, Systematic Review and Meta-analysis

Exercise #3

• Who is familiar with Forest Plots?• Create a Forest Plot

– Based on the 7 RCT’s you reviewed– Each group come down and add their results

to the Forest plot

Favors treatment Favors control

What is the outcome measure?

What is the unit of measure?

Let’s Look at Some Prettier Examples

Forest Plots• The label explains comparison and outcome

Forest Plots• The data for each trial are represented, divided into

experimental and control groups• The calculated effect measure is presented numerically• The % weight given to each study is reported

Forest Plots• The result of each study is plotted graphically with a block• The size of the block for each study is proportional to the % weight. • The horizontal line represents the confidence interval.

• The label above the graph provides information about statistics• The horizontal line at the bottom depicts the scale for the treatment

effect

Forest Plots

Forest Plots• The vertical line in the middle is where treatment and control have

the same effect – THE LINE OF NO DIFFERENCE

Forest Plots• The pooled estimate is given as a diamond shape

– The horizontal width of the diamond is the confidence interval of the pooled estimate

• The pooled estimate is also given numerically

Forest Plots• Heterogeneity: The forest plot displays the results of the Chi square

test and the p value

Heterogeneity

• Inevitably, studies brought together in a systematic review will differ.

• The amount of variability in the results from studies in a systematic review is termed heterogeneity.

• What are some possible sources of heterogeneity?

Measures of Heterogeneity

• Eyeball test• Chi Square• I2 Inconsistency

“eyeball test”

Bordley 2013

Measures of HeterogeneityEyeball test

• Chi Square– Assesses whether observed differences in results

are compatible with chance alone. – A low P value (or a large chi-squared statistic

relative to its degree of freedom) provides evidence of heterogeneity (variation in effect estimates beyond chance).

• I2 Inconsistency– Moves the focus away from testing whether

heterogeneity is present to assessing its impact on the meta-analysis.

Back to Our Case

Validity Criteria for a Systematic Review

• Focused clinical question?• Appropriate study inclusion criteria?• Thorough search?

– unlikely to miss relevant studies?• Quality of included studies assessed? • Heterogeneity?• Sufficient follow-up (duration, subjects

accounted for)? • Generalizable to your clinical scenario?

It All Comes Back to the Search

Assessing Quality of Included Studies

• Why– Garbage in leads to garbage out– Individual study bias affects meta-analysis

result• How: Quality Scores

– 25 scales & 9 checklists (e.g., Jadad)– Cochrane method:

• 7 domains and risk of bias for each– Sensitivity analysis

Exercise #4

APPRAISE THE SYSTEMATIC REVIEW• Focused clinical question?• Appropriate study inclusion criteria?• Thorough search?

– unlikely to miss relevant studies?• Quality of included studies assessed? • Heterogeneity?• Sufficient follow-up (duration, subjects

accounted for)? • Generalizable to your clinical scenario?

What grade would you assign this systematic review?

1 2 3 4 5

0% 0% 0%0%0%

1. A- nearly perfect minimal bias

2. B-not too bad; some bias3. C-mediocre; may have

biased results4. D-garbage in…5. F-garbage out…

3% saline versus 0.9% saline: Length of hospital stay (days)

Figure 1. 3% saline versus 0.9% saline: Length of hospital stay (days)

3% saline versus 0.9% saline: Rate of Hospitalization

Given the information you have, would you give the baby nebulized saline?

1 2 3 4 5

0% 0% 0%0%0%

1. No, just use humidified air (tell the kid's parents to keep the hot shower running in a closed bathroom)

2. No, just use the nebulized Xopenex

3. Yes, use 0.9% nebulized saline4. Yes, use 3% nebulized saline5. I prefer to conduct my own

clinical trial

Summary Points• All reviews are not Systematic Reviews• Systematic review use explicit and reproducible

methods to answer a focused question• A meta-analysis is a mathematical synthesis of

the results of two or more primary studies that addressed the same hypothesis in the same way

• Meta-analysis increases the precision of a result– Decision to perform involves judgment and

quantitative assessment

Hierarchy of Evidence

Extra stuff

• Systematic reviews are the foundation of other evidence based tools

• Finding systematic reviews

Other Applications of Systematic Reviews

• Practice Guidelines– Combine quality of evidence with strength of

recommendation• Cost-benefit analyses (to derive ranges

used in models• Decision analysis (to derive probabilities

used in models)

Practice Guidelines

Other Studies Built Upon Systematic Reviews

Other Systematic Reviews“The Rational Clinical Examination”

Locate systematic reviews using• Cochrane Database of Systematic Reviews (CDSR)

The CDSR is a full-text database containing systematic reviews and protocols (reviews still in progress) of the effects of health care interventions; mainly randomized controlled trials. "Gold Standard" for high quality, systematic reviews.

• Database of Abstracts of Reviews of Effects (DARE)Structured abstracts of good quality systematic reviews from around the world. Complements CDSR by offering a selection of quality assessed reviews in subjects with no Cochrane review. Also part of the Cochrane Library.

• PubMed Systematic ReviewsPubMed Clinical Queries: Systematic Reviews combines your search term(s) with citations identified as systematic reviews, meta-analyses, reviews of clinical trials, evidence-based medicine, consensus development conferences, and guidelines. No evaluation of comparative quality of different reviews on a topic.

• PubMed | Search TipsHint: you can limit to Publication Type: Meta-Analysis

• CINAHL Plus | Search TipsHint: you can limit with Publication Types: Systematic Review

Effective Ways to Keep Up

Increasing Utility of Systematic Reviews

• Cochrane—standardizing methodology• The PRISMA statement

– reporting systematic reviews and meta-analyses of healthcare interventions

• Critical appraisal criteria – JAMA Users Guide to the Medical Literature

• DARE and PubMed filters – Finding systematic reviews can be easy!

Publishing Systematic Reviews

The CONSORT equivalent