systematic review, meta-analysis,...
TRANSCRIPT
Systematic Review, Meta-Analysis, Clinical Practice Guidelines…
Tools, TSH and Tricks of the Trade…
Larry Young
Sheri Keitz
Teaching Objectives
• Understand what systematic reviews can and cannot do for you
• To learn to love forest plots• To review principles of when it is appropriate to
combine studies • Interpret the results of a meta-analysis including
forest plots• Become familiar with the GRADE approach and
its use in creating clinical practice guidelines and implementing evidence
Definitions
• A systematic review is a summary of research that addresses a focused clinical question in a systematic, reproducible manner.
• A meta-analysis is a statistical pooling or aggregation of estimate of effect.
SR MA
Users’ Guide to the Medical Literature. Guyatt et al.
Advantages of Meta-Analyses
• More representative of the total body of evidence
• Save time
• Increase precision of results
• Include a greater range of patients, potentially improving generalizability
• Provide an assessment of confidence in results
The Systematic Review
Hierarchy of Evidence Pyramid
Unsystematic Clinical Experience
Case-Control
Case Series
Observational Studies
(Cohort)
RCT
Systematic Review/Meta-analysis
Ask
Acquire
Appraise
Apply
Assess
Clinical
dilemma
Evidence-based
medicine
cycle
Values &
PreferencesHierarchy of
Evidence
THE BEST WAY TO UNDERSTAND JUST ABOUT ANYTHING IN MEDICINE IS TO REFLECT ON A PATIENT…
The Setting
• You are covering the walk in clinic for one of your colleagues.
• A staff member, who usually sees one of your residents, is here for follow up of her lab results.
Mrs. SR
• 67 year old retired 3rd GRADE teacher
• PMH: hypertension and osteopenia
• Weight gain, dry skin and fatigue for the last 6 months; quality of life has declined – less active, new knee pain
• Normal physical examination
• BP 158/92
• TSH 9.5; normal T3 and T4
• You diagnose subclinical hypothyroidism.
She wonders if thyroid replacement will help with her symptoms
• P : non-pregnant patients with subclinical hypothyroidism
• I : thyroid hormone replacement
• C : no treatment, placebo
• O : quality of life, thyroid related symptoms, HTN, worsening osteopenia
• T : therapy
• T : randomized clinical trial, meta-analysis of RCTs
She wonders if thyroid replacement will help with her symptoms
• P : non-pregnant patients with subclinical hypothyroidism
• I : thyroid hormone replacement
• C : no treatment, placebo
• O : quality of life, thyroid related symptoms, HTN, worsening osteopenia
• T : therapy
• T : randomized clinical trial, meta-analysis of RCTs
Acquire
• 21 studies
• 2192 patients
• RCTs comparing thyroid hormone replacement to placebo or no treatment
Ask
Acquire
Appraise
Apply
Assess
Clinical
dilemma
Evidence-based
medicine
cycle
Values &
PreferencesHierarchy of
Evidence
Appraisal
Let’s appraise the article together… in 5 minutes?
Rea
d A
bst
ract
… 2
m
inu
tes
Assessing the Credibility of the Systematic Review Process
Did the review address a focused clinical question (i.e.
can be framed in PICO format)?
1
Was the search for relevant studies detailed and
exhaustive?
2
Was the risk of bias of the primary studies assessed? 3
Did the review address possible explanations of
between-study differences in results using prespecified
hypotheses?
4
Were selection and assessment of studies reproducible? 5
Did the review describe a process to assess confidence
in effect estimates? (e.g. GRADE tool to assess quality of
the body of evidence)
6
Understanding the Summary Estimate of a Meta-analysis
What is the magnitude of treatment effect? (what is the
pooled estimate?)
7
How precise are the results? (i.e. confidence interval
around the pooled estimate)
8
Rating Confidence in the Estimates (the Quality of a Body of Evidence)How serious is the risk of bias in the body of evidence? 9
Are the results consistent across studies? (i.e.
heterogeneity or inconsistency)
10
Do the results directly apply to my patient? (i.e. PICO,
generalizability, indirectness)
11
Is there a concern about reporting or publication bias? 12
Are there reasons to increase or decrease the confidence
of the rating? (Randomized trials start high and
observational studies start low)
13
Overall, what is the quality of the body of evidence by
outcome? (High, moderate, low, very low)
14
How can I apply the results to my patient care?
Did the review present results that are ready for clinical
application? (e.g. patient important outcomes, absolute
benefit /risk)
15
Are the study patients similar to my patient and are
likely benefits worth potential harms/costs?
16
Ap
pra
isal
… in
yo
ur
bin
der
Assessing the Credibility of the Systematic Review ProcessDid the review address a focused clinical
question (i.e. can be framed in PICO format)?
1
Was the search for relevant studies detailed
and exhaustive?
2
Was the risk of bias of the primary studies
assessed?
3
Did the review address possible explanations
of between-study differences in results using
prespecified hypotheses?
4
Were selection and assessment of studies
reproducible?
5
Did the review describe a process to assess
confidence in effect estimates? (e.g. GRADE
tool to assess quality of the body of
evidence)
6
Appraisal – Did the Meta-Analysts Do a Good Job?
Did the review address a focused clinical question (i.e. can be framed in PICO format)?
Was the search for relevant studies detailed and exhaustive?
Was the risk of bias of the primary studies assessed?
Did the review address possible explanations of between-study differences in results using prespecified hypotheses?
Were selection and assessment of studies reproducible?
Did the review describe a process to assess confidence in effect estimates? (e.g. GRADE tool to assess quality of the body of
evidence)
Assessing the Credibility of the Systematic Review ProcessDid the review address a focused clinical
question (i.e. can be framed in PICO format)?
1
Was the search for relevant studies detailed
and exhaustive?
2
Was the risk of bias of the primary studies
assessed?
3
Did the review address possible explanations
of between-study differences in results using
prespecified hypotheses?
4
Were selection and assessment of studies
reproducible?
5
Did the review describe a process to assess
confidence in effect estimates? (e.g. GRADE
tool to assess quality of the body of
evidence)
6
Appraisal
What are the results
• Everyone: look at Figure 1 (page 1354).
• Let’s look together at the primary outcomes
Page 1354
Page 1354
Should we pool the results?Heterogeneity??? What the fork?
Expert on Heterogeneity
https://www.youtube.com/watch?v=HwqHUSOHPvU
Is pooling sensible?
• A priori determination of whether it makes sense to combine on a principled basis.
• You have not looked at the data yet…
The Common Sense Test
Is pooling sensible?
• MA of all kinds of treatments for all cancers?
• MA of steroid treatments for COPD?
• MA of studies looking at CT-angiogram for diagnosis of pulmonary embolism?
Does combining RESULTS make sense?
• Comes from a review of dots and lines
Favors New Treatment Favors ControlNo Difference
Result 1
Eye-ball test
Favors New Treatment Favors ControlNo Difference
Can we combine?
Favors New Treatment Favors ControlNo Difference
Result 2
Eye-ball test
Favors New Treatment Favors ControlNo Difference
Can we combine?
Favors New Treatment Favors ControlNo Difference
Result 3
Eye-ball test
Favors New Treatment Favors ControlNo Difference
Can we combine?
Okay: What is the “Eye-ball test”?
Eye-ball test
• Visual impression of degree of overlap of each study’s results (find a common neighborhood of truth)
• Visual impression of where the estimates of study “effects” themselves lie (where are the dots)
c
Favors New Treatment Favors ControlNo Difference
Favors New Treatment Favors ControlNo Difference
Back to result 3: if not, why not?
Why might things be different
• P: more or less severe symptoms
• I: types of replacement (and duration)
• C: active control / placebo
• O: different scales for QOL, fatigue, others
• T: treatment question
• T: RCT or other study designs
I2 what?
• From 0 to 100
• The variability that is due to a real difference
• I2 = 0
• I2 = 100
• What is the range of possible values for I2
• Explain in a sentence?
• Is the variability between studies due ONLY to chance?
• Is the variability between studies due ONLY to a real difference in treatment?
Favors New Treatment Favors ControlNo Difference
Result 1: what is I2
Favors New Treatment Favors ControlNo Difference
Result 2: what is I2
Favors New Treatment Favors ControlNo Difference
Result 3: what is I2
Back to Results
BACK TO THE PAPER
BUT HOW MUCH CONFIDENCE CAN WE HAVE IN THIS BODY OF EVIDENCE?
Rating Confidence in the Estimates (the Quality of a Body of Evidence)
How serious is the risk of bias in the body
of evidence?
9
Are the results consistent across studies?
(i.e. heterogeneity or inconsistency)
10
Do the results directly apply to my
patient? (i.e. PICO, generalizability,
indirectness)
11
Is there a concern about reporting or
publication bias?
12
Are there reasons to increase or decrease
the confidence of the rating?
(Randomized trials start high and
observational studies start low)
13
Overall, what is the quality of the body of
evidence by outcome? (High, moderate,
low, very low)
14
How serious is the risk of bias?
Do these look familiar?
Remember, LOW values are GOOD
Rating Confidence in the Estimates (the Quality of a Body of Evidence)
How serious is the risk of bias in the body
of evidence?
9
Are the results consistent across studies?
(i.e. heterogeneity or inconsistency)
10
Do the results directly apply to my
patient? (i.e. PICO, generalizability,
indirectness)
11
Is there a concern about reporting or
publication bias?
12
Are there reasons to increase or decrease
the confidence of the rating?
(Randomized trials start high and
observational studies start low)
13
Overall, what is the quality of the body of
evidence by outcome? (High, moderate,
low, very low)
14
Rating Confidence in the Estimates (the Quality of a Body of Evidence)
How serious is the risk of bias in the body
of evidence?
9
Are the results consistent across studies?
(i.e. heterogeneity or inconsistency)
10
Do the results directly apply to my
patient? (i.e. PICO, generalizability,
indirectness)
11
Is there a concern about reporting or
publication bias?
12
Are there reasons to increase or decrease
the confidence of the rating?
(Randomized trials start high and
observational studies start low)
13
Overall, what is the quality of the body of
evidence by outcome? (High, moderate,
low, very low)
14
Does This Match Our PICOTT?
• P : non-pregnant patients with subclinical hypothyroidism
• I : thyroid hormone replacement
• C : no treatment, placebo
• O : quality of life; thyroid related symptoms, HTN, worsening osteopenia
• T : therapy
• T : randomized clinical trial, meta-analysis of RCTs
Rating Confidence in the Estimates (the Quality of a Body of Evidence)
How serious is the risk of bias in the body
of evidence?
9
Are the results consistent across studies?
(i.e. heterogeneity or inconsistency)
10
Do the results directly apply to my
patient? (i.e. PICO, generalizability,
indirectness)
11
Is there a concern about reporting or
publication bias?
12
Are there reasons to increase or decrease
the confidence of the rating?
(Randomized trials start high and
observational studies start low)
13
Overall, what is the quality of the body of
evidence by outcome? (High, moderate,
low, very low)
14
Is there a concern about reporting or publication bias?
• Positive studies may be as much as 3 times more likely to be published than negative studies.
• Usually exaggerate treatment effect
• Funnel plot
• Look for asymmetry in the included studies
Result
Stu
dy
size
Result
Stu
dy
size
Result
Is there a concern about reporting or publication bias?
Rating Confidence in the Estimates (the Quality of a Body of Evidence)
How serious is the risk of bias in the body
of evidence?
9
Are the results consistent across studies?
(i.e. heterogeneity or inconsistency)
10
Do the results directly apply to my
patient? (i.e. PICO, generalizability,
indirectness)
11
Is there a concern about reporting or
publication bias?
12
Are there reasons to increase or decrease
the confidence of the rating?
(Randomized trials start high and
observational studies start low)
13
Overall, what is the quality of the body of
evidence by outcome? (High, moderate,
low, very low)
14
Are there reasons to increase or decrease the confidence of the rating?
X is GOODSummary of Findings Table
No downgrade; inconsistency due to one trial that showed effect of placebo.
Rating Confidence in the Estimates (the Quality of a Body of Evidence)
How serious is the risk of bias in the body
of evidence?
9
Are the results consistent across studies?
(i.e. heterogeneity or inconsistency)
10
Do the results directly apply to my
patient? (i.e. PICO, generalizability,
indirectness)
11
Is there a concern about reporting or
publication bias?
12
Are there reasons to increase or decrease
the confidence of the rating?
(Randomized trials start high and
observational studies start low)
13
Overall, what is the quality of the body of
evidence by outcome? (High, moderate,
low, very low)
14
For Primary Outcomes: High
Rating Confidence in the Estimates (the Quality of a Body of Evidence)
How serious is the risk of bias in the body
of evidence?
9
Are the results consistent across studies?
(i.e. heterogeneity or inconsistency)
10
Do the results directly apply to my
patient? (i.e. PICO, generalizability,
indirectness)
11
Is there a concern about reporting or
publication bias?
12
Are there reasons to increase or decrease
the confidence of the rating?
(Randomized trials start high and
observational studies start low)
13
Overall, what is the quality of the body of
evidence by outcome? (High, moderate,
low, very low)
14
Rating Confidence in the Estimates (the Quality of a Body of Evidence)
How serious is the risk of bias in the body
of evidence?
9
Are the results consistent across studies?
(i.e. heterogeneity or inconsistency)
10
Do the results directly apply to my
patient? (i.e. PICO, generalizability,
indirectness)
11
Is there a concern about reporting or
publication bias?
12
Are there reasons to increase or decrease
the confidence of the rating?
(Randomized trials start high and
observational studies start low)
13
Overall, what is the quality of the body of
evidence by outcome? (High, moderate,
low, very low)
14
For Primary Outcomes: High
We already did this!
CLINICAL PRACTICE GUIDELINES DEVELOPMENT
Evidence Recommendations
Guideline Development Committee
YOU!
Three Groups
ABlood
pressure
B Fatigue
CBMI
Look at e-Table 4
TABLE VOTE Make recommendation using
handout on your table
YES
NO
WEAK
STRONG
OR OR
Five Ten Minutes
A: Blood Pressure
X is GOODNo difference
Blood PressureMake Recommendation - VOTE
YES
NO
OR
Blood PressureMake Recommendation - VOTE
YES
NO
WEAK
STRONG
OR OR
B: Fatigue
X is GOODDowngrade due to only one trial.
No difference
FatigueMake Recommendation - VOTE
YES
NO
OR
FatigueMake Your Recommendation - VOTE
YES
NO
WEAK
STRONG
OR OR
C: BMI
BMI
X is GOODNo downgrade; inconsistency due to one trial that showed effect of placebo.
No difference
BMIMake Recommendation - VOTE
YES
NO
OR
BMI Make Recommendation- VOTE
YES
NO
WEAK
STRONG
OR OR
Back to Mrs. SR…
• Among nonpregnant adults with subclinical hypothyroidism, the use of thyroid hormone therapy was not associated with improvements in general quality of life or thyroid-related symptoms.
• We advise her that this medication will not improve her symptoms and discuss alternative approaches to address her ongoing concerns.
Anyone Recognize This?
• GRADE: a process for rating the quality of the best available evidence and developing health care recommendations.
• Why did we add this content? – It is a good way to bring many concepts together
and make explicit how evidence can be incorporated on a broader scale
– It is everywhere
GRADE: adopted by over 100 organizations
worldwide
Take Home Messages
• Systematic Reviews and Meta-analyses are needed to inform practice and can be used to drive decisions and actions.
• GRADE is a tool with explicit rules that is transparent and sensible for users of evidence and makers of clinical practice guidelines
• Two components of the GRADE process
– Quality of a body of evidence (certainty, confidence)
– Strength of recommendation: yes/no; weak/ strong
• When determining confidence in results: RCTs start high and Observational studies start low (hierarchy of evidence)
Take Home Messages
• Not all evidence is created equalA Hierarchy of evidence helps us differentiate information more likely to be valid or true
• Evidence alone is never enoughDecisions are made with our patients, guided by values and preferences.
Final Take Home Message:Cookie Monster is awesome