simple hierarchies are (too) simplistic
DESCRIPTION
Simple hierarchies are (too) simplistic. STUDY DESIGN Randomized Controlled Trials Cohort Studies and Case Control Studies Case Reports and Case Series, Non-systematic observations. BIAS. Expert Opinion. Expert Opinion. Schünemann & Bone, 2003. Likelihood of and confidence in an outcome. - PowerPoint PPT PresentationTRANSCRIPT
Simple hierarchies are (too) simplistic
STUDY DESIGN Randomized Controlled
Trials Cohort Studies and
Case Control Studies Case Reports and Case
Series, Non-systematic observations
BIAS
Expert Opinion
Expert Opinion
Schünemann & Bone, 2003
Likelihood of and confidence in an outcome
Determinants of confidence• RCTs • observational studies • 5 factors that can lower quality
1. limitations in detailed study design and execution (risk of bias criteria)
2. Inconsistency (or heterogeneity)3. Indirectness (PICO and applicability)4. Imprecision5. Publication bias
• 3 factors can increase quality1. large magnitude of effect2. opposing plausible residual bias or confounding3. dose-response gradient
1. Design and Execution/Risk of BiasLimitation in observational studies Explanations
Failure to develop and apply appropriate eligibility criteria (inclusion of control population)
• under- or over-matching in case-control studies
• selection of exposed and unexposed in cohort studies from different populations
Flawed measurement of both exposure and outcome
• differences in measurement of exposure (e.g. recall bias in case- control studies)
• differential surveillance for outcome in exposed and unexposed in cohort studies
Failure to adequately control confounding
• failure of accurate measurement of all known prognostic factors
• failure to match for prognostic factors and/or adjustment in statistical analysis
Incomplete or inadequately short follow-up
1. Design and Execution/Risk of BiasLimitations in RCTs
lack of concealment
intention to treat principle violated
inadequate blinding
loss to follow-up
early stopping for benefit
selective outcome reporting
Design and Execution/RoB
From Cates , CDSR 2008
Design and Execution/RoB
Overall judgment required
YesNoDon’t know or undecided
Who believes the risk of bias is of concern?
Detailed study design and execution
Mortality, cancer andanticoagulation
Akl E, Barba M, Rohilla S, Terrenato I, Sperati F, Schünemann HJ. “Anticoagulation for the long term treatment of venous thromboembolism in patients with cancer”. Cochrane Database Syst Rev. 2008 Apr 16;(2):CD006650.
Five trials
YesNoDon’t know or undecided
Who believes the risk of bias is of concern?
2. Inconsistency of results(Heterogeneity)
• if inconsistency, look for explanation– patients, intervention, comparator, outcome
• if unexplained inconsistency lower quality
Reminders for immunization uptake
Inconsistency
• I2
• P-value• Overlap in CI• Difference in point estimates
3. Directness of Evidencegeneralizability, transferability, applicability
• differences in– populations/patients (HIC – L/MIC, patients with HIV – all
patients)– interventions (new fluroquinolones - old)– comparator appropriate (newer antibx – old)– outcomes (important – surrogate; signs and symptoms –
mortality)• indirect comparisons
– interested in A versus B– have A versus C and B versus C– Cryo + antibiotics versus no intervention versus Cryo versus
no intervention
EVIDENCE PROFILEQuestion: Cyrotherapy with antibiotics vs no antibiotics for histologically confirmed CIN
1 All rates presented at 12 months with assumption that events would occur within this time frame.2 Indirect analysis between single arm observational studies
Quality assessment No of patients Effect
Quality ImportanceNo of studies Design Limitations Inconsistency Indirectness Imprecision Other
Cryotherapy with
antibioticsNo
antibioticsRelative(95% CI) Absolute
Major infection (follow-up 12 months1; requiring hospitalisation or blood transfusion)16 observation
al studiesno serious limitations
no serious inconsistency
serious2 no serious imprecision
none0/1600
(0%)10/4573 (0.22%)
RD 0 (0 to 0)
0 fewer per 1000 OOO IMPORTANT
Resource use - not measured
All severe adverse events (follow-up 12 months; (major infections and bleeding, pelvic inflammatory disease, stenosis, etc )17 observation
al studiesno serious limitations
no serious inconsistency
serious2 no serious imprecision
none0/1705
(0%)22/5142 (0.43%)
RD 0 (0 to 0)
0 fewer per 1000 OOO IMPORTANT
4. Publication Bias
• Should always be suspected– Only small “positive” studies– For profit interest– Various methods to evaluate – none perfect, but
clearly a problem
Egger M, Smith DS. BMJ 1995;310:752-54 19
I.V. Mg in acute
myocardial infarction
Publication bias
Meta-analysisYusuf S.Circulation 1993
ISIS-4Lancet 1995
Egger M, Cochrane Colloquium Lyon 2001 20
Funnel plotS
tand
ard
Err
or
Odds ratio0.1 0.3 1 3
3
2
1
0
100.6
Symmetrical:No publication bias
Egger M, Cochrane Colloquium Lyon 2001 21
Funnel plotS
tand
ard
Err
or
Odds ratio0.1 0.3 1 3
3
2
1
0
100.6
Asymmetrical:Publication bias?
0.4
5. Imprecision• Small sample size
– small number of events
• Wide confidence intervals– uncertainty about magnitude of effect
Example: Immunization in children
For systematic reviews
• If the 95% CI excludes a relative risk (RR) of 1.0 and the total number of events or patients exceeds the OIS criterion, precision is adequate. If the 95% CI includes appreciable benefit or harm (we suggest a RR of under 0.75 or over 1.25 as a rough guide) rating down for imprecision may be appropriate even if OIS criteria are met.
Optimal information size
• We suggest the following: if the total number of patients included in a systematic review is less than the number of patients generated by a conventional sample size calculation for a single adequately powered trial, consider rating down for imprecision. Authors have referred to this threshold as the “optimal information size” (OIS)
What can raise quality?
1. large magnitude can upgrade (RRR 50%/RR 2)– very large two levels (RRR 80%/RR 5)– criteria
• everyone used to do badly• almost everyone does well
– parachutes to prevent death when jumping from airplanes
Reminders for immunization uptake
What can raise quality?2. dose response relation
– (higher INR – increased bleeding)– childhood lymphoblastic leukemia
• risk for CNS malignancies 15 years after cranial irradiation• no radiation: 1% (95% CI 0% to 2.1%) • 12 Gy: 1.6% (95% CI 0% to 3.4%) • 18 Gy: 3.3% (95% CI 0.9% to 5.6%)
3. all plausible residual confounding may be working to reduce the demonstrated effect or increase the effect if no effect was observed
All plausible residual confoundingwould result in an overestimate of effect
Hypoglycaemic drug phenformin causes lactic acidosis
The related agent metformin is under suspicion for the same toxicity.
Large observational studies have failed to demonstrate an association– Clinicians would be more alert to lactic acidosis in
the presence of the agent• Vaccine – adverse effects
Quality assessment criteria Study design
Initial quality of a body of evidence
Lower if Higher if Quality of a body of evidence
Randomised trials
High Risk of Bias
Inconsistency
Indirectness
Imprecision
Publication bias
Large effect Dose response All plausible residual confounding & bias -Would reduce a demonstrated effect -Would suggest a spurious effect if no effect was observed
A/High (four plus: )
B/Moderate (three plus: )
Observational studies
Low C/Low (two plus: )
D/Very low (one plus: )
Overall quality of a body of evidence
• The quality of evidence reflects the extent of our confidence that the estimates of an effect are adequate to support a particular decision or recommendation.
• Guideline developers must specify and determine importance of all relevant outcomes
• Overall quality of evidence is based on the lowest quality of all critical outcomes
Meta-analyses of several critical and important outcomes (one PICO)
Better Relative RiskWorse
0.5 0.75 1 1.25 1.5
SAE (critical)
Nausea (important)
Myo. Infarct. (critical)
Mortality (critical) High
Low Due to imprecision and risk of bias
Moderate Due to imprecision
High
Overall Quality of Evidence: Moderate based on critical outcomes
Low
Meta-analyses of several critical outcomes (one PICO)
Better Relative RiskWorse
0.5 0.75 1 1.25 1.5
SAE
Stroke
Dis. Specific QoL
Mortality
Threshold of acceptable harm for strong
recommendation based on sure benefit in mortality and stroke
High
High
Moderate Due to imprecision
High
Overall Quality of Evidence: High
Meta-analyses of several critical outcomes (one PICO)
Better Relative RiskWorse
0.5 0.75 1 1.25 1.5
SAE
Stroke
Dis. Specific QoL
Mortality
Overall Quality of Evidence:
High
Moderate due to risk of bias
High
High
Moderate
Threshold of acceptable harm for strong
recommendation based on sure benefit in mortality and stroke
Interpretation of grades of evidence
• /A/High: Further research is very unlikely to change confidence in the estimate of effect.
• /B/Moderate: Further research is likely to have an important impact on confidence in the estimate of effect and may change the estimate.
• /C/Low: Further research is very likely to have an important impact on confidence in the estimate of effect and is likely to change the estimate.
• /D/Very low: We have very little confidence in the effect estimate: Any estimate of effect is very uncertain.
Agenda8.30-8.45 Introduction to the course8.45-9.15 Overview of guideline development9.15-10.15 The GRADE approach: introduction10.15-10.30 Break10.30-11.30 Asking a key question, specific outcomes – small group work11.30-12.30 The GRADE approach: assessing the quality of evidence12.30-13.15 Lunch13.15-14.45 Grading quality of evidence – small group work14.45-15.15 Moving from evidence to recommendations: theoretical considerations15.15-15.30 Break15.30-16 Making recommendations – small group work16-16.30 Report from small groups and practical considerations of group process
Agenda8.30-8.45 Introduction to the course8.45-9.15 Overview of guideline development9.15-10.15 The GRADE approach: introduction10.15-10.30 Break10.30-11.30 Asking a key question, specific outcomes – small group work11.30-12.30 The GRADE approach: assessing the quality of evidence12.30-13.15 Lunch13.15-14.45 Grading quality of evidence – small group work14.45-15.15 Moving from evidence to recommendations: theoretical considerations15.15-15.30 Break15.30-16 Making recommendations – small group work16-16.30 Report from small groups and practical considerations of group process
MOVING FROM EVIDENCE TO RECOMMENDATIONS
Holger (contributions from Yngve Falck-Ytter)
Strength of recommendation
“The strength of a recommendation reflects the extent to which we can, across the range of patients for whom the recommendations are intended, be confident that desirable effects of a management strategy outweigh undesirable effects.” • Strong or conditional
Evidence based healthcare decisions
Research evidence
Population/societalvalues
and preferences
(Clinical) state and circumstances
Expertise
Haynes et al. 2002
Evidence based healthcare decisions
Research evidence
Population/societalvalues
and preferences
Expertise
Evidence of baseline risk and context for a given population
Schunemann et al. in prep
Evidence based healthcare decisions
Research evidence
Expertise
Evidence of baseline risk and context for a given population
Evidence of population & societal values
& preferences
Schunemann et al. in prep
Evidence based healthcare decisions
Expertise
Evidence of baseline risk and context for a given population
Evidence of population & societal values
& preferences
Evidence of effectsSchunemann et al. in prep
Evidence based healthcare decisions
Expertise
Evidence of baseline risk and context for a given population
Evidence of population & societal values
& preferences
Evidence of effectsSchunemann et al. in prep
From evidence to recommendations
RCTObser-vational study
High level recommen-
dation
Lower level recommen-
dation
Old system
Quality of evidence
Balance between benefits, harms & burdens
Patients’ values &
preferences
GRADE
Resource use
Determinants of the strength of recommendation
Factors that can strengthen a recommendation
Comment
Quality of the evidence The higher the quality of evidence, the more likely is a strong recommendation.
Balance between desirable and undesirable effects
The larger the difference between the desirable and undesirable consequences, the more likely a strong recommendation warranted. The smaller the net benefit and the lower certainty for that benefit, the more likely weak recommendation warranted.
Values and preferences The greater the variability in values and preferences, or uncertainty in values and preferences, the more likely weak recommendation warranted.
Costs (resource allocation) The higher the costs of an intervention – that is, the more resources consumed – the less likely is a strong recommendation warranted
Balancing desirable and undesirable consequences
Conditional
Strong For Against
Effects & $ x
values
Effects & $ x
values
Balancing desirable and undesirable consequences
↑ Allergic reactions
↑ Local skin reactions
↑ Nausea↑ Resources
↑ QoL ↓ Death
↓ Morbidity
↑ herd immunity
Conditional
Strong For Against
Effect x value Effect x value
↑ Allergic
reactions
↑ Local skin
reactions
↑ Nausea
↑ Resources
↑ QoL↓ Death
↓ Morbidity↑ herd
immunityConditional
Strong For Against
Balancing desirable and undesirable consequences
↑ Allergic reactions↑ Local skin reactions
↑ Nausea
↑ Resources↑ QoL ↓ Death
↓ Morbidity
↑ herd immunity
Conditional
Strong For Against
Balancing desirable and undesirable consequences
Balancing desirable and undesirable consequences
↑ Allergic
reactions
↑ Local skin
reactions
↑ Nausea
↑ Resources
↑ QoL
↓ Death
↓
Morbidity
↑ herd
immunity
Conditional
Strong For Against
Balancing desirable and undesirable consequences
↑ Allergic reactions ↑ Local skin
reactions
↑ Nausea
↑ Resources
↑ QoL
↓ Death
↓ Morbidity
↑ herd immunity
Conditional
Strong For Against
Implications of a strong recommendation
• Policy makers: The recommendation can be adapted as a policy in most situations
• Patients: Most people in this situation would want the recommended course of action and only a small proportion would not
• Clinicians: Most patients should receive the recommended course of action
Implications of a conditional recommendation• Policy makers: There is a need for
substantial debate and involvement of stakeholders
• Patients: The majority of people in this situation would want the recommended course of action, but many would not
• Clinicians: Be more prepared to help patients to make a decision that is consistent with their own values/decision aids and shared decision making
Case scenario
A 13 year old girl who lives in rural Indonesia presented with flu symptoms and developed severe respiratory distress over the course of the last 2 days. She required intubation. The history reveals that she shares her living quarters with her parents and her three siblings. At night the family’s chicken stock shares this room too and several chicken had died unexpectedly a few days before the girl fell sick.
Methods – WHO Rapid Advice Guidelines for Avian Flu
Applied findings of a recent systematic evaluation of guideline development for WHO/ACHR
Group composition (including panel of 13 voting members): clinicians who treated influenza A(H5N1) patients infectious disease experts basic scientists public health officers methodologists
Independent scientific reviewers: Identified systematic reviews, recent RCTs, case series, animal
studies related to H5N1 infection
Oseltamivir for Avian FluSummary of findings: • No clinical trial of oseltamivir for treatment of H5N1
patients.• 4 systematic reviews and health technology
assessments (HTA) reporting on 5 studies of oseltamivir in seasonal influenza. – Hospitalization: OR 0.22 (0.02 – 2.16)– Pneumonia: OR 0.15 (0.03 - 0.69)
• 3 published case series. • Many in vitro and animal studies. • No alternative that was more promising at present.• Cost: 40$ per treatment course
Evidence profile
No of studies(Ref) Design Limitations Consistency Directness Other
considerations Oseltamivir Placebo Relative(95% CI )
Absolute(95% CI )
Mortality 0 - - - - - - - - - 9
5(TJ 06)
Randomised trial
No limitations One trial only Major uncertainty (-2)1
Imprecise or sparse data (-1)
- - OR 0.22(0.02 to 2.16)
- Very low
6
0 - - - - - - - - - - 7
5(TJ 06)
Randomised trial
No limitations One trial only Major uncertainty (-2)1
Imprecise or sparse data (-1)2
2/982(0.2%)
9/662(1.4%)
RR 0.149(0.03 to 0.69)
- Very low
8
53
(TJ 06)(DT 03)
Randomised trials
No limitations4 Important inconsistency(-1)5
Major uncertainty (-2)1
- - - HR 1.303
(1.13 to 1.50)-
Very low5
26
(TJ 06)Randomised trials
No limitations -7 Major uncertainty (-2)1
None - - - WMD -0.738
(-0.99 to -0.47)
Low4
0 - - - - - - - - - - 4
0 - - - - - - - - - - 7
09 - - - - - - - - - - 7
311
(TJ 06)Randomised trials
No limitations -12 Some uncertainty (-1)13
Imprecise or sparse data (-1)14
- - OR range15
(0.56 to 1.80)-
Low
0 - - - - - - - - - - 4
I mportance
Summary of findings
Cost of drugs
Outbreak control
Resistance
Serious adverse effects (Mention of significant or serious adverse effects)
Minor adverse effects 10 (number and seriousness of adverse effects)
Viral shedding (Mean nasal titre of excreted virus at 24h)
Duration of disease (Time to alleviation of symptoms/median time to resolution of symptoms – influenza cases only)
Duration of hospitalization
LRTI (Pneumonia - influenza cases only)
Healthy adults:
Hospitalisation (Hospitalisations from influenza – influenza cases only)
Quality assessmentNo of patients Effect
Quality
Oseltamivir - H5N1
-
-
Quality across
endpoints
PICO & Confidence in estimates
Benefits vs harms
Values and preferences
Resource utilization
From evidence to recommendation
Factors that can strengthen a recommendation
Comment
Quality of the evidence Very low quality evidence
Balance between desirable and undesirable effects
Uncertain, but small reduction in relative risk still leads to large absolute effect
Values and preferences Little variability and clear
Costs (resource allocation) Low cost under non-pandemic conditions
Example: Oseltamivir for Avian FluRecommendation: In patients with confirmed or strongly suspected infection with avian influenza A (H5N1) virus, clinicians should administer oseltamivir treatment as soon as possible (strong recommendation, very low quality evidence).
Remarks: This recommendation places a high value on the prevention of death in an illness with a high case fatality. It places relatively low values on adverse reactions, the development of resistance and costs of treatment.
Schunemann et al. The Lancet ID, 2007
?
Other explanationsRemarks: Despite the lack of controlled treatment data for H5N1, this is a strong recommendation, in part, because there is a lack of known effective alternative pharmacological interventions at this time.
The panel voted on whether this recommendation should be strong or weak and there was one abstention and one dissenting vote.
GRADE Grid
Values and preferences
• Implicit value judgments in recommendations
• Trade-offs: example prevention of VTE in surgery– Thrombotic events
• Deep vein thrombosis, pulmonary embolism– Bleeding events
• Gastrointestinal bleeds, operative site bleeds– Inconvenience of injections
• Variability in values and preferences
Case• 77 y/o patient with atrial fibrillation, mild
CHF, HTN, DM and history of stroke (fully recovered)
• Meds: warfarin, antihypertensives, statin, glyburide
• Admitted with nausea/vomiting, then hematemesis; INR 2.5; 1 U blood transfused; EGD: no active bleed, possible Mallory Weiss
• This is his second major bleed since he started warfarin one year ago
CHADS2 score
Acceptable additional bleeds?• Study: Patients at high risk for atrial fibrillation and high
risk of stroke (h/o CHF/MI); internists and cardiologists
• Warfarin decreases risk at cost of increased GI bleeds• Without treatment 100 patients will suffer:
– 12 strokes (six major, six minor), 3 serious GI bleeds in 2 years
• Warfarin would decrease strokes in 100 patients to 4 per 2 years (8 fewer strokes, 4 major, minor)
• How many additional bleeds would you accept in 100 patients over a year, and still be willing to administer/take warfarin?
Strength of recommendation“The strength of a recommendation reflects the extent to which we can, across the range of patients for whom the recommendations are intended, be confident that desirable effects of a management strategy outweigh undesirable effects.”
Understanding values & preferences necessary to trade-off benefits and downsides
Values and preferences should ideally be informed by systematic reviews, but evidence is often sparse
Example recommendation
ACCP AT9 recommendation:In patients undergoing major orthopedic surgery (e.g., total hip replacement), we suggest the use of LMWH in preference to the other agents.Patients who place a high value on avoiding bleeding complications and a low value on its inconvenience are likely to choose a compression device (IPCD) over the drug options.
Considering resource use
• Allocating resources such as cost• In GRADE: cost = outcome. Challenges include:
– Who pays? Individual vs. society?– Varies widely (even within 1 kilometer) – Opportunity cost differ– Often highly political (e.g., USA)
• Resource use considered last as an additional step
Example 1: ipilimumabfor metastatic melanoma
Estimated absolute effects at 2 years
Outcome Relative effect
(95% CI)
control(per 1000)
ipilimumab(per 1000)
Difference(per 1000)
Certainty of the effect
Death HR 0.68(0.55 to
0.85)850 725 125 fewer deaths
(49 to 202 fewer)
High
Serious immune-
related AEs
RR 7.37(4.42 to
12.3)39 289 249 more SAEs
(134 to 441 more)
High
Quality of life
Acquisition cost
Ipilimumab: $120,000 (12 weeks induction (4 injections))Dacarbazine: $400
Simplified cost effectiveness projection
$ or $ per LY
Dacarbazine induction $400
Ipilimumab induction $120,000
Increase in median survival 3.6 mo (0.3 years)
Incremental cost-effectiveness ($/life year)
~$400,000
Not only cost but also resource use
• Emergency room visits, hospitalizations• Severe immune-mediated events (grade 3 and
4): 86/627 patients with hepatic toxicity, 18/627 with enterocolitis; 5 immune-mediated deaths associated with ipilimumab
• Use of other interventions downstream• Include complete resource use in evidence
profile if possible (e.g., hospital days, nurses hours, drug units dispensed, etc.)
Define perspective taken
• Example 1: single regional health system– Health plan / insurance– Country– International: e.g., WHO
• Perspective should be sufficiently broad– Should include downstream effects– Comprehensive view desirable: e.g., societal
• Includes all cost regardless who bears them
Quality of evidence assessment
• Rarely assessed in the context of RCTs used to inform estimate of effect
• Often derived from other sources of evidence
• More confident in the acquisition cost than the estimate of resources used for serious AEs
Modeling resource use• Modeling results in
cost per unit benefits achieved can be helpful– …but sometimes be misleading and prone to
bias– …relies heavily on assumptions – …debate whether indirect costs, such as lost
productivity should be included• Avoid including cost-utility models directly
in the GRADE evidence profile
SummaryFrom evidence to making recommendations
1. Beyond net benefit, GRADE makes transparent the process on how values & preferences and resource use informs decision making
2. Understanding values & preferences is necessary to make trade-offs between benefits and downsides
3. If necessary, resource use considerations can inform direction and strength of recommendations
Conclusions Guidelines should be based on the best available evidence to
be evidence based GRADE is the approach used by over 65 organizations and
gaining acceptance internationally combines what is known in health research methodology and
provides a structured approach to improve communication Does not avoid judgments but provides framework Criteria for evidence assessment across questions and
outcomes Criteria for moving from evidence to recommendations Transparent, systematic
four categories of quality of evidence two grades for strength of recommendations
Transparency in decision making and judgments is key