pplanlan - mcmaster university evidence-based clinical...

Plan Plan • • GRADE background GRADE background

• • two steps two steps – – quality of evidence quality of evidence – – strength of recommendation strength of recommendation

• • quality and strength can differ quality and strength can differ

• • profiles and summary of findings profiles and summary of findings

• • importance of values/preferences importance of values/preferences

Plan Plan • • GRADE background GRADE background • • two steps two steps

– – quality of evidence quality of evidence – – strength of recommendation strength of recommendation

• • evidence profiles evidence profiles

• • an exercise in applying GRADE an exercise in applying GRADE

Plan Plan • • GRADE background GRADE background • • two steps two steps

– – quality of evidence quality of evidence – – strength of recommendation strength of recommendation

• • quality and strength can differ quality and strength can differ

• • profiles and summary of findings profiles and summary of findings





• • application to breast cancer screening application to breast cancer screening – – contrast with USPSTF contrast with USPSTF

Summarizing recommendations Summarizing recommendations

• • clinicians need succinct summaries clinicians need succinct summaries

• • should include should include – – quality of evidence quality of evidence – – summaries of best estimates of effect summaries of best estimates of effect

• • all patient all patient- -important outcomes important outcomes – – strength of recommendations strength of recommendations

• • GRADE working group GRADE working group – – BMJ 2004 and 2008 BMJ 2004 and 2008

• • Is grading recommendations a good Is grading recommendations a good idea? idea?

• • Why? Why?

• • experience with grading experience with grading – – systems used? systems used?

Why Grade Why Grade Recommendations? Recommendations?

• • strong recommendations strong recommendations – – strong methods strong methods – – large precise effect large precise effect – – few down sides of therapy few down sides of therapy

• • weak recommendations weak recommendations – – weak methods weak methods – – imprecise estimate imprecise estimate – – small effect small effect – – substantial down sides substantial down sides

Which grading system to use? Which grading system to use? • • many available many available

– – Australian National and MRC Australian National and MRC – – Oxford Center for Evidence Oxford Center for Evidence- -based Medicine based Medicine – – Scottish Intercollegiate Guidelines (SIGN) Scottish Intercollegiate Guidelines (SIGN) – – US Preventative Services Task Force US Preventative Services Task Force – – American professional organizations American professional organizations

• • AHA/ACC, ACCP, AAP, Endocrine society, etc.... AHA/ACC, ACCP, AAP, Endocrine society, etc....

• • cause of confusion, dismay cause of confusion, dismay

A common international A common international grading system? grading system?

• • GRADE ( GRADE (G Grades of rades of r recommendation, ecommendation, a assessment, ssessment, d development and evelopment and e evaluation) valuation)

• • international group international group – – Australian NMRC, SIGN, USPSTF, WHO, NICE, Australian NMRC, SIGN, USPSTF, WHO, NICE,

Oxford CEBM, CDC, CC Oxford CEBM, CDC, CC

• • ~ 25 meetings over last ten years ~ 25 meetings over last ten years • • (~10 (~10 – – 50 attendants) 50 attendants)

GRADE Uptake GRADE Uptake Agencia sanitaria regionale, Bologna, Italia Agency for Health Care Research and Quality (AHRQ) Allergic Rhinitis and Group - Independent Expert Panel American Association for the study of liver diseases American College of Cardiology Foundation American College of Chest Physicians American College of Emergency Physicians American College of Physicians American Endocrine Society American Society of Gastrointestinal Endoscopy American society of Interventional Pain Physicians American Thoracic Society (ATS) BMJ Clinical Evidence British Medical Journal Canadian Agency for Drugs and Technology in Health Canadian Cardiovascular Society Canadian Task Force on Preventive Health Care Centers for Disease Control Cochrane Collaboration EBM Guidelines Finland Emergency Medical Services for Children National

Resource Center European Association for the Study of the Liver European Respiratory Society European Society of Thoracic Surgeons Evidence-based Nursing Sudtirol, Alta Adiga, Italy Finnish Office of Health Technology Assessment

German Agency for Quality in Medicine Heelth Inspectorate for Scotland Infectious Disease Society of America Japanese Society of Oral and Maxillofacial Radiology Joslin Diabetes Center Journal of Infection in Developing Countries Kaiser Permanente Kidney Disease International Guidelines Organization National and Gulf Centre for Evidence-based Medicine National Institute for Clinical Excellence (NICE) National Kidney Foundation Norwegian Knowledge Centre for the Health Services Ontario MOH Medical Advisory Secretariat Panama and Costa Rica National Clinical Guidelines Program Polish Institute for EBM Scottish Intercollegiate Guideline Network (SIGN) Society of Critical Care Medicine Society of Pediatric Endocrinology Society of Vascular Surgery Spanish Society of Family Practice (SEMFYC) Stop TB Diagnostic Working Group Surviving sepsis campaign Swedish Council on Technology Assessment in Health Care Swedish National Board of Health and Welfare University of Pennsylvania Health System for EB Practice UpToDate WINFOCUS World Allergy Organization World Health Organization (WHO)

What are we grading? What are we grading?

• • two components two components

• • quality of body of evidence quality of body of evidence – – extent to which confidence in estimate of extent to which confidence in estimate of

effect adequate to support decision effect adequate to support decision • • high, moderate, low, very low high, moderate, low, very low

• • strength of recommendation strength of recommendation • • strong and weak strong and weak

What are we grading? What are we grading?

• • two components two components

• • quality of body of evidence quality of body of evidence – – confidence in estimate of effect confidence in estimate of effect

• • high, moderate, low, very low high, moderate, low, very low

• • strength of recommendation strength of recommendation • • strong and weak strong and weak

Interpretation of quality Interpretation of quality • • High quality High quality— — Further research is very unlikely to Further research is very unlikely to

change our confidence in the estimate of effect change our confidence in the estimate of effect • • Moderate quality Moderate quality— — Further research is likely to Further research is likely to

have an important impact on our confidence in the have an important impact on our confidence in the estimate of effect and may change the estimate estimate of effect and may change the estimate

• • Low quality Low quality— — Further research is very likely to Further research is very likely to have an important impact on our confidence in the have an important impact on our confidence in the estimate of effect and is likely to change the estimate of effect and is likely to change the estimate estimate

• • Very low quality Very low quality— — Any estimate of effect is very Any estimate of effect is very uncertain uncertain

Interpretation of quality Interpretation of quality • • High: We are very confident that t High: We are very confident that the true effect lies he true effect lies

close to that of the estimate of the effect. close to that of the estimate of the effect.

• • Moderate: We are moderately confident in the effect Moderate: We are moderately confident in the effect estimate: The true effect is likely to be close to the estimate: The true effect is likely to be close to the estimate of the effect, but there is a possibility that it estimate of the effect, but there is a possibility that it is substantially different. is substantially different.

• • Low: Our confidence in the effect estimate is limited: Low: Our confidence in the effect estimate is limited: The true effect may be substantially different from the The true effect may be substantially different from the estimate of the effect. estimate of the effect.

• • Very low: We have very little confidence in the effect Very low: We have very little confidence in the effect estimate: The true effect is likely to be substantially estimate: The true effect is likely to be substantially different from the estimate of effect. different from the estimate of effect.

Health Care Question (PICO)

Systematic reviews

Studies

Outcomes

Important outcomes

Rate the quality of evidence for each outcome, across studies RCTs start high, observational studies start low ()Study limitations Imprecision Inconsistency of results Indirectness of evidence Publication bias likely

Final rating of quality for each outcome: high, moderate, low, or very low

(+) Large magnitude of effect Dose response Plausible confounders would ↓ effect when an effect is present or ↑ effect if effect is absent

Decide on the direction (for/against) and grade strength (strong/weak*) of the recommendation considering:

Quality of the evidence Balance of desirable/undesirable outcomes

Values and preferences Decide if any revision of direction or strength is

necessary considering: Resource use *also labeled “conditional” or “discretionary”

Rate overall quality of evidence (lowest quality among critical outcomes)

S1 S2 S3 S4

OC1 OC2 OC3 OC4

OC1 OC3 Critical outcomes

OC4

Generate an estimate of effect for each outcome

OC2

S5

Structured question Structured question

• patients: lymphoma patients at risk of developing chemotherapy-induced febrile neutropenia

• granulocyte colony-stimulating (G-CSF)

• alternative not using G-CSF

Structured question Structured question • patients:

– women considering breast cancer screening – age 40-9; 50 to 74; > 75 – no ñ risk genetic mutation chest radiation

• intervention – film mammography

• alternative – no screening

Need to define all patient Need to define all patient- -important outcomes important outcomes and evaluate their importance and evaluate their importance

• desirable consequences – reduction in breast cancer mortality

• undesirable consequences – false positive screening results – invasive procedures from positive results – complications of invasive procedures – unnecessary diagnosis and treatment

Determinants of quality Determinants of quality • • RCTs RCTs start high start high • • observational studies start low observational studies start low

• • what can lower quality? what can lower quality? – – detailed design and execution detailed design and execution – – inconsistency inconsistency – – indirectness indirectness – – imprecision imprecision – – reporting bias reporting bias


• • 5 limitations can lower quality 5 limitations can lower quality • • detailed design and execution detailed design and execution

– – concealment, blinding, loss to follow concealment, blinding, loss to follow- -up up

• • inconsistency inconsistency – – variability in results (heterogeneity) variability in results (heterogeneity)

• • publication bias publication bias

Determinants of quality Determinants of quality • • RCTs start high RCTs start high • • observational studies start low observational studies start low

• • 5 limitations can lower quality 5 limitations can lower quality • • Bias Bias

– – detailed design and execution detailed design and execution • • concealment, blinding, loss to follow concealment, blinding, loss to follow- -up up

– – publication bias publication bias

• • Imprecision Imprecision – – wide confidence intervals wide confidence intervals


• • limitations can lower quality? limitations can lower quality? • • Bias Bias

– – detailed design and execution detailed design and execution • • concealment, blinding, loss to follow concealment, blinding, loss to follow- -up up

• • Imprecision Imprecision – – wide confidence intervals wide confidence intervals

Determinants of quality Determinants of quality

• • RCTs RCTs start high start high

• • observational studies start low observational studies start low

• • limitations can lower quality? limitations can lower quality?

Determinants of quality Determinants of quality • • 5 limitations can lower quality 5 limitations can lower quality

• • risk of bias risk of bias – – concealment, blinding, loss to follow concealment, blinding, loss to follow- -up up

• • imprecision imprecision


• • Indirectness Indirectness • • publication bias publication bias

Determinants of quality Determinants of quality • • 5 limitations can lower quality 5 limitations can lower quality

• • risk of bias risk of bias – – concealment, blinding, loss to follow concealment, blinding, loss to follow- -up up

• • imprecision imprecision


• • publication bias publication bias

Risk of Bias Risk of Bias

• • well established well established – – concealment concealment – – intention to treat principle observed intention to treat principle observed – – blinding blinding – – completeness of follow completeness of follow- -up up

• • more recent more recent – – early stopping for benefit early stopping for benefit – – selective outcome reporting bias selective outcome reporting bias

Breast cancer risk of bias Breast cancer risk of bias • • most trials not concealed most trials not concealed

• • blinding blinding – – ? adjudication of ? adjudication of outome outome – – no other blinding no other blinding

• • ? loss to follow ? loss to follow- -up up

• • all trials rated as all trials rated as “ “fair fair” ” by USPSTF by USPSTF

Consistency of results Consistency of results

• • if inconsistency, look for explanation if inconsistency, look for explanation – – patients, intervention, outcome, methods patients, intervention, outcome, methods

• • judgment of consistency judgment of consistency – – variation in size of effect variation in size of effect – – overlap in confidence intervals overlap in confidence intervals – – statistical significance of heterogeneity statistical significance of heterogeneity – – I I 2 2

Relative Risk with 95% CI for Vitamin D Nonvertebral Fractures

Chapuy Chapuy et al, (2002) 0.85 (0.64, 1.13) et al, (2002) 0.85 (0.64, 1.13)

Pooled Random Effect Model 0.82 (0.69 to 0.98)

p= 0.05 for heterogeneity, I 2 =53%

Chapuy Chapuy et al, (1994) 0.79 (0.69, 0.92) et al, (1994) 0.79 (0.69, 0.92)

Lips et al, (1996) 1.10 (0.87, 1.39) Lips et al, (1996) 1.10 (0.87, 1.39)

Dawson Dawson Hughes et al, (1997) 0.46 (0.24, 0.88) Hughes et al, (1997) 0.46 (0.24, 0.88)

Pfeifer et al, (2000) 0.48 (0.13, 1.78) Pfeifer et al, (2000) 0.48 (0.13, 1.78)

Meyer et al, (2002) 0.92 (0.68, 1.24) Meyer et al, (2002) 0.92 (0.68, 1.24)

Trivedi Trivedi et al, (2003) 0.67 (0.46, 0.99) et al, (2003) 0.67 (0.46, 0.99)

'

'

'

'

'

'

'

'

0.1 1 10

Favours Vitamin D Favours Control

Relative Risk 95% CI

Relative Risk with 95% CI for Vitamin D (NonVertebral Fractures, Dose >400)

Chapuy et al, (1994) 0.70 (0.69, 0.92)

DawsonHughes et al, (1997) 0.46 (0.24, 0.88)

Pfeifer et al, (2000) .48 (0.13, 1.78)

Chapuy et al, (2002) 0.85 (0.64, 1.13)

Trivedi et al, (2003) 0.67 (0.46, 0.99)

Pooled Random Effect Mode 0.75 (0.63 to 0.89)

p= 0.26 for heterogeneity, I 2 =24%

'

'

'

'

'

'

0.1 1 10



Relative Risk with 95% CI for Vitamin D (NonVertebral Fractures, Dose = 400)

Lips et al (1996) 1.10 (0.87, 1.39)

Meyer et al (2002) 0.92 (0.68, 1.24)

Pooled Random Effect Mode 1.03 (0.86 to 1.24)

p = 0.35 heterogeneity, I 2 =0%

'

'

'

0.1 1 10



Meta-analysis of women 39 to 49 years of age

Should Should we we believe believe sub sub- -group group analysis analysis? ?

• • within within- -study study comparison comparison? ? No No • • large large difference difference in in effect effect Borderline Borderline • • unlikely unlikely chance chance Yes Yes, p = 0.006 , p = 0.006 • • consistent consistent across across studies studies Yes Yes • • a priori a priori hypothesis hypothesis Yes Yes • • one one of of small small number number hypotheses hypotheses Yes Yes • • biologically biologically compelling compelling Yes Yes

• • shall shall we we believe believe sub sub- -group group analysis analysis? ?

Directness of Evidence Directness of Evidence

• • differences in differences in – – patients patients – – interventions interventions – – comparators comparators

• • differences in outcomes differences in outcomes – – surrogates surrogates

Quality judgments: Directness Quality judgments: Directness

• • populations populations – – older, sicker or more co older, sicker or more co- -morbidity morbidity

• • interventions interventions – – new statins versus old new statins versus old

• • outcomes outcomes – – important versus surrogate outcomes important versus surrogate outcomes – – glucose control versus CV events glucose control versus CV events

Flatulence

Figure 6: Hierarchy of outcomes according to their patient Figure 6: Hierarchy of outcomes according to their patient- -importance to assess the importance to assess the effect of phosphate lowering drugs in patients with renal failur effect of phosphate lowering drugs in patients with renal failure and hyperphophatemia e and hyperphophatemia

Importance of endpoints

Critical for decision making

Important, but not critical for decision making

Of low patient importance 2

5

Pain due to soft tissue Calcification / function 6

Fractures 7

Myocardial infarction 8

Mortality 9

3

4

1

Coronary calcification

Ca 2+ /P Product

Surrogates of declining importance

Bone density

Ca 2+ /P Product

Soft tissue calcification

Ca 2+ /P Product

Lower by one level for

indirectness

Lower by two levels for

indirectness

Alendronate Risedronate

Placebo

Directness Directness interested in A versus B interested in A versus B

available data A available data A vs vs C, B C, B vs vs C C

Issues in directness breast Issues in directness breast cancer screening cancer screening

• • estimated screening effects in over 75 estimated screening effects in over 75 – – could also use observational studies could also use observational studies

• • no direct comparisons of screening no direct comparisons of screening intervals intervals

Imprecision Imprecision

• • small sample size small sample size – – small number of events small number of events

• • wide confidence intervals wide confidence intervals – – uncertainty about magnitude of effect uncertainty about magnitude of effect

• • primary criterion: primary criterion: – – would decisions differ at ends of CI would decisions differ at ends of CI

Meta-analysis of women 39 to 49 years of age

25% RRR, reduction in breast cancer deaths 1/1,000

4% RRR, reduction in breast cancer deaths 1/6,000

Imprecision Imprecision • • small sample size small sample size

– – small number of events small number of events

• • wide confidence intervals wide confidence intervals – – uncertainty about magnitude of effect uncertainty about magnitude of effect

• • problems problems – – analogy to stopping early analogy to stopping early – – lack of prognostic balance lack of prognostic balance

• • solution: optimal information size solution: optimal information size – – # of pts from conventional sample size # of pts from conventional sample size

calculation calculation – – specify CER, effect, specify CER, effect, α α, , β β, , Δ Δ

Fluoroquinolone prophylaxis in neutropenia: infection-related mortality

Total number of events: 47

Fluoroquinolone prophylaxis in neutropenia: infection-related mortality

sample size 1,002 sample size 1,002 α α 0.05, 0.05, β β 0.20, 0.20, Δ Δ 0.25, N = 6,000 0.25, N = 6,000

Stroke Stroke – – Fixed Effects Fixed Effects

Downgrading for precision Downgrading for precision

• • if OIS not met rate down for imprecision if OIS not met rate down for imprecision – – unless very large unless very large ss ss (? > 1,000 per group) (? > 1,000 per group)

• • if OIS met and CI exclude RR = 1, don if OIS met and CI exclude RR = 1, don‛‛t t downgrade downgrade

• • if OIS met and CI includes, RR = 1, if OIS met and CI includes, RR = 1, downgrade only if RR < 0.75 or > 1.25 downgrade only if RR < 0.75 or > 1.25

Precision Precision • • atrial fib at risk of stroke atrial fib at risk of stroke

• • warfarin increases serious warfarin increases serious gi gi bleeding bleeding – – 3% per year 3% per year

• • 1,000 patients 1 less stroke 1,000 patients 1 less stroke – – 30 more bleeds for each stroke prevented 30 more bleeds for each stroke prevented

• • 1,000 patients 100 less strokes 1,000 patients 100 less strokes – – 3 strokes prevented for each bleed 3 strokes prevented for each bleed

• • where is your threshold? where is your threshold? – – how many strokes in 100 with 3% bleeding? how many strokes in 100 with 3% bleeding?

0 1.0%

0 0.5% 1.0%

Example: Example: clopidogrel clopidogrel or ASA? or ASA? • • pts with threatened stroke pts with threatened stroke

• • RCT of RCT of clopidogrel clopidogrel vs vs ASA ASA – – 19,185 patients 19,185 patients

• ischaemic stroke, MI, or vascular death compared – 939 events (5∙32%) clopidogrel – 1021 events (5∙83%) with aspirin

• RR 0.91 (95% CI 0.83 – 0.99) (p=0∙043)

• • downgrade for precision? downgrade for precision?

0 1.0%

Clopidogrel or ASA for threatened vascular events

RCT 19,185 patients

1.7% - 0.9 – 0.1%

RR 0.91 (95% CI 0.83 – 0.99)

0

Non-inferiority

Avoiding misleading conclusions Avoiding misleading conclusions

• • analogy analogy – – SR accumulating data just as single trial SR accumulating data just as single trial

• • risk of spurious effect by stopping early risk of spurious effect by stopping early

• • how to avoid how to avoid – – insist on sufficient data insist on sufficient data – – optimal information size optimal information size

Avoiding misleading conclusions Avoiding misleading conclusions

• • optimal information size optimal information size – – # of pts from conventional sample size # of pts from conventional sample size

calculation calculation – – specify CER, effect, specify CER, effect, α α, , β β

• • alternative, number of events alternative, number of events – – ? 300 ? ? 300 ?

Criteria for NOT downgrading Criteria for NOT downgrading • • CI narrow enough to permit confident CI narrow enough to permit confident

recommendation for or against recommendation for or against

• • if positive benefit outcome, if positive benefit outcome, safeguard against false positive safeguard against false positive – – OIS OIS or or- - number threshold met (300) number threshold met (300)

Publication bias Publication bias

• • high likelihood could lower quality high likelihood could lower quality

• • when to suspect when to suspect • • number of small studies number of small studies • • industry sponsored industry sponsored

What can raise quality? What can raise quality? • • large magnitude can upgrade one level large magnitude can upgrade one level

– – very large two levels very large two levels

• • common criteria common criteria – – everyone used to do badly everyone used to do badly – – almost everyone does well almost everyone does well – – quick action quick action

• • hip replacement for hip osteoarthritis hip replacement for hip osteoarthritis

• • mechanical ventilation in respiratory failure mechanical ventilation in respiratory failure

Dose Dose- -response gradient response gradient

• • childhood lymphoblastic leukemia childhood lymphoblastic leukemia

• • risk for CNS malignancies 15 years after risk for CNS malignancies 15 years after cranial irradiation cranial irradiation

• no radiation: 1% (95% CI 0% to 2.1%) • 12 Gy: 1.6% (95% CI 0% to 3.4%) • 18 Gy: 3.3% (95% CI 0.9% to 5.6%).

Observational study starts low Observational study starts low What can move up? What can move up?

• plausible confounders strengthen association

• FP hospitals higher mortality than NFP hospitals – NFP treat sicker patients – FP treat wealthier patients

Quality assessment criteria Quality assessment criteria

Overall level of evidence Overall level of evidence • • most systems just use evidence about most systems just use evidence about

primary benefit outcome primary benefit outcome

• • but what about others (risk)? but what about others (risk)?

• • what to do? what to do?

• • options options – – ignore all but primary ignore all but primary – – weakest of any outcome weakest of any outcome – – some blended approach some blended approach – – weakest of critical outcomes weakest of critical outcomes

Quality Assessment

Summary of Findings

Quality Relative Effect (95% CI)

Absolute risk difference

Outcome Number of participants (studies)

Risk of Bias Consistency Directness Precision Reportin

g Bias

Myocardial infarction

10,125 (9)

No serious limitations

No serious imitations



Not detected High 0.71

(0.57 to 0.86) 1.5% fewer

(0.7% fewer to 2.1% fewer)

Mortality 10,205 (7)


Possiblly inconsistent

No serious limitations Imprecise Not

detected Moderate or low

1.23 (0.98 – 1.55)

0.5% more (0.1% fewer to 1.3% more)

Stroke 10,889 (5)

No serious limitaions



Possible imprecision

Not detected

Moderate or High

2.21 (1.37 – 3.55)

0.5% more (0.2% more to 1.3% more0

Beta blockers in non‐cardiac surgery

Quality Assessment Summary of Findings

Quality

Relative Risk

(95% CI) pvalue

Illustrative risks

Outcome No. of patients (studies)

Risk of Bias Inconsistency Indirectness Imprecision Publication Bias

Example control rate

Associated risk with PVL

Hospital mortality

1,664 (9)

Inability to blind. 2 trials stopped early with few events and large effects; were also confounded by ‘open lung’ strategies.

p = 0.07 I 2 = 45.6% Varied populations, interventions. Not robust in sensitivity analyses

Direct Precise Undetected Moderate (due

to inconsistency)

0.82 (0.68 – 0.99) p = 0.04

40% 32.8% (27.2 – 39.6)

Barotrauma 1,497 (7) Inability to blind.

p = 0.24 I 2 = 25.3% Varied populations, interventions

Direct Imprecise Undetected Moderate (due

to imprecision)

0.90 (0.66 – 1.24) p = 0.53

NS NS

Paralysis 1,202 (5) Inability to blind.

p = 0.004 I 2 = 59% Varied populations, interventions, measurements

Direct Precise Not assessed

Moderate (due to

inconsistency)

1.37 (1.04 – 1.82) p = 0.03

30% 41.1% (31.2 – 54.6)

Dialysis 173 (2) Inability to blind.

p = 0.26 I 2 = 22.8% Varied populations, interventions

Direct Imprecise Not assessed

Moderate (due to

imprecision)

1.76 (0.79 – 3.90) p = 0.16

NS NS

Pressure limited ventilation

Quality Assessment Summary of Findings

Quality

Relative Risk

(95% CI) pvalue

Illustrative risks

Outcome No. of patients (studies)

Risk of Bias Inconsistency Indirectness Imprecision Publication Bias

control rate

vaccinated rate

Zoster episodes

38,546 (1) No serious risk only one study Direct Precise Undetected High not reported

11.12 per 1,000 patient years

5.42 (difference

5.7 per 1,000 ptyears (p< 0.001)

Post herpetic neuralgia

38,546 (1)

No serious risk only one study

Direct Precise Undetected High not reported

1.38 per 1,000

patient- years

0.46 (difference

0.92 per 1,000 pt-

years (p< 0.001)

Serious adverse events

38,546 (1) No serious risk only one study Direct Precise Undetected High Not

reported

13 per 1,000 19 (difference

6 per 1,000)

Zoster vaccine

Population No. of participants (trials) †

Higher PEEP

Lower PEEP

Adjusted Relative Risk (95% CI; P-value) ‡

Adjusted Absolute Risk Difference (95% CI)

Quality

Patients with ARDS

1892 (3) 324/951 (34.1%)

368/941 (39.1%)

0.90 (0.81 to 1.00; 0.049)

-3.9% (-7.4% to -0.04%) High

Patients without ARDS

404 (3) 50/184 (27.2%)

41/220 (18.6%)

1.37 (0.98 to 1.92; 0.065)

6.9% (-0.4% to 17.1%) Moderate (imprecision)

High versus low PEEP in ALI and ARDS

Whipples procedure pancreatic cancer with or without duodenectomy

Patients or population: Anyone taking a long flight (lasting more than 6 hours) Settings: International air travel Intervention: Compression stockings 1

Comparison: Without stockings

Outcomes Illustrative comparative risks* (RANGE OF UNCERTAINTY)

Relative effect (95% CI)

Number of participants (studies)

Quality of the evidence (GRADE)

Comments

Assumed risk Corresponding risk

Without stockings With stockings (95% CI)

Symptomatic deep vein thrombosis (DVT)

See comment See comment Not estimable 2637 (9 studies)

See comment 0 participants developed symptomatic DVT in these studies.

Symptomatic deep vein thrombosis – surrogate symptomless deep vein thrombosis

Low risk population 2 RR 0.10 (0.04 to 0.25)

2637 (9 studies)

⊕⊕⊕ Moderate 3

Estimates of control group asymptomatic thrombosis from the primary studies range from 15 per 1,000 in low risk patients to 25 per 1,000 in high risk patients

5 per 10,000 0.5 per 10,000 (0 to 1)

High risk population 2

18 per 10,000 1.8 per 10,000 (0.5 to 4)

Superficial vein thrombosis

13 per 1000 6 per 1000 (2 to 15)

RR 0.45 (0.18 to 1.13)

1804 (8 studies)

⊕⊕⊕ Moderate 4

Diagnostic tests

• same logic as for treatment

• judges quality of evidence NOT for accuracy, but for change in patient-important outcome

• ideally establish through RCT – focus on patient-important outcome – screening RCTs (breast cancer, colon cancer)

• most of time not available – new complexities, process in evolution

Study designs

Study designs II

Test accuracy is a surrogate for patient important outcomes

• When clinicians think about diagnostic tests, they focus on their accuracy

• Underlying assumption: obtaining a better idea of whether a target condition is present or absent will result in superior patient management and improved outcome.

Example of new test and reference test or strategy

Putative benefit of new test

Diagnostic accuracy Patient Outcomes and expected impact on management for the following test outcomes

Sensitivity Specificity

True positives

False positives

True negatives

False negatives

Helical CT for renal calculus compared with intravenous pyeolgram

Detection of more (but smaller) calculi

greater equal Presumed influence on patient important outcomes

Certain benefit for larger stones, for smaller stones the benefit is less clear and unnecessary treatment can result

Likely detriment from unnecessary additional invasive tests

Almost certain benefit from avoiding unnecessary tests

Likely detriment for large stones, less certain for small stones More testing

Directness of the evidence (test results) for patient‐ important outcomes

Some uncertainty

No uncertainty

No uncertainty

Major uncertainty

Balance between presumed patient outcomes, complications and cost: Less complications and downsides compared to IVP would support the new test’s usefulness, but the balance between desirable and undesirable effect not clear in view of the uncertain consequences of identifying smaller stones.

Strength of recommendations Strength of recommendations • • degree of confidence that desirable degree of confidence that desirable

effects of adhering to recommendation effects of adhering to recommendation outweigh undesirable effects. outweigh undesirable effects.

• • strong recommendation strong recommendation – – benefits clearly outweigh risks/hassle/cost benefits clearly outweigh risks/hassle/cost – – risk/hassle/cost clearly outweighs benefit risk/hassle/cost clearly outweighs benefit

Strength of recommendations Strength of recommendations

• • degree of confidence that desirable degree of confidence that desirable effects of adhering to recommendation effects of adhering to recommendation outweigh undesirable effects. outweigh undesirable effects.


• • what can downgrade strength? what can downgrade strength?

Strength of Recommendation Strength of Recommendation


• • what can downgrade strength? what can downgrade strength?

• • low quality evidence low quality evidence

• • close balance between up and downsides close balance between up and downsides

Grades Translations Strong recommendations • Just do it • Virtually all well informed individuals would want

the intervention and only a small proportion would not

• Most individuals should receive the intervention • Use of the intervention according to the

guideline could be used as a quality criterion or performance indicator

Grades Translations Weak recommendation • Examine the evidence yourself • The majority of well informed individuals

would want the intervention, but a substantial proportion would not

• Many but not all individuals should receive the intervention

• The intervention is not a candidate for a quality criterion or performance indicator.

Risk/Benefit tradeoff Risk/Benefit tradeoff

• • aspirin after myocardial infarction aspirin after myocardial infarction – – 25% reduction in relative risk 25% reduction in relative risk – – side effects minimal, cost minimal side effects minimal, cost minimal – – benefit obviously much greater than benefit obviously much greater than

risk/cost risk/cost

• • warfarin in low risk atrial fibrillation warfarin in low risk atrial fibrillation – – warfarin reduces stroke warfarin reduces stroke vs vs ASA by 50% ASA by 50% – – but if risk only 1% per year, ARR 0.5% but if risk only 1% per year, ARR 0.5% – – increased bleeds by 1% per year increased bleeds by 1% per year

Strength of Recommendations • Resuscitate fast in septic patient

- do it!

• Prone ventilation in failing patient with ARDS – Probably do it – Probably do not do it

Strength of Strength of Recommendations Recommendations

Aspirin after MI Aspirin after MI – – do it do it

Warfarin rather than ASA in Warfarin rather than ASA in Afib Afib -- -- probably do it probably do it -- -- probably don probably don‛ ‛t do it t do it

Significance of strong Significance of strong vs vs weak weak • • variability in patient preference variability in patient preference

– – strong, almost all same choice (> 90%) strong, almost all same choice (> 90%) – – weak, choice varies appreciably weak, choice varies appreciably

• • interaction with patient interaction with patient – – strong, just inform patient strong, just inform patient – – weak, ensure choice reflects values weak, ensure choice reflects values

• • use of decision aid use of decision aid – – strong, don strong, don‛‛t bother t bother – – weak, use the aid weak, use the aid

• • quality of care criterion quality of care criterion – – strong, consider strong, consider – – weak, don weak, don‛‛t consider t consider

USPSTF documents - clinical summary - supporting article (decision analysis) - evidence summary - recommendations

For women age 40 – 49 we suggest NOT screening. (Weak recommendation based, moderate quality evidence. 2B)

Values and Preferences: This recommendation places a relatively low value on a very small, uncertain mortality decrease and reflects concerns with false positive results, unnecessary biopsies, and unnecessary dx of breast cancer Women who place a higher value on a small reduction in breast cancer mortality and are less concerned about the undesirable consequences will choose screening

For women age 50 - 74 we suggest screening. (Weak recommendation, moderate quality evidence. 2B)

Women who do not place a high value on a small reduction in breast cancer mortality and are concerned about false positive results, unnecessary biopsies, and unnecessary diagnosis of breast cancer will decline screening

Using GRADE no insufficient - no recommendation or recommend for or against

Breast self-examination: We recommend against breast self-examination. (strong recommendation, high/moderate quality evidence)

Weak recommendation Weak recommendation

• • practice will vary practice will vary – – according to what? according to what?

• • interpretation of evidence interpretation of evidence – – clopidogrel clopidogrel in stroke in stroke

• • patients patients‛‛ values and preferences values and preferences – – atrial fibrillation atrial fibrillation

• • inclination to gamble (risk aversion) inclination to gamble (risk aversion) – – HRT HRT

When evidence is low quality When evidence is low quality

• • choice more preference dependent choice more preference dependent

• • risk aversion risk aversion

• • steroids for pulmonary fibrosis steroids for pulmonary fibrosis – – low quality evidence in support of low quality evidence in support of

benefit benefit – – high quality evidence of toxicity high quality evidence of toxicity

When evidence is low quality When evidence is low quality • • recommendation to the hopeful patient recommendation to the hopeful patient

– – I I‛‛m likely to deteriorate m likely to deteriorate – – if something might work, let if something might work, let‛‛s try it s try it – – damn the torpedoes damn the torpedoes

• • recommendation to the fearful patient recommendation to the fearful patient – – doctor, you mean you know it doctor, you mean you know it‛‛s toxic s toxic

• • diabetes, skin changes, body diabetes, skin changes, body habitus habitus, infection, , infection, osteoporosis osteoporosis

– – you don you don‛‛t know for sure it works? t know for sure it works? – – are you crazy? are you crazy?

• • weak recommendation mandated weak recommendation mandated

Strong recommendation Strong recommendation when evidence is low quality? when evidence is low quality? • • known benefit, strong recommendation for known benefit, strong recommendation for

one of two alternatives one of two alternatives – – antipyretics in children with chickenpox antipyretics in children with chickenpox – – but which one: ASA or acetaminophen but which one: ASA or acetaminophen

• • benefit: high quality evidence of benefit: high quality evidence of equivalence equivalence

• • harm: low quality evidence that harm harm: low quality evidence that harm differs appreciably differs appreciably – – Reye syndrome from ASA Reye syndrome from ASA

• • strong recommendation for acetaminophen strong recommendation for acetaminophen

Strong recommendation when Strong recommendation when evidence is low quality? evidence is low quality?

• • Blastomycosis Blastomycosis – – low quality evidence low quality evidence amphotericin amphotericin more more

effective than effective than itraconazole itraconazole – – high quality evidence more toxic high quality evidence more toxic

• • patients with life threatening patients with life threatening blasto blasto – – life and death situation life and death situation – – strong recommendation for strong recommendation for ampho ampho

Strong recommendation when Strong recommendation when evidence is low quality? evidence is low quality?

• • head to toe CT scanning head to toe CT scanning – – prevent cancer deaths prevent cancer deaths

• • very low quality evidence of benefits very low quality evidence of benefits • • moderate quality evidence re risks, moderate quality evidence re risks,

high re costs high re costs • • strong recommendation against strong recommendation against

Presentation Presentation • • strong and weak strong and weak

– – discomfort with discomfort with “ “weak weak” ” – – alternative wording: discretionary, conditional alternative wording: discretionary, conditional

• • strong strong – – “ “we recommend we recommend”… ”…

• • discretionary discretionary – – “ “we suggest we suggest…” …”

• • never never – – we recommend (or suggest) you consider we recommend (or suggest) you consider… …

• • always: quality of evidence and grade always: quality of evidence and grade

When (not to) GRADE When (not to) GRADE “ “good to remind/alert good to remind/alert” ”

• • if no systematic review undertaken if no systematic review undertaken

• • no sensible person would consider contrary no sensible person would consider contrary – – We recommend that the patient, and the clinician We recommend that the patient, and the clinician

responsible for the patient responsible for the patient‛‛s care, should be made s care, should be made aware of any change in a prescribed medication, aware of any change in a prescribed medication, including change to a generic drug including change to a generic drug

• • very general (not sufficiently specific) very general (not sufficiently specific) – – We suggest that long We suggest that long- -term maintenance term maintenance

immunosuppression immunosuppression be tailored to individual patient be tailored to individual patient‛‛s s adverse events or risk of adverse events adverse events or risk of adverse events

Explicit comparator Explicit comparator

• • we recommend hourly urine volume we recommend hourly urine volume measurement for at least 24 hours measurement for at least 24 hours – – in contrast to every 2 hours, every 3 in contrast to every 2 hours, every 3… …? ?

• • we suggest measuring serum creatinine in we suggest measuring serum creatinine in all all KTRs KTRs at least at least – – daily for 7 days daily for 7 days – – 2 to 3 X per week for weeks 2 to 4 2 to 3 X per week for weeks 2 to 4 – – every 2 weeks for months 4 to 6 every 2 weeks for months 4 to 6

Value and preference Value and preference statements statements

• • underlying values and preferences underlying values and preferences always present always present

• • sometimes crucial sometimes crucial

• • important to make explicit important to make explicit

Values and preferences Values and preferences

Stroke guideline: patients with TIA clopidogrel over aspirin (Grade 2B).

Underlying values and preferences: This recommendation to use clopidogrel over aspirin places a relatively high value on a small absolute risk reduction in stroke rates, and a relatively low value on minimizing drug expenditures.

Values and preferences Values and preferences

peripheral vascular disease: aspirin be used instead of clopidogrel (Grade 2A).

Underlying values and preferences: This recommendation places a relatively high value on avoiding large expenditures to achieve small reductions in vascular events.

Flavanoids for Hemorrhoids Flavanoids for Hemorrhoids • • venotonic venotonic agents agents

– – mechanism unclear, increase venous return mechanism unclear, increase venous return

• • popularity popularity – – 90 90 venotonics venotonics commercialized in France commercialized in France – – none in Sweden and Norway none in Sweden and Norway – – France 70% of world market France 70% of world market

• • possibilities possibilities – – French misguided French misguided – – rest of world missing out rest of world missing out

Systematic Review Systematic Review • • 14 trials, 1432 patients 14 trials, 1432 patients • • key outcome key outcome

– – risk not improving/persistent symptoms risk not improving/persistent symptoms – – 11 studies, 1002 patients, 375 events 11 studies, 1002 patients, 375 events – – RR 0.4, 95% CI 0.29 to 0.57 RR 0.4, 95% CI 0.29 to 0.57

• • minimal side effects minimal side effects

• • is France right? is France right? • • what is the quality of evidence? what is the quality of evidence?

What can lower quality? What can lower quality?

• • risk of bias risk of bias – – lack of detail re concealment lack of detail re concealment – – questionnaires not validated questionnaires not validated

• • indirectness indirectness – – no problem no problem

• • inconsistency, need to look at the inconsistency, need to look at the results results

Review : Phlebotonics for hemorrhoids Comparison: 01 Venotonics vs placebp Outcome: 08 Overall improvement: no improvement/some improvement

Study RR (random) Weight RR (random) or subcategory log[RR] (SE) 95% CI % 95% CI

01 Up to seven days Chauvenet 0.8916 (0.2376) 12.67 0.41 [0.26, 0.65] Cospite 2.2073 (0.6117) 5.51 0.11 [0.03, 0.36] Thanapongsathorn 0.4308 (0.2985) 11.18 0.65 [0.36, 1.17]

Subtotal (95% CI) 29.36 0.37 [0.18, 0.77] Test for heterogeneity: Chi² = 6.92, df = 2 (P = 0.03), I² = 71.1% Test for overall effect: Z = 2.67 (P = 0.008)

02 Up to four w eeks Annoni F 1.6094 (0.7073) 4.50 0.20 [0.05, 0.80] Clyne MB 0.9943 (0.3983) 8.94 0.37 [0.17, 0.81] Pirard J 1.1712 (0.3086) 10.94 0.31 [0.17, 0.57] Thanapongsathorn 1.1087 (1.1098) 2.18 0.33 [0.04, 2.91] Thorp 0.2624 (0.3291) 10.46 1.30 [0.68, 2.48] Titapan 0.8916 (0.3691) 9.56 0.41 [0.20, 0.85] Wijayanegara 0.5978 (0.1375) 14.97 0.55 [0.42, 0.72]

Subtotal (95% CI) 61.54 0.48 [0.32, 0.72] Test for heterogeneity: Chi² = 13.87, df = 6 (P = 0.03), I² = 56.7% Test for overall effect: Z = 3.57 (P = 0.0004)

03 Further than four w eeks Godeberg 1.7719 (0.3906) 9.10 0.17 [0.08, 0.37]

Subtotal (95% CI) 9.10 0.17 [0.08, 0.37] Test for heterogeneity: not applicable Test for overall effect: Z = 4.54 (P < 0.00001)

Total (95% CI) 100.00 0.40 [0.29, 0.57] Test for heterogeneity: Chi² = 28.66, df = 10 (P = 0.001), I² = 65.1% Test for overall effect: Z = 5.14 (P < 0.00001)

0.001 0.01 0.1 1 10 100 1000

Favours treatment Favours control

Publication bias? Publication bias?

• • size of studies size of studies – – 40 to 234 patients, most around 100 40 to 234 patients, most around 100

• • all industry sponsored all industry sponsored

Review : Phlebotonics for hemorrhoids Comparison: 01 Venotonics vs placebp Outcome: 08 Overall improvement: no improvement/some improvement

0.001 0.01 0.1 1 10 100 1000

0.0

0.4

0.8

1.2

1.6

RR (fixed)

What can lower quality? What can lower quality? • • detailed design and execution detailed design and execution

– – lack of detail re concealment lack of detail re concealment – – questionnaires not validated questionnaires not validated

• • inconsistency inconsistency – – almost all show positive effect, trend almost all show positive effect, trend – – heterogeneity p < 0.001; I heterogeneity p < 0.001; I 2 2 65.1% 65.1%

• • indirectness indirectness • • imprecision imprecision

– – RR 0.4, 95% CI 0.29 to 0.57 RR 0.4, 95% CI 0.29 to 0.57

• • reporting bias reporting bias – – 40 to 234 patients, most around 100 40 to 234 patients, most around 100

Recommendation Recommendation

• • for clinician for clinician – – offer to patient offer to patient – – don don‛‛t offer to patient? t offer to patient?

• • strength of recommendation strength of recommendation – – strong or weak strong or weak

• • for the funding body for the funding body – – publicly funded publicly funded – – not publicly funded not publicly funded – – strong or weak? strong or weak?

Is France right? Is France right?

• • recommendation recommendation – – yes yes – – no against use no against use

• • strength strength – – strong strong – – weak weak

Resource Use Resource Use

• • why not cost? why not cost? - - may lead to focus on cost of intervention may lead to focus on cost of intervention

rather than downstream resource use rather than downstream resource use - - resource use emphasizes alternative resource use emphasizes alternative

uses of resources (opportunity cost) uses of resources (opportunity cost)

Resource Use just another Resource Use just another outcome? outcome?

• • yes and no yes and no

• • who benefits? who benefits? – – other outcomes usually clear other outcomes usually clear – – costs borne by different payers costs borne by different payers

• • across societies and within (age) across societies and within (age)

• • some argue costs aren some argue costs aren‛‛t relevant to t relevant to clinicians when third party payer clinicians when third party payer

Why resource use different Why resource use different

• • costs vary much more than other outcomes costs vary much more than other outcomes – – across jurisdictions across jurisdictions – – within jurisdictions within jurisdictions – – over time over time

• • even when resource use the same, even when resource use the same, implications may differ implications may differ – – year year‛‛s supply of expensive drug s supply of expensive drug – – nurses nurses‛‛ salary in U.S., 6 in Poland, 30 in China salary in U.S., 6 in Poland, 30 in China

Why resource use different Why resource use different • • opportunity cost differs by perspective opportunity cost differs by perspective

• • hospital pharmacy, fixed budget hospital pharmacy, fixed budget – – new expensive drug, clear what give up new expensive drug, clear what give up

• • envelope public spending envelope public spending – – more on health, less on education, social more on health, less on education, social

services services – – will refraining from spending on drugs will refraining from spending on drugs

really mean more for other services? really mean more for other services? – – should envelope include military spending? should envelope include military spending?

Implications Implications • • unbearable lightness of resource use unbearable lightness of resource use • • consider balance of desirable and consider balance of desirable and

undesirable before considering undesirable before considering resource use resource use

• • may decide not to consider resource may decide not to consider resource use at all use at all – – intervention not useful intervention not useful – – desirable consequences >>>> undesirable desirable consequences >>>> undesirable – – relevant only when difference small relevant only when difference small

Similarities with other Similarities with other outcomes outcomes

• • only consider important resource use only consider important resource use

• • need estimate of difference between need estimate of difference between trt trt and control and control

• • explicit judgments about the quality of the explicit judgments about the quality of the evidence, special judgments evidence, special judgments – – perspective perspective – – how to judge quality of evidence how to judge quality of evidence – – ? use of economic model ? use of economic model

Evidence summary Evidence summary • • includes quality of evidence, summary includes quality of evidence, summary

of findings of findings – – “ “balance sheet balance sheet” ”, special form of grade , special form of grade

profile profile

• • resource use and not just costs resource use and not just costs – – can judge whether resource use can judge whether resource use

applicable to local setting applicable to local setting – – focus on focus on coss coss relevant to them relevant to them

(pharmacy) (pharmacy) – – apply unit costs to local setting apply unit costs to local setting

Example question Example question • • patients patients

– – women with pre women with pre- -eclampsia eclampsia

• • intervention intervention – – intravenous magnesium intravenous magnesium

• • RCT done in 33 countries RCT done in 33 countries – – over 9,000 patients over 9,000 patients

• • for presentation of resource use evidence for presentation of resource use evidence need to specify perspective need to specify perspective – – health system health system

Quality assessment

Studies Design Limitations Inconsistency Indirectness Imprecision No of patients

Relative effect

(95% CI)

Quality

Eclampsia

Duley 2003 RCT No one trial only No No 9,992 RR 0.41 (0.290,58)

High

Maternal death

Duley 2003 RCT No one trial only No Imprecision 9,992 RR 0.54 (0.261,10)

Moderate

Quality assessment

Design Limita tions

Inconsis tency

Indirect ness

Impre cision

Resources Costs per patient

Studies per patient (US $; year 2001)

Place bo MgSO4 Placebo MgSO4

Magnesium sulphate

High GNI 0 6 0 20

Simon 2005 Middle GNI RCT No one trial only No No 0 6 0 3 High

Low GNI 0 6 0 5

Administration of the drug

High GNI 0 1 0 66

Simon 2005 Middle GNI RCT No one trial only No No 0 1 0 14 High

Low GNI 0 1 0 8

Other hospital resources a, b

High GNI Large

variationre sources c

NA NA 12,839 12,818

Simon 2005 Middle GNI RCT No one trial only No NA NA 1,412 1,416 Modera

te

Low GNI NA NA 155 157

Outcomes

Typical control group risk

Typical absolute effect (95% CI)

Relative effect (95% CI)

Nr. of participants (studies)

Quality of the evidence

Comments

Clinical outcomes

Eclampsia Severe RR 0.41 (0.29 0.58)

11,444 ⊕⊕⊕⊕ High

27 per 1,000 16 fewer per 1,000 (11 to 19)

Not severe

15 per 1,000 9 fewer per 1,000 (6 to 11)

Maternal death Severe RR 0.54 (0.26 1.10)

10,795 ⊕⊕⊕ Moderate 2

6 per 1,000 3 fewer per 1,000 (0.6 more to 4 fewer)

Not severe

3 per 1,000 1 fewer per 1,000 (0.3 more to 2 fewer)

Side effects 46 per 1,000 3 196 more per 1,000 (165 to 231)

RR 5.26 (4.59 6.03)

9.992 ⊕⊕⊕⊕ High

Mostly flushing. Other side effects include nausea, vomiting, slurred speech, muscle weakness, dizziness, drowsiness, confusion and headache.

Magnesium sulphate ampoules

0 6 10 ml. ampoules per woman

9.996 ⊕⊕⊕⊕ High

Cost High GNI Middle GNI Low GNI

$20 more per patient $ 3 more per patient $ 5 more per patient

Administration of magnesium sulphate

0 1 per woman 9.996 ⊕⊕⊕⊕ High


$66 per patient $14 per patient $ 8 per patient

Resources for administering magnesium sulphate included midwife time (main cost), intravenous cannula/needle, syringe, IV fluids, drug.

Other hospital resources Varied widely 9.996 ⊕⊕⊕ Moderate 5

There was large variation in the use of other hospital resources in both intervention and control groups.


$12,839 $ 1,416 $ 157

$20 less per woman (0 to 60)

$ 4, less per woman (0 to 10)

$ 2 less per woman (1 to 3)

Other hospital costs have been adjusted based on the influence of eclampsia to control for the many other factors that influenced these costs.

Resource use from the perspective of the health system

control grop difference trt vs control

Issues in resource use Issues in resource use • • broad perspective desirable broad perspective desirable

– – narrow perspectives ignore much resource use narrow perspectives ignore much resource use – – users can pick costs relevant to them users can pick costs relevant to them – – either health care system or societal either health care system or societal

• • indirect costs controversial indirect costs controversial

• • indirect evidence of resources use indirect evidence of resources use – – costs only reported costs only reported – – RCT but doesn RCT but doesn‛‛t reflect practice t reflect practice

• • ulcer prevention everyone gets repeat endoscopy ulcer prevention everyone gets repeat endoscopy

Quality of evidence for Quality of evidence for resource use resource use

• • rules basically the same rules basically the same – – RCTs RCTs start high, observational low start high, observational low

• • may need multiple sources of evidence may need multiple sources of evidence – – RCTs RCTs may not fully report resource use may not fully report resource use

• • variation across settings variation across settings – – RCT may not reflect real world RCT may not reflect real world – – time frame may extend beyond trial time frame may extend beyond trial

• • different quality for different resources different quality for different resources – – mag mag sulphate sulphate versus hospital resources versus hospital resources

Formal economic models Formal economic models • • limitations limitations

– – supported by industry, biased supported by industry, biased – – setting specific setting specific – – reduces transparency reduces transparency – – if evidence low quality, speculative if evidence low quality, speculative – – often many assumptions often many assumptions

• • solution: develop own model solution: develop own model – – OK if you are NICE with lots of resources OK if you are NICE with lots of resources

• • even so, don even so, don‛‛t include in profile t include in profile

Costs versus affordability Costs versus affordability

• • intervention may be intervention may be “ “cost cost- -effective effective” ” – – $10,000 per $10,000 per qaly qaly gained gained

• • but if applicable to huge proportion but if applicable to huge proportion of population, may still be of population, may still be unaffordable unaffordable

healthy asymptomatic postmenopausal healthy asymptomatic postmenopausal qomwn qomwn: : HRT in 1992? HRT in 1992?

Possible benefits Possible benefits – – CHD, Hip fracture, Colorectal cancer CHD, Hip fracture, Colorectal cancer

Possible harms Possible harms – – Breast cancer Breast cancer – – Stroke Stroke – – Thrombosis Thrombosis – – Gall bladder disease Gall bladder disease

Can GRADE lead to change?

Evidence profile: Quality assessment Evidence profile: Quality assessment Oestrogen + progestin for prevention Oestrogen + progestin for prevention

in 1992 (before WHI and HERS) in 1992 (before WHI and HERS)

Oestrogen + progestin versus usual care

Oestrogen + progestin for Oestrogen + progestin for prevention after WHI and HERS prevention after WHI and HERS

Postulate

• major work in preparing guideline/HTA assessment is systematic review

• If already doing this, GRADE framework should add little

• history: Rolls-Royce and Volkswagen

VW and RR VW and RR appraoches appraoches

• • Rolls Royce (NICE) Rolls Royce (NICE) – – systematic review for every outcome systematic review for every outcome – – production of evidence profiles production of evidence profiles – – involvement of multiple constituencies involvement of multiple constituencies

• • including patients including patients – – inclusion of economic analysis inclusion of economic analysis

• • cost $1 million per guideline cost $1 million per guideline

MOPED GRADE MOPED GRADE • • UpToDate UpToDate

– – 5,000 graded recommendations 5,000 graded recommendations

• • generate PICO (informal) generate PICO (informal) – – no formal rating of outcome importance no formal rating of outcome importance

• • use of existing reviews, primary studies use of existing reviews, primary studies – – no new evidence syntheses no new evidence syntheses

• • quality for key outcomes quality for key outcomes – – 5 reasons rating down, 3 up 5 reasons rating down, 3 up – – no new evidence profiles, no new evidence profiles, SoF SoF tables tables

• • recommendations recommendations – – strong or weak, consider 3 factors strong or weak, consider 3 factors – – value and preference statements value and preference statements

ACCP • formal structured questions

• no formal rating of outcome importance – trying to change

• hit-and miss systematic reviews – largely only available ones

• hit-and-miss individual study evidence summaries

• rare evidence profiles – trying to change

VW approach VW approach

• • take systematic reviews if available take systematic reviews if available

• • if not, review key, accessible evidence if not, review key, accessible evidence

• • no meta no meta- -analysis if not done analysis if not done

• • no evidence profiles no evidence profiles

• • small group make expert small group make expert judgement judgement

Conclusion Conclusion

• • clinicians, policy makers need summaries clinicians, policy makers need summaries – – quality of evidence quality of evidence – – strength of recommendations strength of recommendations

• • explicit rules explicit rules – – transparent, informative transparent, informative

• • GRADE GRADE – – simple, transparent, systematic simple, transparent, systematic – – increasing wide adoption increasing wide adoption

pplanlan - mcmaster university evidence-based clinical...

Documents