how grade could help to implement the evidence holger schünemann, md, phd professor and chair,...

47
HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor of Medicine Michael Gent Chair in Healthcare Research McMaster University, Hamilton, Canada

Upload: ophelia-lindsey

Post on 25-Dec-2015

212 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE

Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & BiostatisticsProfessor of MedicineMichael Gent Chair in Healthcare ResearchMcMaster University, Hamilton, Canada

Page 2: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

ContentPulmonary rehabilitation compared to usual community care for COPD with recent exacerbation

Bibliography: Puhan M, et al. Pulmonary rehabilitation following exacerbations of chronic obstructive pulmonary disease. Cochrane Database of Systematic Reviews 2010, Issue 11.

Outcomes No of Participants(studies)Follow up

Quality of the evidence(GRADE)

Relative effect(95% CI)

Anticipated absolute effects

Risk with Usual community care

Risk difference with Pulmonary rehabilitation (95% CI)

Hospital admission

250(6 studies)3-18 months

⊕⊕⊕⊕HIGH

OR 0.22 (0.08 to 0.58)

405 per 1000 275 fewer per 1000(from 122 fewer to 353 fewer)

Mortality 110(3 studies)3-48 months

⊕⊕⊕⊝MODERATE2

due to imprecision

OR 0.28 (0.1 to 0.84)

Low1

100 per 1000 70 fewer per 1000(from 15 fewer to 89 fewer)

High1

500 per 1000 281 fewer per 1000(from 43 fewer to 409 fewer)

Quality of life (CRQ) dyspneaChronic Respiratory Questionnaire3. Scale from: 1 to 7.

258(5 studies)12 and 76 weeks

⊕⊕⊕⊝MODERATE4

due to imprecision

The mean quality of life (sgrq) total in the control groups was3.1

The mean quality of life (crq) dyspnea in the intervention groups was0.97 higher(0.35 to 1.58 higher)

Quality of life (SGRQ) totalSt George's Respiratory Questionnaire5. Scale from: 0 to 100.

127(3 studies)12 and 26 weeks

⊕⊕⊕⊝MODERATE4

due to imprecision

The mean quality of life (sgrq) total in the control groups was50

The mean quality of life (sgrq) total in the intervention groups was9.88 lower(5.37 to 14.4 lower)

Ambulation (as measured by 6 min walking distance)distance in meters6

299(6 studies)1 - 208 weeks7

⊕⊕⊕⊝MODERATE4,8

due to imprecision

The mean ambulation (as measured by 6 min walking distance) in the intervention groups was77.7 higher(12.21 to 143.2 higher)

Resource use - not reported

- - - See footnote See footnote

Systematic Reviews & GRADE

Evidence & judgments

Recommendation, health policy & implementation

Page 3: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

ContentPulmonary rehabilitation compared to usual community care for COPD with recent exacerbation

Bibliography: Puhan M, et al. Pulmonary rehabilitation following exacerbations of chronic obstructive pulmonary disease. Cochrane Database of Systematic Reviews 2010, Issue 11.

Outcomes No of Participants(studies)Follow up

Quality of the evidence(GRADE)

Relative effect(95% CI)

Anticipated absolute effects

Risk with Usual community care

Risk difference with Pulmonary rehabilitation (95% CI)

Hospital admission

250(6 studies)3-18 months

⊕⊕⊕⊕HIGH

OR 0.22 (0.08 to 0.58)

405 per 1000 275 fewer per 1000(from 122 fewer to 353 fewer)

Mortality 110(3 studies)3-48 months

⊕⊕⊕⊝MODERATE2

due to imprecision

OR 0.28 (0.1 to 0.84)

Low1

100 per 1000 70 fewer per 1000(from 15 fewer to 89 fewer)

High1

500 per 1000 281 fewer per 1000(from 43 fewer to 409 fewer)

Quality of life (CRQ) dyspneaChronic Respiratory Questionnaire3. Scale from: 1 to 7.

258(5 studies)12 and 76 weeks

⊕⊕⊕⊝MODERATE4

due to imprecision

The mean quality of life (sgrq) total in the control groups was3.1

The mean quality of life (crq) dyspnea in the intervention groups was0.97 higher(0.35 to 1.58 higher)

Quality of life (SGRQ) totalSt George's Respiratory Questionnaire5. Scale from: 0 to 100.

127(3 studies)12 and 26 weeks

⊕⊕⊕⊝MODERATE4

due to imprecision

The mean quality of life (sgrq) total in the control groups was50

The mean quality of life (sgrq) total in the intervention groups was9.88 lower(5.37 to 14.4 lower)

Ambulation (as measured by 6 min walking distance)distance in meters6

299(6 studies)1 - 208 weeks7

⊕⊕⊕⊝MODERATE4,8

due to imprecision

The mean ambulation (as measured by 6 min walking distance) in the intervention groups was77.7 higher(12.21 to 143.2 higher)

Resource use - not reported

- - - See footnote See footnote

Systematic Reviews & GRADE

Evidence & judgments

Recommendation, health policy & implementation

Examples and summary from leading/co-leading 15 guideline projects

• 11 World Health Organization

• World Allergy Organization, Allergic Rhinitis in Asthma, American Thoracic Society (2)

Page 4: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Guideline development Process

Page 5: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Systematic review

Guideline development

PICO

OutcomeOutcomeOutcomeOutcome

Formulate

question

Rate

importa

nce

Critical

Important

Critical

Not important

Create

evidence

profile with

GRADEpro

Summary of findings & estimate of effect for each outcome

Grade overall quality of

evidence across outcomes based

on lowest quality of critical outcomes

Panel

Randomization increases initial

quality

1. Risk of bias2. Inconsisten

cy3. Indirectnes

s4. Imprecision5. Publication

bias

Gra

de d

own

Gra

de u

p 1. Large effect

2. Dose response

3. Confounders

Rate quality

of evidence

for each

outcomeSelect

outcomes

Very low

LowModerate

High

Formulate recommendations:• For or against (direction)• Strong or conditional/weak

(strength)

By considering: Quality of evidence Balance benefits/harms Values and preferences

Revise if necessary by considering: Resource use (cost)

• “We recommend using…”• “We suggest using…”• “We recommend against using…”• “We suggest against using…”

Outcomes

across

studies

Page 6: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Outcome generation and selection• Multidisciplinary panels

– Researchers, epidemiologists, public health officers, methodologists, patient representatives…

• Delphi process – 3 rounds

1. List of possible outcomes from literature– Panel members review and add

2. List of all outcomes grouped by theme– Panel members rate importance

3. Final agreement and results

Page 7: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor
Page 8: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Mortality from cervical cancer8.2

Cervical cancer Incidence8.3

Detected CIN 2,37. 9

Major Infections (requiring hospital admission and antibiotics, e.g. PID)6.0

Maternal bleeding5.8

Premature delivery5.7

Fertility5.4

Identification of STIs (benefit)5.0

Minor infections (requiring outpatient treatment only)3.8

Outcomes for screening on a scale of 1 (not important) to 9 (critical)

Page 9: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Challenges and advantages of this approach

• Often starting with many outcomes

• Experts initially focused on what they know from research studies

• Requires detailed explanations

• ↓ Participation of panel members

• Perspective taken

• Complete• Everyone involved

– Independent ratings• Numerical estimates• Well documented and

kept record• Transparent• Reduces work

Page 10: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

WHO influenza guidelines

• New guideline on pharmacological management of influenza– Previously few randomized trials

• Low quality evidence for many outcomes (imprecision)• Industry sponsored – publication bias• Not all outcomes

• Review of observational studies– To inform guidelines

Page 11: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor
Page 12: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Methods

• Standard systematic review– MEDLINE, EMBASE, CENTRAL, CINAHL, SIGLE, the Chinese

Biomedical Literature Database, Panteleimon and LILACS for relevant studies up to November 2010

– contacted pharmaceutical companies and international agencies– RevMan 5.1

• 10 PICO → recommendations approach– Outcomes determined through Delphi process previously

• QoE according to GRADE approach– GRADEpro (www.gradeworkinggroup.org)– Risk of bias using modified Ottawa Newcastle scale

Page 13: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

ResultsQuality assessment Summary of Findings

Participants(studies)

Risk of bias Inconsistency Indirectness Imprecision Publication bias

Overall quality of evidence

Study event rates (%) Relative effect(95% CI)

Anticipated absolute effects

With no antiviral treatment

With oseltamivir Risk with no antiviral treatment Absolute effect with Oseltamivir (95% CI)

Mortality

681(3 studies)

no serious risk of bias

no serious inconsistency

no serious indirectness

no serious imprecision

undetected1 ⊕⊕⊝⊝LOW1

59/242 (24.4%)

31/439 (7.1%)

adj OR 0.23 (0.13 to 0.43) 240 deaths per 1000 172 fewer deaths per 1000

(from 120 to 201 fewer)

1557(9 studies)

serious2 no serious inconsistency

no serious indirectness

no serious imprecision

undetected1 ⊕⊝⊝⊝VERY LOW1,2

due to risk of bias

61/320 (19.1%)

228/1237 (18.4%)

OR 0.51 (0.23 to 1.14)3 240 deaths per 1000 101 fewer deaths per 1000

(from 172 fewer to 25 more)

Hospitalisation

150710(5 studies)

no serious risk of bias

no serious inconsistency

no serious indirectness

no serious imprecision

undetected4 ⊕⊕⊝⊝LOW4

1238/100585 (1.2%)

431/50125 (0.86%)

adj OR 0.75 (0.66 to 0.89)

12 hospitalisations per 1000 3 fewer hospitalisations per 1000(from 1 to 4 fewer)

242762(6 studies)

serious2 no serious inconsistency

no serious indirectness

no serious imprecision

undetected4 ⊕⊝⊝⊝VERY LOW2,4

due to risk of bias

1738/146410 (1.2%)

1086/96352 (1.1%)

OR 0.75 (0.66 to 0.86)

12 hospitalisations per 1000 3 fewer hospitalisations per 1000(from 2 to 4 fewer)

ICU admissions/mechanical ventilation/respiratory failure

1032(6 studies5)

Serious5 serious6 no serious indirectness

no serious imprecision

undetected1 ⊕⊝⊝⊝VERY LOW1,6

due to risk of bias, inconsistency

- 200/1032 (19.4%)

- -

Complications - Pneumonia

150466(3 studies)

no serious risk of bias

serious6 no serious indirectness

no serious imprecision

undetected4 ⊕⊝⊝⊝VERY LOW4,6

due to inconsistency

2111/100449 (2.1%)

647/50017 (1.3%)

adj OR 0.83 (0.59 to 1.16)

21 pneumonias per 1000 4 fewer pneumonias per 1000(from 9 fewer to 3 more)

265276(6 studies)

serious2 serious6 no serious indirectness

no serious imprecision

undetected4 ⊕⊝⊝⊝VERY LOW2,4,6

due to risk of bias, inconsistency

3244/166256 (2%)

1273/99020 (1.3%)

OR 0.64 (0.46 to 0.88)

20 pneumonias per 1000 7 fewer pneumonias per 1000(from 2 to 10 fewer)

Question: Should oseltamivir vs. no antiviral treatment be used for influenza (follow-up: 30 days)?

Page 14: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

ContentPulmonary rehabilitation compared to usual community care for COPD with recent exacerbation

Bibliography: Puhan M, et al. Pulmonary rehabilitation following exacerbations of chronic obstructive pulmonary disease. Cochrane Database of Systematic Reviews 2010, Issue 11.

Outcomes No of Participants(studies)Follow up

Quality of the evidence(GRADE)

Relative effect(95% CI)

Anticipated absolute effects

Risk with Usual community care

Risk difference with Pulmonary rehabilitation (95% CI)

Hospital admission

250(6 studies)3-18 months

⊕⊕⊕⊕HIGH

OR 0.22 (0.08 to 0.58)

405 per 1000 275 fewer per 1000(from 122 fewer to 353 fewer)

Mortality 110(3 studies)3-48 months

⊕⊕⊕⊝MODERATE2

due to imprecision

OR 0.28 (0.1 to 0.84)

Low1

100 per 1000 70 fewer per 1000(from 15 fewer to 89 fewer)

High1

500 per 1000 281 fewer per 1000(from 43 fewer to 409 fewer)

Quality of life (CRQ) dyspneaChronic Respiratory Questionnaire3. Scale from: 1 to 7.

258(5 studies)12 and 76 weeks

⊕⊕⊕⊝MODERATE4

due to imprecision

The mean quality of life (sgrq) total in the control groups was3.1

The mean quality of life (crq) dyspnea in the intervention groups was0.97 higher(0.35 to 1.58 higher)

Quality of life (SGRQ) totalSt George's Respiratory Questionnaire5. Scale from: 0 to 100.

127(3 studies)12 and 26 weeks

⊕⊕⊕⊝MODERATE4

due to imprecision

The mean quality of life (sgrq) total in the control groups was50

The mean quality of life (sgrq) total in the intervention groups was9.88 lower(5.37 to 14.4 lower)

Ambulation (as measured by 6 min walking distance)distance in meters6

299(6 studies)1 - 208 weeks7

⊕⊕⊕⊝MODERATE4,8

due to imprecision

The mean ambulation (as measured by 6 min walking distance) in the intervention groups was77.7 higher(12.21 to 143.2 higher)

Resource use - not reported

- - - See footnote See footnote

Systematic Reviews & GRADE

Evidence & judgments

Recommendation, health policy & implementation

• Systematic and transparent approach

• Transparently lay out rationale for recommendations

• Manage COI

Page 15: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Judgments/explanations

1 Although we did not downgrade, publication bias cannot be excluded and is of concern.2 Studies not adjusted for potential confounding factors.3 Significant differences in effect for pandemic versus seasonal influenza (see subgroup analyses table).4 Publication bias a concern since large studies had for-profit funding and weighted heavily in analyses.5 No independent comparison group.6 High heterogeneity among studies.7 Measured in select patients in trials.

Page 16: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Getting from evidence to recommendations - GRADE

Recommendations are based on judgments:– Quality of evidence (confidence in estimates of

effect)– Balance between benefits and downsides– Values and preferences– Resource use

But judgments need to be based on the best available evidence and transparent

Page 17: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Balancing desirable and undesirable consequences

↑ Allergic reactions

↑ Local skin reactions

↑ Nausea↑ Resources

↑ QoL ↓ Death

↓ Morbidity

↑ herd immunity

Conditional

Strong For Against

Page 18: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

↑ Allergic

reactions

↑ Local skin

reactions

↑ Nausea

↑ Resources

↑ QoL↓ Death

Morbidity↑ herd

immunityConditional

Strong For Against

Balancing desirable and undesirable consequences

Page 19: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

↑ Allergic reactions

↑ Local skin reactions

↑ Nausea

↑ Resources↑ QoL↓ Death

↓ Morbidity

↑ herd immunity

Conditional

Strong For Against

Balancing desirable and undesirable consequences

Page 20: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Balancing desirable and undesirable consequences

↑ Allergic

reactions

↑ Local skin

reactions

↑ Nausea

↑ Resources

↑ QoL

↓ Death

Morbidity

↑ herd

immunity

Conditional

Strong For Against

Page 21: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Balancing desirable and undesirable consequences

↑ Allergic reactions ↑ Local skin

reactions

↑ Nausea

↑ Resources

↑ QoL

↓ Death

↓ Morbidity

↑ herd immunity

Conditional

Strong For Against

Page 22: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

ContentPulmonary rehabilitation compared to usual community care for COPD with recent exacerbation

Bibliography: Puhan M, et al. Pulmonary rehabilitation following exacerbations of chronic obstructive pulmonary disease. Cochrane Database of Systematic Reviews 2010, Issue 11.

Outcomes No of Participants(studies)Follow up

Quality of the evidence(GRADE)

Relative effect(95% CI)

Anticipated absolute effects

Risk with Usual community care

Risk difference with Pulmonary rehabilitation (95% CI)

Hospital admission

250(6 studies)3-18 months

⊕⊕⊕⊕HIGH

OR 0.22 (0.08 to 0.58)

405 per 1000 275 fewer per 1000(from 122 fewer to 353 fewer)

Mortality 110(3 studies)3-48 months

⊕⊕⊕⊝MODERATE2

due to imprecision

OR 0.28 (0.1 to 0.84)

Low1

100 per 1000 70 fewer per 1000(from 15 fewer to 89 fewer)

High1

500 per 1000 281 fewer per 1000(from 43 fewer to 409 fewer)

Quality of life (CRQ) dyspneaChronic Respiratory Questionnaire3. Scale from: 1 to 7.

258(5 studies)12 and 76 weeks

⊕⊕⊕⊝MODERATE4

due to imprecision

The mean quality of life (sgrq) total in the control groups was3.1

The mean quality of life (crq) dyspnea in the intervention groups was0.97 higher(0.35 to 1.58 higher)

Quality of life (SGRQ) totalSt George's Respiratory Questionnaire5. Scale from: 0 to 100.

127(3 studies)12 and 26 weeks

⊕⊕⊕⊝MODERATE4

due to imprecision

The mean quality of life (sgrq) total in the control groups was50

The mean quality of life (sgrq) total in the intervention groups was9.88 lower(5.37 to 14.4 lower)

Ambulation (as measured by 6 min walking distance)distance in meters6

299(6 studies)1 - 208 weeks7

⊕⊕⊕⊝MODERATE4,8

due to imprecision

The mean ambulation (as measured by 6 min walking distance) in the intervention groups was77.7 higher(12.21 to 143.2 higher)

Resource use - not reported

- - - See footnote See footnote

Systematic Reviews & GRADE

Evidence & judgments

Recommendations, health policy & implementation

Page 23: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Question/Recommendation: Should contacts of new or recurrent TB cases be investigated? Population: In people (at normal risk) who had contact with new or recurrent cases of TB (does)

Intervention: contact investigation Comparison: no investigation Setting (if relevant): high income countries

Decision domain: Judgment Summary of reason for judgment Explanation Subdomains influencing judgment

Quality of evidence (QoE) Is there high or moderate quality evidence? The higher the quality of evidence, the more likely is a strong recommendation

Yes No

There is very low quality evidence from observational studies that are moderate risk of bias for the critical outcomes.

QoE for benefits: Very low QoE for harms: Harms not explicitly evaluated QoE for resource use: Resource use not explicitly evaluated Key reasons for down- or upgrading? Risk of bias was a reason for downgrading for most critical outcomes All critical outcomes measured? Harms and resources not explicitly evaluated

Balance of benefits versus harms and burdens Are you confident that the benefits outweigh the harms and burden or vice versa? The larger the difference between the benefits and harms and the certainty around that difference, the more likely is a strong recommendation. The smaller the net benefit or net harm and the lower the certainty for that net effect, the more likely is a conditional/weak recommendation.

Yes No

There is considerable benefit while little clinical harm or downsides are expected

The yield for all tuberculosis (bacteriologically-confirmed and clinically diagnosed) was 4.5% of contacts investigated.

Latent tuberculosis infection was found in 51.4% of contacts investigated.

The yield for all tuberculosis (bacteriologically-confirmed and clinically diagnosed) was 7.0 % of pediatric contacts investigated.

Latent tuberculosis infection was found in 40.4% % of pediatric contacts investigated.

Baseline risk for benefits and harm and burden?

Is the baseline risk similar across subgroups?

Should there be separate recommendations for subgroups?

Relative risk for benefits and harms: Are the relative benefits large?

Yes, the relative benefits are probably large. Are the relative harms large?

No, the relative harms are probably small. Recommendations for other groups are made separately, pediatric and adult index cases were considered together. Requirement for modeling:

Is there a lot of extrapolation and modeling required for these outcomes?

Yes, modeling is required.

Page 24: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Values and preferences Are you confident about the assumed or identified relative values and are they similar across the target population? The more certainty or similarity in values and preferences, the more likely a strong recommendation.

Yes No

Benefits much higher valued than expected minor harms.

A high value was placed on avoiding consequences of TB, dissemination of TB and mortality. A low value was placed on possible adverse events. There is likely little variability and panel is quite certain

Perspective taken: Patients or public Source of values: Guideline panels assessment Source of variability if any: Not a lot of variability Method for determining values satisfactory for this recommendation: Yes, given the expected small variability and difference between guideline panel and patients.

Resource implications Are the resources worth the expected net benefit from following the recommendation? The lower the cost of an intervention compared to the alternative, and other costs related to the decision – that is, the fewer resources consumed – the more likely is a strong recommendation in favour of that intervention.

Yes No

Resources required are worth the net benefit considering the benefit on mortality and new TB cases.

There are resources required to conduct contact investigation but these resources are worth the expected benefits and downstream treatment costs.

What are the cost per resource unit? Feasibility: Opportunity cost: Differences across settings:

Overall strength of recommendation

?Strong/conditional?

The guideline panel recommends that contacts of patients with TB who are at normal risk be investigated. (NOTE: this is a hypothetical recommendation developed for this article and not intended for clinical decision making.)

Depending on contact investigation strategy used, resource utilization and implications will vary. Opportunity cost may be high.Feasibility is dependent on existing and well functioning programs.Resources worth in smear positive index cases

Page 25: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Quality assessment No of patients Effect

Quality ImportanceNo. of studies Design Limitations Inconsistency Indirectness Imprecision Other Cryotherapy LEEP Relative

(95% CI)Absolute effect at 1 year

(95% CI)

Recurrence CIN2–3 (follow-up 12 months randomized trials; 3–85 months observational studies)1 randomized

trialsno serious limitations

no serious inconsistency

no serious indirectness seriousa,b none 12/161 (7.5%)

4/168 (2.4%)

OR 3.3 (1.04 to 10.46)

51 more per 1000(from 1 to 179 more) O CRITICAL

3 observational studies

no serious limitations

no serious inconsistency

no serious indirectness no serious imprecision

none

2227/14 387(15.5%)

319/7454(4.3%)

OR 2.66 (1.89 to 3.75)

OO CRITICAL

2.4%c 37 more per 1000(from 20 to 60 more)

Cervical cancer (follow-up 12 months randomized trials; 3–85 months to 26 years observational studies)1 randomized

trialsno serious limitations

no serious inconsistency

no serious indirectness very seriousa none 0/200(0%)

0/200(0%) — 0 fewer per 1000d OO CRITICAL

2 observational studies

no serious limitations

no serious inconsistency

no serious indirectness no serious imprecision

none 2/679(0.3%)

3/3350(0.1%) — 0 fewer per 1000e OO CRITICAL

Treatment unacceptable to women (follow-up 2 weeks; acceptability question)1 randomized

trialsno serious limitations

no serious inconsistency

no serious indirectness very seriousf none 15/170 (8.8%)

8/186 (4.3%)

OR 2.15 (0.89 to 5.22)

45 more per 1000(from 5 fewer to 147 more) OO CRITICAL

All severe adverse events (follow-up mean 12–16 months; stenosis and PID)2 randomized

trialsno serious limitations

no serious inconsistency

no serious indirectness very seriousf none3/300 (1%)

2/298f (0.67%) — 0.4 more per 1000

(from 8 fewer to 9 more) OO CRITICAL

All severe adverse events (follow-up 33 months; PID, plug syndrome, stenosis, blood transfusion) 5 randomized

trialsno serious limitations

no serious inconsistency

serioush serioush none 136 480OR 0.53

(0.1 to 2.88)

OO CRITICAL 4%i 18 fewer per 1000

(from 36 fewer to 67 more)

All severe adverse events (follow-up 12 months; PID, stenosis, major bleeding)9 observational

studies serious limitationsj

no serious inconsistency

seriousi seriousf none 1/2233(0%)

38/960(4%)a — 10 fewer per 1000

(from 20 fewer to 0) OOO CRITICAL

Should cryotherapy versus LEEP be used in women with histologically confirmed cervical intraepithelial neoplasia?

Page 26: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Recommendation

• In settings where LEEP is available and accessible, and women present with CIN lesions extending into the cervical canal, the expert panel suggests treatment with LEEP over cryotherapy (conditional recommendation, OO quality evidence)

• Remarks: The benefits of LEEP were greater than those of cryotherapy, and the harms were fewer in these women. However, since there are greater resource implications for LEEP than cryotherapy, and thus LEEP is not available in all settings, a conditional recommendation was made.

Page 27: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Implications of a conditional/weak recommendation

• Patients: The majority of people in this situation would want the recommended course of action, but many would not

• Clinicians: Be more prepared to help patients to make a decision that is consistent with their own values/decision aids and shared decision making

• Policy makers: There is a need for substantial debate and involvement of stakeholders

Page 28: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Implications of a strong recommendation

• Patients: Most people in this situation would want the recommended course of action and only a small proportion would not

• Clinicians: Most patients should receive the recommended course of action

• Policy makers: The recommendation can be adapted as a policy in most situations , can be used as quality indicator/performance measure

Page 29: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Only two of six performance measures seemed reasonable

Page 30: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

WHO evaluation & feedback

• WHO staff & guideline review committee members (GRC) invited to feedback (Jan 2011) about using GRADE approach– what worked– what did not work

• Group discussion (NGT), 11 + 2

Page 31: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Summary of feedback

• Transparency of the GRADE process helps• Requirement for a good Chair

– methods knowhow to move the process• For observational studies we need better/

¿different? summaries (e.g. narrative versions) • Integration and elicitation of values and

preferences was frequently challenging• For global guidelines: issues around the

description of baseline risks and applicability across countries require work

Page 32: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Summary of feedback• Variability in baseline risk → weak or conditional

recommendations should follow • A description of the modifying factors and the layout

of the evidence could be a great benefit and will facilitate implementation

• Measures should be taken to streamline the timing of the development of guidelines

• Working with centers, training, and capacity building of these centers who collaborate with WHO a priority for implementing GRADE

• Impact evaluation (of current process for development of guidelines) should take place

Page 33: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Conclusions• (WHO) guidelines should be based on the best

available evidence to be evidence based• GRADE not avoid judgments but provides

framework• combines what is known in health research methodology

and provides an approach to improve communication

• GRADE process works – is it better?• Change in culture towards the use of evidence• Transparency in decision making and judgments is

key

Page 34: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Thanks

• Nancy Santesso, Andy Oxman, Suzanne Hill• WHO staff who participated in providing

feedback

Page 35: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor
Page 36: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Results - PRISMA

Studies awaiting assessment

(n = 6)•Studies awaiting translation (1)•Papers could not obtain in full (5)

Records identified through database searching (all study

designs)EMBASE, MEDLINE = 9873

SIGLE = 7CINAHL = 1062

LILACS = 19COCHRANE = 301

Chinese Biomedical Literature Database = 914

Panteleimon = 12(Total n = 12176)

Additional records identified through

other sourcesPharmaceutical

companies(n = 12)

Reference lists of relevant papers

(n=15)Records after duplicates removed(n = 7456)

Records screened(n = 7483)

Records excluded(n = 6563)

Full-text articles assessed for eligibility

(n = 920 )

Full-text articles excluded (n = 825)

Excluded for•Not influenza or influenza like illness•Fewer than 25 people•Randomised controlled trial, or not an observational study•Not antiviral agent•Antiviral agents analysed together•Prophylaxis•No outcomes reported

Studies includedN = 89

Question•51 + 5 studies •7 studies•6 studies•0 studies•8•0 studies•16•0 studies•1 study•2 studiesNote: one study may be relevant to multiple questions

Page 37: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Results

Study or Subgroup

Hanshaoworakul 2009Liem 2009 (1)McGeer 2009 (2)

Total (95% CI)

Heterogeneity: Tau² = 0.00; Chi² = 1.58, df = 2 (P = 0.45); I² = 0%Test for overall effect: Z = 4.63 (P < 0.00001)

log[Odds Ratio]

-2.040221-0.941609-1.309333

SE

0.587394160.751132390.4270348

Total

3155569

439

Total

13012

100

242

Weight

28.5%17.5%54.0%

100.0%

IV, Random, 95% CI

0.13 [0.04, 0.41]0.39 [0.09, 1.70]0.27 [0.12, 0.62]

0.23 [0.13, 0.43]

oseltamivir no treatment Odds Ratio

(1) Adjusted for neutropenia and hospital admission(2) Does not specify what was adjusted for

Odds RatioIV, Random, 95% CI

0.1 0.2 0.5 1 2 5 10Favours oseltamivir Favours no treatment

Study or Subgroup

Chemaly 2007Estenssoro 2010Hien 2009Huang 2009Li 2010Liem 2009McGeer 2009Siston 2010 (1)Xi 2009

Total (95% CI)

Total eventsHeterogeneity: Tau² = 0.70; Chi² = 16.76, df = 7 (P = 0.02); I² = 58%Test for overall effect: Z = 1.64 (P = 0.10)

Events

0150

520

188

2124

228

Total

253282517

1185568

476125

1237

Events

352108

3453

61

Total

884

572712

1007430

320

Weight

5.2%13.4%8.6%7.3%

14.4%18.9%17.3%14.9%

100.0%

M-H, Random, 95% CI

0.03 [0.00, 0.69]0.51 [0.12, 2.15]0.25 [0.03, 2.24]

7.47 [0.63, 88.02]Not estimable

0.24 [0.06, 0.92]0.26 [0.11, 0.60]0.64 [0.23, 1.74]2.14 [0.60, 7.64]

0.51 [0.23, 1.14]

Oseltamivir No treatment Odds Ratio

(1) Pregnant women

Odds RatioM-H, Random, 95% CI

0.001 0.1 1 10 1000Favours oseltamivir Favours no treatment

Should oseltamivir versus no treatment be used to treat influenza?Mortality (adjusted)

Mortality (unadjusted)

Page 38: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Recommendation

- The Guidelines Group recommends that TB

programs/clinicians use/do not use

fluoroquinolones in the treatment of all

patients with MDR (Strong (conditional)

recommendation/ low (very low, low, moderate,

high) grade of evidence)

Page 39: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Example: Oseltamivir for Avian Flu

Recommendation: In patients with confirmed or strongly suspected infection with avian influenza A (H5N1) virus, clinicians should administer oseltamivir treatment as soon as possible (strong recommendation, very low quality evidence).

Remarks: This recommendation places a high value on the prevention of death in an illness with a high case fatality. It places relatively low values on adverse reactions, the development of resistance and costs of treatment.

Schunemann et al. The Lancet ID, 2007

Page 40: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Other explanationsRemarks: Despite the lack of controlled treatment data for H5N1, this is a strong recommendation, in part, because there is a lack of known effective alternative pharmacological interventions at this time.

The panel voted on whether this recommendation should be strong or weak and there was one abstention and one dissenting vote.

Page 41: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Recommendation: In settings where LEEP/LLETZ is available and accessible, the expert panel suggests treatment with LEEP/LLETZ over cryotherapy

Population: Women with histologically confirmed CINIntervention: Cryotherapy versus LEEPFactor Decision Explanation High or moderate evidence(is there high- or moderate-quality evidence?) The higher the quality of evidence, the more likely is a strong recommendation.

Yes No ÅÅOO

There is moderate-quality evidence from both randomized and observational controlled studies for recurrence rates. However, there is low-quality evidence for other outcomes that were considered critical and important for decision-making (e.g. severe adverse events, cervical cancer). There is uncertainty for fertility and other obstetric outcomes, and HIV acquisition/transmission was not measured.

Certainty about the balance of benefits versus harms and burdens(is there certainty?) The larger the difference between the desirable and undesirable consequences and the certainty around that difference, the more likely is a strong recommendation. The smaller the net benefit and the lower the certainty for that benefit, the more likely is a conditional/ weak recommendation.

Yes No

Benefits of LEEP were greater, and harms were fewer or similar

Recurrence rates of CIN1, CIN2–3 and all CINs are probably greater with cryotherapy:o CIN2–3, odds ratio (OR) 3.3 (CI 1.04 to 10.46)o CIN1, OR 2.74 (CI 0.62 to 12.07)o All CIN, OR 2.14 (CI 1.05 to 4.33).

Cryotherapy may be less acceptable to patients than LEEP. There may be little difference in serious adverse events

between cryotherapy and LEEP, but there may be fewer minor adverse events (such as pain) with cryotherapy.

It is unclear whether there is a difference in fertility/obstetric outcomes.

High value was placed on CIN recurrence, serious adverse events and acceptability to the patient.

Low value was placed on minor adverse events.

Certainty in or similar values (is there certainty or similarity?) The more certainty or similarity in values and preferences, the more likely is a strong recommendation.

Yes No

Similar values across women

There is not a lot of variability The panel felt secure in assuming the populations value

Resource implications(are resources worth expected benefits?) The lower the cost of an intervention compared to the alternative that is considered and other costs related to the decision – that is, fewer resources consumed – the more likely is a strong recommendation.

Yes No

More resources required for LEEP

Need for more skilled providers to perform LEEP Need for more or expensive equipment/supplies for LEEP;

electricity supply for LEEP Need for local anaesthesia with LEEP

Overall strength of recommendationConditional

Page 42: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor
Page 43: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

MethodsTypes of participants• We included studies in all populations with influenza or influenza like-illness. Types of intervention• Oseltamivir, zanamivir, amantadine or rimantadine in any dose or by any route.Type of outcome measures• We determined a priori to report on the following outcomes because they were

judged to be important or critical for decision making:• Mortality, Hospitalisation, ICU Admission, mechanical ventilation and respiratory

failure, Duration of hospitalization, Time to alleviation of symptoms, Time to return to normal activity, Complications

• Critical adverse events (e.g. major psychotic disorders, encephalitis, stroke and seizure),

• Important adverse events (e.g. pain in extremities, clonic twitching, body weakness, dermatological changes such as uticaria and rash)

• Viral shedding and Resistance

Page 44: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

GRADE Uptake World Health Organization Allergic Rhinitis in Asthma Guidelines (ARIA) American Thoracic Society American College of Physicians European Respiratory Society European Society of Thoracic Surgeons British Medical Journal Infectious Disease Society of America American College of Chest Physicians UpToDate® National Institutes of Health and Clinical Excellence (NICE) Scottish Intercollegiate Guideline Network (SIGN) Cochrane Collaboration Infectious Disease Society of America Clinical Evidence Agency for Health Care Research and Quality (AHRQ) Partner of GIN Over 60 major organizations

Page 45: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

GRADE Uptake World Health Organization Allergic Rhinitis in Asthma Guidelines (ARIA) American Thoracic Society American College of Physicians European Respiratory Society European Society of Thoracic Surgeons British Medical Journal Infectious Disease Society of America American College of Chest Physicians UpToDate® National Institutes of Health and Clinical Excellence (NICE) Scottish Intercollegiate Guideline Network (SIGN) Cochrane Collaboration Infectious Disease Society of America Clinical Evidence Agency for Health Care Research and Quality (AHRQ) Partner of GIN Over 60 major organizations

Page 46: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Recommendation 1a

• The panel recommends that people who had household contact with smear positive or M/XDR TB index cases be investigated for active TB (strong recommendation, very low quality evidence).

Page 47: HOW GRADE COULD HELP TO IMPLEMENT THE EVIDENCE Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor

Recommendation 1b

• The panel recommends that people who had household contact with TB index cases who are children younger than 5 years of age be investigated for active TB (strong recommendation, very low quality evidence).