Grading evidence and Grading evidence and recommendationsrecommendations

Holger Schünemann Holger Schünemann

Andy Oxman Andy Oxman

for the for the GRADE Working GroupGRADE Working Group

Professional good intentions and Professional good intentions and plausible theories are plausible theories are insufficientinsufficient

for selecting policies and for selecting policies and practices for protecting, practices for protecting,

promoting and restoring healthpromoting and restoring health..

Iain Chalmers

How can we judge the How can we judge the extent of our confidence extent of our confidence

that adherence to that adherence to aa recommendation will do recommendation will do more good than harm?more good than harm?

GRADEGRADE

Grades of Recommendation Assessment, Development

and Evaluation

What do you know about What do you know about GRADE?GRADE?

Have prepared a guidelineHave prepared a guideline Read the BMJ paper Read the BMJ paper

Have prepared a systematic review and a Have prepared a systematic review and a summary of findings tablesummary of findings table

Have attended a GRADE meeting, workshop or talkHave attended a GRADE meeting, workshop or talk

About GRADEAbout GRADE

o Began as informal working group in 2000Began as informal working group in 2000o Researchers/guideline developers with Researchers/guideline developers with

interest in methodologyinterest in methodologyo Aim: to develop a Aim: to develop a commoncommon system for system for

grading the quality of evidence and the grading the quality of evidence and the strength of recommendations that is sensible strength of recommendations that is sensible and to explore the range of interventions and and to explore the range of interventions and contexts for which it might be useful*contexts for which it might be useful*

o 12 meetings (~10 – 35 attendants)12 meetings (~10 – 35 attendants)o Evaluation of existing systems and reliability*Evaluation of existing systems and reliability*o Workshops at Cochrane Colloquia, WHO and Workshops at Cochrane Colloquia, WHO and

GIN since 2000GIN since 2000

*Grade Working Group. CMAJ 2003, BMJ 2004, BMC 2004, BMC 2005

GRADE Working GroupGRADE Working GroupDavid Atkins, chief medical officerDavid Atkins, chief medical officeraa Dana Best, assistant professorDana Best, assistant professorbb Peter A Briss, chiefPeter A Briss, chiefcc Martin Eccles, professorMartin Eccles, professordd Yngve Falck-Ytter, associate directorYngve Falck-Ytter, associate directoree Signe Flottorp, researcherSigne Flottorp, researcherff Gordon H Guyatt, professorGordon H Guyatt, professorgg Robin T Harbour, Robin T Harbour, quality and information quality and information

directordirector h h Margaret C Haugh, methodologistMargaret C Haugh, methodologistii David Henry, professorDavid Henry, professorjj Suzanne Hill, senior lecturerSuzanne Hill, senior lecturer jj Roman Jaeschke, clinical professorRoman Jaeschke, clinical professorkk Gillian Leng, guidelines programme directorGillian Leng, guidelines programme director ll Alessandro Liberati, professorAlessandro Liberati, professormm Nicola Magrini, directorNicola Magrini, directornn

James Mason, professorJames Mason, professordd Philippa Middleton, honorary research fellowPhilippa Middleton, honorary research fellowoo Jacek Mrukowicz, executive directorJacek Mrukowicz, executive directorpp Dianne O’Connell, senior epidemiologistDianne O’Connell, senior epidemiologistqq Andrew D Oxman, directorAndrew D Oxman, directorff Bob Phillips, associate fellowBob Phillips, associate fellowrr Holger J Schünemann, associate professorHolger J Schünemann, associate professorg,sg,s Tessa Tan-Torres Edejer, medical Tessa Tan-Torres Edejer, medical

officer/scientistofficer/scientisttt Helena Varonen, associate editorHelena Varonen, associate editoruu Gunn E Vist, researcherGunn E Vist, researcherff John W Williams Jr, associate professorJohn W Williams Jr, associate professorvv Stephanie Zaza, Stephanie Zaza, project directorproject directorww

a)a) Agency for Healthcare Research and Quality, Agency for Healthcare Research and Quality, USA USA b)b) Children's National Medical Center, Children's National Medical Center, USA USAc) Centers for Disease Control and Prevention, c) Centers for Disease Control and Prevention, USAUSAd) University of Newcastle upon Tyne, d) University of Newcastle upon Tyne, UKUKe) German Cochrane Centre, e) German Cochrane Centre, GermanyGermanyf) Norwegian Centre for Health Services, f) Norwegian Centre for Health Services, NorwayNorwayg) McMaster University, g) McMaster University, CanadaCanadah) Scottish Intercollegiate Guidelines Network, h) Scottish Intercollegiate Guidelines Network, UKUKi) Fédération Nationale des Centres de Lutte i) Fédération Nationale des Centres de Lutte Contre le Cancer, Contre le Cancer, FranceFrancej) University of Newcastle, j) University of Newcastle, AustraliaAustraliak) McMaster University, k) McMaster University, CanadaCanadal) National Institute for Clinical Excellence, l) National Institute for Clinical Excellence, UKUKm) m) Università di Modena e Reggio Emilia, Università di Modena e Reggio Emilia, ItalyItalyn)n) Centro per la Valutazione della Efficacia della Centro per la Valutazione della Efficacia della Assistenza Sanitaria, Assistenza Sanitaria, ItalyItalyo) Australasian Cochrane Centre, o) Australasian Cochrane Centre, Australia Australia p) Polish Institute for Evidence Based Medicine, p) Polish Institute for Evidence Based Medicine, PolandPolandq) The Cancer Council, q) The Cancer Council, AustraliaAustraliar) r) Centre for Evidence-based Medicine, Centre for Evidence-based Medicine, UKUKs)s) National Cancer Institute, National Cancer Institute, ItalyItalyt) World Health Organisation, t) World Health Organisation, Switzerland Switzerland u) Finnish Medical Society Duodecim, u) Finnish Medical Society Duodecim, Finland Finland v) Duke University Medical Center, v) Duke University Medical Center, USA USA w) w) Centers for Disease Control and Prevention, Centers for Disease Control and Prevention, USAUSA

Why guidelines?Why guidelines?

users looking for different things

just tell me what to do (recommendation)

what to do, and on strong or weak grounds– recommendation and grade

recommend, grade, evidence summary, values– systematic review, value statement

evidence from individual studies

Grading SystemGrading System

current profusion: can there be consensus?current profusion: can there be consensus?

trade-off benefits and riskstrade-off benefits and risks– do it (or don’t do it)do it (or don’t do it)– probably do it (or probably don’t do it)probably do it (or probably don’t do it)

quality of underlying evidencequality of underlying evidence– high quality (well done RCT)high quality (well done RCT)– intermediate (quasi-RCT)intermediate (quasi-RCT)– low (well done observational)low (well done observational)– very low (anything else)very low (anything else)

Moving downMoving down

poor RCT design, implementationpoor RCT design, implementation– inconsistencyinconsistency

indirectindirect– A vs B, but have A to C, B to CA vs B, but have A to C, B to C– patients, interventions, outcomespatients, interventions, outcomes

reporting biasreporting bias

reporting biasreporting bias

Moving upMoving up

magnitude of effectmagnitude of effect

dose-responsedose-response

biases favor controlbiases favor control– for-profit, not-for-profitfor-profit, not-for-profit

When to make a recommendation?When to make a recommendation?

– nevernever patient values differpatient values differ just lay out benefits and risksjust lay out benefits and risks

– when evidence strong enoughwhen evidence strong enough when very weak, too uncertainwhen very weak, too uncertain

– clinicians need guidanceclinicians need guidance intense study demands decisionintense study demands decision

Why bother about grading?Why bother about grading?

People draw conclusions about thePeople draw conclusions about the– quality of evidencequality of evidence– strength of recommendationsstrength of recommendations

Systematic and explicit approaches can helpSystematic and explicit approaches can help– protect against errorsprotect against errors– resolve disagreementsresolve disagreements– facilitate critical appraisalfacilitate critical appraisal– communicate informationcommunicate information

However, there is wide variation in currently However, there is wide variation in currently used approachesused approaches

Who is confused? Who is confused?

Evidence Evidence RecommendationRecommendation

II-2II-2 BB C+ C+ 11 StrongStrong Strongly Strongly

recommendedrecommended

OrganizatioOrganizationn

USPSTFUSPSTF ACCPACCP GCPSGCPS

Still not confused?Still not confused?

EvidenceEvidenceRecommendationRecommendation

BB Class IClass I C+ C+ 11 IVIV CC

OrganizatioOrganizationn

AHAAHA ACCPACCP SIGNSIGN

Recommendation for use of oral Recommendation for use of oral anticoagulation in patients with atrial anticoagulation in patients with atrial fibrillation and rheumatic mitral valve fibrillation and rheumatic mitral valve diseasedisease

Guidelines development processGuidelines development process

Example ACCPExample ACCP• First ACCP guidelines in 1986 (J. Hirsh; J. First ACCP guidelines in 1986 (J. Hirsh; J.

Dalen)Dalen)• Initially aimed at consensus Initially aimed at consensus • Methodologists involved since beginning Methodologists involved since beginning • Now formally convening every 2 to 3 Now formally convening every 2 to 3

yearsyears• > 200.000 copies in 2001> 200.000 copies in 2001• Seventh conference held in 2003Seventh conference held in 2003• 87 panel members, 22 chapters 87 panel members, 22 chapters • Across subspecialtiesAcross subspecialties• 565 recommendations, 230 new565 recommendations, 230 new• Evidence Based RecommendationsEvidence Based Recommendations

Evidence – recommendation: transparent link

Explicit inclusion criteria Comprehensive search

Standardized consideration of study quality

Conduct/use meta-analysis

Grade recommendations

Acknowledge values and preferences underlying recommendations

What makes guidelines evidence based (in 2005)?

Schünemann et al. Chest 2004

Schünemann et al. Chest 2004

Schünemann HJ et al. Chest 2004

Transparent link between evidence and recommendations

&Explicit inclusion criteria

Table 1 Eligibility Criteria

Section Inclusion Criteria

Population Intervention(s) or Exposure

Outcome Methodology

… … … … …

4.1. Patients with unstable

angina, MI, TIA and non-acute stroke

Any antiplatelet agent compared with placebo

or one or more other antiplatelet agents (s);

Death Stroke or recurrent stroke Other vascular events

Randomized controlled trials

4.2 Patients with

cardioembolic stroke Oral anticoaluation

Death Stroke or recurrent stroke

Randomized controlled trials

… … … … …

Albers et al. Chest 2004

Quality of evidenceQuality of evidence

The extent to which one can be confident that an estimate The extent to which one can be confident that an estimate of effect or association is correct. of effect or association is correct.

It depends on the:It depends on the:– study designstudy design (e.g. RCT, cohort study) (e.g. RCT, cohort study)– study quality/limitationsstudy quality/limitations (protection against bias; (protection against bias;

e.g. concealment of allocation, blinding, follow-up)e.g. concealment of allocation, blinding, follow-up)– consistency of resultsconsistency of results– directness of the evidencedirectness of the evidence including the including the

populationspopulations (those of interest versus similar; for (those of interest versus similar; for example, older, sicker or more co-morbidity)example, older, sicker or more co-morbidity)

interventionsinterventions (those of interest versus similar; for (those of interest versus similar; for example, drugs within the same class)example, drugs within the same class)

outcomesoutcomes (important versus surrogate outcomes) (important versus surrogate outcomes) comparisoncomparison (A - C versus A - B & C - B) (A - C versus A - B & C - B)

Quality of evidenceQuality of evidence

The quality of the evidence (i.e. our confidence) may also The quality of the evidence (i.e. our confidence) may also be REDUCEDbe REDUCED when there is: when there is:

Sparse or imprecise dataSparse or imprecise data Reporting biasReporting bias

The quality of the evidence (i.e. our confidence) may be The quality of the evidence (i.e. our confidence) may be INCREASEDINCREASED when there is: when there is:

A strong association A strong association A dose response relationshipA dose response relationship All plausible confounders would have reduced the All plausible confounders would have reduced the

observed effect observed effect All plausible biases would have increased the observed All plausible biases would have increased the observed

lack of effectlack of effect

Quality assessment criteriaQuality assessment criteria

Quality of evidence

Study design Lower if Higher if

High Randomised trial

Moderate

Low Observational study

Very low Any other evidence

Study quality: -1 Serious limitations -2 Very serious limitations -1 I mportant inconsistency Directness: -1 Some uncertainty -2 Major uncertainty -1 Sparse or imprecise data -1 High probability of reporting bias

Strong association: +1 Strong, no plausible confounders +2 Very strong, no major threats to validity +1 Evidence of a Dose response gradient +1 All plausible confounders would have reduced the eff ect

Categories of qualityCategories of quality

HighHigh: Further research is very unlikely to : Further research is very unlikely to change our confidence in the estimate of change our confidence in the estimate of effect. effect.

ModerateModerate: Further research is likely to have : Further research is likely to have an important impact on our confidence in the an important impact on our confidence in the estimate of effect and may change the estimate of effect and may change the estimate.estimate.

LowLow: Further research is very likely to have : Further research is very likely to have an important impact on our confidence in the an important impact on our confidence in the estimate of effect and is likely to change the estimate of effect and is likely to change the estimate.estimate.

Very lowVery low: Any estimate of effect is very : Any estimate of effect is very uncertain.uncertain.

Judgements about the overall Judgements about the overall quality of evidencequality of evidence

Most systems not explicitMost systems not explicit

Options:Options:– strongest outcomestrongest outcome– primary outcomeprimary outcome– benefitsbenefits– weightedweighted– separate grades for benefits and harmsseparate grades for benefits and harms– no overall gradeno overall grade– weakest outcomeweakest outcome

Based on lowest of all the Based on lowest of all the criticalcritical outcomes outcomes Beyond the scope of a systematic reviewBeyond the scope of a systematic review

Strength of recommendationStrength of recommendation

The extent to which one can be confident that The extent to which one can be confident that adherence to a recommendation will do more adherence to a recommendation will do more good than harm. good than harm.

trade-offstrade-offs (the relative value attached to (the relative value attached to the expected benefits, harms and costs)the expected benefits, harms and costs)

quality of the evidencequality of the evidence translation of the evidencetranslation of the evidence into practice into practice

in a specific settingin a specific setting uncertainty about baseline riskuncertainty about baseline risk

Judgements about the balance Judgements about the balance between benefits and harmsbetween benefits and harms

Before considering cost and making a Before considering cost and making a recommendationrecommendation

For a specified setting, taking into For a specified setting, taking into account issues of translation into account issues of translation into practicepractice

Clarity of the trade-offs Clarity of the trade-offs between benefits and the between benefits and the

harms harms the estimated size of the effect for the estimated size of the effect for

each main outcomeeach main outcome the precision of these estimatesthe precision of these estimates the relative value attached to the the relative value attached to the

expected benefits and harmsexpected benefits and harms important factors that could be important factors that could be

expected to modify the size of the expected to modify the size of the expected effects in specific settings; expected effects in specific settings; e.g. proximity to a hospitale.g. proximity to a hospital

Balance between benefits and Balance between benefits and harmharm

Net benefitsNet benefits:: The intervention does The intervention does more good than harm.more good than harm.

Trade-offsTrade-offs:: There are important There are important trade-offs between the benefits and trade-offs between the benefits and harms.harms.

Uncertain net benefitsUncertain net benefits:: It is not It is not clear whether the intervention does clear whether the intervention does more good than harm.more good than harm.

Not net benefitsNot net benefits:: The intervention The intervention does not do more good than harm.does not do more good than harm.

Judgements about Judgements about recommendationsrecommendations

This should include considerations of This should include considerations of costs; i.e. “Is the net gain (benefits-costs; i.e. “Is the net gain (benefits-harms) worth the costs?”harms) worth the costs?” Do itDo it Probably do it Probably do it No recommendationNo recommendation Probably don’t do itProbably don’t do it Don’t do itDon’t do it

Should healthy asymptomatic postmenopausal women Should healthy asymptomatic postmenopausal women have been given oestrogen + progestin for prevention in have been given oestrogen + progestin for prevention in

1992?1992? Quality of evidence across studies forQuality of evidence across studies for

– CHDCHD– Hip fractureHip fracture– Colorectal cancerColorectal cancer– Breast cancerBreast cancer– StrokeStroke– ThrombosisThrombosis– Gall bladder diseaseGall bladder disease

Quality of evidence across critical outcomesQuality of evidence across critical outcomes Balance between benefits and harmsBalance between benefits and harms RecommendationsRecommendations

Will GRADE lead to change?

Evidence profile: Quality assessmentEvidence profile: Quality assessment Oestrogen + progestin for prevention in Oestrogen + progestin for prevention in

1992 (before WHI and HERS)1992 (before WHI and HERS)

Oestrogen + progestin versus usual care

Oestrogen + progestin for Oestrogen + progestin for prevention after WHI and HERSprevention after WHI and HERS

Further developmentsFurther developments

Diagnostic testsDiagnostic tests ComplexityComplexity CostsCosts (Equity)(Equity) Empirical evaluationsEmpirical evaluations

GRADE for diagnostic testsGRADE for diagnostic testsQuality of evidence Study design Lower if * High Cross-sectional (or cohort)

studies of patients with diagnostic uncertainty with direct comparison

Moderate Low Anything else Very low

Study limitations (including representativeness of population, choice of gold standard, incomplete performance of tests, independence of test interpretation) -1 Serious limitations -2 Very serious limitations -1 Important inconsistency Directness -1-Some uncertainty -2-Major uncertainty -1 Sparse or imprecise data -1 High probability of reporting bias

GRADE ProfilerGRADE Profiler

GRADE profiler (GRADEpro)GRADE profiler (GRADEpro)

Empirical evaluationsEmpirical evaluations

Critical appraisal of other systemsCritical appraisal of other systems Pilot test + sensibilityPilot test + sensibility ““Case law” + practical experienceCase law” + practical experience Guidance for judgementsGuidance for judgements

– Single studiesSingle studies– Sparse data or imprecise dataSparse data or imprecise data

AgreementAgreement Validity?Validity? Comparisons with other systemsComparisons with other systems Alternative presentationsAlternative presentations

Comparison of GRADE and other systemsComparison of GRADE and other systems

Explicit definitionsExplicit definitions Explicit, sequential judgementsExplicit, sequential judgements Components of qualityComponents of quality Overall qualityOverall quality Relative importance of outcomesRelative importance of outcomes Balance between health benefits and harmsBalance between health benefits and harms Balance between incremental health benefits Balance between incremental health benefits

and costsand costs Consideration of equityConsideration of equity Evidence profilesEvidence profiles International collaborationInternational collaboration SoftwareSoftware Consistent judgements?Consistent judgements? Communication?Communication?

Who is interested in GRADEWho is interested in GRADE WHOWHO American Endocrine SocietyAmerican Endocrine Society American College of Chest Physicians American College of Chest Physicians

(ACCP)(ACCP) Italian National Cancer InstituteItalian National Cancer Institute Clinical EvidenceClinical Evidence Norwegian Centre for Health ServicesNorwegian Centre for Health Services UpToDateUpToDate Close relationship with Cochrane Close relationship with Cochrane

CollaborationCollaboration American Society of Clinical Oncology American Society of Clinical Oncology

(ASCO)(ASCO)

We will serve the public more responsibly and ethically

when research designed to reduce the likelihood that we will be misled by

bias and the play of chance has become an expected element of professional and policy making

practice, not an optional add-on.

Iain Chalmers

A prerequisiteA prerequisite

Practitioners and policy makers must Practitioners and policy makers must make much clearer that make much clearer that

they need rigorous evaluative research they need rigorous evaluative research to help ensure that they do more good to help ensure that they do more good

than harm. than harm.

Iain Chalmers

Questions?Questions?

Taking account of costsTaking account of costs

Include Include important (disaggregated)important (disaggregated) costs in costs in evidence summaries and balance sheets when evidence summaries and balance sheets when relevantrelevant– May be useful to aggregate and value (in monetary terms)May be useful to aggregate and value (in monetary terms)– Always include disaggregated resource utilisationAlways include disaggregated resource utilisation– Note when important information is missingNote when important information is missing– Published cost-effectiveness analyses are rarely helpful Published cost-effectiveness analyses are rarely helpful

Assess the quality of the evidence for important Assess the quality of the evidence for important costs (consumption of costs (consumption of resourcesresources) as for other ) as for other effects (Were quantities measured reliably?)effects (Were quantities measured reliably?)

If costs are If costs are criticalcritical to a decision, low quality to a decision, low quality evidence can lower the overall quality of evidenceevidence can lower the overall quality of evidence

Costs are negotiable (the Costs are negotiable (the valuevalue of resources) of resources) There are many possible There are many possible criteria for making a criteria for making a

recommendationrecommendation

Should activated protein C be Should activated protein C be given to patients in severe given to patients in severe

sepsis?sepsis?

An example with costsAn example with costs

GRADE evidence profile: GRADE evidence profile: Activated Protein C for sepsisActivated Protein C for sepsis

Name:Name: Jaeschke and SchunemannJaeschke and Schunemann Date: Date: September 2004 September 2004 Question: Question: Should APC be used for severe sepsis?Should APC be used for severe sepsis? Setting:Setting: ICU in ParisICU in Paris Baseline risk:Baseline risk: Severe sepsis or septic shock > 24 hSevere sepsis or septic shock > 24 h References: References: Effectiveness: Effectiveness: Bernard 2001. Bernard 2001.

Efficacy and safety of recombinant human activated Efficacy and safety of recombinant human activated protein C for severe sepsis. NEJM 2001; 344:699 and protein C for severe sepsis. NEJM 2001; 344:699 and Manns 2002. An economic evaluation of activated protein Manns 2002. An economic evaluation of activated protein C treatment for severe sepsis. NEJM 2002;347:993.C treatment for severe sepsis. NEJM 2002;347:993.

Cost-effectiveness: Manns 2002. An economic evaluation Cost-effectiveness: Manns 2002. An economic evaluation of activated protein C treatment for severe sepsis. NEJM of activated protein C treatment for severe sepsis. NEJM 2002;347:993.2002;347:993.

Possible criteria for making a Possible criteria for making a recommendationrecommendation

Treatment effectTreatment effect Adverse effectsAdverse effects CostCost Cost-effectivenessCost-effectiveness EquityEquity Seriousness of the problemSeriousness of the problem Administrative restrictionsAdministrative restrictions

Quality assessmentQuality assessment

Summary of findingsSummary of findings