Harmonizing levels of evidence: Harmonizing levels of evidence: The Grading of The Grading of

Recommendations Assessment, Recommendations Assessment, Development and Evaluation Development and Evaluation

(GRADE) working group(GRADE) working group

Holger Schünemann, Andy Holger Schünemann, Andy Oxman, Gordon Guyatt for the Oxman, Gordon Guyatt for the

GRADE working groupGRADE working group

GRADEGRADE

Grades of Recommendation Assessment, Development

and Evaluation

Why bother about grading?Why bother about grading?

People draw conclusions about thePeople draw conclusions about the– quality of evidencequality of evidence– strength of recommendationsstrength of recommendations

Systematic and explicit approaches can helpSystematic and explicit approaches can help– protect against errorsprotect against errors– resolve disagreementsresolve disagreements– facilitate critical appraisalfacilitate critical appraisal– communicate informationcommunicate information

However, there is wide variation in currently However, there is wide variation in currently used approachesused approaches

Who is confused? Who is confused?

EvidenceEvidenceRecommendationRecommendation

BB Class IClass I C+ C+ 11 IVIV CC

OrganizatioOrganizationn

AHAAHA ACCPACCP SIGNSIGN

Recommendation for use of oral Recommendation for use of oral anticoagulation in patients with atrial anticoagulation in patients with atrial fibrillation and rheumatic mitral valve fibrillation and rheumatic mitral valve diseasedisease

About GRADEAbout GRADE

o Began as informal working group in 2000Began as informal working group in 2000o Researchers/guideline developers with Researchers/guideline developers with

interest in methodologyinterest in methodologyo Aim: to develop a system for grading the Aim: to develop a system for grading the

quality of evidence and the strength of quality of evidence and the strength of recommendations that is sensible and reliable recommendations that is sensible and reliable and to explore the range of interventions and and to explore the range of interventions and contexts for which it might be useful*contexts for which it might be useful*

o 12 meetings (~10 – 35 attendants)12 meetings (~10 – 35 attendants)o Evaluation of existing systems and reliability*Evaluation of existing systems and reliability*o Workshops at Cochrane Colloquia, WHO and Workshops at Cochrane Colloquia, WHO and

GIN since 2000GIN since 2000

*Grade Working Group. CMAJ 2003, BMJ 2004, BMC 2004, BMC 2005

GRADE Working GroupGRADE Working GroupDavid Atkins, chief medical officerDavid Atkins, chief medical officeraa Dana Best, assistant professorDana Best, assistant professorbb Peter A Briss, chiefPeter A Briss, chiefcc Martin Eccles, professorMartin Eccles, professordd Yngve Falck-Ytter, associate directorYngve Falck-Ytter, associate directoree Signe Flottorp, researcherSigne Flottorp, researcherff Gordon H Guyatt, professorGordon H Guyatt, professorgg Robin T Harbour, Robin T Harbour, quality and information quality and information

directordirector h h Margaret C Haugh, methodologistMargaret C Haugh, methodologistii David Henry, professorDavid Henry, professorjj Suzanne Hill, senior lecturerSuzanne Hill, senior lecturer jj Roman Jaeschke, clinical professorRoman Jaeschke, clinical professorkk Gillian Leng, guidelines programme directorGillian Leng, guidelines programme director ll Alessandro Liberati, professorAlessandro Liberati, professormm Nicola Magrini, directorNicola Magrini, directornn

James Mason, professorJames Mason, professordd Philippa Middleton, honorary research fellowPhilippa Middleton, honorary research fellowoo Jacek Mrukowicz, executive directorJacek Mrukowicz, executive directorpp Dianne O’Connell, senior epidemiologistDianne O’Connell, senior epidemiologistqq Andrew D Oxman, directorAndrew D Oxman, directorff Bob Phillips, associate fellowBob Phillips, associate fellowrr Holger J Schünemann, associate professorHolger J Schünemann, associate professorg,sg,s Tessa Tan-Torres Edejer, medical Tessa Tan-Torres Edejer, medical

officer/scientistofficer/scientisttt Helena Varonen, associate editorHelena Varonen, associate editoruu Gunn E Vist, researcherGunn E Vist, researcherff John W Williams Jr, associate professorJohn W Williams Jr, associate professorvv Stephanie Zaza, Stephanie Zaza, project directorproject directorww

a)a) Agency for Healthcare Research and Quality, Agency for Healthcare Research and Quality, USA USA b)b) Children's National Medical Center, Children's National Medical Center, USA USAc) Centers for Disease Control and Prevention, c) Centers for Disease Control and Prevention, USAUSAd) University of Newcastle upon Tyne, d) University of Newcastle upon Tyne, UKUKe) German Cochrane Centre, e) German Cochrane Centre, GermanyGermanyf) Norwegian Centre for Health Services, f) Norwegian Centre for Health Services, NorwayNorwayg) McMaster University, g) McMaster University, CanadaCanadah) Scottish Intercollegiate Guidelines Network, h) Scottish Intercollegiate Guidelines Network, UKUKi) Fédération Nationale des Centres de Lutte i) Fédération Nationale des Centres de Lutte Contre le Cancer, Contre le Cancer, FranceFrancej) University of Newcastle, j) University of Newcastle, AustraliaAustraliak) McMaster University, k) McMaster University, CanadaCanadal) National Institute for Clinical Excellence, l) National Institute for Clinical Excellence, UKUKm) m) Università di Modena e Reggio Emilia, Università di Modena e Reggio Emilia, ItalyItalyn)n) Centro per la Valutazione della Efficacia della Centro per la Valutazione della Efficacia della Assistenza Sanitaria, Assistenza Sanitaria, ItalyItalyo) Australasian Cochrane Centre, o) Australasian Cochrane Centre, Australia Australia p) Polish Institute for Evidence Based Medicine, p) Polish Institute for Evidence Based Medicine, PolandPolandq) The Cancer Council, q) The Cancer Council, AustraliaAustraliar) r) Centre for Evidence-based Medicine, Centre for Evidence-based Medicine, UKUKs)s) University at Buffalo, University at Buffalo, USAUSAt) World Health Organisation, t) World Health Organisation, Switzerland Switzerland u) Finnish Medical Society Duodecim, u) Finnish Medical Society Duodecim, Finland Finland v) Duke University Medical Center, v) Duke University Medical Center, USA USA w) w) Centers for Disease Control and Prevention, Centers for Disease Control and Prevention, USAUSA

Guideline development processGuideline development process

Quality of evidenceQuality of evidenceThe extent to which one can be confident that an estimate The extent to which one can be confident that an estimate of effect or association is correct. of effect or association is correct.

It depends on the:It depends on the:– study designstudy design (e.g. RCT, cohort study) (e.g. RCT, cohort study)– study quality/limitationsstudy quality/limitations (protection against bias; (protection against bias;

e.g. concealment of allocation, blinding, follow-up)e.g. concealment of allocation, blinding, follow-up)– consistency of resultsconsistency of results– directness of the evidencedirectness of the evidence including the including the

populationspopulations (those of interest versus similar; for (those of interest versus similar; for example, older, sicker or more co-morbidity)example, older, sicker or more co-morbidity)

interventionsinterventions (those of interest versus similar; (those of interest versus similar; for example, drugs within the same class)for example, drugs within the same class)

outcomesoutcomes (important versus surrogate (important versus surrogate outcomes)outcomes)

comparisoncomparison (A - C versus A - B & C - B) (A - C versus A - B & C - B)

Quality of evidenceQuality of evidence

The quality of the evidence (i.e. our confidence) may be The quality of the evidence (i.e. our confidence) may be REDUCEDREDUCED when there is: when there is:

Sparse or imprecise dataSparse or imprecise data Reporting biasReporting bias

The quality of the evidence (i.e. our confidence) may be The quality of the evidence (i.e. our confidence) may be INCREASEDINCREASED when there is: when there is:

A strong association A strong association A dose response relationshipA dose response relationship All plausible confounders would have reduced the All plausible confounders would have reduced the

observed effect observed effect All plausible biases would have increased the observed All plausible biases would have increased the observed

lack of effectlack of effect

Quality assessment criteriaQuality assessment criteria

Quality of evidence

Study design Lower if Higher if

High Randomised trial

Moderate

Low Observational study

Very low Any other evidence

Study quality: -1 Serious limitations -2 Very serious limitations -1 I mportant inconsistency Directness: -1 Some uncertainty -2 Major uncertainty -1 Sparse or imprecise data -1 High probability of reporting bias

Strong association: +1 Strong, no plausible confounders +2 Very strong, no major threats to validity +1 Evidence of a Dose response gradient +1 All plausible confounders would have reduced the eff ect

Categories of qualityCategories of quality

HighHigh: Further research is very unlikely to : Further research is very unlikely to change our confidence in the estimate of change our confidence in the estimate of effect. effect.

ModerateModerate: Further research is likely to have : Further research is likely to have an important impact on our confidence in the an important impact on our confidence in the estimate of effect and may change the estimate of effect and may change the estimate.estimate.

LowLow: Further research is very likely to have : Further research is very likely to have an important impact on our confidence in the an important impact on our confidence in the estimate of effect and is likely to change the estimate of effect and is likely to change the estimate.estimate.

Very lowVery low: Any estimate of effect is very : Any estimate of effect is very uncertain.uncertain.

Judgements about the overall Judgements about the overall quality of evidencequality of evidence

Most systems just use evidence about primary Most systems just use evidence about primary benefit/outcomebenefit/outcome

But what about other outcomes (downsides)?But what about other outcomes (downsides)? Options:Options:

– ignore all but the primary outcomeignore all but the primary outcome– basing it on the evidence for benefitsbasing it on the evidence for benefits– some blended approachsome blended approach– having separate grades for benefits and having separate grades for benefits and

harmsharms– weakest of any outcomeweakest of any outcome

Based on lowest of all the Based on lowest of all the criticalcritical outcomes outcomes Beyond the scope of a systematic reviewBeyond the scope of a systematic review

Judgements about the balance Judgements about the balance between benefits and harmsbetween benefits and harms

Before considering cost and making a Before considering cost and making a recommendationrecommendation

For a specified setting, taking into For a specified setting, taking into account issues of translation into account issues of translation into practicepractice

Clarity of the trade-offs Clarity of the trade-offs between benefits and the between benefits and the

harms harms The estimated size of the effect for The estimated size of the effect for

each main outcomeeach main outcome The precision of these estimatesThe precision of these estimates The relative value attached to the The relative value attached to the

expected benefits and harmsexpected benefits and harms Important factors that could be Important factors that could be

expected to modify the size of the expected to modify the size of the expected effects in specific settings; expected effects in specific settings; e.g. proximity to a hospitale.g. proximity to a hospital

Strength of recommendationStrength of recommendation

The extent to which one can be confident that The extent to which one can be confident that adherence to a recommendation will do more adherence to a recommendation will do more good than harm. good than harm.

trade-offstrade-offs (the relative value attached to (the relative value attached to the expected benefits, harms and costs)the expected benefits, harms and costs)

quality of the evidencequality of the evidence translation of the evidencetranslation of the evidence into practice into practice

in a specific settingin a specific setting uncertainty about baseline riskuncertainty about baseline risk

Judgements about Judgements about recommendationsrecommendations

This should include considerations of This should include considerations of costs; i.e. “Is the net gain (benefits-costs; i.e. “Is the net gain (benefits-harms) worth the costs?”harms) worth the costs?” Do itDo it Probably do it Probably do it No recommendationNo recommendation Probably don’t do itProbably don’t do it Don’t do itDon’t do it

Challenges for GRADEChallenges for GRADE

GRADE for diagnostic tests, costGRADE for diagnostic tests, cost Dissemination/buy inDissemination/buy in

– simple to dosimple to do– easy to understand and useeasy to understand and use

Operationalise all stepsOperationalise all steps Tool and manualTool and manual

GRADE profiler (GRADEpro)GRADE profiler (GRADEpro)

GRADE IT!GRADE IT!

Should healthy asymptomatic postmenopausal women Should healthy asymptomatic postmenopausal women have been given oestrogen + progestin for prevention in have been given oestrogen + progestin for prevention in

1992?1992? Quality of evidence across studies forQuality of evidence across studies for

– CHDCHD– Hip fractureHip fracture– Colorectal cancerColorectal cancer– Breast cancerBreast cancer– StrokeStroke– ThrombosisThrombosis– Gall bladder diseaseGall bladder disease

Quality of evidence across critical outcomesQuality of evidence across critical outcomes Balance between benefits and harmsBalance between benefits and harms RecommendationsRecommendations

Will GRADE lead to change?

Evidence profile: Quality assessmentEvidence profile: Quality assessment Oestrogen + progestin for prevention before Oestrogen + progestin for prevention before

WHI and HERSWHI and HERS

Oestrogen + progestin versus usual care

Oestrogen + progestin for Oestrogen + progestin for prevention after WHI and HERSprevention after WHI and HERS

GRADE for diagnostic testsGRADE for diagnostic testsQuality of evidence Study design Lower if * High Cross-sectional (or cohort)

studies of patients with diagnostic uncertainty with direct comparison

Moderate Low Anything else Very low

Study limitations (including representativeness of population, choice of gold standard, incomplete performance of tests, independence of test interpretation) -1 Serious limitations -2 Very serious limitations -1 Important inconsistency Directness -1-Some uncertainty -2-Major uncertainty -1 Sparse or imprecise data -1 High probability of reporting bias

What do you know about What do you know about GRADE?GRADE?

Have prepared a guidelineHave prepared a guideline Read the BMJ paper Read the BMJ paper

Have prepared a systematic review and a Have prepared a systematic review and a summary of findings tablesummary of findings table

Have attended a GRADE meeting, workshop or talkHave attended a GRADE meeting, workshop or talk