grade: implications for teaching ebm

PlanPlan

• GRADE backgroundGRADE background

• two stepstwo steps– quality of evidencequality of evidence– strength of recommendationstrength of recommendation

• evidence profilesevidence profiles

• an exercise in applying GRADEan exercise in applying GRADE

Why Grade Why Grade Recommendations?Recommendations?

• strong recommendationsstrong recommendations– strong methods strong methods – large precise effect large precise effect – few down sides of therapyfew down sides of therapy

• weak recommendationsweak recommendations– weak methodsweak methods– imprecise estimateimprecise estimate– small effectsmall effect– substantial down sidessubstantial down sides

Which grading system to Which grading system to use?use?

• many availablemany available– Australian National and MRCAustralian National and MRC– Oxford Center for Evidence-based Oxford Center for Evidence-based

MedicineMedicine– Scottish Intercollegiate Guidelines (SIGN)Scottish Intercollegiate Guidelines (SIGN)– US Preventative Services Task ForceUS Preventative Services Task Force– American professional organizationsAmerican professional organizations

• AHA/ACC, ACCP, AAP, Endocrine society, etc....AHA/ACC, ACCP, AAP, Endocrine society, etc....

• cause of confusion, dismaycause of confusion, dismay

A common international A common international grading system?grading system?

• GRADE (GRADE (GGrades of rades of rrecommendation, ecommendation, aassessment, ssessment, ddevelopment and evelopment and eevaluation)valuation)

• international groupinternational group– Australian NMRC, SIGN, USPSTF, WHO, NICE, Australian NMRC, SIGN, USPSTF, WHO, NICE,

Oxford CEBM, CDC, CCOxford CEBM, CDC, CC

• ~ 25 meetings over last ten years~ 25 meetings over last ten years• (~10 – 50 attendants)(~10 – 50 attendants)

GRADE UptakeGRADE UptakeAgencia sanitaria regionale, Bologna, Italia Agency for Health Care Research and Quality (AHRQ)Allergic Rhinitis and Group - Independent Expert PanelAmerican Association for the study of liver diseasesAmerican College of Cardiology FoundationAmerican College of Chest PhysiciansAmerican College of Emergency PhysiciansAmerican College of PhysiciansAmerican Endocrine Society American Society of Gastrointestinal EndoscopyAmerican society of Interventional Pain PhysiciansAmerican Thoracic Society (ATS)BMJ Clinical Evidence British Medical Journal Canadian Agency for Drugs and Technology in HealthCanadian Cardiovascular SocietyCanadian Task Force on Preventive Health CareCenters for Disease ControlCochrane Collaboration EBM Guidelines Finland Emergency Medical Services for Children National

Resource Center European Association for the Study of the LiverEuropean Respiratory SocietyEuropean Society of Thoracic SurgeonsEvidence-based Nursing Sudtirol, Alta Adiga, ItalyFinnish Office of Health Technology Assessment

German Agency for Quality in MedicineHeelth Inspectorate for ScotlandInfectious Disease Society of America Japanese Society of Oral and Maxillofacial Radiology Joslin Diabetes CenterJournal of Infection in Developing CountriesKaiser PermanenteKidney Disease International Guidelines Organization National and Gulf Centre for Evidence-based MedicineNational Institute for Clinical Excellence (NICE)National Kidney FoundationNorwegian Knowledge Centre for the Health ServicesOntario MOH Medical Advisory SecretariatPanama and Costa Rica National Clinical Guidelines Program

Polish Institute for EBMScottish Intercollegiate Guideline Network (SIGN)Society of Critical Care MedicineSociety of Pediatric Endocrinology Society of Vascular SurgerySpanish Society of Family Practice (SEMFYC) Stop TB Diagnostic Working GroupSurviving sepsis campaign Swedish Council on Technology Assessment in Health CareSwedish National Board of Health and Welfare University of Pennsylvania Health System for EB Practice UpToDate WINFOCUSWorld Allergy OrganizationWorld Health Organization (WHO)

What are we grading?What are we grading?

• two componentstwo components

• quality of body of evidencequality of body of evidence– extent to which confidence in estimate extent to which confidence in estimate

of effect adequate to support decisionof effect adequate to support decision• high, moderate, low, very lowhigh, moderate, low, very low

• strength of recommendationstrength of recommendation• strong and weakstrong and weak

Determinants of qualityDeterminants of quality

• RCTs start highRCTs start high

• observational studies start low observational studies start low

• limitations can lower quality?limitations can lower quality?

Determinants of qualityDeterminants of quality• 5 limitations can lower quality5 limitations can lower quality

• risk of biasrisk of bias– concealment, blinding, loss to follow-upconcealment, blinding, loss to follow-up

• imprecisionimprecision

• inconsistencyinconsistency– variability in results (heterogeneity)variability in results (heterogeneity)

• publication biaspublication bias

Quality judgments: Quality judgments: DirectnessDirectness

• populations populations – older, sicker or more co-morbidityolder, sicker or more co-morbidity

• interventions interventions – new statins versus oldnew statins versus old

• outcomes outcomes – important versus surrogate outcomesimportant versus surrogate outcomes– glucose control versus CV eventsglucose control versus CV events

Alendronate

Risedronate

Placebo

DirectnessDirectness

interested in A versus B interested in A versus B available data A vs C, B vs Cavailable data A vs C, B vs C

What can raise quality?What can raise quality?

• large magnitude can upgrade one levellarge magnitude can upgrade one level– very large two levelsvery large two levels

• common criteriacommon criteria– everyone used to do badlyeveryone used to do badly– almost everyone does wellalmost everyone does well– quick actionquick action

• hip replacement for hip osteoarthritiship replacement for hip osteoarthritis

• mechanical ventilation in respiratory failuremechanical ventilation in respiratory failure

Quality assessment criteriaQuality assessment criteria

Quality Assessment

Summary of Findings

QualityRelative Effect

(95% CI)

Absolute risk difference

OutcomeNumber of

participants(studies)

Risk of Bias

Consistency Directness PrecisionReportin

g Bias

Myocardial infarction

10,125(9)

No serious limitations

No serious imitations

No serious limitations

No serious limitations

Not detected

High0.71

(0.57 to 0.86)1.5% fewer

(0.7% fewer to 2.1% fewer)

Mortality10,205

(7)No serious limitations

Possiblly inconsistent

No serious limitations

ImpreciseNot

detectedModerate

or low1.23

(0.98 – 1.55)

0.5% more(0.1% fewer

to 1.3% more)

Stroke10,889

(5)No serious limitaions

No serious limitations

No serious limitations

Possible imprecision

Not detected

Moderate or High

2.21(1.37 – 3.55)

0.5% more (0.2% more to

1.3% more0

Beta blockers in non-cardiac surgery

Strength of RecommendationStrength of Recommendation

• strong recommendationstrong recommendation– benefits clearly outweigh risks/hassle/costbenefits clearly outweigh risks/hassle/cost– risk/hassle/cost clearly outweighs benefitrisk/hassle/cost clearly outweighs benefit

• what can downgrade strength?what can downgrade strength?

• low quality evidence low quality evidence

• close balance between up and downsidesclose balance between up and downsides

Risk/Benefit tradeoffRisk/Benefit tradeoff

• aspirin after myocardial infarctionaspirin after myocardial infarction– 25% reduction in relative risk 25% reduction in relative risk – side effects minimal, cost minimalside effects minimal, cost minimal– benefit obviously much greater than benefit obviously much greater than

risk/costrisk/cost

• warfarin in low risk atrial fibrillationwarfarin in low risk atrial fibrillation– warfarin reduces stroke vs ASA by 50%warfarin reduces stroke vs ASA by 50%– but if risk only 1% per year, ARR 0.5%but if risk only 1% per year, ARR 0.5%– increased bleeds by 1% per yearincreased bleeds by 1% per year

Strength of Strength of RecommendationsRecommendations

Aspirin after MI – do itAspirin after MI – do it

Warfarin rather than ASA in Afib Warfarin rather than ASA in Afib -- probably do it-- probably do it

-- probably don’t do it-- probably don’t do it

Significance of strong vs Significance of strong vs weakweak

• variability in patient preferencevariability in patient preference– strong, almost all same choice (> 90%)strong, almost all same choice (> 90%)– weak, choice varies appreciablyweak, choice varies appreciably

• interaction with patientinteraction with patient– strong, just inform patientstrong, just inform patient– weak, ensure choice reflects valuesweak, ensure choice reflects values

• use of decision aiduse of decision aid– strong, don’t botherstrong, don’t bother– weak, use the aidweak, use the aid

• quality of care criterionquality of care criterion– strong, considerstrong, consider– weak, don’t considerweak, don’t consider

Flavanoids for Flavanoids for HemorrhoidsHemorrhoids

• venotonic agentsvenotonic agents– mechanism unclear, increase venous returnmechanism unclear, increase venous return

• popularitypopularity– 90 venotonics commercialized in France90 venotonics commercialized in France– none in Sweden and Norwaynone in Sweden and Norway– France 70% of world marketFrance 70% of world market

• possibilitiespossibilities– French misguidedFrench misguided– rest of world missing outrest of world missing out

Systematic ReviewSystematic Review

• 14 trials, 1432 patients14 trials, 1432 patients

• key outcomekey outcome– risk not improving/persistent symptomsrisk not improving/persistent symptoms– 11 studies, 1002 patients, 375 events11 studies, 1002 patients, 375 events

– RR 0.4, 95% CI 0.29 to 0.57RR 0.4, 95% CI 0.29 to 0.57

• minimal side effectsminimal side effects

• is France right?is France right?

• what is the quality of evidence?what is the quality of evidence?

What can lower quality?What can lower quality?

• risk of biasrisk of bias– lack of detail re concealmentlack of detail re concealment– questionnaires not validatedquestionnaires not validated

• indirectness – no problemindirectness – no problem

• inconsistency, need to look at the inconsistency, need to look at the resultsresults

Review : Phlebotonics for hemorrhoidsComparison: 01 Venotonics vs placebp Outcome: 08 Overall improvement: no improvement/some improvement

Study RR (random) Weight RR (random)or sub-category log[RR] (SE) 95% CI % 95% CI

01 Up to seven daysChauvenet -0.8916 (0.2376) 12.67 0.41 [0.26, 0.65] Cospite -2.2073 (0.6117) 5.51 0.11 [0.03, 0.36] Thanapongsathorn -0.4308 (0.2985) 11.18 0.65 [0.36, 1.17]

Subtotal (95% CI) 29.36 0.37 [0.18, 0.77]Test for heterogeneity: Chi² = 6.92, df = 2 (P = 0.03), I² = 71.1%Test for overall effect: Z = 2.67 (P = 0.008)

02 Up to four w eeksAnnoni F -1.6094 (0.7073) 4.50 0.20 [0.05, 0.80] Clyne MB -0.9943 (0.3983) 8.94 0.37 [0.17, 0.81] Pirard J -1.1712 (0.3086) 10.94 0.31 [0.17, 0.57] Thanapongsathorn -1.1087 (1.1098) 2.18 0.33 [0.04, 2.91] Thorp 0.2624 (0.3291) 10.46 1.30 [0.68, 2.48] Titapan -0.8916 (0.3691) 9.56 0.41 [0.20, 0.85] Wijayanegara -0.5978 (0.1375) 14.97 0.55 [0.42, 0.72]

Subtotal (95% CI) 61.54 0.48 [0.32, 0.72]Test for heterogeneity: Chi² = 13.87, df = 6 (P = 0.03), I² = 56.7%Test for overall effect: Z = 3.57 (P = 0.0004)

03 Further than four w eeksGodeberg -1.7719 (0.3906) 9.10 0.17 [0.08, 0.37]

Subtotal (95% CI) 9.10 0.17 [0.08, 0.37]Test for heterogeneity: not applicableTest for overall effect: Z = 4.54 (P < 0.00001)

Total (95% CI) 100.00 0.40 [0.29, 0.57]Test for heterogeneity: Chi² = 28.66, df = 10 (P = 0.001), I² = 65.1%Test for overall effect: Z = 5.14 (P < 0.00001)

0.001 0.01 0.1 1 10 100 1000

Favours treatment Favours control

Publication bias?Publication bias?

• size of studiessize of studies– 40 to 234 patients, most around 10040 to 234 patients, most around 100

• all industry sponsoredall industry sponsored

Review : Phlebotonics for hemorrhoidsComparison: 01 Venotonics vs placebp Outcome: 08 Overall improvement: no improvement/some improvement

0.001 0.01 0.1 1 10 100 1000

0.0

0.4

0.8

1.2

1.6

RR (fixed)

What can lower quality?What can lower quality?

• detailed design and executiondetailed design and execution– lack of detail re concealmentlack of detail re concealment– questionnaires not validatedquestionnaires not validated

• inconsistencyinconsistency– almost all show positive effect, trendalmost all show positive effect, trend– heterogeneity p < 0.001; Iheterogeneity p < 0.001; I22 65.1% 65.1%

• indirectnessindirectness

• imprecisionimprecision

– RR 0.4, 95% CI 0.29 to 0.57RR 0.4, 95% CI 0.29 to 0.57

• reporting biasreporting bias– 40 to 234 patients, most around 10040 to 234 patients, most around 100

Is France right?Is France right?

• recommendationrecommendation– yesyes

– no against useno against use

• strengthstrength– strong strong – weakweak

ConclusionConclusion

• clinicians, policy makers need summariesclinicians, policy makers need summaries– quality of evidencequality of evidence– strength of recommendationsstrength of recommendations

• explicit rulesexplicit rules– transparent, informativetransparent, informative

• GRADEGRADE– simple, transparent, systematicsimple, transparent, systematic– increasing wide adoptionincreasing wide adoption