grade approach for making recommendations in clinical practice guidelines #wac2015

Carlos A. Cuello Garcia MD, PhD(c) Health Research Methodology Program Department of Clinical Epidemiology and Biostatistics

Guideline development

Levels of evidence and the GRADE approach

DISCLOSURE• currently working at

MacGRADE centre & Cochrane GRADEing group

• Cochrane author

• I have received support from the WAO for travel / meetings

• no other COIs related to this talk

GRADEingMethods

OUTLINE

• what are clinical practice guidelines?

• why should we use them?

• what are recommendations and levels of

evidence?

• how to interpret a CPG recommendation?

ATOPIC DERMATITIS

EASY! PIMECROLIMUS

EASY! PIMECROLIMUS

WHAT ABOUT CANCER REPORTED?

WE BETTER STAY WITH TOPICAL

CORTICOSTEROIDS

WE BETTER STAY WITH TOPICAL

CORTICOSTEROIDS

PIMECROLIMUS IS BETTER THAN TCS

PIMECROLIMUS OR TOPICAL CORTICOSTEROIDS?

Meanwhile, at the Ministry of Health…

CLINICAL PRACTICE GUIDELINES

๏ statements that include recommendations

intended to optimize patient care that are

informed by a systematic review of evidence

and an assessment of the benefits and harms

of alternative care options.

IOM · AHRQ 2011

CLINICAL RECOMMENDATION

EVIDENCE

PATIENT VALUES

ACCEPTA-BILITY

BENEFITS VS HARMS COSTS

IN PATIENTS (CLINICAL) GUIDELINES

DECISION

EVIDENCE

PATIENT VALUES

ACCEPTA-BILITY

BENEFITS VS HARMS FEASIBILITY

EQUITY RESOURCE USE

IN PUBLIC HEALTH GUIDELINES

guidelines we can trust๏ Based on a

systematic review ofthe existing evidence

1

guidelines we can trust๏ Developed by a

knowledgeable,multidisciplinarypanel of experts andrepresentatives fromkey affected groups

2

guidelines we can trust๏ Considers important

patient subgroupsand patient

preferences

3

guidelines we can trust๏ Based on an explicit

and transparentprocess thatminimizes distortions,biases, and conflictsof interest

4

guidelines we can trust๏ Provides a clear

explanation of the logicalrelationships betweenalternative care options andhealth outcomes, andprovide ratings of both thequality of evidence and thestrength of

recommendations

5

guidelines we can trust๏ It is revised as

appropriate whenimportant newevidence warrantsmodifications ofrecommendations.

6


or DECISION

EVIDENCE

PATIENT VALUES

ACCEPTA-BILITY

BENEFITS VS HARMS FEASIBILITY

EQUITY RESOURCE USE


or DECISION

EVIDENCE

RESEARCH GENERATION

PUBLISHING

RESEARCH GENERATION

PUBLISHING

systematic reviews

clinical practice guidelines

CHALLENGES AHEAD

why we need trustworthy clinical practice guidelines?

๏ Every year $100 billion dollars are invested in biomedical research

๏ Only 10% is used to test treatments

Chalmers 2009

$

๏ only half of researchers use a systematic review that has been previously done on their topic

Chalmers I, Glasziou P. Lancet 2009;374:86-9 Cooper NJ, et al Clin Trials 2005;2:260–4.

50%

๏ only half of researchers use adequate tools to ensure the quality of their work


CONSORT

๏ not all research is registered, reported, or published…

Cressey D. Secrets of trial data revealed. Nature. 2013 Oct 10;502(7470):154-5

RESEARCH GENERATION

and many times we overlook research priorities and patient important outcomes


100 participants

parachute placebo

100 participants

36.5 (2.1) 36.7 (1.9)

4.3 (1.2) 6.1(1.3)

25 (3) 22(2.4)

Temp ºC –mean (SD)

Fear scale –mean (IQR)

Serum adrenaline (mcg/dL) – mean

(SD)

parachute placebo

100 participants

36.5 (2.1) 36.7 (1.9)

4.3 (1.2) 6.1(1.3)

25 (3) 22(2.4)

Temp ºC –mean (SD)

Fear scale –mean (IQR)

Serum adrenaline (mcg/dL) – mean

(SD)

p=0.00001

parachute placebo

antiarrhythmicsPatients with MI

ecainide flecainide placebo

↓ ↓

↑↑

arrhythmias

death ??

Epstein AE, et al. JAMA 1993

outcomes Patients with asthma

nitrous oxide placebo

O2 sat

FEV1

Length of hospital stay

Death

so, do you recommend..?


๏ pimecrolimus or topicalcorticosteroids for atopic dermatitis?



๏ nitrous oxide for severe asthma?



๏ nitrous oxide for severe asthma?๏ probiotics for allergy prevention?



๏ nitrous oxide for severe asthma?๏ probiotics for allergy prevention?๏ epinephrine for anaphylactic shock?

expert recommendations

cross-sectional

case series, case reports

case-control

cohort

Randomized trial

Systematic review

CTFPHE 1979


cross-sectional


case-control

cohort

Randomized trial

Systematic review

CTFPHE 1979

I good

II fair

III poor

EVIDENCE


cross-sectional


case-control

cohort

Randomized trial

Systematic review

CTFPHE 1979

I good

II fair

III poor

EVIDENCE

A

B

C

RECOMMENDATION

D

I want to jump from this airplane

do you recommend a parachute?

Grading of Recommendations Assessment, Development and Evaluation


cross-sectional case series

case-control

cohort

RCT


case-control

cohort

RCT RCT

Observational

experience is not evidence,

it’s a tool

By default...


case-control

cohort

RCT RCT

Observational

HIGH QUALITY

LOW QUALITY

QUALITY OF EVIDENCE

๏ You can also call it: ๏ certainty of evidence ๏ confidence in the estimates

Balshem H, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol. 2011;64(4):401-6

QUALITY OF EVIDENCE

Balshem H, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol. 2011;64(4):401-6

SYSTEMATIC REVIEWCLINICAL PRACTICE

GUIDELINE

the extent of our confidence that the estimates of the effect are correct

the extent of our confidence that the estimates of the effect are adequate to support a particular decision or recommendation

QUALITY OF EVIDENCE

HIGH We are very confident that the true effect lies close to that of the estimate of the effect

We are moderately confident in the effect estimate: The true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially

different

Our confidence in the effect estimate is limited: The true effect may be substantially different from the

estimate of the effect

We have very little confidence in the effect estimate: The true is likely to be substantially different from the

estimate of effect

MODERATE

LOW

VERY LOW

how is the quality (confidence) of the evidence determined?

what would make you loose confidence?

๏ Overall, probiotics reduced the risk of eczema in infants when given to their pregnant mothers

๏ RR=0.72 (95% CI 0.61 to 0.85)

๏ 3’509 participants, 15 trials

QUALITY OF EVIDENCE

risk of bias

STUDY 1 STUDY 2 STUDY 3

QUALITY OF EVIDENCE

inconsistency

risk of bias


QUALITY OF EVIDENCE

inconsistency

indirectn

ess

P I C O

?

risk of bias


QUALITY OF EVIDENCE

inconsistency

indirectn

ess

P I C O

?

imprecision

risk of bias


QUALITY OF EVIDENCE

inconsistency

indirectn

ess

P I C O

?

imprecision

publication biasrisk of bias


By default...

cross-sectional case series, etc.

case-control

cohort

RCT RCT

Observational

HIGH QUALITY

LOW QUALITY

By default...

RCT

Observational

HIGH QUALITY

LOW QUALITY

RCT

HIGH

MODERATE

LOW

VERY LOW

QUALITY OF EVIDENCE

RCT

HIGH

MODERATE

LOW

VERY LOW

QUALITY OF EVIDENCErisk of bias

HIGH

MODERATE

LOW

VERY LOW

QUALITY OF EVIDENCE

RCT

risk of bias

HIGH

MODERATE

LOW

VERY LOW

QUALITY OF EVIDENCE

RCT

risk of bias

inconsistency

HIGH

MODERATE

LOW

VERY LOW

QUALITY OF EVIDENCE

RCT

risk of bias

inconsistency

imprecision

HIGH

MODERATE

LOW

VERY LOW

QUALITY OF EVIDENCE

RCT

risk of bias

inconsistency

imprecision

indirectn

ess

HIGH

MODERATE

LOW

VERY LOW

QUALITY OF EVIDENCE

RCT

risk of bias

inconsistency

imprecision

indirectn

ess

publication bias

VERY LOW

QUALITY OF EVIDENCE

RCT

risk of bias

inconsistency

imprecision

indirectn

ess

publication bias

OBSERVATIONAL

HIGH

MODERATE

LOW

VERY LOW

QUALITY OF EVIDENCE

OBSERVATIONAL

HIGH

MODERATE

LOW

VERY LOW

QUALITY OF EVIDENCE

strong association

HIGH

MODERATE

LOW

VERY LOW

QUALITY OF EVIDENCE

OBSERVATIONAL

strong association

dose response

HIGH

MODERATE

LOW

VERY LOW

QUALITY OF EVIDENCE

OBSERVATIONAL

strong association

dose response

opposing residual

confounding

HIGH

QUALITY OF EVIDENCE

OBSERVATIONAL

strong association

dose response

opposing residual

confounding

from evidence to recommendations

Quality of evidence

Strength of a recommendation

the extent of our confidence that the estimates of an effect are adequate to support a particular decision or recommendation

the extent to which one can be confident that the desirable consequences of an intervention outweigh its undesirable consequences

NO

YES

YES

NO

stre

ngth

of a

rec

omm

enda

tion

direction of a recommendation

for

against

NO

YES

YES

NO

stre

ngth

of a

rec

omm

enda

tion


for

against

STRONG

NO

YES

YES

NO

stre

ngth

of a

rec

omm

enda

tion


for

against

CONDITIONAL

STRONG

patients

clinicians

policy makers

Most individuals in this situation would want

the recommended course of action, and

only a small proportion would not

Most individuals should receive the intervention.

Adherence to this recommendation could

be used as a quality criterion or

performance indicator

The recommendation can be adopted as

policy in most situations.

STRONG CONDITIONAL

The majority of individuals in this situation would want the suggested course of action, but many would not

Different choices will be appropriate for individual patients. Must help each patient arrive at a management decision consistent with his or her values and preferences.

Policy making will require substantial debate and involvement of various stakeholders.

FRAMEWORKS

evidence to recommendation

just 4 columns & conclusions

CRITERIA

CRITERIA

Problem

Quality of evidence

Benefits & harms

Values

Resource use

Equity

Acceptability

Feasibility

CRITERIA

Problem

Quality of evidence

Benefits & harms

Values

Resource use

Equity

Acceptability

Feasibility

JUDGEMENTSRESEARCH EVIDENCE

ADDITIONAL INFORMATION

CRITERIA

Problem

Quality of evidence

Benefits & harms

Values

Resource use

Equity

Acceptability

Feasibility



what is to be considered?

CRITERIA

Problem

Quality of evidence

Benefits & harms

Values

Resource use

Equity

Acceptability

Feasibility



what do we consider about

it?

CRITERIA

Problem

Quality of evidence

Benefits & harms

Values

Resource use

Equity

Acceptability

Feasibility



what do we consider about

it?

��

CRITERIA

Problem

Quality of evidence

Benefits & harms

Values

Resource use

Equity

Acceptability

Feasibility



based on what evidence?

CONCLUSIONS

Balance of consequences

Decision /recommendation

Justification

Implementation considerations

Monitoring

Evaluation

Research priorities

example

Explanations Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013

Generic EtD framework 1

Evidence to decision framework

Question 1: Should probiotics vs. no probiotics be used in pregnant women?

Population:!pregnant!women!

Option:!probiotics!Comparison:!no!probiotics!Setting:3outpatient!Perspective:3individual!patient

Background: The$intestinal$microbiome$could$play$an$important$role$in$the$immune$system$maturation,$and$it$has$been$suggested$that$early6life$probiotic$administration,$whether$directly$to$the$infant$or$in$their$mothers$breast$milk,$may$reduce$the$risk$of$allergies$in$childhood.$The$objective$of$this$question$is$to$evaluate$the$impact$of$probiotics$administered$to$the$expecting$mothers$on$their$infant.

Subgroup considerations: subpopulation of women at high risk for allergy in a child

CRITERIA JUDGEMENTS RESEARCH EVIDENCE ADDITIONAL INFORMATION

PR

OB

LEM

Is the problem a priority?

No Probably no Uncertain Probably yes Yes

X

Allergic diseases represent a spectrum of health conditions and a worldwide burden in different populations. (1)

Are a large number of people affected?

No Probably no Uncertain Probably yes Yes

X

As many as 40% of the worldwide population is affected by any type of allergy. In infants prevalence depends highly on the allergic status of their parents, being approximately of 10% in those without an allergic parent or sibling, versus 20% to 30% in those with the atopic background in their relatives. (2)

Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013



CRITERIA JUDGEMENTS RESEARCH EVIDENCE ADDITIONAL CONSIDERATIONS

VA

LUE

S

Is there important uncertainty or variability about how much people value the main outcomes?

Important uncertainty

or variability

Possibly important

uncertainty or

variability

Probably no

important uncertainty

or variability

No important

uncertainty or

variability

No known undesirable outcomes

X

Detailed judgements

The relative importance or values of the main outcomes of interest:

Outcome Relative importance Certainty of the evidence Eczema critical low

Asthma/wheezing critical low

Food allergy critical low

Adverse effects critical low

We judged that the outcomes eczema, asthma and food allergy are critical for people. The adverse outcomes are probably of high importance and the burden of taking daily pills is limited. Some immunocompromised women might not accept the risk.

BE

NE

FIT

S &

HA

RM

S O

F T

HE

OP

TIO

NS

What is the overall certainty of the evidence of effectiveness?

No included studies Very low Low Moderate High

X

Summary of findings:

Outcome With [intervention]

Without [intervention]

Difference (per 100) (95%CI)

Relative effect (RR)

(95%CI)

Certainty of the

evidence (GRADE)

Eczema (follow-up 1 to 5 years)

365/1520 (24%)

484/1515 (31.9%)

9 fewer per 100 (from 4 fewer to 13

fewer)

RR 0.72 (0.6 to 0.86)

⊕⊝⊝⊝ VERY LOW

Asthma/wheezing (follow-up 2 to 7 years)

143/992 (14.4%)

139/982 (14.2%)

0 fewer per 100 (from 3 fewer to 3

more)

RR 0.97 (0.77 to 1.22)

⊕⊕⊝⊝ LOW

Food allergy (follow-up 1 to 2 years)

36/279 (12.9%)

41/284 (14.4%)

1 more per 100 (from 3 fewer to 8

more)

RR 1.08 (0.73 to 1.59)


Adverse effects 101/394 (25.6%)

88/397 (22.2%)

3 more per 100 (from 4 fewer to 12

more)

RR 1.13 (0.82 to 1.52)


Link to detailed evidence profile Subgroup considerations: Link(s) to summary of findings and judgments for subgroups

The data are indirect for all outcomes because they are primarily derived from studies that looked at mixed exposure in women during pregnancy and breastfeeding and of infants after birth. Only 1 RCT assessed the effect on eczema in pregnant women only: RR 0.88 (0.63 to 1.22); RD 5 fewer per 100 (from 14 fewer to 9 more) 5 RCTs included pregnant women + later breastfeeding mothers: RR 0.5 (0.4 to 0.63); RD 21 fewer per 100 (from 15 fewer to 25 fewer) 5 RCTs included pregnant women + infants after birth (follow-up 1 to 5 years): RR 0.87 (0.72 to 1.04), RD 4 fewer per 100 (from 8 fewer to 1 more) 3 RCTs included pregnant women + subsequently breastfeeding + infants (follow-up 3 to 4 years): RR 0.78 (0.49 to 1.24); RD 7 fewer per 100 (from 17 fewer to 8 more) No effects were observed on asthma/wheezing and food allergy.

How substantial are the desirable anticipated effects?

Don’t know

Not important

Somewhat important

Moderately important

Very important

Varies

X

X

Detailed judgements

There was some disagreement among panel members whether the effect is somewhat or moderately important.

How substantial are the undesirable anticipated effects?

Don’t know

Very important

Moderately important

Somewhat important

Not important

Varies

X

Detailed judgements

No serious adverse effects, and no difference in mild adverse effects between the groups.

Do the desirable effects outweigh the undesirable effects?

No Probably No

Don’t know Probably Yes

Yes Varies

X

Detailed judgements





RE

SO

UR

CE

US

E

How large are the resource requirements?

Large costs

Moderate costs

Small Moderate savings

Large savings

Varies

X

Detailed judgements

Prices are likely to vary substantially depending on the setting. This may be a particularly important consideration in low and middle-income countries. A level and type of insurance may play a substantial role as well. From a health systems point of view it might also be cost effective given that probiotic would be used for 9 months and cost of treatment of eczema may be distributed across many years.

Extremely limited research evidence (internet searches of drug prices)

Bifidobacterium bifidum (cost per person per year US$) Dose: 1 pill

each day

Lactobacillus gg (cost per person per

year US$) 1 pill each day

North-America

Average $181.16 $341.6

South-America

Average $174.3 $286

Europe

Average $167.86 $251.56

Fewer office visits would occur as a result of eczema if the effects on eczema were true.

How large is the incremental cost relative to the net benefit?

Very large ICER

Large ICER

Moderate ICER

Small ICER

Savings Varies

Detailed judgements

No research evidence

If eczema was reduced the intervention might be cost-effective given fewer office visits (between $17,400 and $34,100 to treat 100 people for 1 year or ¾ of that for 9 months) preventing 9 cases of eczema. In most studies probiotics were used in the last trimester of pregnancy, which, if used this same way, might reduce the cost per pregnant woman.

EQ

UIT

Y What would

be the impact on health inequities?

Increased Probably increased

Uncertain Probably reduced

Reduced Varies

X

Detailed judgements


In some settings it may be important to consider equity as the access may depend on socioeconomic status of the country or setting where coverage will depend on policymakers.

AC

CE

PT

AB

ILIT

Y

Is the option acceptable to key stakeholders?

No Probably No

Uncertain Probably Yes

Yes Varies

X

Detailed judgements






FE

AS

IBIL

ITY

Is the option feasible to implement?

No Probably No

Uncertain Probably Yes

Yes Varies

X

Detailed judgements





Recommendation Should probiotics vs. no probiotics be used in pregnant women (exposing their children in utero)?

Overall balance of consequences Undesirable consequences clearly outweigh desirable

consequences

Undesirable consequences probably outweigh desirable

consequences

The balance of desirable and undesirable consequences indicates

they are very similar*

Desirable consequences probably outweigh undesirable

consequences

Desirable consequences

clearly outweigh undesirable

consequences

! ! ! X !

We recommend against the option or for the alternative

We suggest not to use the option or to use the alternative

We suggest using the option We recommend the option

! ! X !

Panel decisions 3 panel members with potential COI recused themselves from participating in formulating the recommendation. Consensus was obtained from the rest of the team.

Recommendation (text) The guideline panel suggests using probiotics in pregnant women at high risk for allergy in their children (conditional recommendation, very low quality evidence).

Remarks and justification Most studies commenced probiotics in the last trimester of pregnancy. The very low quality evidence for adverse effects indicates that our confidence in the absence of increased adverse effects is low. Future research is needed (see definitions of very low quality) e.g., generalizing to immune-compromised children

Subgroup considerations Women with high risk of allergy in their children Women with average risk of allergy in their children

Implementation considerations This recommendation is based on trials investigating the probiotics or mixtures of probiotic listed below. We have not found a difference between these different probiotics, but that does not mean there is no difference

Monitoring and evaluation considerations

Research priorities Develop instruments for evaluating the risk of allergy in children as the family history predicts only about 30% of the population risk. There is some evidence that first child is at higher risk for allergy than subsequent children. Long-term follow-up of long-term effects. No direct evidence for the use of probiotics in formula – this should be evaluated in future research and is an unmet need.


OUTLINE

• what are clinical practice guidelines?

• why should we use them?

• what are recommendations and levels of

evidence?

• how to interpret a CPG recommendation?

tools / more readings

• guidelinedevelopment.org

• cebgrade.mcmaster.ca

• gradeing.cochrane.org

GRADEingMethods

Thank you! [email protected]@CharlieNeck

�

grade approach for making recommendations in clinical practice guidelines #wac2015

Health & Medicine