Developing guidelines What’s the difference in

process if you use GRADE? Nancy Santesso, RD, MLIS, PhD

McMaster University

Department of Health Research Methods, Evidence and Impact

19 January 2017

Disclosures

• Deputy Director of Cochrane Canada

www.canada.cochrane.org

• Member of the GRADE Working Group

www.gradeworkinggroup.org

Standard guideline development process

Establish group Decide questions

and outcomes Gather evidence and synthesise

Assess evidence Make

recommendation

Standard guideline development process – Another full session for people interested in doing a GRADE assessment

Establish group Decide questions

and outcomes Gather evidence and synthesise

Assess evidence Make

recommendation

Materials: available on the website

• Section about questions and outcomes

• Section about pulling out the data for outcomes

• Section about how to present synthesised data for making recommendations

GRADE approach

Background information

Good practice statements

Recommendation

Evidence to support recommendation

Recommendations are ACTION statements describing what to do For women in the first trimester, we suggest medical abortion as a first option rather than surgical abortion.

For women with confirmed ectopic pregnancy in the first trimester, we suggest surgical abortion as a first option rather than medical.

For pregnant women who are Rh negative who undergo medical abortion before 49 days from last menstrual period, we recommend to not administer Rh immunoglobulin.

Recommendations are NOT…

”Treatment X leads to these complications…”

“Clinicians should use evidence based treatments…”

What are ‘summary statements’? There is no strong evidence supporting routine antibiotic prophylaxis for medical abortion. (Level II-2) Something like…We suggest to not provide antibiotic prophylaxis for medical abortion.

Why is it not a recommendation?

What should a recommendation look like?

Institute of Medicine Standards

National Guideline Clearing House

Recommendation

Summary statements

Detailed summary of evidence

Very particular about framing the question and choosing outcomes Ask questions when…

• it’s not clear what clinicians should do – guidance is needed

• there is some controversy about what to do

• there is some evidence (e.g. not a very new procedure not available in practice yet)

Should

A or B

be used in

people with X?

Materials: Brainstorming questions

1. Do prophylactic antibiotics reduce the risk of post abortion endometritis among women having first/second trimester surgical abortion?

a) What prophylactic antibiotic regimens are associated with reduced risk of post abortion endometritis among women having first/second trimester abortion?

b) What are the risks associated with prophylactic antibiotics?

Should

A or B

be used in

people with X?

What is the A? What is the B? Who are the people?

Antibiotic prophylaxis No prophylaxis Women having 1st/2nd …

What about the risks? Where do they fit in?

1. Do prophylactic antibiotics reduce the risk of post abortion endometritis among women having first/second trimester surgical abortion?

a) What prophylactic antibiotic regimens are associated with reduced risk of post abortion endometritis among women having first/second trimester abortion?

b) What are the risks associated with prophylactic antibiotics?

Should

antibiotic prophylaxis or

no prophylaxis

be used in

women in 1st/2nd trimester…?

You will decide based on all the benefits and harms of each treatment; and burden, resource use, acceptability, feasibility

In women undergoing second-trimester medical abortion, does feticide confer any benefits over not inducing fetal death?

INSTEAD

In women undergoing second-trimester medical abortion, should feticide or not inducing fetal death be performed? Consider benefits, harms, resources, etc.

Should A versus B be used for medical abortion in women during the first trimester

Abortion completed with intended method

Time to completed abortion

Haemorrhage requiring blood transfusion

Haemorrhage not requiring transfusion

Surgical abortion

Side effects (GI)

Pain

Hospitalisation

Satisfaction

Acceptability

Many outcomes and more!

Decide ahead of time which ones are important to

decision making

Selective use of outcomes in recommendations

• Similar to selective reporting of outcomes in trials

• Recommendations about whether to use a treatment or not, are often made on selective use of outcomes

• We should decide ahead of time what are the most important outcomes that will determine the recommendation

Does the balance of benefits and harms favour A or B?

Choose 5-7 important outcomes – look very hard for

that evidence

Use time wisely Good decision making

Medical method A

• Increase benefits

• Increase minor infections

Medical method B

• Increase benefits

• Increase minor bleeding

Advocates for B may argue that minor infections are a more critical side effect and so should not recommend medical method A

Decide ahead of time what outcomes are important to avoid bias

Establish important outcomes early Quick survey or discussion/email

Standard guideline development process

Establish group Decide questions

and outcomes Gather evidence and synthesise

Assess evidence Make

recommendation

• Brainstorm questions • Format in “should A versus B be used in patients with X”

• Brainstorm important outcomes (include benefits and risks) • Rate the importance of outcomes to limit to 5-7 outcomes

GRADEd recommendation? Best practice statement? Or background?

Good Practice Statements: How to identify?

Questions to decide if is better as a good practice statement

Good practice statement

Is the message really necessary in regard to actual

health care practice?

Yes many clinicians are not doing this

Questions to decide if is better as a good practice statement

Good practice statement

Is the message really necessary in regard to actual

health care practice?

Yes many clinicians are not doing this

After consideration of all relevant outcomes and

potential downstream consequences, will implementing

the good practice statement result in large net positive

consequences.

Yes, serious consequences of complications can

be avoided and information can be provided with

little to no harm but great benefit

Questions to decide if is better as a good practice statement

Good practice statement

Is the message really necessary in regard to actual

health care practice?

Yes many clinicians are not doing this

After consideration of all relevant outcomes and

potential downstream consequences, will implementing

the good practice statement result in large net positive

consequences.

Yes, serious consequences of complications can

be avoided and information can be provided with

little to no harm but great benefit

Is collecting and summarizing the evidence a poor use

of a guideline panel’s limited time and energy

(opportunity cost is large)?

We would likely have to contact hospitals for data

about women who have come for urgent

care…knowledge of women and awareness,

current practices of clinicians

Questions to decide if is better as a good practice statement

Good practice statement

Is the message really necessary in regard to actual

health care practice?

Yes many clinicians are not doing this

After consideration of all relevant outcomes and

potential downstream consequences, will implementing

the good practice statement result in large net positive

consequences.

Yes, serious consequences of complications can

be avoided and information can be provided with

little to no harm but great benefit

Is collecting and summarizing the evidence a poor use

of a guideline panel’s limited time and energy

(opportunity cost is large)?

We would likely have to contact hospitals for data

about women who have come for urgent

care…knowledge of women and awareness,

current practices of clinicians

Is there a well-documented clear and explicit rationale

connecting the indirect evidence?

Yes, we can link practice, women’s knowledge,

visits to urgent care…

What components of the history and physical are indicated in a post-abortion follow-up visit?

• Should this be a recommendation?

• Should this be a good practice statement?

• Or is this simply background information about doing a history and physical?

Is the message really necessary in regard to

actual health care practice?

Standard guideline development process

Establish group Decide questions

and outcomes Gather evidence and synthesise

Assess evidence Make

recommendation

Systematic review of the evidence – labour intensive

• Search for reviews and/or studies

• Select studies

• Extract data from studies and assess studies

• Synthesise and assess evidence

Search for reviews and/or studies – strategic process

• Start searching for previously conducted reviews (and possibly update those reviews) – watch out for non-systematic reviews!

• Conduct specific searches to limit the number of papers

• Restrict to randomised controlled trials when possible

• Restrict to large observational studies when possible

Extract data from studies and assess studies

• Start by organising data immediately by outcomes – only do the 5-7 outcomes that were ranked as important

• Decide whether a meta-analysis or statistical synthesis of the data will be done

• If not, see example in handout

Medical abortion with A versus B in first trimester

We found 8 studies, some studies measured some of our outcomes

• Study 1 measured completed abortion and found X, it also measured haemmorhage and found Y. For pain, it found Z.

• Study 2 measured completed abortion and found X.

• Study 3 found that pain was reduced, and minor haemorrhage was increased.

Study and outcome Results Risk of bias of study

Completed abortion

Study 1 50/50 with A; 43/51 with B

Study 2 73/84 with A; 70/85 with B

Study 3 …

Haemorhage (no transfusion)

Study 1 All bleeding reported: 2/50 with A; 4/51 with B

Study 2 5/84 with A; 4/85 with B

…

Medical abortion with A versus B in first trimester

We found 8 studies, some studies measured some of our outcomes

Study and outcome Results Risk of bias of study

Completed abortion

Study 1 50/50 with A; 43/51 with B

Study 2 73/84 with A; 70/85 with B

Study 3 …

Haemorhage (no transfusion)

Study 1 All bleeding reported: 2/50 with A; 4/51 with B

Study 2 5/84 with A; 4/85 with B

…

Next Extract data from studies and assess studies

Synthesise and assess evidence • See materials and table

Part One: Make a

summary

Part One: Make a

summary

Part Two: Assess the evidence -

GRADE

GRADE: What to consider when assessing evidence?

• Do I believe the results from these studies? Risk of bias

• Are the results consistent across studies? Inconsistency

• Are the results applicable to my question? Indirectness

• Is this effect precise enough? Imprecision

• Are these all of the studies? Publication bias

• Plus large effect, dose response, opposing confounding

Other systems

We know that we consider a few more criteria

What about the number of people in

the studies?

What about results that are totally

inconsistent across the studies?

This is about evidence for one outcome!

• Do I believe the results from these studies? Risk of bias

• Are the results consistent across studies? Inconsistency

• Are the results applicable to my question? Indirectness

• Is this effect precise enough? Imprecision

• Are these all of the studies? Publication bias

• Plus large effect, dose response, opposing confounding

Prophylactic antibiotics with medical abortion

Outcome Evidence

Risk of infection 5 studies, 324 pregnant women

Completion 4 studies, 285 pregnant women

Gastrointestinal side effects

8 studies, 566 pregnant women

… 3 studies, 200 pregnant women

…

Prophylactic antibiotics with medical abortion

Outcome Effects of immediate

Absolute risk difference Certainty in effect

Risk of infection RR 0.71 (0.45 to 1.12) 9 fewer infections per 1000

HIGH

Completion RR 1.14 (0.82 to 1.59)

LOW

Gastrointestinal side effects

RR 2.00 (1.44 to 2.59) 120 more GI events per 1000

MODERATE

…

2 more completions per 1000

We are very certain about 9 fewer infections, and less certain about the GI events, and uncertain about completion – suggest antibiotics

Prophylactic antibiotics with medical abortion

Outcome Effects of immediate

Absolute risk difference Certainty in effect

Risk of infection RR 0.71 (0.45 to 1.12) 9 fewer infections per 1000

LOW

Completion RR 1.14 (0.82 to 1.59)

LOW

Gastrointestinal side effects

RR 2.00 (1.44 to 2.59) 120 more GI events per 1000

HIGH

…

2 more completions per 1000

We are uncertain about 9 fewer infections, and very certain about the GI events, and uncertain about completion – suggest no antibiotics

Important to consider the effects but also certainty in those effects when making

recommendations

More about how to GRADE in another webinar

Making recommendations

In women undergoing second-trimester medical abortion, should feticide or not inducing fetal death be performed? Consider benefits, harms, resources, etc.

In women undergoing second-trimester medical abortion, should feticide or not inducing fetal death be performed?

How large are the benefits of this medication? How large are the harms? What is the balance of benefits and harms? Will it save money or cost money to use it? Is it acceptable to provide? Is it feasible to provide? Based on the answers, the guideline group decides whether to recommend it

Strength of recommendation does not rely on the quality of the

evidence

Just one part

In women undergoing second-trimester medical abortion, should feticide or not inducing fetal death be performed?

How large are the benefits of this medication? SMALL – HIGH CERTAINTY How large are the harms? TRIVIAL – HIGH CERTAINTY What is the balance of benefits and harms? PROBABLY FAVOURS FETICIDE Will it save money or cost money to use it? LARGE COSTS Is it acceptable to provide? YES Is it feasible to provide? NOT FEASIBLE IN MOST SETTINGS WE SUGGEST NOT INDUCING FETAL DEATH

We suggest… We recommend… Based on these criteria

• How large are the benefits of this medication?

• How large are the harms?

• What is the balance of benefits and harms?

• Will it save money or cost money to use it?

• Is it acceptable to provide?

• Is it feasible to provide?

Making recommendations: Balance of all factors

For Against

Benefits and Harms Certainty in evidence

Values Acceptability

Feasibility Equity Costs

For Against

Should we recommend feticide or not inducing death? Weak/conditional recommendation against…

Weak/conditional recommendation

AGAINST A

For Against

Should we recommend feticide or not inducing death? Strong recommendation against…

Strong recommendation

AGAINST A

Additional information about interpretation

Standard guideline development process

Establish group Decide questions

and outcomes Gather evidence and synthesise

Assess evidence Make

recommendation

Standard guideline development process – Another full session for people interested in doing a GRADE assessment

Establish group Decide questions

and outcomes Gather evidence and synthesise

Assess evidence Make

recommendation

More information available on the SOGC website