can we teach critical appraisal the gate approach rod jackson university of auckland, nz september...

Can we teach critical appraisal

‘the GATE approach’

Rod Jackson

University of Auckland, NZ

September 2010

in 30 minutes!

GATE: a Graphic Appraisal Tool for Epidemiology

1 picture, 2 formulas &

3 acronyms

A picture: the GATE frame

the shape of every epidemiological study

©

Participants

Exposure Group Comparison Group

OutcomesTime

P

E C

O

T

The PECOT acronym: the 5 parts of every epidemiological study

All epidemiological studies can be hung on the GATE frame

2 formulas: study analyses

1. Occurrence of disease

= Numerator ÷ Denominator ÷ Time

2. Random error (95% Confidence Interval)

= 1.96 x Standard Error

D

N

P

E C

OT

P

E

C

O

T

Recruitment

Allocation

Maintenance

Measurement of outcomes

blind or

objective

The RAMMbo* acronym: assessing study bias

* Paul Glasziou

Participants

Exposure Group Comparison Group

OutcomesTime

P

E C

O

T

GATE frame picture & PECOT acronym

Describe a study’s design by hanging PECOT on the frame

Participants

Study Setting

Eligible Participants

ParticipantsP

Fill in for

DVT in long-haul flights.Lancet 2001; 357:1485-9

Page 95 in Glasziou et al

ParticipantsStudy Setting: volunteers, UK, ? 1990s

Eligible Participants: no previous DVT, > 50 yrs, planned economy air travel 2 sectors > 8 hours

Participants: 200, mean age 61-62 years

P

DVT in long-haul flights: Lancet 2001;357:1485-9

Exposure & Comparison Groups

Exposure or Intervention Group

(EG)

Comparison or Control Group

(CG)EG CG

Fill in for

DVT in long-haul flights.Lancet 2001; 357:1485-9

Page 95 in Glasziou et al

Exposure & Comparison Groups

Exposure or Intervention Group

(EG):Below knee compression

stockings

Comparison or Control Group

(CG):no stockings

100 100

115 116

Outcomes (O)

Outcomes (O)Oa b

c d

yes

no

‘Dis-ease’

Fill in for

DVT in long-haul flights.Lancet 2001; 357:1485-9

Page 95 in Glasziou et al

Outcomes (O)

Outcomes (O)O

a= 0

b= 12

c d

yes

noDVT

100 100

Time (T)

T

incidence

prevalence

Fill in for

DVT in long-haul flights.Lancet 2001; 357:1485-9

Page 95 in Glasziou et al

Time (T)

T= from take-off to 2 days post

final flight

incidence

Outcome: e.g. death

0 0

Time (T)

T = at post-flight assessment (<48 hrs)

prevalence

Outcome: number with

DVTs 0 12

Study design: GATE frame & PECOT

Participants

Exposure Group Comparison Group

OutcomesTime

P

E C

O

TEvery epidemiological study hangs on the GATE frame

Participants

Exposure Group:stockings

Comparison Group:no stockings

Outcomes:DVT

Time:at post flight assess

231

100 100

DVT in long-haul flights: Lancet 2001;357:1485-9

115 116

Setting: UK volunteers – 479 consideredEligible: > 50 yrs, no DVT, flying > 8 hrs

0 12

2 formulas: study analyses

1. Occurrence of disease

= Numerator ÷ Denominator ÷ Time

2. Random error (95% Confidence Interval)

= 1.96 x Standard Error

D

N

Denominator (Participants)D

N Numerator (Outcomes)

O = N÷D

All epidemiological studies involve measuring the OCCURRENCE of disease

Occurrence = Numerator ÷ Denominator

GATE study analyses

P

EG CG

O

Exposure Group (EG)

Numerator 1: a

Comparison Group (CG)

Overall Denominator

a b

c d

Numerator 2: b

Describe a study’s analyses by hanging the numbers on the frame

Denominator 1: Denominator 2:

Occurrence = N ÷ D

P

EG CG

O

Denominator 1:Exposure Group

EG

Numerator 1:a

Denominator 2:Comparison Group

CG

a b

c d

Numerator 2:b

Exposure Group Occurrence:EGO = a ÷ EG

Comparison Group Occurrence:CGO = b ÷ CG

Calculate EGO & CGO for the outcome ‘DVT’ post long-haul flight

P

EG CG

Oa b

c d

Denominator 1:Exposure Group

EG = 100

Numerator 1:a = 3*

Denominator 2:Comparison Group

CG = 100

Numerator 2:b = 12

EGO = 3/100 people at post flight assessment

CGO = 12/100= people at post flight assessment

* a = 0 but have used 3 to illustrate calculations on next slide

Describing differences between occurrences

Relative difference or Relative Risk = EGO ÷ CGO

Absolute Difference or Risk Difference = EGO - CGO

Number Needed To Treat (NNT) = 1 ÷ RD

Fill in for

DVT in long-haul flights.Lancet 2001; 357:1485-9

Page 95 in Glasziou et al

Describing differences between occurrences

Relative difference or Relative Risk = EGO ÷ CGO

Absolute Difference or Risk Difference = EGO - CGO

Number Needed To Treat (NNT) = 1 ÷ RD

= 3/100 ÷ 12/100 = 3/12 = 0.25

= 3/100 - 12/100 = - 9/12 = - 7.5 /100

= 1 ÷ (- 7.5 /100) = - 100/7.5 = 13.3

Analyses

it’s all about EGO & CGO

P

E C

OT

P

E

C

O

T

Recruitment

Allocation

Maintenance

Measurement of outcomes

blind or

objective

The RAMMbo acronym: assessing study bias

RAMMbo

E C

OT

appropriate Recruitment processes?P

• Study setting & eligibility criteria well described?

e.g. Recruit random/representative sample OR consecutive eligibles OR volunteers from advertisements• Participants representative of eligibles?• Prognostic/risk profile appropriate to

study question?

P

Study setting

Eligible people

RAMMbo

E C

OT

appropriate Recruitment processes?P

• Study setting & eligibility criteria well described?

Recruited volunteers from advertisements; eligibility criteria well described• Participants representative of eligibles?479 considered, 248 excluded - probably• Prognostic/risk profile appropriate to

study question? yes

P

Study setting

Eligible people

EG CG

OT

P

RCT: Allocate randomly by investigators (e.g drugs)

EG CG

OT

P

Cohort: Allocate by measurement (e.g. smoking)

RAMMbo: A is for Allocation

were EG & CG similar at baseline?

EG CG

OT

P

RCT: Allocate randomly by investigators using sealed envelopes

RAMMbo: A is for Allocation

EG & CG similar at baseline except more women in stocking group

RAMMbo

EG CG

OT

good Maintenance?did most participants remain in allocated groups (EG &

CG)

P

Participants &/or investigators blind to exposure (and comparison exposure)?

Compliance high & similar in EG & CG?Contamination low & similar in EG & CG?Co-interventions low & similar in EG & CG?

Completeness of follow-up high & similar in EG & CG?

RAMMbo

100 100

OT

good Maintenance?did most participants remain in allocated groups (EG &

CG)

P

Participants &/or investigators blind to exposure (and comparison exposure)? no

Compliance high & similar in EG & CG? dk*Contamination low & similar in EG & CG? dkCo-interventions low & similar in EG & CG? dk

Completeness of follow-up high & similar in EG & CG? probably ok, lost < 15%* dk = don’t know

115 116

RAMMbo

EG CG

OT

Measurement of outcomesblind or objective?

P

If outcome measurements not Objective (eg. automated or definitive) were investigators blind to exposure (and comparison exposure)

RAMMbo

EG CG

OT

Measurement of outcomesblind or objective?

P

Outcome measurements reasonably objective (ultra-sound)Technician was blind to exposure (although some participants may have had a stocking line)

The 4 (GATE) study biases

P

E C

OT

Recruitment bias

Allocation bias

Maintenance bias

Measurement (of outcomes) bias

P

E C

OT

Recruitment ok

Allocation ok

Maintenance ok

Measurement (of outcomes) ok

DVT in long-haul flights: Lancet 2001;357:1485-9

2 formulas: study analyses

1. Occurrence of disease

= Numerator ÷ Denominator ÷ Time

2. Random error (95% Confidence Interval)

= 1.96 x Standard Error

D

N

2 formulas: study analyses

1. Occurrence of disease

= Numerator ÷ Denominator ÷ Time

2. Random error (95% Confidence Interval)

= 1.96 x Standard Error

EGO = 0%; 95% CI = 0% - 3%

CGO = 10%; 95% CI = 4.8% - 16%

D

N

Excel CATs & paper Gate-lites

There is a GATE for every study designwww.epiq.co.nz

the 3rd acronym:SRs & meta-analyses

• Find appropriate studies?

• Appraise selected studies?

• Include only valid studies?

• Total-up (synthesise) appropriately?

• Heterogeneity adequated addressed?

In the 19th century we made great advances in health through the provision of clean, clear water; in the 21st century

we will make the same advances through clean, clear (systematically

reviewed) information.Muir Gray

Systematic reviews are one of the greatest ideas of modern thought.

They should be celebrated.

Ben Goldacre

Muhammad Shafique Sajid et al. Knee-length graduated compression stockings for thromboprophylaxis in air travellers: A meta-analysis. Int J Angiol. 2008; 17: 119–123.

Nine trials studying participants using KL stockings were analyzed. Forty-six of 1261 participants randomly assigned to the control group developed deep vein thrombosis (DVT), compared with two of 1237 participants (0.16%) in the KL stockings group. …..There was an absolute difference of 3.4% in the incidence of DVT, in favour of KL stockings. The number needed to treat with KL stockings to avoid one case of DVT was 29.4. However, there was significant heterogeneity among trials. The RR for DVT was 0.08 in high-risk participants and 0.14 in low- to medium-risk participants.

GATE can also be used to frame the first 4 steps of EBP

The 4 steps of EBP1. Ask a focused question.

2. Access (systematically search for) epidemiological evidence to help answer question.

3. Appraise evidence found for its validity, effect size, precision (ideally all the relevant evidence)

4. Apply the evidence:

a. amalgamate the valid evidence with other relevant information (patient/community values, clinical/health issues, & policy context) to make a good decision; and

b. Act (implement) the decision in practice

Participants

Exposure Group Comparison Group

OutcomesTime

P

E C

O

T

the PECOT acronym: the 5 parts of every epidemiological study

All epidemiological studies can be hung on the GATE frame

EBP Step 1: ASK - turn your question into 5 parts (PECOT)

1. Participants (the patient problem)

2. Exposure (e.g. a therapy)

3. Comparison (there is always an alternative! - another therapy or no treatment…

4. Outcome (e.g. a disease you want to prevent or manage)

5. Time frame (over which you expect a result)

EBP Step 2: ACCESS the evidence – use PECOT to identify search terms

1. Participants (the patient problem)

2. Exposure (e.g. a therapy)

3. Comparison (there is always an alternative! - another therapy or no treatment…

4. Outcome (e.g. a disease you want to prevent or manage)

5. Time frame (over which you expect a result)

P

E C

OT

P

E

C

O

T

Recruitment

Allocation

Maintenance

Measurement of outcomes

blind or

objective

EBP Step 3: APPRAISE the evidence – using PECOT & RAMMbo

EBP Step 4: APPLY the evidence by: a. AMALGAMATING the relevant information & making

an evidence-based decision:’ the X-factor

©

Epidemiologic evidence

Clinical / health considerations

Policy issues

Patient / community preferences

X-factor: making evidence-based decisions

expertise: ‘putting it all together’ the art of practice

Step 4b

ACT

Step 5: EBP 360°1. Ask a focused question.

2. Access (systematically search for) epidemiological evidence to help answer question.

3. Appraise evidence AND then meta-analyse (systematically review) ALL relevant valid evidence.

4. Apply the best evidence:

a. amalgamate the valid evidence with other relevant information to make a good decision; and

b. ACT on your decision5. AUDIT your practice (i.e. check your actual practice –

‘actions’ - against ‘best’ evidence-based practice)

= EBP + Quality Improvement