how can we reduce diagnostic error? - fmh€¦ · how can we reduce diagnostic error? ... accuracy...

+ HOW CAN WE REDUCE

DIAGNOSTIC ERROR?

GEOFF NORMAN, PhD

McMaster University

+ Strategies to reduce errors

GENERAL INSTRUCTIONS

Slowing down, be attentive, be systematic etc.

HEURISTIC - BASED STRATEGIES

“Cognitive Forcing Strategies” (Croskerry)

KNOWLEDGE- BASED STRATEGIES

During Diagnostic Process

Reflection (Mamede & Schmidt, Monteiro)

As a Learning Strategy

+

“…errors of intuitive judgment involve failures of both

systems: System 1, which generated the error, and System 2,

which failed to detect and correct it.”

D. Kahnemann, 2004

+

`“Most errors occur with [System] 1 and may to

some extent be expected whereas [System] 2

errors are infrequent and unexpected…”

P. Croskerry, 2009

+ GENERAL INSTRUCTIONS

“What can be done about biases? …The short answer is that

little can be achieved without a considerable investment of

effort… S1 is not readily educable…”

“The way to block errors that originate in System 1 is simple in

principle: recognize that you are in a conceptual minefield,

slow down, and ask for reinforcement from System 2.

[emphasis ours]“

(Thinking fast and slow, p.417 ).

+

Do errors result from rapid pattern recognition (System 1)

processes;

Does slower, more methodical problem-solving reduce

errors?

+ Norman, Brooks, Rosenthal, 1998

Accuracy and Time in Dermatology

100 slides in 20 categories

Students, clerks, residents, GPs, Dermatologist

Accuracy and Response Time

Accuracy by Educational

Level

0

10

20

30

40

50

60

70

80

90

100

Stud

ent

Clerk

Res

iden

tGP

Der

mto

logist

% c

orre

ct

Response time by

Educational Level

0

5

10

15

20

25

30

Stud

ent

Clerk

Res

iden

tGP

Der

mto

logist

Res

ponse

Tim

e

CorrectIncorrectD K

+ Sherbino, 2010

Accuracy and Time in Internal Medicine

75 Canadian PGY2 internal medicine

20 written I.M. cases

“Proceed as rapidly as you can but try not to make any

mistakes”

+ Rapid Instructions

You are to make your diagnosis and type it

in as quickly but as accurately as possible.

Case information will appear on one

screen, and you click on a button to go to

the diagnosis screen. You may spend as

much time as you wish reading the case

information, but remember that you only

have 30 minutes to complete all the cases.

R = -.55

ACCURACY VS. READING TIME

+

Accuracy is associated with

shorter time

Longer times reflect

uncertainty

+ Norman et al., 2011

Controlled trial

Rapid reasoning (2010) vs. Systematic approach (2011)

20 cases

PGY2 residents

96 Rapid (2010)

108 Systematic (2011)

+ Systematic Instructions

Be careful and thorough. Try not to skip

anything. Consider all the data. Take as long

as you want on that screen, but when you click

on button to go to diagnosis screen, you can’t

go back to the information. There is a counter

in the upper right corner that tells you how

many cases you’ve done. You are to consider

all the data and then make your diagnosis and

type it in.

+ Comparison of Rapid and

Systematic Instructions

+ Comparison of Rapid and

Systematic Instructions

+ Heuristic Based Strategies

+ Cognitive Forcing Strategies

“Becoming alert to the influence of bias requires maintaining

keen vigilance and mindfulness of one’s own thinking. When a

bias is identified by a decision-maker, a deliberate decoupling

from the intuitive mode is required so that corrective

“mindware” can be engaged from the analytical mode.”

Croskerry, 2013

+ CAN CLINICIANS IDENTIFY WHEN

A BIAS IS PRESENT?

Zwaan et al, 2016

+ Bias Definitions

Anchoring The tendency to perceptually lock onto

salient features in the patient’s initial

presentation too early in the diagnostic

process, and failing to adjust this initial

impression in the light of later information.

Availability Heuristic The disposition to judge things as being more

likely or frequently occurring, if they readily

come to mind. Thus recent experience with a

disease may inflate the likelihood of its being

diagnosed. Conversely, if a disease has not

been seen for a long time (i.e. is less available)

it may be underdiagnosed.

Base Rate Neglect The tendency to ignore the true prevalence of

a disease, either inflating or reducing its base

rate, and distorting Bayesian reasoning

Confirmation Bias The tendency to look for confirming data to

support a diagnosis rather than look for

disconfirming evidence to refute it, despite

the latter often being more persuasive and

definitive

Premature Closure The tendency to apply premature closure to

the decision-making process, accepting a

diagnosis before it has been fully verified.

The consequences of the bias are reflected in

the maxim - ”when the diagnosis is made, the

thinking stops.”

Representativeness Bias The tendency to look for prototypical

manifestations of disease. Restraining

decision-making along pattern-recognition

lines leads to atypical variants being missed.

+ Subjects

Members of Society to Improve Diagnosis in Medicine

(mailing list)

Practicing physicians

Initial approach and consent (n = 75)

Web based administration (n =37)

+ Methods

Created 50/50 cases

Two approximately equally likely diagnoses

Experimental manipulation of test positive / negative

Measurement

Was a diagnostic error committed?

Which of the following biases were present?

+ PE or Pneumonia

History of Present Illness A 43-year old woman was brought to the emergency

department by her husband at 0200 in the morning because of shortness of breath.

The dyspnea occurred suddenly at 1100 pm …The patient complained of nausea …

She has had no recent surgery.

Past Medical History

…The ECG demonstrates non-specific ST depression in V3-V6.

A Chest X-ray was ordered to diagnose pneumonia.

The chest x-ray demonstrated an opacity …The patient was prescribed a course of

antibiotics, …and instructed to follow-up …

Two days later, the patient was seen in the clinic for follow-up with

her primary care physician. She reported marked improvement in

her chest pain and shortness of breath, as well as resolution of

her fevers and chills. She was instructed to complete her course

of antibiotics.

+ PE or Pneumonia

History of Present Illness A 43-year old woman was brought to the emergency

department by her husband at 0200 in the morning because of shortness of breath.

The dyspnea occurred suddenly at 1100 pm …The patient complained of nausea …

She has had no recent surgery.

Past Medical History ….The ECG demonstrates non-specific ST depression in V3-V6.

A Chest X-ray was ordered to diagnose pneumonia.The chest x-ray demonstrated an

opacity …The patient was prescribed a course of antibiotics, …and instructed to

follow-up …

Two days later, the patient was seen in the clinic for follow-up with

her primary care physician. She reported continued chest pain

and shortness of breath, and several episodes of hemoptysis. A

CT Pulmonary Angiogram was ordered to diagnose a

pulmonary embolism. This demonstrated a pulmonary

embolism in the segmental pulmonary artery of her left lower

lobe. A heparin drip was started and the patient was admitted to

the hospital.

DIAGNOSIS A DIAGNOSIS B

Pneumonia Pulmonary embolism

Acute MI Type A aortic dissection

Tubo-ovarian abscess Appendicitis

Subarachnoid hemorrhage Meningitis

Kidney stone Type B aortic dissection

Pyelonephritis AAA

Pancreatitis Cholecystitis

Cellulitis DVT

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%P

erc

en

t o

f R

es

po

ns

es

Specific Bias with Consistent and Inconsistent Outcome

Consistent Inconsistent

Consistent Inconsistent Relative

Increase

KAPPA*

Anchoring 35% 70% 100% 0.0

Availability 25% 55% 120% .025

Confirmation 35% 62% 77% .024

Base Rate 11% 28% 154% .063

Premature Closure 39% 88% 125% .046

Representativeness 26% 45% 73% .044

+ COGNITIVE DEBIASING –

Effect on Errors

Smith and Slack (2015)

Sherbino et al., (2012, 2013)

+

(Smith and Slack, 2015)

19 Fam Med residents

Debiasing workshop

Evaluation by preceptor with actual

patients before and after

+ Accuracy and Ability to Recognize

Bias

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

PRE POST

Accuracy

Recognition

+

(Sherbino et al., 2013)

198 students

Intervention 145

Control 46

Instruction on two biases

Search satisficing

Locate the second lesion on X ray / ECG

Availability

Identify the rare diagnosis

Test on 6 cases

Near transfer, far transfer, “False positive”

+ Search Satisficing

0%

20%

40%

60%

80%

100%

120%

CFS Control

PRIMARY

SECONDARY

+ Availability

0%

20%

40%

60%

80%

100%

120%

CFS Control

Common Inc

Uncommon Corr

+ SUMMARY

Cognitive debiasing strategies:

No apparent agreement on presence of absence of specific

biases when cases are not preselected to illustrate a bias

Students can learn to identify biases in contrived situations (e.g.

Reilly, 2013; Ogdie, 2012)

No effect of cognitive debiasing training on diagnostic errors

+

“ If you have not heard about myasthenia

gravis, you cannot cognitively debias your

way into that diagnosis. You can spend all

day in system 2 and collect more and more

information, but if you do not have a well-

developed illness script that contains

atypical manifestations of heart failure, you

will never recognise it. In the realm of expert

performance, knowledge is king.”

Gurpreet Dhaliwal, BMJ Qual Saf, 2016

+ KNOWEDGE-BASED STRATEGIES

During the Diagnostic Process

Reflection

- Structured

- Self-initiated

As a Learning Strategy

+ Structured Reflection

(Mamede & Schmidt)

With case description in front of them:

Write down most likely diagnosis

Write down alternative diagnoses

List findings

Supporting

Against

Not present

Rank diagnoses in order of likelihood

+

(Psych Res 2010)

34 PGY2 residents

50 medical students

6 complex, 6 easy cases

Analytic reflective reasoning vs.

Write down first thing that comes to mind

+ Reflection vs. “First thing”

Residents

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Simple Complex

Reflection

First thing

+ Reflection vs. “First thing”

Students

0%

10%

20%

30%

40%

50%

60%

70%

Simple Complex

Reflection

First thing

+

Mamede and Schmidt, 2012

46 Year 4 med students

4 learning cases (2/2) , 6 test cases (2/2/2)

Immediate test / Delayed (1 week)

+ Reflection vs. “First thing”

Immediate and Delayed

0%

10%

20%

30%

40%

50%

60%

70%

Immediate Delayed

Reflection

First thing

+ Do clinicians know when to reflect?

+ Self-initiated Reflection (Monteiro, 2013)

47 residents

27 PGY1, 15 PGY2, 23 PGY3

16 cases

First Impression (Fast) then Reflection

After first pass through cases, review case again and either confirm or

revise diagnosis

+

Pass 1, 746 diagnoses / 316 correct

Pass 2, 62 diagnoses (8%) revised

+ Effect of Revision on individual

diagnoses

0

0.2

0.4

0.6

0.8

1

1.2

1.4

Pass 1 Pass 2

Revised

Not Revised

+ Effect of Revision overall

0

5

10

15

20

25

30

35

40

45

50

Before revision After revision

+ Summary

Structured reflection appears to have positive effect, of

about 10%.

However, results are apparently affected by context, expertise,

case difficulty.

“Reflection” strategy very time consuming

Spontaneous reflection has small overall effect

8% revise diagnosis

Minimal overall increase (1.3%)

+ Instruction in Clinical Reasoning

Whole case, low fidelity (Durning)

Mixed Practice (Hatala, 1999)

Explicit Direction for S1 and S2 (Norman,

Brooks, Colle, 2000)

+ Durning, 2011

133 Year 2 med students

Instruction on 3 cases using:

Written case

DVD video case

Standardized Patient

Test using:

OSCE

Written knowledge quiz

Video quiz

+ Effect of Paper vs. DVD vs Std Pt

Instruction on Performance

0

10

20

30

40

50

60

70

80

OSCE Written Video

Paper

DVD

StdPt

+ Mixed vs. Blocked Practice

In the face of ambiguous features (which are subject to

reinterpretation), and multiple categories, students must learn

the features which discriminate one category from another, not

those which support a particular category

+ Mixed vs. Blocked Practice

(Hatala, 2000)

ECG Diagnosis -- 3 categories

6 examples / category

Blocked

Review, then 6 examples/category

Mixed

Review, 2/category, 12 (4 x 3) practice

TEST

6 new ECGs

Accuracy -- %

0

5

10

15

20

25

30

35

40

45

50

Mixed Blocked

+ Explicit Instruction to Use S1 and S2 Norman, Brooks Colle, 2000

32 Undergrad Psychology students

11 disorders, rules + examples

Test -- 10 new ECGs

+ Instructions

Think of the first thing that comes to mind, then consider

features

vs.

Gather all the data then arrive at diagnosis

Diagnostic Accuracy

System 1 + 2 System 2

Diagnostic Accuracy

Resident

Clerk

System 1 + 2 System 2

+ CONCLUSION (1)

WORKPLACE-BASED (REAL TIME) STRATEGIES

Error reduction based on cognitive biases is ineffective

Experts cannot agree on specific bias

Instruction based on identification of bias does not reduce error

Error reduction based on reorganizing knowledge has small effect, but requires self-recognition of error

+ CONCLUSION (2)

INSTRUCTIONAL STRATEGIES

A number of strategies to improve teaching

of reasoning

Simple cases

Mixed practice

Combined reasoning (S1 and S2)