dubrowski draft

The different uses of simulation:

From education to product testing

Adam Dubrowski, PhDLearning and Research Institutes

Wilson CentreDepartment of Paediatrics, UofT

www.wordle.net

Chaos in the BrickyardForscher, BK. Chaos in the Brickyard. Science. 1963 Oct 18; 142(3590): 339.

Applying the Scientific Method and underpinning research methodology to generate evidence that will be used to build simulation based education programs.

General Aim

• Models of Medical Education• Introduction to Simulation• Working Model• Components of Simulation

– Simulators– Educators– Organization

• Future Directions

Outline

Apprenticeship Model

Trainer

Trainee

Pat

ien

t C

are

Shift in responsibility

Ashley, 2000

Apprenticeship Model

Trainer

Trainee

Pat

ien

t C

are

Gap

Ashley, 2000

Apprenticeship Model

Trainer

Trainee

Pat

ien

t C

are

More people die in a given year as a result of medical errors ( 44,000) than from motor vehicle accidents (43,458), breast cancer (42,297), or AIDS (16,516).

"To Err is Human, Building a Safety Health System" (2000).

Apprenticeship Model

Trainer

Trainee

Pat

ien

t C

are

Simulation

Simulation Based Education

Learn fasterRemember longer

Be safer

The replication of a task or an event for the purpose of

education and/or assessment.

Simulation Based Education

Simulation Based Education

Workingmodel

Dubrowski, A., Brydges, R., Satterthwaite, L., Xeroulis, G., Classe, R. American Journal of Surgery. in press.

Busy Environment: Multitasking

Simulation Based Education

Technical Performance

Communication

Environment

Clinical Knowledge

Attentional capacity threshold

Additional Learning

Simulation Based Education

Simulation Based Education

Technical Performance

Communication

Environment

Clinical Knowledge

Additional Learning

Attentional capacity threshold

Simulation Based Education

Technical Performance

Communication

Environment

Clinical Knowledge

Additional Learning

Attentional capacity threshold

Simulation Based Education

Technical Performance

Communication

Environment

Clinical Knowledge

Additional Learning

Attentional capacity threshold

Simulation Based Education

Technical Performance

Communication

Environment

Clinical Knowledge

Additional Learning

Attentional capacity threshold

Simulation Based Education

Evidence

Kurahashi et al. Surgery. 2010

Study 1: Does it work?

Can simulation based training help in pre-training?

Practice: Trained SkillsTest

WrittenTest

1 week rest

Study 1: Does it work?

No practice: Novice

Skills Test: Expert Opinion

05

10152025

Written Test: Score

05

10152025

TrainedNovice Novice

TrainedNovice Novice

Study 1: Does it work?

simulation performanceimproves

Study 1: Does it work?

learningenables

Study 1: Does it work?

simulation

Simulation Based Education

How to use it?

Simulation Based Education

Components

Simulators Educators

Organization

SimulatorsInformation Flow

Framework

Brydges et al., Am J Surg. 2007, Sidhu et al., J Vasc Surg. 2007

Study 2: Realism

Brydges et al., Am J Surg. 2007, Sidhu et al., J Vasc Surg. 2007

High fidelity

Animal model

Low fidelity

Study 2: Realism

Brydges et al., Am J Surg. 2007, Sidhu et al., J Vasc Surg. 2007

Exp

ert

Op

inio

n

05

10152025

Senior

05

10152025

Novice

SyntheticCadaver

Exp

ert

Op

inio

n

Pre-training baseline

SyntheticCadaver

Study 2: Realism

simulation appropriate modelsrequires

Study 3: EducatorsErrors in learning

Xeroulis et al., Surgery. 2007

Study 3: Educators

Xeroulis et al., Surgery. 2007

Exp

ert

Op

inio

n

0

5

10

15

20

25

Post-training Retention

1 monthPre-training baseline

Errors allowedErrors not allowedIndependent learning

Study 3: Educators

simulation educated educatorsrequires

Organization

Moulton et al., Ann Surg. 2007

Optimal schedules

Study 4: Organization

Moulton et al., Ann Surg. 2007

1 2 3 4

1 hour block

1 month

1 2 3 4

1 hour block

1 weekrest

1 month

distributed practice

massed practice

1 2 3 41 2 3 41 2 3 41 2 3 41 2 3 4

T

T

Transfer

Study 4: Organization

Moulton et al., Ann Surg. 2007

distributedmassed

Exp

ert

O

pin

ion

30

20

10

0 Pre-test

Microsurgical drill

Post-test

Rat

Transfer

Study 4: Organization

simulation effective curricularequires

Summary of Results

simulation appropriate modelsrequires

simulation educated educatorsrequiressimulation effective curricularequires

Implications

Trainer

Trainee

Pat

ien

t C

are

Medical Education = Evidence Based Education

Simulation for Assessment

FrameworkPreferenceFor learningOf learningTesting

Framework

Results

Behavior

Learning

Reaction

Kirkpatrick

Simulation for Assessment

Results

Behavior

Learning

Reaction

Kirkpatrick

Simulation for Assessment

Results

Behavior

Learning

Reaction

Kirkpatrick

Does

Shows How

Knows How

Knows

Miller

Simulation for Assessment

Results

Behavior

Learning

Reaction

Kirkpatrick

Does

Shows How

Knows How

Knows

Miller

Moore

Simulation for Assessment

Assessment of what?

Assessment of Performance

Performance

Assessment of Performance

PGY 1-5 Gen Surgery residentsKnot tying: surface (easy) and depth (difficult)Assessed with expert-based (GRS) and computer-based (motion analysis device) assessments

Brydges et al., J Am Coll Surg. 2008

Assessment of Performance

T

T

T

T T T

T T TT

400

200

100

0

5

4

3

2

1

0

T TT

T T

T T T TT

1000

800

600

400

200

0T

T

T

T TSurgical Volume

Expert-basedComputer-based

1 2 3 4 5

DepthSurface

Brydges et al., J Am Coll Surg. 2008

Assessment of Performance

simulation valid assessmentsrequires

Brydges et al., J Am Coll Surg. 2008

Assessment of Performance

simulation realistic contextprovides

Brydges et al., J Am Coll Surg. 2008

Assessment of Performance

Porte et al., AJS. 2007

Assessment for Learning

Computer

Practice

PreTest

RetentionTest

1 week restComputer + Crit

Expert

PostTest

Porte et al., AJS. 2007

Assessment for Learning

T T

T

T

T TT

T T

Pre Post Retention

Exp

ert

O

pin

ion

20

10

0

Computer

Computer + Crit

Expert

Porte et al., AJS. 2007

Assessment for Learning

Porte et al., AJS. 2007

assessments for learning: choices

Assessment for Learning

Assessment of Learning

Learning

Building Skills, Changing Practice: Simulator Training for Hand Hygiene Protocols

Canadian Institutes of Health ResearchPartnerships for Health System Improvement

A. McGeer, MA. Beduz, A. Dubrowski

Assessment of Learning

Purpose

• Current models of knowledge delivery about proper hand hygiene rely on didactic session

• Transfer to practice is low

Assessment of Learning

PurposeAssessment of Learning

Results

Behavior

Learning

Reaction

Does

Shows How

Knows How

Knows

PurposeAssessment of Learning

Results

Behavior

Learning

Reaction

Does

Shows How

Knows How

Knows

PurposeAssessment of Learning

Results

Behavior

Learning

Reaction

Does

Shows How

Knows How

Knows

Recruitment

Clinical monitoring (audits)

Clinical monitoring (audits)

Δ in Behavior

Δ in PerformanceReaction

Education(Sim/

traditional)

PurposeAssessment of Learning

Sim

Sim

• Simulation Design Scale• Educational Practices Questionnaire

PurposeAssessment of LearningFre

qu

en

cy

100

50

0

Disagree StronglyAgree

90

66

Pre-test Post-testHand Hygiene Compliance(average of all opportunities; all 4 moments)

62% (n=558

opportunities)

64% (n=550

opportunities)

Average Adequacy(Excellent/ Satisfactory/ Unsatisfactory)

Satisfactory (n=252

observations)

Satisfactory (n=282

observations)

Average Duration

13 seconds (n=254

observations)

13 seconds(n=252

observations)

# Extra Washes 140 90

PurposeAssessment of Learning

Bef-Pat/Env Bef-Asp Aft-Bfl Aft-Pat/Env0

10

20

30

40

50

60

70

80

90

100 98

24

51

9292

4448

93

PreTest (n=558)Post Test (n=550)

Moment 1 Moment 2 Moment 3 Moment 4

PurposeAssessment of Learning

*

• No transfer of knowledge

• No changes in performance, no learning

• Ineffective education and not transfer to practice

PurposeAssessment of Learning

Assessment for Testing

Product and Process testing

Patient handoffs

Assessment for Testing

Patient handoffs Checklist

Assessment for Testing

Is it an effective tool?Does it disturb the flow?What is the frequency of use?What is the most optimal format?

Assessment for Testing

Assessment for Testing

Porte et al., AJS. 2007

simulation with process testinghelps

Assessment for Testing

Summary

Summary

Best available evidence

Ongoing assessment

Educator’s experience

• Simulators• Educators• Schedule

Summary

Best available evidence

Ongoing assessment

Educator’s experience

• Simulators• Educators• Schedule

• Performance• For learning• Of learning• Other uses

Summary

Best available evidence

Ongoing assessment

Educator’s experience

• Simulators• Educators• Schedule

• Performance• For learning• Of learning• Other uses

Simulation may not be the

answer

End