dubrowski draft
TRANSCRIPT
The different uses of simulation:
From education to product testing
Adam Dubrowski, PhDLearning and Research Institutes
Wilson CentreDepartment of Paediatrics, UofT
www.wordle.net
Chaos in the BrickyardForscher, BK. Chaos in the Brickyard. Science. 1963 Oct 18; 142(3590): 339.
Applying the Scientific Method and underpinning research methodology to generate evidence that will be used to build simulation based education programs.
General Aim
• Models of Medical Education• Introduction to Simulation• Working Model• Components of Simulation
– Simulators– Educators– Organization
• Future Directions
Outline
Apprenticeship Model
Trainer
Trainee
Pat
ien
t C
are
Shift in responsibility
Ashley, 2000
Apprenticeship Model
Trainer
Trainee
Pat
ien
t C
are
Gap
Ashley, 2000
Apprenticeship Model
Trainer
Trainee
Pat
ien
t C
are
More people die in a given year as a result of medical errors ( 44,000) than from motor vehicle accidents (43,458), breast cancer (42,297), or AIDS (16,516).
"To Err is Human, Building a Safety Health System" (2000).
Apprenticeship Model
Trainer
Trainee
Pat
ien
t C
are
Simulation
Simulation Based Education
Learn fasterRemember longer
Be safer
The replication of a task or an event for the purpose of
education and/or assessment.
Simulation Based Education
Simulation Based Education
Workingmodel
Dubrowski, A., Brydges, R., Satterthwaite, L., Xeroulis, G., Classe, R. American Journal of Surgery. in press.
Busy Environment: Multitasking
Simulation Based Education
Technical Performance
Communication
Environment
Clinical Knowledge
Attentional capacity threshold
Additional Learning
Simulation Based Education
Simulation Based Education
Technical Performance
Communication
Environment
Clinical Knowledge
Additional Learning
Attentional capacity threshold
Simulation Based Education
Technical Performance
Communication
Environment
Clinical Knowledge
Additional Learning
Attentional capacity threshold
Simulation Based Education
Technical Performance
Communication
Environment
Clinical Knowledge
Additional Learning
Attentional capacity threshold
Simulation Based Education
Technical Performance
Communication
Environment
Clinical Knowledge
Additional Learning
Attentional capacity threshold
Simulation Based Education
Evidence
Kurahashi et al. Surgery. 2010
Study 1: Does it work?
Can simulation based training help in pre-training?
Practice: Trained SkillsTest
WrittenTest
1 week rest
Study 1: Does it work?
No practice: Novice
Skills Test: Expert Opinion
05
10152025
Written Test: Score
05
10152025
TrainedNovice Novice
TrainedNovice Novice
Study 1: Does it work?
simulation performanceimproves
Study 1: Does it work?
learningenables
Study 1: Does it work?
simulation
Simulation Based Education
How to use it?
Simulation Based Education
Components
Simulators Educators
Organization
SimulatorsInformation Flow
Framework
Brydges et al., Am J Surg. 2007, Sidhu et al., J Vasc Surg. 2007
Study 2: Realism
Brydges et al., Am J Surg. 2007, Sidhu et al., J Vasc Surg. 2007
High fidelity
Animal model
Low fidelity
Study 2: Realism
Brydges et al., Am J Surg. 2007, Sidhu et al., J Vasc Surg. 2007
Exp
ert
Op
inio
n
05
10152025
Senior
05
10152025
Novice
SyntheticCadaver
Exp
ert
Op
inio
n
Pre-training baseline
SyntheticCadaver
Study 2: Realism
simulation appropriate modelsrequires
Study 3: EducatorsErrors in learning
Xeroulis et al., Surgery. 2007
Study 3: Educators
Xeroulis et al., Surgery. 2007
Exp
ert
Op
inio
n
0
5
10
15
20
25
Post-training Retention
1 monthPre-training baseline
Errors allowedErrors not allowedIndependent learning
Study 3: Educators
simulation educated educatorsrequires
Organization
Moulton et al., Ann Surg. 2007
Optimal schedules
Study 4: Organization
Moulton et al., Ann Surg. 2007
1 2 3 4
1 hour block
1 month
1 2 3 4
1 hour block
1 weekrest
1 month
distributed practice
massed practice
1 2 3 41 2 3 41 2 3 41 2 3 41 2 3 4
T
T
Transfer
Study 4: Organization
Moulton et al., Ann Surg. 2007
distributedmassed
Exp
ert
O
pin
ion
30
20
10
0 Pre-test
Microsurgical drill
Post-test
Rat
Transfer
Study 4: Organization
simulation effective curricularequires
Summary of Results
simulation appropriate modelsrequires
simulation educated educatorsrequiressimulation effective curricularequires
Implications
Trainer
Trainee
Pat
ien
t C
are
Medical Education = Evidence Based Education
Simulation for Assessment
FrameworkPreferenceFor learningOf learningTesting
Framework
Results
Behavior
Learning
Reaction
Kirkpatrick
Simulation for Assessment
Results
Behavior
Learning
Reaction
Kirkpatrick
Simulation for Assessment
Results
Behavior
Learning
Reaction
Kirkpatrick
Does
Shows How
Knows How
Knows
Miller
Simulation for Assessment
Results
Behavior
Learning
Reaction
Kirkpatrick
Does
Shows How
Knows How
Knows
Miller
Moore
Simulation for Assessment
Assessment of what?
Assessment of Performance
Performance
Assessment of Performance
PGY 1-5 Gen Surgery residentsKnot tying: surface (easy) and depth (difficult)Assessed with expert-based (GRS) and computer-based (motion analysis device) assessments
Brydges et al., J Am Coll Surg. 2008
Assessment of Performance
T
T
T
T T T
T T TT
400
200
100
0
5
4
3
2
1
0
T TT
T T
T T T TT
1000
800
600
400
200
0T
T
T
T TSurgical Volume
Expert-basedComputer-based
1 2 3 4 5
DepthSurface
Brydges et al., J Am Coll Surg. 2008
Assessment of Performance
simulation valid assessmentsrequires
Brydges et al., J Am Coll Surg. 2008
Assessment of Performance
simulation realistic contextprovides
Brydges et al., J Am Coll Surg. 2008
Assessment of Performance
Porte et al., AJS. 2007
Assessment for Learning
Computer
Practice
PreTest
RetentionTest
1 week restComputer + Crit
Expert
PostTest
Porte et al., AJS. 2007
Assessment for Learning
T T
T
T
T TT
T T
Pre Post Retention
Exp
ert
O
pin
ion
20
10
0
Computer
Computer + Crit
Expert
Porte et al., AJS. 2007
Assessment for Learning
Porte et al., AJS. 2007
assessments for learning: choices
Assessment for Learning
Assessment of Learning
Learning
Building Skills, Changing Practice: Simulator Training for Hand Hygiene Protocols
Canadian Institutes of Health ResearchPartnerships for Health System Improvement
A. McGeer, MA. Beduz, A. Dubrowski
Assessment of Learning
Purpose
• Current models of knowledge delivery about proper hand hygiene rely on didactic session
• Transfer to practice is low
Assessment of Learning
PurposeAssessment of Learning
Results
Behavior
Learning
Reaction
Does
Shows How
Knows How
Knows
PurposeAssessment of Learning
Results
Behavior
Learning
Reaction
Does
Shows How
Knows How
Knows
PurposeAssessment of Learning
Results
Behavior
Learning
Reaction
Does
Shows How
Knows How
Knows
Recruitment
Clinical monitoring (audits)
Clinical monitoring (audits)
Δ in Behavior
Δ in PerformanceReaction
Education(Sim/
traditional)
PurposeAssessment of Learning
Sim
Sim
• Simulation Design Scale• Educational Practices Questionnaire
PurposeAssessment of LearningFre
qu
en
cy
100
50
0
Disagree StronglyAgree
90
66
Pre-test Post-testHand Hygiene Compliance(average of all opportunities; all 4 moments)
62% (n=558
opportunities)
64% (n=550
opportunities)
Average Adequacy(Excellent/ Satisfactory/ Unsatisfactory)
Satisfactory (n=252
observations)
Satisfactory (n=282
observations)
Average Duration
13 seconds (n=254
observations)
13 seconds(n=252
observations)
# Extra Washes 140 90
PurposeAssessment of Learning
Bef-Pat/Env Bef-Asp Aft-Bfl Aft-Pat/Env0
10
20
30
40
50
60
70
80
90
100 98
24
51
9292
4448
93
PreTest (n=558)Post Test (n=550)
Moment 1 Moment 2 Moment 3 Moment 4
PurposeAssessment of Learning
*
• No transfer of knowledge
• No changes in performance, no learning
• Ineffective education and not transfer to practice
PurposeAssessment of Learning
Assessment for Testing
Product and Process testing
Patient handoffs
Assessment for Testing
Patient handoffs Checklist
Assessment for Testing
Is it an effective tool?Does it disturb the flow?What is the frequency of use?What is the most optimal format?
Assessment for Testing
Assessment for Testing
Porte et al., AJS. 2007
simulation with process testinghelps
Assessment for Testing
Summary
Summary
Best available evidence
Ongoing assessment
Educator’s experience
• Simulators• Educators• Schedule
Summary
Best available evidence
Ongoing assessment
Educator’s experience
• Simulators• Educators• Schedule
• Performance• For learning• Of learning• Other uses
Summary
Best available evidence
Ongoing assessment
Educator’s experience
• Simulators• Educators• Schedule
• Performance• For learning• Of learning• Other uses
Simulation may not be the
answer
End