![Page 1: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/1.jpg)
From Learning Objectives to Outcomes
Assessment During SimulationPaul E. Phrampus
DirectorWinter Institute for Simulation, Education and
Research (WISER)University of Pittsburgh
![Page 2: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/2.jpg)
• Why• What• When• Where• How
Assessment
![Page 3: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/3.jpg)
The Future of Simulation………
![Page 4: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/4.jpg)
What is a simulation for?
1. Education2. Assessment3. Fun4. 1 and 2
![Page 5: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/5.jpg)
TESTING TEACHING
The Simulation Advantage
Psychomotor SkillsPsychomotor Skills
Decision MakingDecision Making
Skills and DecisionsSkills and Decisions
Individuals Individuals Teams Teams
![Page 6: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/6.jpg)
What is “Simulation”….. Really…..
![Page 7: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/7.jpg)
The Job of Healthcare Educators
Psychomotor Skills
Communications Skills
Professionalism Skills
Decision MakingBase Knowledge
Teamwork Skills
![Page 8: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/8.jpg)
Why is this so hard?
![Page 9: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/9.jpg)
Learning Objective
• You have to have a clear reason of why you are running any simulation.
![Page 10: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/10.jpg)
Lets Watch
![Page 11: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/11.jpg)
• What is most important aspect to debrief?
![Page 12: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/12.jpg)
Where does a simulation fit?
• It depends on what your doing!
![Page 13: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/13.jpg)
EACH Sim is part of a bigger plan!
![Page 14: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/14.jpg)
What is this?
![Page 15: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/15.jpg)
Does it belong here?
![Page 16: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/16.jpg)
Course (Global) objectives v sim objectives
![Page 17: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/17.jpg)
Global Objectives
• Course Objectives– 1. Demonstrate a proper patient assessment– 2. Demonstrate following the emergency protocol– 3. Demonstrate Airway Management Skills
![Page 18: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/18.jpg)
Course Schedule / Flow
• Sim 1• Sim 2• Lecture• Skills Lab• Sim 3• Sim 4
Course Objectives
Course Objectives
Session Objective1.Ventilate w BVM2.Recognize Narcotic Overdose3.Activate Protocol
Session Objective1.Ventilate w BVM2.Recognize Narcotic Overdose3.Activate Protocol
![Page 19: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/19.jpg)
How do you determine “success” of a simulation?
Students
liked...
Learn
ing objec...
Students
passe...
Not S
ure
25% 25%25%25%1. Students liked it2. Learning objectives
achieved3. Students passed the
scenario4. Not Sure
![Page 20: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/20.jpg)
How do you determine “success” of a simulation?
Instructor Number 1
Instructor Number 2
Instructor Number 3
![Page 21: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/21.jpg)
How do you determine “success” of a simulation?
Instructor Number 1 Instructor Number 2 Instructor Number 3
Measure Here
![Page 22: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/22.jpg)
You can only keep track of so many things at one time
![Page 23: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/23.jpg)
How many of you feel like this ?
![Page 24: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/24.jpg)
Tools for Success
• Make it easy for your faculty to stay on track
• “In my opinion…..”
![Page 25: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/25.jpg)
Assessment Methodology– Identify methods for measuring core aspects in
simulation education / assessment– Discuss development and deployment of
measurement tools
![Page 26: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/26.jpg)
Student Learning
OutcomesCurriculum
Teaching & Learning Assessment Evaluation
![Page 27: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/27.jpg)
Assessment and Evaluation
• Assessment and Evaluation often used interchangeably
• Assessment – learner outcomes• Evaluation – course or program outcomes
![Page 28: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/28.jpg)
Performance Assessment
![Page 29: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/29.jpg)
Rate the Performance
• Pass or Fail
• Score Performance
• Procedural Check
![Page 30: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/30.jpg)
Assessment
• Why• What• When• Where• How
Rosen, MA et al. Measuring Team Performance in Simulation-Based Training: Adopting Best Practices for Healthcare. Sim Healthcare 3:2008;33–41.
![Page 31: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/31.jpg)
ACTION
PERFORMANCE
COMPETENCE
KNOWLEDGE
Miller GE, The assessment of clinical skills/competence/performance. Academic Med. 1990 Sep;65(9):S63-7.
Assessment of clinical skills/competence/performance
![Page 32: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/32.jpg)
Performance Assessment
• Basic to performance – Do they know it and know how?
• Competence – Can they do it?
• Performance – Do they do it?
![Page 33: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/33.jpg)
DOES
SHOWS HOW
KNOWS HOW
KNOWS
Knows When
Miller GE, The assessment of clinical skills/competence/performance. Academic Med. 1990 Sep;65(9):S63-7.
Assessment of clinical skills/competence/performance
![Page 34: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/34.jpg)
DOES
SHOWS HOW
KNOWS HOW
KNOWS
Knows When
Miller GE, The assessment of clinical skills/competence/performance. Academic Med. 1990 Sep;65(9):S63-7.
Assessment of clinical skills/competence/performance
Knowledge Tests
MCQs, Essay, Oral exam
Clinical Based Tests
Pt Mgt Qs, Essay, Oral
Performance Tests
OSCE, TaskDemo, SIM/SP
Practice Review
Audits, Video, UC-SP, 360O
![Page 35: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/35.jpg)
Assessment• Formative Assessment
– Lower stakes assessment– One of several assessments over time of course or program– May be evaluative, diagnostic, or prescriptive– Often results in remediation or progression to next level
• Summative Assessment– Higher stakes assessment– Generally final course or program assessment– Primary purpose is performance evaluation– Often results in a Go-No Go outcome
![Page 36: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/36.jpg)
Assessment
• Assessors– Instructors– Observers– SIM/Patients– Peers– Participants
• Instruments– AV annotation– Pencil/Paper– Scantron form– TabletPC or PDA
• Assessment data sources– Live observation– Video recording– SIM device logs
![Page 37: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/37.jpg)
STUDENT
EVALUATOR
ASSESSMENTPATIENT
Variables in Assessment
![Page 38: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/38.jpg)
Assessment Validity
• Validity – – Degree to which an assessment tool measures what it is
intended to measure– Construct validity
– Accuracy of tool to measure intended construct
• Concurrent validity• Content validity• Predicative validity
![Page 39: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/39.jpg)
Assessment Reliability• Test Reliability –
– consistency of a test to yield the same measure over time (does not assure validity)
– Test-Retest Reliabilityconsistency among different administrations
– Parallel Forms Reliabilityto assure that memory effects do not occur use a different pre- and posttest
– Inter-Rater Reliability two or more observers rate the same subjects with strong positive correlation of their ratings
![Page 40: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/40.jpg)
STUDENT
EVALUATOR
ASSESSMENTPATIENT
Variables in Assessment
![Page 41: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/41.jpg)
Assessment Improvement
• Improving Reliability and Validity– Base assessment on outcome/objectives >
event triggers > observable behavior > behavioral rating > assess against competencies
– Define <low – medium – high> performance– Use of Rubric or rating metric– Use (video) training examples of performance– Employ a quality assurance/improvement system
![Page 42: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/42.jpg)
Assessment Based on Objectives and Events
Rosen, MA, et al A measurement Tool for Simulation –Based Training in Emergency Medicine: The Simulation Module for Assessment of Resident Targeted Event Responses (SMARTER) Approach. Sim Healthcare 3:2008;170-179.
![Page 43: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/43.jpg)
Event – Behavior – Scale - Report
Rosen, MA, et al. Measuring Team Performance in Simulation –Based Training: Adapting Best Practices for Healthcare. Sim Healthcare 3:2008;33-41.
![Page 44: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/44.jpg)
Assessment Metrics
• Procedural or Check List assessment• Global Rating assessment
![Page 45: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/45.jpg)
Assessment Metrics• Procedural or Check List assessment
BCLS Y N
Open Airway
Check Breathing
BCLS Y N
Open Airway(< 5 sec of LOC)
Check Breathing(< 5 sec of Airway)
BCLS Y N
Open Airway
Check Breathing
A
Rating Score +1 -1 0 *Assist
![Page 46: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/46.jpg)
Assessment Metrics• Global Rating assessment
Code Blue P F
CPR and
ACLS
Code Blue
CPR<1(low) - 5(Hi)> points
ACLS<1(low)- 5(Hi)> points
Code Blue H M
CPR
ACLS
L
Rating Score +1 -1 0
Pts.
![Page 47: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/47.jpg)
Assessment Metrics• Procedural or Check List assessment• Global Rating assessment
< EXAMPLES>
• Pros and Cons to each• Multiple modes & points vs. Single assessment• QA/QI Metric – collective vs single assessment
![Page 48: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/48.jpg)
Brett-Fleegler, MB et al. A simulator-based tool that assesses pediatric resident resuscitation competency. Pediatrics 121(3):2008;597-603.
![Page 49: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/49.jpg)
Gaba, DM et al. Assessment of Clinical Performance during Simulated Crises Using Both Technical and Behavioral Ratings. Anesthesiology, 89(1)July 1998:8-18.
![Page 50: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/50.jpg)
Guise, JM et al. Validation of a Tool to Measure and Promote Clinical Teamwork. Sim Healthcare 3:2008;217–223.
![Page 51: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/51.jpg)
Interrater Reliability
![Page 52: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/52.jpg)
Evaluator9
Evaluator8
Evaluator7
Evaluator6
Evaluator5
Evaluator4
Evaluator3
Evaluator2
Evaluator 1
5 ScenarioVideos
Novices
Interrater Reliability of Data Collected with SimMan During Difficult Airway Simulations
Andrus, Phrampus, Wang
![Page 53: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/53.jpg)
Interrater Reliability of Data Collected with SimMan During Difficult Airway Simulations
0
0.2
0.4
0.6
0.8
1
1.2
9 Raters
6 Raters
3 rookies
Andrus, Phrampus, Wang
![Page 54: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/54.jpg)
Interrater Reliability of Data Collected with SimMan During Difficult Airway Simulations
• CONCLUSIONS: – Simulation instructors exhibited good overall
inter-rater agreement but showed poor agreement for some key airway tasks.
– Variability in instructor ratings may influence real-time evaluation of simulation performance.
– Likely could be improved with careful instructor orientation / training
Andrus, Phrampus, Wang
![Page 55: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/55.jpg)
Evaluation
• Individual Trainee Evaluation– Per episode performance– Course of the Day
![Page 56: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/56.jpg)
Evaluation
• Group Trainee Evaluation– Benchmark
![Page 57: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/57.jpg)
Evaluation
• Instructor Evaluation– Per episode performance– Measure Against Benchmarks
![Page 58: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/58.jpg)
Testing Points
T I M E
![Page 59: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/59.jpg)
Behind the Scenes
![Page 60: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/60.jpg)
Behind the Scenes
![Page 61: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/61.jpg)
Behind the Scenes
![Page 62: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/62.jpg)
Behind the Scenes
![Page 63: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/63.jpg)
Behind the Scenes
![Page 64: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/64.jpg)
Summary
• Know why you are doing the sim• Each sim is part of a bigger picture• Design tools for consistency• Make it easy for your faculty!
![Page 65: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/65.jpg)
Questions and discussion