from learning objectives to outcomes assessment during simulation
DESCRIPTION
From Learning Objectives to Outcomes Assessment During Simulation. Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER) University of Pittsburgh. Assessment. Why What When Where How. The Future of Simulation………. What is a simulation for?. Education - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/1.jpg)
From Learning Objectives to Outcomes
Assessment During SimulationPaul E. Phrampus
DirectorWinter Institute for Simulation, Education and
Research (WISER)University of Pittsburgh
![Page 2: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/2.jpg)
• Why• What• When• Where• How
Assessment
![Page 3: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/3.jpg)
The Future of Simulation………
![Page 4: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/4.jpg)
What is a simulation for?
1. Education2. Assessment3. Fun4. 1 and 2
![Page 5: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/5.jpg)
TESTING TEACHING
The Simulation Advantage
Psychomotor SkillsPsychomotor Skills
Decision MakingDecision Making
Skills and DecisionsSkills and Decisions
Individuals Individuals Teams Teams
![Page 6: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/6.jpg)
What is “Simulation”….. Really…..
![Page 7: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/7.jpg)
The Job of Healthcare Educators
Psychomotor Skills
Communications Skills
Professionalism Skills
Decision MakingBase Knowledge
Teamwork Skills
![Page 8: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/8.jpg)
Why is this so hard?
![Page 9: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/9.jpg)
Learning Objective
• You have to have a clear reason of why you are running any simulation.
![Page 10: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/10.jpg)
Lets Watch
![Page 11: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/11.jpg)
• What is most important aspect to debrief?
![Page 12: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/12.jpg)
Where does a simulation fit?
• It depends on what your doing!
![Page 13: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/13.jpg)
EACH Sim is part of a bigger plan!
![Page 14: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/14.jpg)
What is this?
![Page 15: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/15.jpg)
Does it belong here?
![Page 16: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/16.jpg)
Course (Global) objectives v sim objectives
![Page 17: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/17.jpg)
Global Objectives
• Course Objectives– 1. Demonstrate a proper patient assessment– 2. Demonstrate following the emergency protocol– 3. Demonstrate Airway Management Skills
![Page 18: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/18.jpg)
Course Schedule / Flow
• Sim 1• Sim 2• Lecture• Skills Lab• Sim 3• Sim 4
Course Objectives
Course Objectives
Session Objective1.Ventilate w BVM2.Recognize Narcotic Overdose3.Activate Protocol
Session Objective1.Ventilate w BVM2.Recognize Narcotic Overdose3.Activate Protocol
![Page 19: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/19.jpg)
How do you determine “success” of a simulation?
Students
liked...
Learn
ing objec...
Students
passe...
Not S
ure
25% 25%25%25%1. Students liked it2. Learning objectives
achieved3. Students passed the
scenario4. Not Sure
![Page 20: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/20.jpg)
How do you determine “success” of a simulation?
Instructor Number 1
Instructor Number 2
Instructor Number 3
![Page 21: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/21.jpg)
How do you determine “success” of a simulation?
Instructor Number 1 Instructor Number 2 Instructor Number 3
Measure Here
![Page 22: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/22.jpg)
You can only keep track of so many things at one time
![Page 23: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/23.jpg)
How many of you feel like this ?
![Page 24: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/24.jpg)
Tools for Success
• Make it easy for your faculty to stay on track
• “In my opinion…..”
![Page 25: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/25.jpg)
Assessment Methodology– Identify methods for measuring core aspects in
simulation education / assessment– Discuss development and deployment of
measurement tools
![Page 26: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/26.jpg)
Student Learning
OutcomesCurriculum
Teaching & Learning Assessment Evaluation
![Page 27: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/27.jpg)
Assessment and Evaluation
• Assessment and Evaluation often used interchangeably
• Assessment – learner outcomes• Evaluation – course or program outcomes
![Page 28: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/28.jpg)
Performance Assessment
![Page 29: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/29.jpg)
Rate the Performance
• Pass or Fail
• Score Performance
• Procedural Check
![Page 30: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/30.jpg)
Assessment
• Why• What• When• Where• How
Rosen, MA et al. Measuring Team Performance in Simulation-Based Training: Adopting Best Practices for Healthcare. Sim Healthcare 3:2008;33–41.
![Page 31: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/31.jpg)
ACTION
PERFORMANCE
COMPETENCE
KNOWLEDGE
Miller GE, The assessment of clinical skills/competence/performance. Academic Med. 1990 Sep;65(9):S63-7.
Assessment of clinical skills/competence/performance
![Page 32: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/32.jpg)
Performance Assessment
• Basic to performance – Do they know it and know how?
• Competence – Can they do it?
• Performance – Do they do it?
![Page 33: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/33.jpg)
DOES
SHOWS HOW
KNOWS HOW
KNOWS
Knows When
Miller GE, The assessment of clinical skills/competence/performance. Academic Med. 1990 Sep;65(9):S63-7.
Assessment of clinical skills/competence/performance
![Page 34: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/34.jpg)
DOES
SHOWS HOW
KNOWS HOW
KNOWS
Knows When
Miller GE, The assessment of clinical skills/competence/performance. Academic Med. 1990 Sep;65(9):S63-7.
Assessment of clinical skills/competence/performance
Knowledge Tests
MCQs, Essay, Oral exam
Clinical Based Tests
Pt Mgt Qs, Essay, Oral
Performance Tests
OSCE, TaskDemo, SIM/SP
Practice Review
Audits, Video, UC-SP, 360O
![Page 35: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/35.jpg)
Assessment• Formative Assessment
– Lower stakes assessment– One of several assessments over time of course or program– May be evaluative, diagnostic, or prescriptive– Often results in remediation or progression to next level
• Summative Assessment– Higher stakes assessment– Generally final course or program assessment– Primary purpose is performance evaluation– Often results in a Go-No Go outcome
![Page 36: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/36.jpg)
Assessment
• Assessors– Instructors– Observers– SIM/Patients– Peers– Participants
• Instruments– AV annotation– Pencil/Paper– Scantron form– TabletPC or PDA
• Assessment data sources– Live observation– Video recording– SIM device logs
![Page 37: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/37.jpg)
STUDENT
EVALUATOR
ASSESSMENTPATIENT
Variables in Assessment
![Page 38: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/38.jpg)
Assessment Validity
• Validity – – Degree to which an assessment tool measures what it is
intended to measure– Construct validity
– Accuracy of tool to measure intended construct
• Concurrent validity• Content validity• Predicative validity
![Page 39: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/39.jpg)
Assessment Reliability• Test Reliability –
– consistency of a test to yield the same measure over time (does not assure validity)
– Test-Retest Reliabilityconsistency among different administrations
– Parallel Forms Reliabilityto assure that memory effects do not occur use a different pre- and posttest
– Inter-Rater Reliability two or more observers rate the same subjects with strong positive correlation of their ratings
![Page 40: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/40.jpg)
STUDENT
EVALUATOR
ASSESSMENTPATIENT
Variables in Assessment
![Page 41: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/41.jpg)
Assessment Improvement
• Improving Reliability and Validity– Base assessment on outcome/objectives >
event triggers > observable behavior > behavioral rating > assess against competencies
– Define <low – medium – high> performance– Use of Rubric or rating metric– Use (video) training examples of performance– Employ a quality assurance/improvement system
![Page 42: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/42.jpg)
Assessment Based on Objectives and Events
Rosen, MA, et al A measurement Tool for Simulation –Based Training in Emergency Medicine: The Simulation Module for Assessment of Resident Targeted Event Responses (SMARTER) Approach. Sim Healthcare 3:2008;170-179.
![Page 43: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/43.jpg)
Event – Behavior – Scale - Report
Rosen, MA, et al. Measuring Team Performance in Simulation –Based Training: Adapting Best Practices for Healthcare. Sim Healthcare 3:2008;33-41.
![Page 44: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/44.jpg)
Assessment Metrics
• Procedural or Check List assessment• Global Rating assessment
![Page 45: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/45.jpg)
Assessment Metrics• Procedural or Check List assessment
BCLS Y N
Open Airway
Check Breathing
BCLS Y N
Open Airway(< 5 sec of LOC)
Check Breathing(< 5 sec of Airway)
BCLS Y N
Open Airway
Check Breathing
A
Rating Score +1 -1 0 *Assist
![Page 46: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/46.jpg)
Assessment Metrics• Global Rating assessment
Code Blue P F
CPR and
ACLS
Code Blue
CPR<1(low) - 5(Hi)> points
ACLS<1(low)- 5(Hi)> points
Code Blue H M
CPR
ACLS
L
Rating Score +1 -1 0
Pts.
![Page 47: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/47.jpg)
Assessment Metrics• Procedural or Check List assessment• Global Rating assessment
< EXAMPLES>
• Pros and Cons to each• Multiple modes & points vs. Single assessment• QA/QI Metric – collective vs single assessment
![Page 48: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/48.jpg)
Brett-Fleegler, MB et al. A simulator-based tool that assesses pediatric resident resuscitation competency. Pediatrics 121(3):2008;597-603.
![Page 49: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/49.jpg)
Gaba, DM et al. Assessment of Clinical Performance during Simulated Crises Using Both Technical and Behavioral Ratings. Anesthesiology, 89(1)July 1998:8-18.
![Page 50: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/50.jpg)
Guise, JM et al. Validation of a Tool to Measure and Promote Clinical Teamwork. Sim Healthcare 3:2008;217–223.
![Page 51: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/51.jpg)
Interrater Reliability
![Page 52: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/52.jpg)
Evaluator9
Evaluator8
Evaluator7
Evaluator6
Evaluator5
Evaluator4
Evaluator3
Evaluator2
Evaluator 1
5 ScenarioVideos
Novices
Interrater Reliability of Data Collected with SimMan During Difficult Airway Simulations
Andrus, Phrampus, Wang
![Page 53: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/53.jpg)
Interrater Reliability of Data Collected with SimMan During Difficult Airway Simulations
0
0.2
0.4
0.6
0.8
1
1.2
9 Raters
6 Raters
3 rookies
Andrus, Phrampus, Wang
![Page 54: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/54.jpg)
Interrater Reliability of Data Collected with SimMan During Difficult Airway Simulations
• CONCLUSIONS: – Simulation instructors exhibited good overall
inter-rater agreement but showed poor agreement for some key airway tasks.
– Variability in instructor ratings may influence real-time evaluation of simulation performance.
– Likely could be improved with careful instructor orientation / training
Andrus, Phrampus, Wang
![Page 55: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/55.jpg)
Evaluation
• Individual Trainee Evaluation– Per episode performance– Course of the Day
![Page 56: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/56.jpg)
Evaluation
• Group Trainee Evaluation– Benchmark
![Page 57: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/57.jpg)
Evaluation
• Instructor Evaluation– Per episode performance– Measure Against Benchmarks
![Page 58: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/58.jpg)
Testing Points
T I M E
![Page 59: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/59.jpg)
Behind the Scenes
![Page 60: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/60.jpg)
Behind the Scenes
![Page 61: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/61.jpg)
Behind the Scenes
![Page 62: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/62.jpg)
Behind the Scenes
![Page 63: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/63.jpg)
Behind the Scenes
![Page 64: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/64.jpg)
Summary
• Know why you are doing the sim• Each sim is part of a bigger picture• Design tools for consistency• Make it easy for your faculty!
![Page 65: From Learning Objectives to Outcomes Assessment During Simulation](https://reader035.vdocument.in/reader035/viewer/2022062500/56814f31550346895dbcc22b/html5/thumbnails/65.jpg)
Questions and discussion