2012.07.02 developing and evaluating performance in simulator scenarios

Steven Yule PhD STRATUS Center for Medical Simulation

Brigham & Women’s Hospital, Boston

Department of Surgery, Harvard Medical School

2nd July 2012

Developing scenarios and evaluating performance in medical simulation

The challenge: Defining how learners should act, think or feel differently as a result of your simulation program

1. Basics of scenario development

2. Performance evaluation

Basics of scenario development

Checklist for designing a simulation scenario

① Define aim

② Needs assessment & learning objectives

③ Learning outcomes (for evaluation)

④ Develop a matrix

⑤ Work on evaluation strategy

⑥ Identify resources and people required

⑦ Programming and software

⑧ Dry run

⑨ Pilot with real participant

⑩ Run program

Performance evaluation

To successfully evaluate learners, you need to complete these three tasks as part of the scenario design process:

#1 Consider the evaluation options

#2 Review learning objectives

#3 Select strategy

Task # 1: Consider evaluation options • Written record

• Log

• Attendance

• Case Presentations

• Direct Observations

• Written exam

• Oral examination

» Structured (oral patient management problem)

» Short answer

• Videotape and audio tape review

• Simulated Patient

• Objective Structured Clinical Examination

Task # 2: Review learning objectives and select the most appropriate evaluation method according to these criteria:

Practical

Economical

Objective

Comprehensive

Valid

Reliable

Feasibile

Task # 3: Select strategy and implement

• How are the data to be collected?

• By what mechanism or by who?

For example:

• “Leadership behaviors will be assessed during the

scenario from video by blinded expert observers

using the Surgical Leadership Inventory”

• “Confidence in maintaining an airway during an

anaesthetic crisis will be measured by pre- and post-

course survey”

A worked example…

Categories

Elements

Situation Awareness Gathering Information Understanding Information Projecting and anticipating future state

Decision Making Considering options Selecting and communicating option Implementing and reviewing decisions

Communication and Teamwork

Exchanging information Establishing a shared understanding Co-ordinating team

Leadership Setting and maintaining standards Coping with pressure Supporting others

NOTSS skills taxonomy v1.2

Yule et al (2006) Medical Education, 40, 1098-1104

NOTSS rating form

Simulation coaching

NOTSS coaching significantly improved skills in a randomized controlled trial of 16 senior surgical trainees

Simulation-based team training (2012-2015)

Designing good simulation scenarios is impossible without clear learning outcomes and an evaluation strategy: summary

1. Decide what you want people to do differently after your session

2. Use a structured method of scenario design

3. Select most appropriate evaluation method for learning outcomes using criteria

Contact » syule@partners.org » http://scholar.harvard.edu/yule/