Slide 1

Developing an Effective Simulation Lab How to Evaluate Residents Skills with Simulation Ann Van Heest MD Director of Education Professor, University of MN Dept of Orthopedic Surgery

Slide 2

Simulation Skills Lab Skills Acquisition Competency Testing

Slide 3

How do we measure competence? Idea Face validity: Does the test is going to measure what it was intended to measure? Construct validity: Can the test differentiate between experts and novices? Reliability: Inter-rater and Intra-rater

Slide 4

OUTLINE Matching the Right educational level for residents level of learning 1. Upper Extremity Surgical Skills Competency Testing 2. Articular Fracture Reduction Model 3. Basic Arthroscopy Skills Box 4. G1 Basic Skills Mandate

Slide 5

Upper Extremity Surgical Skills Competency Testing

Slide 6

Slide 7

Milestone Project Milestone Topics (n=16) ACL injury Ankle Arthritis Ankle Fracture Carpal Tunnel Syndrome Diabetic foot Shaft fx. Femur/tibia Distal Radius fracture Elbow fracture Medical Knowledge and Patient Care Hip and Knee OA Hip fracture Meniscal tear Metastatic bone Rotator Cuff Injury Septic arthritis (peds) Pediatric Supracondylar Humerus fracture Degenerative Spine

Slide 8

Slide 9

Slide 10

6th Lab Session: May 17, 2013 Carpal Tunnel Cadaveric Station 2 board-certified hand surgeon graders Detailed Checklist Global Rating Scale Pass / Fail Trigger Release Cadaveric Station 2 board-certified hand surgeon graders Detailed Checklist Global Rating Scale Pass / Fail Volar Approach Distal Radius Cadaveric Station board-certified hand surgeon grader Detailed Checklist Global Rating Scale Pass / Fail Biomechanical Testing Load to Failure Distal Radius Fixation with C-arm Radiographic Scores Debrief Pass/Fail Debrief Pass/Fail Post Test Evaluation Debrief

Slide 11

Motor Skills Testing: 3 Bench Stations 1.Global Rating Scale 2.Detailed Checklist 3.Adverse Events 4.Pass/Fail 5.Time

Slide 12

Slide 13

n =285742 Program Year Global Rating Carpal Tunnel p = 0.040

Slide 14

n =285742 Program Year Carpal Tunnel Release Detailed Checklist (100pts) p = 0.002

Slide 15

Pass/Fail Assessment Did the surgeon achieve the goal of surgery?

Slide 16

n =0 / 21 / 75 / 07 / 04 / 02 / 0 Program Year Pass / Fail Assessment p < 0.001

Slide 17

Distal Radius Fracture Fixation

Slide 18

Conclusions This study reports that a surgeons ability to release a trigger finger does not correlate specifically to his or her ability to perform a carpal tunnel release or to perform plate fixation of a radius fracture. The results of this study would indicate that, for 3 different surgical simulations representing procedures of varying complexity, assessments by a single assessment tool is not adequate. To completely understand a residents abilities, assessment by checklist (understanding the steps of the surgery), global rating scales (assessment of basic surgical skills in light of lesser or greater complexity surgeries), and pass/fail assessment (examination of adverse events) are all necessary components.

Slide 19

Competence High Stakes Exam G3 level Pass vs Needs more practice Competency advancement, not social advancement DEMONSTRATING COMPETENCE IS GRADUATION REQUIREMENT

Slide 20

OUTLINE Matching the Right educational level for residents level of learning 1. Upper Extremity Surgical Skills Competency Testing 2. Articular Fracture Reduction Model 3. Basic Arthroscopy Skills Box 4. G1 Basic Skills Mandate

Slide 21

Surgical Simulation Training Program for Articular Fracture Surgery Jenniefer Y. Kho, MD University of Iowa Hospitals and Clinics Karam MD, Ohrt GT, Thomas GW, Yehyawi TM, Lafferty PM, Anderson DD, Marsh JL

Slide 22

Aims Develop a comprehensive surgical simulation training program utilizing an articular fracture model Investigate whether simulator training can improve performance in junior residents

Slide 23

Methods 6 PGY-1 and 6 PGY-2 residents

Slide 24

Methods Interventio n (n=6) PretestTrainingPosttest Control (n=6) Pretest No Training Posttest 6 PGY-1 and 6 PGY-2 residents

Slide 25

Sawbones Fracture Model with Barium coated articular surface

Slide 26

Slide 27

Video capture

Slide 28

Methods - Intervention On-line cognitive module Video review with traumatologist Skills module Interventio n (n=6) PretestTrainingPosttest

Slide 29

Methods Outcome measures OSATS global rating scale Articular reduction Hand motion Fluoroscopy time, radiation dose Face Validity (Questionnaire)

Slide 30

Results OSATS GLOBAL RATING SCALE Control OSATS global rating score p=0.06p= 0.68 Intervention

Slide 31

Results FLUOROSCOPY TIME Fluoroscopy time (s) Pretest Posttest Control Intervention p= 0.03p= 0.16

Slide 32

Results Control Intervention ARTICULAR STEP--OFF p= 0.22p= 0.69 Articular step-off (mm) Pretest Posttest

Slide 33

Results Pretest Posttest Control Intervention Cumulative hand distance (m) CUMULATIVE HAND DISTANCE p= 0.8p= 1.0

Slide 34

Results Face validity Scale ranges from 1 (most) to 5 (least realistic) How realistic was this exercise overall? 1 2 3 4 5 How realistic was the bone model visually? 1 2 3 4 5 How realistic was the movement of the synthetic fracture fragments compared to that of real bony fragments? 1 2 3 4 5 How realistic was the feel of the synthetic bone during instrumentation compared to that of real bone? 1 2 3 4 5 Overall average score 2.75 Acceptable, but room for improvement

Slide 35

Discussion Improved OSATS score, decrease in fluoroscopy time and radiation exposure Articular step-off, hand motion no different Junior residents with little to no articular fracture experience may benefit from short-term dedicated training

Slide 36

Collaboration with University of Minnesota Incorporation into surgical skills curriculum at the University of Iowa Potential for expansion to other programs

Slide 37

OUTLINE Matching the Right educational level for residents level of learning 1. Upper Extremity Surgical Skills Competency Testing 2. Articular Fracture Reduction Model 3. Basic Arthroscopy Skills Box 4. G1 Basic Skills Mandate

Slide 38

Low-Fidelity Arthroscopic Simulation Can Differentiate Between Experts and Novices Jon Braman MD, Robert Sweet, MD David Hananel, SimPortal Lab Paula Ludewig, Ph.D Ann VanHeest, MD

Slide 39

Panel of Experts Visualization Triangulation Object Manipulation

Slide 40

Triangulation 10 LED lights 10 switches Randomly order lights (training) Pre-determined order lights (testing) Time Errors: Alarm if touches metal VIDEO

Slide 41

Object Manipulation 3 bars on left with rings 3 bars on right with rings Transfer rings from left to right (Switch scope) Time Errors: Dropped rings VIDEO

Slide 42

Triangulation p = 0.0013 p = 0.0073

Slide 43

Object Manipulation p = 0.0190 Errors: Drops p=1.0

Slide 44

Basic Arthroscopic Skills Construct Validity Face Validity G1 level skills: Readiness for OR Multi-Center Trial Planned Competency Testing: By year in training Translation to OR Scope Skills

Slide 45

ABOS Surgical Simulation Mandate formal instruction in basic surgical skills, provided longitudinally or as a dedicated non-orthopaedic surgery rotation will be required. Skills training will need to be designed to integrate with subsequent post graduate years and should prepare the PGY 1 to participate in orthopedic surgery cases. Skills training will need a dedicated space and a curriculum which must include: Goals and objectives and assessment metrics. Skills used in the outpatient management of injured patients including splinting, casting, application of traction devices and other types of immobilization. Basic operative skills including soft tissue management, suturing, bone management, arthroscopy, flouroscopy and use of basic orthopaedic equipment.

Slide 46

ABOS/AOA-CORD/AAOS Develop a modular outline of surgical skills curriculum 3 meetings (Sept, March, May) Posted on ABOS website with access to all residency programs.

Slide 47

1. Sterile Technique and Operating Room Set-Up 2.Knot Tying & Suturing 3.Basic Microsurgical Suturing 4.Soft Tissue Handling Techniques 5.Casting and Splinting 6.Traction 7.Compartment Syndrome 8.Bone Handling Techniques - Osteotomy 9.Fluoroscopic Knowledge and Skills 10.K-Wire Techniques 11.Techniques Basic to Internal Fixation of Fractures 12.Principles and Techniques of Fracture Reduction 13.External Fixation 14.Basic Arthroscopy Skills 15.Basic Arthroplasty Skills 16.Joint Sspiration and Injection 17.Patient Safety, Team Training, Obtaining Consent

Slide 48

Example of Module

Slide 49

How to measure Competency? 1.Upper Extremity Surgical Skills 2.Articular Fracture Reduction Model 3. Basic Arthroscopy Skills Box 4. G1 Basic Skills Modules Idea Face validity: Does the test measure what it was intended to measure? Construct validity: Can the test teach the desired operative skills? Reliability: Inter-rater and Intra-rater

Slide 50