meaningful integration of direct observation into resident ... · direct observation in medical...
TRANSCRIPT
![Page 1: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/1.jpg)
Meaningful Integration of Direct Observation into Resident
Evaluation
Lynelle Boamah, MD, MEd Natalie Burman, DO, MEd
Shellie Kendall, MD Matt Zackoff, MD
Melissa Klein, MD, MEd
![Page 2: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/2.jpg)
Disclosures
We have no personal financial relationships to disclose
![Page 3: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/3.jpg)
∗ Describe benefits of incorporating direct observations into resident evaluation
∗ Perform Direct Observation utilizing structured tools
∗ Map outcomes from direct observations to reportable pediatric sub-competencies
∗ Develop a specific plan to implement direct observations into your training program
Objectives
![Page 4: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/4.jpg)
Workshop Roadmap
∗ Introductions ∗ Practice Observation ∗ Brief Didactic ∗ Intro to Tools ∗ Small Group Application ∗ Large Group Discussion ∗ Wrap-up
![Page 5: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/5.jpg)
Does
Shows How
Knows How
Knows
Miller’s triangle of clinical competence (Miller GE. The assessment of clinical skills/ competence/performance. Acad Med 1990; 65: s63-7)
![Page 6: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/6.jpg)
If you didn’t see it, it may not have happened!
![Page 7: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/7.jpg)
∗ Questions used to obtain history ∗ Physical exam and procedural
techniques ∗ Communication ∗ Professional demeanor ∗ Helping patient to navigate the
healthcare system ∗ Trainee efficiency
What you don’t see
![Page 8: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/8.jpg)
∗ Provides trainees opportunity to demonstrate application of what they have learned
∗ Allows timely, specific, credible feedback
∗ Enhances quality of summative evaluations
∗ Provides valuable information for CCC’s
Why DOs are important
Lane JL, Gottlieb RP. Structured clinical observations: a method to teach clinical skills with limited time and financial resources. Pediatrics. 2000 Apr;105(4 Pt 2):973-7
![Page 9: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/9.jpg)
∗ Milestones are anchored in direct observation of behaviors
∗ Allow for more meaningful assessment
∗ Enhance ability to identify struggling trainees earlier
NAS Requirement
ACGME Program Requirements for Graduate Medical Education in Pediatrics, 2013. The Pediatrics Milestone Project, 2013.
![Page 10: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/10.jpg)
Clinical Competency Committee
Presentation
EvaluationsOSCE
Mini-CEX
SCO
Rotation Evaluation
Scho
larly
Activ
ity
CEX
Simulation
OSTE
![Page 11: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/11.jpg)
![Page 12: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/12.jpg)
SCO
![Page 13: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/13.jpg)
What is an OSTE?
![Page 14: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/14.jpg)
∗ Describes a spectrum of observable behaviors along a continuum from novice to expert
∗ Each behavior is related to a skill that can be mapped to a competency
∗ Used to assess a wide range of teaching (i.e. bedside, classroom)
OSTE
![Page 15: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/15.jpg)
Teaching Assessment OSTE
1. Identify the Competency 2. Foundation in consensus competencies 3. Define an EPA
![Page 16: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/16.jpg)
Teaching Assessment OSTE
4. Determine skills/traits 5. Characterize observable behaviors 6. Align behaviors with gradient of competency
![Page 17: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/17.jpg)
OSTE
![Page 18: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/18.jpg)
Activity Introduction
In the next 30 minutes:
1. Role play using the OSTE to assess a teaching scenario
2. Complete a SCO on an outpatient scenario
3. Discuss observed Sub-competencies
![Page 19: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/19.jpg)
∗ Teaching Scenario – build a burrito ∗ Teacher and 3 students
∗ Expert who works at Chipotle
∗ Minimally knowledgeable to competent - like to eat burritos
∗ Novice – never heard of or eaten a burrito
∗ 2-3 observers at the table
Activity #1 OSTE Application
![Page 20: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/20.jpg)
∗ Outpatient continuity clinic teaching scenario
∗ Review the video-taped visit ∗ Rate the learner with the SCO ∗ Discuss findings at your table
Activity #2 SCO Application
![Page 21: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/21.jpg)
In your small groups…
Consider the following:
∗ Successes using observations in your program
∗ Barriers to using in your home program
∗ Challenges to implementing a systematic direct observation program
Report out to large groups
![Page 22: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/22.jpg)
The Benefits
∗ What really happened in the room? ∗ History taking skills
∗ Physical exam –appropriate, done correct
∗ Communication - was the “hidden agenda” addressed?
∗ Interpersonal/Interaction
∗ Counseling skills ∗ Did family understand the diagnosis, plan and
follow-up?
∗ Professionalism
![Page 23: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/23.jpg)
The Challenges
∗ Time, time, time ∗ While observing, not seeing other patients ∗ Feedback after observation
∗ Patient schedule and clinic flow ∗ Office/hospital room configuration ∗ Observation in large group i.e. rounds ∗ Resident discomfort/stress ∗ Preceptor discomfort/faculty development ∗ Concern of parental perception ∗ Determining optimal number
![Page 24: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/24.jpg)
How have you overcome some of these barriers?
How to get faculty to buy in?
![Page 25: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/25.jpg)
Overcoming Barriers - Outpatient
∗ Observational room with one way mirror ∗ If you have unlimited funds☺
∗ Video recordings of encounters ∗ Think cheap – iPhones work well ∗ Review with learner
∗ Learner review on own for self-reflection
![Page 26: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/26.jpg)
∗ Standardized tool ∗ Preceptor becomes comfortable with tool ∗ Standardized objective information collected
∗ Multiple short observations (SCO) ∗ Learner becomes more comfortable
∗ Less time intensive
∗ Less impact on patient flow
∗ Observe different aspects of exam at different points over time
Overcoming Barriers - Outpatient
![Page 27: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/27.jpg)
Overcoming Barriers - Inpatient
∗ Assign roles to different learners ∗ Get multiple learners involved in the
observation
∗ Multiple people observing rounding encounters
∗ Tools for presentation content ∗ Tools for rounding process
![Page 28: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/28.jpg)
Questions?
![Page 29: Meaningful Integration of Direct Observation into Resident ... · Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009](https://reader036.vdocument.in/reader036/viewer/2022071104/5fddc58a66fa466ff401ce06/html5/thumbnails/29.jpg)
∗ Fromme HB, Karani R, Downing SM. Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med. 2009 Aug;76(4):365-71.
∗ Hamburger EK, Cuzzi S, Coddington DA, Allevi AM, Loprieato J, Moon R, Yu C, Lane JL. Observation of resident clinical skills: outcomes of a program of direct observation in the continuity clinic setting. Acad Pediatr. 2011 Sept-Oct:11(5):394-402.
∗ Lane JL, Gottlieb RP. Structured clinical observations: a method to teach clinical skills with limited time and financial resources. Pediatrics. 2000;105(4): 973-977.
∗ Lu WH, Mylona E, Lane S, Wertheim WA, Baldelli P, Williams PC. Faculty development on professionalism and implementation of Objective Structured Teaching Exercises (OSTEs). Med Teach. 2014; 36(10):876-882.
∗ Steinert Y, Cruess S, Cruess R, Snell L. 2005. Faculty development for teaching and evaluating professionalism: From programme design to curriculum change. Med Educ 39:127–136.
∗ Trowbridge RL, Snydman LK, Skolfield J, Hafler J, Bing-You RG. 2011. A systematic review of the use and effectiveness of the objective structured teaching encounter. Med Teach 33:893–903.
∗ Stone S, Mazor K, Devaney-O’Neil S, Starr S, Ferguson W, Wellman S, Jacobson E, Hatem DS, Quirk M. 2003. Development and implementation of an objective structured teaching exercise (OSTE) to evaluate improvement in feedback skills following a faculty development workshop. Teach Learn Med 15:7–13.
∗ Williams RG, Dunnington GL, Mellinger JD, Klamen DL. 2014. Placing Constraints on the Use of the ACGME Milestones: A Commentary on the Limitations of Global Performance Ratings. Acad Med. Oct 7. [Epub ahead of print]
References