Bret Simon, PhD

Moving toward milestones: Innovative approaches to

assessing resident performance

Michael M. Johnson, MD, UTHSCSA Christopher Nabors, MD, PhD, NYMC

Bret Simon, PhD, UTHSCSA

Bret Simon, PhD

Session Goal

To explore the potential for incorporating creative, innovative approaches to resident assessment into graduate medical education while maintaining ACGME and other performance standards.

2

Bret Simon, PhD

Your Goals?

3

Bret Simon, PhD

Reflection

If you could design the best resident assessment process for your location, what would it look like? Consider…

• What would be the role of faculty? • What would be the role of residents? • How would progress be determined? • What terms would you use to describe it? • Other thoughts or questions?

4

Bret Simon, PhD

Pair / Share

In pairs or triads, discuss your vision of the ideal assessment process.

5

Operationalizing Milestones …. at New York Medical College/Westchester Medical Center

Chris Nabors, MD Assistant Professor of Medicine Associate Program Director November 7, 2012

Green ML, Aagaard EM, Caverzagie KJ, et al. Charting the road to competence: Developmental milestones for internal medicine residency training. Journal of Graduate Medical Education. 2009;1(1):5-20.

Core Competency & Milestone Abbreviation

Month

P - A1 Document and report clinical information truthfully 1 P - A2 Follow formal policies 1 P - A3 Accept personal errors and honestly acknowledge them 6 P - A4 Uphold ethical expectations of research and scholarly activity 36 P - B1 Demonstrate empathy and compassion to all patients 3 P - B2 Demonstrate a commitment to relieve pain and suffering 3

P - B3 Provide support (physical, psychological, social, and spiritual) for dying patients and their families 24

P - B4 Provide leadership for a team that respects patient dignity and autonomy 24

P - C1 Communicate constructive feedback to other members of the health care team 12

P - C2 Recognize, respond to, and report impairment in colleagues or substandard care via peer review process 18

P - D1 Respond promptly and appropriately to clinical responsibilities including but not limited to calls and pages 1

P - D2 Carry out timely interactions with colleagues, patients, and their designated caregivers 6

P - E1 Recognize and manage obvious conflicts of interest, such as caring for family members and professional associates as patients 6

P - E2 Maintain ethical relationships with industry 30 P - E3 Recognize and manage subtler conflicts of interest 30 P - F1

Dress and behave appropriately

1

Implementation Outline

Curriculum Review and Modification (2009) Evaluation Framework (2010+…)

a. Landmarks & Entrustable Professional Activities b. Ward Evaluations

Consensus Based Emphasis on Direct Observation Time-Referenced Faculty and Non-Faculty Conversion of select forms

Type of Rotation/Evaluation Questionnaire Names

PGY1 Administrative Internship, SBP Global Pharmacy, Case Management, Quality Improvement, Nutrition, Research & System General Wards & Clinics Global and Chart Review

Floor, Night Medicine, Floor Teaching, Morning Report, Admitting, Clinic , Chart Review (Open and Closed)

Wards And Clinics Direct Observation C E X-Handoff , C E X-Morning Sign-out , CEX Emergency, CEX Inpatient, CEX Morning Rounds, CEX NLM search, CEX (Clinic), CEX (Ward)

Elective Rotations Global Faculty Evaluation of House Officer, Electives

Pulmonary Rotations Global Pulmonary Elective, MICU Day and Night

3600 Evaluations Global Peer, Self, Nursing Nursing, Patient Sat (ABIM+), Patient Sat (EIP), Peer, Self, Resident Supervisor

PGY2 Administrative Internship, SBP Global

Pharmacy, Case Management, Quality Improvement, Nutrition, Research & System

General Wards & Clinics Global and Chart Review Floor, Night Medicine, Floor Teaching, Consult, Research & Systems), Clinic, Chart Review (Open and Closed)

Wards And Clinics Direct Observation CEX Inpatient (m12-18), CEX Inpatient (m18-24),C E X Rounds, C E X Morning Sign-out, CEX admission, CEX Clinic ,CEX code, CEX mock code, CEX Rapid Response, CEX NLM search

Elective Rotations Global Faculty Evaluation of House Officer

Pulmonary Rotations Global Pulmonary Elective, MICU Day and Night

3600 Evaluations Global Peer, Self, Nursing Nursing, Patient Sat (ABIM+), Patient Sat (EIP), Peer, Self, Resident Supervisor

PGY3 Administrative Internship, SBP Global Pharmacy, Case Management, Quality Improvement, Nutrition, Research & System

General Wards & Clinics Global and Chart Review Admitting, Clinic , Research & Systems, Consults, Chart Reviews

Wards And Clinics Direct Observation CEX code, CEX admission, CEX Clinic, CEX mock code, CEX Rapid Response, CEX - (inpatient)

Elective Rotations Global Faculty Evaluation of House Officer

Pulmonary Rotations Global Pulmonary Elective, MICU Day and Night

3600 Evaluations Global Peer, Self, Nursing Nursing, Patient Sat (ABIM+), Patient Sat (EIP), Peer, Self, Resident Supervisor

Developmental Trajectory Using Milestones and EPA’s Precocious Fully Competent Delayed Attainment 1 2 3 4

5

Start of Training (PGY-1) End of Training (PGY-3)

N Engl J Med 2012; 366: 1051-6

A key element of the NAS is the measurement and reporting of outcomes through the educational milestones…

N Engl J Med 2012; 366: 1051-6

Patient Care 1 - Gather and synthesize essential and accurate information to define each patient’s clinical problem(s). 2 - Develops and achieves comprehensive management plan for each patient. 3 - Manages patients with progressive responsibility and independence. 4 - Skill in performing procedures. 5 - Request and provide consultative care.

Medical Knowledge 6 - Clinical knowledge. 7 - Knowledge of diagnostic testing and procedures. Systems-Based Practice 8 - Works effectively within an interprofessional team (e.g. peers, consultants, nursing, ancillary professionals and other support personnel). 9 - Recognizes system error and advocates for system improvement. 10 - Identifies forces that impact the cost of health care, advocates for, and practices cost-effective care. 11 - Transitions patients effectively within and across health delivery systems. Practice-Based Learning and Improvement 12 - Monitors practice with a goal for improvement. 13 - Learns and improves via performance audit. 14 - Learns and improves via feedback. 15 - Learns and improves at the point of care.

Professionalism 16 - Professional and respectful interactions with patients and their caregivers. 17 – Professional and respectful interactions within an interprofessional team (e.g. peers, consultants, nursing, ancillary professionals and support personnel). 18 - Accepts responsibility and follows through on tasks. 19 - Responds to each patient’s unique characteristics and needs. 20 - Integrity and ethical behavior in professional conduct.

Interpersonal and Communication Skills 21 - Communicate effectively with patients and caregivers. 22 - Communicate effectively in interprofessional teams (e.g. peers, consultants, nursing, ancillary professionals and other support personnel). 23 – Appropriate utilization and completion of health records. used with Permission of ABIM

ABIM Draft Milestones-Based-Narratives

Milestones Comprising Narrative Unacceptable Ready for unsupervised practice

Aspirational

+ SBP-C1 Recognize health system forces that increase the risk for error including barriers to optimal patient care (12) + SBP-C3 Dialogue with care team members to identify risk for and prevention of medical error (12) + SBP-C2 Identify, reflect on, and learn from critical incidents such as near misses and preventable medical errors (12) + PBLI-F1 Respond welcomingly and productively to action plans feedback from all members of the health care team including faculty, peer residents, students, nurses, allied health workers, patients, and their advocates (12) + SBP-C6 Partner with other health care professionals to identify, propose improvement opportunities within the system (36) + SBP-C4 Understand mechanisms for analysis and correction of systems errors (24) + PBLI-A5 Engage in a quality improvement intervention (36) + SBP-C5 Demonstrate ability to understand and engage in a system-level quality improvement intervention (36) + PBLI-H2 Integrate teaching, feedback, and evaluation with supervision of interns’ and students’ patient care (24) + SBP-A1 Understand unique roles and services provided by local health care delivery systems

Recognizes but ignores a potential risk for error within the system that may impact the care of a patient Does not accept feedback or demonstrate a willingness to change behavior in order to reduce the risk for error

Does not recognize the potential for system error Makes patient care decisions that increase the potential for error that may cause a patient harm Resistant to feedback about patient care decisions that may lead to error or otherwise cause harm

Recognizes the potential for error within the system (12) Identifies obvious or critical causes of error and notifies supervisor accordingly (12) Makes decisions that could lead to error which are otherwise corrected by the system or supervision. Willing to receive feedback about patient care decisions that may lead to error or otherwise cause harm (12)

Identifies systemic causes of medical error and navigates them to provide safe patient care (12) Recognizes the potential risk for error in the immediate system and takes necessary steps to mitigate that risk (12) Reflects upon and learns from own critical incidents that may lead to medical error (12) Engages with other team members to identify and propose quality improvement interventions designed to prevent or reduce medical error (36) Activates formal system resources to investigate and mitigate real or potential medical error (24)

Actively engages in a formal quality improvement activity (36) Requests system leadership to formally engage in quality assurance and quality improvement activities Viewed as a leader in identifying and advocating for the prevention of medical error Teaches others regarding the importance of recognizing and mitigating system error (24)

Draft Narrative May 2012, used with permission of ABIM □ □ □ □ □

Milestones-Based-Narratives to Assess Developmental Progression

9. SBP Recognizes system error and advocates for system improvement

Guiding Faculty Observations and Ratings of Performance

Appropriately recognized signs of an acute abdomen in patient who walked in following laparoscopic surgery at outside facility and recommended urgent ED referral, imaging and surgical consultation

Exam revealed guarding, rebound tenderness

Synthesized information to attain appropriate plan

Resident Self-Assessment

Tools – Mobile Handoff EPA

Readiness for Mobile Handoff

Thank you for your kindness and attention

Bret Simon, PhD

Dr. Michael M. Johnson University of Texas Health Science Center at San Antonio Associate Director for Internal Medicine Residency Curriculum Co-director

24

Bret Simon, PhD

Measurement Dilemma

25

Highly Specific Global

Utility of Measure

Bret Simon, PhD

Innovative Evaluation 1. Using the handout provided, identify a

general or specific competence that is of interest to you.

2. As a group, identify one or more strategies that could be used to assess competency in this area.

3. How is this an improvement over existing assessment methods

4. What impediments would you anticipate encountering if attempting to implement

26

Bret Simon, PhD

27