mmi to the rescue! - appd residency interview challenges: multiple mini interview (mmi) to the...
TRANSCRIPT
OVERCOMING RESIDENCY INTERVIEW CHALLENGES: MULTIPLE MINI INTERVIEW
(MMI) TO THE RESCUE!
Mona Hanna-Attisha, MD MPH
Crystal Cederna-Meko, PsyD
Franklin Trimm, MD
Sophia Goslings, MD
Rita Harper
DISCLOSURES
none
OVERVIEW Introduction to MMI
The problem with traditional interviewing
MMI in action
Selecting characteristics
Developing a station
Faculty development
Pearls & Pitfalls
INTRODUCTION TO MMI
MULTIPLE MINI INTERVIEW (MMI)
Interview approach developed at McMaster’s University
Series of short stations applicants rotate through Standardized questions, activities, scenarios, role plays
1-3 pre-specified characteristics measured by each station
1-2 blinded raters per station
5-12 stations each lasting 6-12 minutes in duration; brief breaks between stations
Raters can be anyone; trained in characteristic(s) prior to interview day
Behaviorally anchored, likert scale ratings for each characteristic
Data compiled with other applicant materials to inform resident selection
BENEFITS
Program
• More applicants in less time
• Reduced faculty burden
• Anyone can rate!
• Reduced financial costs
• Early measurement/screening of milestones, program valued characteristics
• Enhanced reliability & validity
• Flexibility in what is measured
• Better informed resident selection
• Cutting-edge/evidence-based
Applicant
• More opportunities for a first impression
• Increased applicant-faculty engagement
• Exposure to more faculty members
• Perceived fairness, transparency, accurate self-presentation
• Generally fun and enjoyable process
SHOW ME THE EVIDENCE
Feasible
Reliable & (preliminarily) Valid
Accepted by interviewers and applicants
APPLICATION IN GRADUATE MEDICAL EDUCATION
Medical Schools
Medical Residencies
Veterinary Schools
PA & pharmacy programs
Dental Schools
Most well
studied
MODIFIED MMI FOR MEDICAL RESIDENCY
More faculty development
Consistency in raters; core faculty as raters
Incorporation of milestones into station content
Extensive rating form development – milestone-esque rating
Applicant education on MMI
Maintenance of 1 traditional interview (program director)
Inclusion of recruitment and Q&A opportunities within each station
THE PROBLEM WITH TRADITIONAL INTERVIEWING
MMI IN ACTION
MMI AT USA CHILDREN’S & WOMEN’S HOSPITAL
We interview: To fill 12 residency slots
170 applicants in one interview season
10 total interview days
How do we accomplish this???
By interviewing:
10 applicants per day on a half day of interviews
20 applicants per day on a full day of interviews
Dividing those applicants into 2 or 4 groups of 5
Welcome Session &
Presentation -
Program Director
Lunch with Residents
Tour of USA
Children’s and
Women’s Hospital
MMI Sessions Coachability Ethics & Professionalism Team Work Multicultural Identity & Competence
Program Director
Tour of USA
Children’s and
Women’s Hospital
MMI Sessions Coachability Ethics & Professionalism Team Work Multicultural Identity & Competence
Program Director
GROUP A GROUP B
GROUPS A & B
GROUPS A & B
HALF DAY AT USA
Lunch with Residents
Return to Hotel GROUPS A B C & D
TRANSITION TO
FULL DAY
Begin Interview
Day
GROUPS A & B
GROUPS C & D
Welcome Session &
Presentation -
Program Director
Return to Hotel
Tour of USA
Children’s and
Women’s Hospital
MMI Sessions Coachability Ethics & Professionalism Team Work Multicultural Identity & Competence
Program Director
Tour of USA
Children’s and
Women’s Hospital
MMI Sessions Coachability Ethics & Professionalism Team Work Multicultural Identity & Competence
Program Director
GROUP C GROUP D
GROUPS C & D
GROUPS C & D
FULL DAY AT USA
MMI INTERVIEW SESSIONS •PD presents an overview of our MMI process during the large group presentation
•PC presents detailed description of how the MMI process works to small groups prior to beginning the interviews:
Interview with the PD and 4 additional faculty members.
Only the PD will have reviewed application, faculty members will have only the basics.
5 doors labeled “Station 1” through “Station 5”.
At the sound of the bell proceed into station to meet with faculty member.
You will have 8 minutes in each room, the bell will sound when time is up.
After a 2 minute break in the hall, the bell will sound to enter the next room.
This will continue until you have interviewed in all 5 stations.
The entire interview session will only take 1 hour to complete
CHALLENGES
• Interviewee anxiety
• Scheduling faculty development
• When scheduled interviewer has an emergency and can’t interview
• Fewer interview date options for applicants
• Initial year feedback: Some applicants felt there was too little time with faculty during interviews
• Faculty development needs to be continuous
SUCCESSES
• Improved faculty willingness to interview
• Reduced number of interview days
• Decreased time spent scheduling faculty
• Improved resident involvement
• Provided a more holistic approach to the interview and selection process
• Filled earlier in our rank order list than we have in year’s past
MMI AT HURLEY CHILDREN’S HOSPITAL/MICHIGAN STATE UNIVERSITY
•Small community-based, university-affiliated program in Flint, Michigan
•7 pediatric residents/year (increased from 5/yr)
•Historically pre-matched 1-2/year
Needed to interview more people in less time and get better info!
•Started MMI in 2012-2013 season; just completed third MMI season
MMI AT HURLEY CHILDREN’S HOSPITAL/MICHIGAN STATE UNIVERSITY
Interview Season
•7 interview days
•Up to 20 applicants per day separated into 4 groups
•Maximally interview 140 applicants
Interview Day Activities
•Orientation to residency
•Tour
•MMI
•Case Presentation
•Lunch
FULL-DAY SCHEDULE AT A GLANCE
Duration Item Duration Item
AM 8:30-9:30 60 Presentation 8:30-9:30 60 Presentation
9:30-10:30 9:30-9:42 12 Station #1 9:30-10:30 60 Tour/Break
9:42-9:54 12 Station #2 10:30-10:40 10 Break
9:54-10:06 12 Station #3 10:40-11:40 10:40-10:52 12 Station #1
10:06-10:18 12 Station #4 10:52-11:04 12 Station #2
10:18-10:30 12 Station #5 11:04-11:16 12 Station #3
11:16-11:28 12 Station #4
10:30-10:40 10 Break 11:28-11:40 12 Station #5
10:40-11:40 60 Tour/Break
11:40-12:30 60 Lunch 11:40-12:30 60 Lunch
12:30-1 30 Case Conference 12:30-1 30 Case Conference
Duration Item Duration Item
PM 11:40-12:30 60 Lunch 11:40-12:30 60 Lunch
12:30-1 30 Case Conference 12:30-1 30 Case Conference
1-2:00 60 Presentation 1-2:00 60 Presentation
2-3:00 2-2:12 12 Station #1 2-3:00 60 Tour/Break
2:12-2:24 12 Station #2 3-3:10 10 Break
2:24-2:36 12 Station #3 3:10-4:10 3:10-3:22 12 Station #1
2:36-2:48 12 Station #4 3:22-3:34 12 Station #2
2:48-3 12 Station #5 3:34-3:46 12 Station #3
3:46-3:58 12 Station #4
3-3:10 10 Break 3:58-4:10 12 Station #5
3:10-4:10 60 Tour/Break
Group 1 (5 candidates) Group 2 (5 candidates)
Group 4 (5 candidates)Group 3 (5 candidates)
PRE-MMI PREP Applicant MMI PREP
• Interview confirmation letter mentions “you will be interviewing with many faculty”
• PD presents an overview (history, evidence, details) of MMI process during the large group presentation
• Individual itineraries for each applicant
• PD presentation is only 30 min, rest of time is for group Q&A (30min)
Program MMI PREP
• All rating forms pre-completed with applicants names, date, and interviewer (templates/mail merge)
• Pre-filled binder for each station (includes faculty schedule with applicant name/times, rating forms sorted in applicant order, station card, etc)
• 5 doors labeled “Station 1” through “Station 5”, extra bathrooms and signs, etc
• Reserve (larger) rooms in advance
• Almost one-year heads up of tentative interview dates for faculty schedules
• Solicit resident/chief resident tour guides
MMI STATIONS
One hour for five stations:
12 minute stations include 10 minutes for interview & 2 minutes to find next station (and for rater to complete form); loud seasonal bell!
Program director is a traditional interview station (only PD has reviewed application)
4 MMI stations, characteristics have included:
Teamwork, Ethics & Professionalism, Cultural Competence, Social Determinant Awareness, Self-Awareness, Empathy, & Emotional Intelligence
POST-MMI WORK
For each applicant:
Input 5 station scores & comments into ERAS (custom field)
Input 5 Interpersonal Skills and Verbal Communication scores into ERAS
For selection/rank meeting, share all applicant ERAS-info (scores, school, etc) with committee, as well as station output.
MMI OUTPUT
Rank Name Int Medical School of Graduation1 2CK 2CS 4 1 2 3 5 Vbl AVG ICS AVG
1 Daisy 10/28 Michigan State University College of Human Medicine230 252 Pass 9.0 9.0 7.0 8.0 5.0 7.3 7.8
2 Minnie 10/28 Michigan State University College of Human Medicine229 249 Pass 9.0 7.5 5.0 8.0 4.0 7.1 7.1
3 Mickey 11/4 Michigan State University College of Human Medicine224 249 Pass 9.0 7.5 5.5 7.0 7.0 6.9 7.6
4 Elsa 11/4 Michigan State University College of Human Medicine236 229 Pass 8.5 8.0 9.0 8.0 7.0 7.7 8.1
5 Cinderella 10/28 Michigan State University College of Human Medicine220 236 Pass 8.0 9.0 7.0 7.0 7.0 7.8 7.7
6 Ariel 10/28 Michigan State University College of Human Medicine226 254 Pass 9.0 7.5 4.5 8.0 8.0 8.0 7.6
7 Jasmine 10/28 Michigan State University College of Human Medicine243 236 Pass 8.5 6.5 7.0 7.0 4.0 6.9 7.1
8 Belle 11/11 Michigan State University College of Human Medicine248 251 8.5 7.0 5.5 7.0 8.0 6.7 7.2
… … … …
… … … … … … … … … …
82 Pluto 1/6 Michigan State University College of Human Medicine252 264 Pass 5.5 6.0 5.5 6.0 4.0 4.6 5.7
83 Tinker 1/6 Michigan State University College of Human Medicine223 230 Pass 5.0 4.0 5.0 6.0 4.0 4.8 5.9
84 Aurora 12/16 Michigan State University College of Human Medicine235 230 Pass 4.5 6.0 5.5 7.0 3.0 4.9 5.2
85 Ursula 11/11 Michigan State University College of Human Medicine234 255 Pass 4.0 5.0 5.0 8.0 7.0 6.2 5.8
86 Goofy 10/28 Michigan State University College of Human Medicine236 247 Pass 5.0 4.0 5.5 5.0 4.5 3.8 5.1
92 Anna 12/9 Michigan State University College of Human Medicine211 236 Pass 4.5 5.0 4.5 6.0 6.0 5.4 5.4
93 Olaf 1/6 Michigan State University College of Human Medicine241 234 Pass 5.0 6.0 4.0 5.0 6.0 5.6 6.5
94 Sven 12/16 Michigan State University College of Human Medicine220 222 Pass 4.5 6.0 4.0 7.0 7.0 5.8 6.2
97 Kristoff 12/2 Michigan State University College of Human Medicine244 245 Pass 4.5 5.0 6.0 8.0 6.0 4.9 6.0
98 Hans 12/16 Michigan State University College of Human Medicine238 249 Pass 4.5 5.0 4.0 2.5 4.0 3.7 4.9
CHALLENGES & SUCCESSES
CHALLENGES
•Culture change
•Learning curve
•Organization
•Faculty wanted in
•Fewer interview dates
SUCCESSES
•Fewer interview dates (save time & $)!
•More objective/transparent
•Innovative
•Residents “screened” and ranked for program-valued characteristics
SELECTING CHARACTERISTICS
CHARACTERISTIC SELECTION
a.k.a.
•Non-cognitive, non-technical competencies
•Personality traits, abilities
•Core interests, values, motivators
Guidelines for selection
•Predictive of important outcomes
•Encouraged by GME literature
•Valued by ACGME and/or measured by the milestones
•Valued by your program, institution
•Easily, reliably, validly measureable
SMALL GROUP BREAKOUT
Identify characteristics for measurement
CHARACTERISTICS IDENTIFIED
Provide 1-2 characteristics identified & rationale for selection
COMMON CHARACTERISTICS
Altruism
Conscientiousness
Critical and creative thinking
Empathy
Emotional intelligence
Ethics
Health advocacy
Interpersonal & communication skills
Leadership
Multicultural competence
Problem solving
Professionalism
Self awareness
Self initiated learning
Sound judgment
Teamwork
OVERLAP WITH MILESTONES
Altruism
Conscientiousness
Critical and creative thinking
Empathy
Emotional intelligence
Ethics
Health advocacy
Interpersonal & communication skills
Leadership
Multicultural competence
Problem solving
Professionalism
Self awareness
Self-initiated learning
Sound judgment
Teamwork
DEVELOPING A STATION
COMPONENTS OF AN MMI STATION
Introductions
Vignette / Setting
Open-ended question(s)
Probing and/or clarifying questions(s)
Scoring profile with behavioral anchors
Open comments
Additional generic scoring profiles, as desired
ACTIVITY: DEVELOP AN MMI STATION
Characteristic: TEAMWORK
Outline of station
Introduction
Vignette
Open-ended question(s)
Probing / clarifying question(s)
Scoring profile
SHARE STATION ACTIVITY
FACULTY DEVELOPMENT
FACULTY CONCERNS
What questions and concerns have you identified?
COMMONLY ENCOUNTERED FACULTY CONCERNS
Change is hard
• Why change?
• What’s wrong with what we do now?
• But I like…
• Knowing who I’m interviewing
• Asking what I want
• More spontaneity
• More time with candidates
M-M-What?
• What is MMI?
• Why is it better?
• How exactly is it more reliable? Valid?
• How does it fit into resident selection?
• Is it working? How do you know?
• Is that really enough time?
• Must I stick to the script?
I want in!
• I have an idea for a station activity!
• I have an idea for a characteristic
• Why can’t everybody be a rater?
• Can I just be a rater for a half day?
MULTI-METHOD APPROACH TO FACULTY DEVELOPMENT
Before MMI
• Large group faculty development (PEC)
• Small group (raters by station)
Formative Feedback
• Individual and group feedback (raters only)
Summative Feedback
• Individual and group feedback (raters only)
• Large group feedback from survey data
LARGE GROUP FACULTY DEVELOPMENT
Experiential
• Problems with current approach (traditional interview)
• Introduction to the solution - MMI
Didactic
• Brief introduction to MMI & incorporation into resident selection
• Review of (anticipated) faculty development
Q&A
FACULTY DEVELOPMENT BY STATION
NRMP Rules & Confidentiality
Characteristic review
Rating form & Answer key (where applicable) Introductions
Station activity
Rating scale
Elicit questions, comments, concerns Revise
Offer to role play, review
Final version sent out 1-2 weeks prior
Time spent 30-45 minutes face-to-face
Attendees PD or APD
Station raters
Materials NRMP Rules
Rating forms
Answer sheets
SAMPLE: PROFESSIONALISM STATION
FORMATIVE AND SUMMATIVE FEEDBACK
Review Rating Forms
Use of rating scale
Rating trends
Solicit feedback from raters
What’s working and what’s not
Behavioral anchors - concerns
Provide specific feedback
Strengths, positives
Areas of concern
Recommended course of action
Adjust forms as indicated
Inform of changes
LARGE GROUP FEEDBACK
Review of survey data results after interview season (PEC meeting)
Strengths
Opportunities for growth
OUTCOME
Increased awareness, collaboration, & faculty input
Increased buy-in
Improved reliability and validity of ratings
Better data to inform rank order determinations
MMI: CONCLUSIONS, PEARLS & PITFALLS
Q&A
CONTACT US
Mona Hanna-Attisha, MD MPH [email protected]
Crystal Cederna-Meko, PsyD [email protected]
Franklin Trimm, MD [email protected]
Sophia Goslings, MD [email protected]
Rita Harper [email protected]