feedback credibility in a formative postgraduate osce: effects of … · 2019-04-06 · feedback...
TRANSCRIPT
Feedback credibility in a formative postgraduate OSCE: Effects of examiner type
Lynfa Stroud, Matt Sibbald, Denyse Richardson Heather McDonald-Blumer, Rodrigo B. Cavalcanti Oct 1, 2016
I do not have an affiliation (financial or otherwise) with a
pharmaceutical, medical device or communications organization.
Je n’ai aucune affiliation (financière ou autre) avec une
entreprise pharmaceutique, un fabricant d’appareils
médicaux ou un cabinet de communication.
2
Background
• Greater emphasis on the perspective of the feedback recipient
• Importance of credibility of feedback provider
• Longitudinal relationships
Ericsson KA. Acad Med. 2004 / Archer JC. Med Educ. 2010 / Sargeant J. Med Educ. 2011 / Eva KW. Adv in Health Sci Educ. 2012 /
Watling C. Med Educ. 2014 / Watling C. Medical Educ. 2012 / Eva K. Acad Med. 2010.
3
Background
• Formative OSCEs as learning opportunities
• Prior exploratory study:
Faculty examiners rated more credible than SP examiners
Generalists more credible than specialists
4
Study Question
What factors affect residents’ perceptions of the credibility of
their examiners during a formative OSCE?
Faculty examiners versus SP examiners
Faculty examiners who are:
generalists
specialty-congruent with station content
specialty-incongruent with station content
5
Formative OSCE
• 5 stations x 12 minutes each
Clinical cases x 2 – faculty
Physical exam – faculty
Communication – standardized patients (+faculty)
• 5 point global rating scale (blind)
• 4 minutes for formative feedback end of each station
7
192 RESIDENTS
OSCE Track 1
Clinical and PE: Faculty Mix
Communication: SP Feedback
107 RESIDENTS
OSCE Track 2
Clinical and PE: Faculty Mix
Communication: “Faculty” Feedback
85 RESIDENTS
Post-OSCE Questionnaire
Study Design
Post-OSCE Questionnaire
• Demographics
• Overall, how would you rate the:
CREDIBILITY of the examiner providing feedback on this station?
9
Analysis
• Resident and examiner demographics
• Univariate analysis
• Multivariable analysis
• Credibility of examiner as dependent variable
10
Univariate Analysis
RESIDENTS CREDIBILITY p
Gender M 6.25
0.03 F 6.41
Level
PGY 1 – Faculty 6.46
0.002
PGY 2 – Faculty 6.27
PGY 3 - Faculty 6.16
PGY1 – SP 5.68
0.132 PGY2 – SP 5.39
PGY3 - SP 5.00 14
EXAMINERS CREDIBILITY p
Gender
M - Faculty 6.45 <0.001
F – Faculty 6.19
M - SP 5.61 0.08
F - SP 5.13
Type Faculty 6.27
<0.001 SP 5.37
Faculty
Generalist 6.48
<0.001 Specialist – Congruent 6.56
Specialist - Incongruent
5.96
Familiarity Yes 6.43
0.45 No 6.32 15
Multivariable Analysis – Faculty Only
VARIABLE MEAN F p
PGY Level
1 = 6.32
5.3 0.005 2 = 6.30
3 = 6.07
Examiner Gender M = 6.35
11.7 0.001 F = 6.11
Examiner Congruity
Generalist = 6.39
32.5 <0.001 Congruent = 6.44
Incongruent = 5.86
Examiner Familiarity Y = 6.43
1.4 0.26 N = 6.32 16
Multivariable Analysis - Communication
VARIABLE MEAN F p
PGY Level
1 = 5.98
1.85 0.16 2 = 5.84
3 = 5.58
SP Gender M = 5.96
3.15 0.08 F = 5.65
Examiner Type SP = 5.36
25.44 <0.001 Faculty = 6.24
17
Discussion – Main Findings
18
• Residents rated the credibility of all examiners highly in a
formative OSCE
• Some characteristics of about learners and examiners may
influence credibility judgments in small, but significant ways:
Training level
Gender
Discussion – Main Findings
• Greater influence on credibility judgements by examiner
congruency with station and by examiner type:
Specialty-congruent versus specialty-incongruent
» Effect of expertise?
Faculty versus SP
» Unlikely content issue
» Unlikely domain issue
» More acceptable from familiar?
19
Acknowledgments
Co-investigators
Matthew Sibbald
Denyse Richardson Heather McDonald-Blumer Rodrigo B. Cavalcanti
Research Assistance Lisa St. Amant
Funding University of Toronto, Department of Medicine,
Education Scholarship & Research Grant
20
• Download the ICRE App,
• Visit the evaluation area in the Main Lobby, near Registration, or
• Go to: http://www.royalcollege.ca/icre-evaluations to complete the session evaluation.
Help us improve. Your input matters.
• Téléchargez l’application de la CIFR
• Visitez la zone d’évaluation dans le hall principal, près du comptoir d’inscription, ou
• Visitez le http://www.collegeroyal.ca/evaluations-
cifr afin de remplir une évaluation de la séance.
Aidez-nous à nous améliorer. Votre opinion compte!
You could be entered to win 1 of 3 $100 gift cards.
Vous courrez la chance de gagner l’un des trois chèques-cadeaux d’une valeur de 100.
22
Example Exam
Tues May 26 PGY2
TRACK 1 TRACK 2
Hemolytic Anemia Hematologist Generalist
Hyper-parathyroid Geriatrician Endocrinologist
Diplopia Neurology Gastroenterology
Error Disclosure Standardized Patient Faculty
Simulator NA NA
24
Limitations
1. Single institution, single program
2. Ceiling effect
3. Unclear if / how affected utilization of feedback
25
Feedback Characteristics
Faculty only Stations Faculty
Specific 5.86
Quality 6.03
Credibility of Examiner 6.31
Communication Stations Faculty SP
Specific 5.84 5.33
Quality 5.99 5.40
Credibility of Examiner 6.27 5.37
26
Effect of Gender Congruency
Resident
Examiner F M Grand Total
F 6.28 6.11 6.19
M 6.56 6.39 6.45
Grand Total 6.41 6.25 6.31
27 p=0.37
Examiner vs SP Score - Communication
MEAN SD RANGE r p
Examiner Score 3.43 0.98 1-5
0.39 <0.001
SP Score 3.99 0.80 2-5
28