Karen Hauer, MDKaren Hauer, MD
Professor of Medicine Professor of Medicine
Gold Headed Cane Endowed Education ChairGold Headed Cane Endowed Education Chair
University of California, San FranciscoUniversity of California, San Francisco
Beyond Clinical Performance Beyond Clinical Performance Assessment: Assessment:
Feedback, Learning Goals, and Feedback, Learning Goals, and RemediationRemediation
2010 2010 MaatschMaatsch Lecture Lecture
Thank youThank you
The purpose of the Jack L. The purpose of the Jack L. MaatschMaatsch Visiting Visiting Scholar in Medical Education fund is:Scholar in Medical Education fund is:
to stimulate and support interaction to stimulate and support interaction around ideas and projects in medical around ideas and projects in medical education, with primary focus on the education, with primary focus on the development and assessment of clinical development and assessment of clinical competence related to the full span of competence related to the full span of professional training.professional training.
Medical TrainingMedical Training Modern Medical TrainingModern Medical Training
Remediation
OutlineOutline
Clinical performance assessmentClinical performance assessment
FeedbackFeedback
RemediationRemediation
Feedback Feedback
Learning goalsLearning goals The importance of formative assessmentThe importance of formative assessment Clinical performance assessment in the Clinical performance assessment in the
workplace workplace
(More on) feedback(More on) feedback
Is this doctor competent?Is this doctor competent?
Assessment of CompetenceAssessment of Competence
““Professional competence: the habitual and Professional competence: the habitual and judicious use of communication, knowledge, judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the values, and reflection in daily practice for the benefit of the individual and community being benefit of the individual and community being servedserved””
Developmental continuum from novice Developmental continuum from novice competent competent expertexpert
Epstein RM, Epstein RM, HundertHundert EM. JAMA 2002EM. JAMA 2002
CarracioCarracio et al. et al. AcadAcad Med 2008Med 2008
A series of studies A series of studies supported by the supported by the Josiah Macy, Jr. Josiah Macy, Jr. FoundationFoundation
National Studies of Medical Student National Studies of Medical Student Clinical Skills Assessment and Clinical Skills Assessment and
RemediationRemediation
Karen E. Hauer, MDKaren E. Hauer, MD
Carol S. Hodgson, PhDCarol S. Hodgson, PhD
David Irby, PhDDavid Irby, PhD
Kathleen Kerr, BAKathleen Kerr, BA
Patricia OPatricia O’’Sullivan, Sullivan, EdDEdD
VarunVarun SaxenaSaxena, MD, MD
Arianne Arianne TeheraniTeherani, PhD, PhD
Forces Influencing Clinical Forces Influencing Clinical Skills AssessmentSkills Assessment
19901990’’s: Josiah Macy, Jr. Foundation funded s: Josiah Macy, Jr. Foundation funded 6 regional consortia for undergraduate SP 6 regional consortia for undergraduate SP assessment (California CPX consortium)assessment (California CPX consortium)
1999: ACGME endorses 6 competencies, 1999: ACGME endorses 6 competencies, emphasizes emphasizes outcomesoutcomes of graduate trainingof graduate training
2004: National Board added licensing 2004: National Board added licensing requirement requirement ‐‐ USMLE Step 2 CSUSMLE Step 2 CS
PurposePurpose
To characterize clinical skills assessment To characterize clinical skills assessment programs nationallyprograms nationally
To explore how USMLE Step 2 CS will impact To explore how USMLE Step 2 CS will impact inin‐‐house clinical skills assessment programshouse clinical skills assessment programs
To identify methods used for standard To identify methods used for standard setting and remediationsetting and remediation
Results: Clinical Skills Assessment Results: Clinical Skills Assessment ProgramsPrograms
Timing in Curriculum
Clinical Skills Assessment Program
During/end 1st year 66 (73%)
During/end 2nd year 82 (90%)
During 3rd year 64 (70%)
End of 3rd or during 4th year 76 (84%)
Hauer et al. Acad Med 2005
Feedback to studentsFeedback to students
A critical purpose of the inA critical purpose of the in‐‐house examhouse examAll schools gave all students some feedback All schools gave all students some feedback
Debate about optimal balance of formative Debate about optimal balance of formative and summative feedbackand summative feedback
Hauer, Hauer, AcadAcad Med 2006Med 2006
Lowest scoring students received Lowest scoring students received the most feedbackthe most feedback
It gives me an opportunity to take a student who has an odd affect and now there’s data. Instead of me just saying, ‘Gee, this may be a problem; you don’t make good eye contact,’ I can say now, ‘This is a problem. You’re not making good eye contact. It’s been noted in
your clerkships and, here on the OSCE, you did poorly.
Barriers to giving students feedbackBarriers to giving students feedback
Maintaining exam securityMaintaining exam security
Limited faculty time Limited faculty time
Resources: balancing goals for teaching, Resources: balancing goals for teaching, formative and summative assessmentformative and summative assessment
Focusing students on concepts, not cases or Focusing students on concepts, not cases or checklistschecklists
Risks of competencyRisks of competency‐‐based based assessmentassessment
Focus on minimally acceptable performanceFocus on minimally acceptable performance
No motivation toward lifelong learningNo motivation toward lifelong learning
Emphasis on discrete tasks rather than Emphasis on discrete tasks rather than complex performancecomplex performance
Wash handsWash hands
Say Say ““that must be hardthat must be hard””
Ask patient what she thinks is going onAsk patient what she thinks is going on
Student perspective: preparing for Student perspective: preparing for USMLE Step 2 CSUSMLE Step 2 CS
Barrier to feedback: faculty skillBarrier to feedback: faculty skill
Redefining feedbackRedefining feedback
Current feedback models are limitedCurrent feedback models are limited
ReductionistReductionist
Hierarchical, diagnosticHierarchical, diagnostic
EducatorEducator‐‐driven, onedriven, one‐‐wayway
Archer, Med Archer, Med EducEduc 20102010
Student performance problemsStudent performance problemsand remediationand remediation
Remediation as feedback for the students Remediation as feedback for the students most in needmost in need
Student performance problemsStudent performance problems
Types of (interTypes of (inter‐‐related) problems in clinical related) problems in clinical performance assessmentsperformance assessments
1.1. Technique errorsTechnique errors
2.2. Cognitive: knowledge, clinical Cognitive: knowledge, clinical reasoningreasoning
3.3. NonNon‐‐cognitive: communication, cognitive: communication, professionalismprofessionalism
Hauer, et al. Acad Med 2007
NonNon‐‐cognitive problems: cognitive problems: communication and professionalismcommunication and professionalism
Poor insight or resistance to exam processPoor insight or resistance to exam process
Lack of respect/empathyLack of respect/empathy
Blaming test environment rather than Blaming test environment rather than oneone’’s own deficienciess own deficiencies
Most challenging to remediate Most challenging to remediate ‐‐maladaptive maladaptive personality stylespersonality styles
Problems Problems receivingreceiving the feedbackthe feedback
ImplicationsImplications
Insight crucial for remediationInsight crucial for remediation
Promoting reflection may facilitate Promoting reflection may facilitate studentsstudents’’ ability to incorporate changes ability to incorporate changes in their approach to patients in their approach to patients
Remediation techniquesRemediation techniquesClinical
• Preceptorships• Remediation within clerkships
• Special clinical rotations
Independent
• Independent study (Web module, reading)
• Student reviews exam recording alone
Precepted video review•Precepted review of exam recording
Organized group activities
• Practice with SPs
• Skills workshops, or group discussions
Saxena V et al. AcadMed 2009
Use of Remediation ActivitiesUse of Remediation Activities
1 = never, 3 = half the time, 5 = always Saxena V et al. Acad Med 2009
Confidence in exam and remediationConfidence in exam and remediation
I am confident in: Mean
our exam scores 4.1
our ability to diagnose student problems 4.0
our ability to remediate history taking, physical exam 3.8
our ability to remediate communication 3.6
our ability to remediate clinical reasoning 3.2
our ability to remediate knowledge 3.1
our ability to remediate professionalism 2.9
I am satisfied with our school’s remediation process 3.5
1 = strongly disagree, 5 = strongly agree
Much uncertainty about remediationMuch uncertainty about remediation
Schools invest significant resources in Schools invest significant resources in analyzing why a student failedanalyzing why a student failed Confidence in diagnosis is highConfidence in diagnosis is high
Confidence in remediation is lowConfidence in remediation is low
Schools use resource nonSchools use resource non‐‐intensive remediation intensive remediation activities (activities (PreceptedPrecepted Video Review) most, and Video Review) most, and resource intensive activities (Clinical Activities) leastresource intensive activities (Clinical Activities) least
Institutions that required retest are most satisfied Institutions that required retest are most satisfied with their remediation process.with their remediation process.
Challenges of remediating late in Challenges of remediating late in medical schoolmedical school
I passed all of my core clerkshipsI passed all of my core clerkships……
I am busy doing my I am busy doing my subinternshipsubinternship……
I have to do 4 visiting I have to do 4 visiting ““auditionaudition”” rotationsrotations……
Residency applications are dueResidency applications are due……
Is there a better way?Is there a better way?
Remediation of Physicians across the Continuum from Medical School to Practice:
a Thematic Review of the Literature
KE Hauer, A KE Hauer, A CicconeCiccone, TR , TR HenzelHenzel, P , P KatsufrakisKatsufrakis, SH Miller, WA , SH Miller, WA Norcross, MA Norcross, MA PapadakisPapadakis, DM Irby, DM Irby
AcadAcad Med 2009Med 2009
Narrative literature review of Narrative literature review of remediationremediation
Study criteriaStudy criteria
1.1. Deficiencies in an individualDeficiencies in an individual’’s s performance identified through an performance identified through an assessment process assessment process
2.2. Attempt is made to provide Attempt is made to provide remedial educationremedial education
3.3. Reassessment after remediationReassessment after remediation
13 published studies of remediation 13 published studies of remediation
Deficit Identification Intervention Outcome
UME Knowledge, clinical skills
Written or SP exam
Faculty video review, SP practice, tutorials
Retest –knowledge or SP exam
GME Knowledge In-training exam Individual study plan, mentor, clinical rotations
Exam scores
CME MD practice Chart review, interviews
Individualized Chart or practice review
Using the learning sciences to guide Using the learning sciences to guide remediationremediation
Knowledge deficit Help build strong knowledge structures and representations (schema, scripts, exemplars, prototypes)
Knowledge and skills deficits
deliberate (conscious and focused) practice with feedback
Professionalism problem
explicit instruction, guided practice, mentored reflection, observation and interaction with role models
Competence Assessment:Multi-modal Assessment
Diagnosis of Deficiencyand Development of
Individualized Learning Plan
Instruction/Remediation Activities with Deliberate Practice, Feedback and Reflection
Focused Reassessment and Certification of Competence
Mentoring and
Coaching
Costs of remediationCosts of remediation
•• Resource intensive interventions for a small Resource intensive interventions for a small number of learners number of learners
•• Creating efficienciesCreating efficiencies
–– ““referral centersreferral centers”” –– collaborative across collaborative across institutionsinstitutions
–– Span UME/GME/CMESpan UME/GME/CME
–– Promote learner Promote learner selfself‐‐assessment assessment of skillsof skills
Feedback and Feedback and formative assessmentformative assessment
Formative assessment Formative assessment
1.1. teacher and learner clarify expectations: learning teacher and learner clarify expectations: learning intentions and criteria for successintentions and criteria for success
2.2. teacher provides feedback teacher provides feedback
3.3. learner is activated as owner of own learning (selflearner is activated as owner of own learning (self‐‐assessment, learning goals)assessment, learning goals)
Black, Black, WiliamWiliam. . EducEduc Assess Assess EvalEval Acc. 2009 Acc. 2009
Lifelong learningLifelong learning
Hojat et al, Medical Teacher, 2003
The inaccuracy of selfThe inaccuracy of self‐‐assessmentassessment
Low AverageLow Average HighHigh
Actual abilityActual ability
Davis D et al, JAMA 2006Davis D et al, JAMA 2006
Self‐assessedability
Feedback complexity affects Feedback complexity affects effectiveness of feedback effectiveness of feedback
Verification (right/wrong, % correct)Verification (right/wrong, % correct)
Right answer, tryRight answer, try‐‐again until rightagain until right
Elaborated Elaborated
Hints/cues, error analysisHints/cues, error analysis
Informative tutoringInformative tutoring
Shute VJ. Review Shute VJ. Review EducEduc RschRsch 2008 2008
Recipient characteristics affect Recipient characteristics affect effectiveness of feedbackeffectiveness of feedback
Shute VJ, Review Educ Rsch 2008
High performers benefit from feedback that is:High performers benefit from feedback that is:
DelayedDelayed
Facilitative (hints, cues, challenges)Facilitative (hints, cues, challenges)
Low performers benefit from feedback that is:Low performers benefit from feedback that is:
ImmediateImmediate
DirectiveDirective
Providing scaffoldingProviding scaffolding
How do our own students hear How do our own students hear and incorporate feedback from and incorporate feedback from
the CPX?the CPX?
StudentsStudents’’ selfself‐‐assessment after a assessment after a standardized patient examstandardized patient exam
Feedback Self‐assessment
1 None • Poor correlation with actual performance• Students did rank order their skills accurately (history, exam, communication)
2 Individual video review
• Students downgraded their performance rating, particularly the low performers
• Still a poor correlation between self‐assessed and actual performance
3 Individual video review + performance benchmarks
• Improved accuracy of self‐assessment – modest correlation
• Correlation improved most for the low performers
Srinivasan et al, Med Educ 2007
Influences on learning goalsInfluences on learning goals
All feedback is filtered through the recipient’s own self‐assessment
Which factors influence students’ learning goals after OSCE cases AND feedback (faculty scores, comments)?
Student, case, faculty feedback?
Students’ self‐assessments most strongly informed learning goals
Eva et al, Eva et al, AcadAcad Med 2010Med 2010
Goal orientationGoal orientation
“I care about thi
s
material and
want to be a
superb doctor”
KH1
Goal orientationGoal orientationKH2
Shepard, Educ Res 2009
Curricular goals in the era of Curricular goals in the era of competency based assessmentcompetency based assessment
Goal: to standardize learning outcomes Goal: to standardize learning outcomes and assess competencies over time AND and assess competencies over time AND individualize the curriculumindividualize the curriculum……Could argue forCould argue for a structured supplemental curriculum for a structured supplemental curriculum for practice and feedbackpractice and feedback
shift from remediation to individualized shift from remediation to individualized curriculumcurriculum
Cooke, Irby, OCooke, Irby, O’’Brien. Educating Physicians: A Call forBrien. Educating Physicians: A Call forReform of Medical School and Residency. 2010. Reform of Medical School and Residency. 2010.
What if we move from a paradigm What if we move from a paradigm of remediation to one of of remediation to one of formative assessment and formative assessment and
individualized learning plans?individualized learning plans?
Changing our approach to Changing our approach to remediationremediation
Predicting Failing Performance on a Predicting Failing Performance on a Standardized Patient Exam (CPX)Standardized Patient Exam (CPX)
•• Which medical school assessment measures Which medical school assessment measures identify students at risk of failing CPX?identify students at risk of failing CPX?
•• Retrospective caseRetrospective case‐‐control studycontrol study
•• Results: students flagged even once in Results: students flagged even once in student progress meetings or clerkships student progress meetings or clerkships ‐‐> >
•• increased risk of failing CPX in communicationincreased risk of failing CPX in communication
•• no predictors of failing H & Pno predictors of failing H & P
Chang et al. Chang et al. AcadAcad Med 2009Med 2009
Clinical Skills Guidance ProgramClinical Skills Guidance Program
Goals:Goals:
Partner with the student progress Partner with the student progress committee and advisory college mentorscommittee and advisory college mentors
Create a competencyCreate a competency‐‐based based supplemental curriculum for H&P and supplemental curriculum for H&P and communication for atcommunication for at‐‐risk students risk students through the clinical core yearthrough the clinical core year
Curriculum activitiesCurriculum activities
Student completes written reflection and own Student completes written reflection and own learning goals learning goals with advisory college mentor with advisory college mentor and clinical skills guidance facultyand clinical skills guidance faculty
1.1. Student watches faculty narrated video casesStudent watches faculty narrated video cases
2.2. Student Student ‐‐ clinical skills guidance faculty: clinical skills guidance faculty: individual performance review, learning plan individual performance review, learning plan (Gap analysis)(Gap analysis)
3.3. Standardized patient workshopStandardized patient workshop
4.4. HighHigh‐‐stakes standardized patient exam (CPX)stakes standardized patient exam (CPX)
The importance of formative The importance of formative assessmentassessment
Feedback must be conceptualized as a Feedback must be conceptualized as a supported sequential process rather than a supported sequential process rather than a series of unrelated events. series of unrelated events.
Requires established, longerRequires established, longer‐‐term term professional relationshipsprofessional relationships
Archer. Med Archer. Med EducEduc 20102010
Next steps:Next steps:feedback in the workplacefeedback in the workplace
SocioculturalSociocultural theory of learning: learn through active theory of learning: learn through active participation in work of a communityparticipation in work of a community
Clinical workplace learning: oriented toward the Clinical workplace learning: oriented toward the central goal of helping patientscentral goal of helping patients
guided participation to engage learners in patient guided participation to engage learners in patient care, challenge to perform at higher levels of care, challenge to perform at higher levels of competencecompetence
supervisor adapts level of support and feedback to supervisor adapts level of support and feedback to learnerlearner’’s needss needs
Dornan T et al. Med Educ 2007
Faculty and student perceptions of Faculty and student perceptions of evaluation in two clerkship modelsevaluation in two clerkship models
Traditional block clerkships and longitudinal Traditional block clerkships and longitudinal integrated clerkshipintegrated clerkship
Longitudinal clerkship:Longitudinal clerkship: Multiple simultaneous Multiple simultaneous preceptorshipspreceptorships (developmental)(developmental)
Longitudinal preceptors (relationships)Longitudinal preceptors (relationships)
Brief structured clinical observations (bedside teaching, Brief structured clinical observations (bedside teaching, feedback)feedback)
Survey of preceptors, students in each Survey of preceptors, students in each clerkship modelclerkship model
MazottiMazotti L et al. Med L et al. Med EducEduc, in press, in press
McGaghieMcGaghie WC et al. WC et al. AcadAcad Med 1995Med 1995
Greater satisfaction and confidence Greater satisfaction and confidence with evaluation in the LICwith evaluation in the LIC
Preceptors and students favored evaluation Preceptors and students favored evaluation in the LIC on three factors (pin the LIC on three factors (p≤≤.01):.01):
1.1. validity of evaluation processvalidity of evaluation process
2.2. quality of clinical skill evaluationquality of clinical skill evaluation
3.3. willingness to provide constructive feedbackwillingness to provide constructive feedback
Informed selfInformed self‐‐assessmentassessment
-> Conditions: environment, relationships, personal attributes
-> Tensions
Sargeant et al, Acad Med 2010
Lessons learned from aLessons learned from aclinician/administrator/researcherclinician/administrator/researcher
1.1. Collaborate; build strong teamsCollaborate; build strong teams
2.2. Clinical and teaching venues are a Clinical and teaching venues are a ‘‘lablab’’
3.3. Recognize unanswered questionsRecognize unanswered questions
4.4. Intersect theory with dayIntersect theory with day‐‐toto‐‐day educational day educational workwork
5.5. Plan ahead to study curricular interventionsPlan ahead to study curricular interventions
6.6. Look outside your comfort zone Look outside your comfort zone –– other other learners, other fieldslearners, other fields
SummarySummary
Psychometrically sound summative assessment, Psychometrically sound summative assessment, while important, does little in isolation to advance while important, does little in isolation to advance learninglearning
For low performers, surprisingly little evidence For low performers, surprisingly little evidence guides remediationguides remediation
Formative assessment promotes learning for all Formative assessment promotes learning for all learnerslearners
Promote individualized articulation of next stepsPromote individualized articulation of next steps
Facilitate constant feedback loopsFacilitate constant feedback loops
ReferencesReferences
Archer JC. State of the science in health professional educationArcher JC. State of the science in health professional education: : effective feedback. Med effective feedback. Med EducEduc 2010; 44:1012010; 44:101‐‐8.8.
Black P, Black P, WiliamWiliam D. Developing the theory of formative D. Developing the theory of formative assessment. assessment. EducEduc Assess Assess EvalEval Acc 2009. 21:5Acc 2009. 21:5‐‐31. 31.
CarraccioCarraccio CL, Benson BJ, Nixon LJ, CL, Benson BJ, Nixon LJ, DerstineDerstine PL.EvaPL.Eva KW. From the KW. From the educational bench to the clinical bedside: translating the Dreyfeducational bench to the clinical bedside: translating the Dreyfus us developmental model to the learning of clinical skills. developmental model to the learning of clinical skills. AcadAcad Med Med 2008;83:7612008;83:761‐‐7. 7.
Chang A, Chang A, BoscardinBoscardin C, Chou CL, C, Chou CL, LoeserLoeser H, Hauer KE. Predicting H, Hauer KE. Predicting failing performance on a standardized patient clinical failing performance on a standardized patient clinical performance examination: the importance of communication and performance examination: the importance of communication and professionalism skills deficits. professionalism skills deficits. AcadAcad Med. 2009;84:S101Med. 2009;84:S101‐‐4.4.
Cooke M, Irby DM, OCooke M, Irby DM, O’’Brien BC. Educating physicians: a call for Brien BC. Educating physicians: a call for reform in the twentieth century. Josiahreform in the twentieth century. Josiah‐‐Bass. 2010Bass. 2010
ReferencesReferences Davis DA, Davis DA, MazmanianMazmanian PE, PE, FordisFordis M, Van Harrison R, Thorpe KE, M, Van Harrison R, Thorpe KE,
Perrier L . Accuracy of physician selfPerrier L . Accuracy of physician self‐‐assessment compared with assessment compared with observed measures of competence. JAMA. 2006;296:1094observed measures of competence. JAMA. 2006;296:1094‐‐1102.1102.
Epstein RM, Epstein RM, HundertHundert EM. Defining and assessing professional EM. Defining and assessing professional competence. JAMA. 2002;287:226competence. JAMA. 2002;287:226‐‐35.35.
Eva KW, Munoz J, Hanson MD, Walsh A, Wakefield J. Which Eva KW, Munoz J, Hanson MD, Walsh A, Wakefield J. Which factors, personal or external, most influence studentsfactors, personal or external, most influence students’’generation of learning goals? generation of learning goals? AcadAcad Med. 2010;85:S102Med. 2010;85:S102‐‐5.5.
Hauer KE, Hauer KE, HodgsonHodgson CS, Kerr KM, CS, Kerr KM, TeheraniTeherani A, A, IrbyIrby DM. DM. A A national study of medical student clinical skills assessment. national study of medical student clinical skills assessment. Academic Medicine. 2005;80:S25Academic Medicine. 2005;80:S25‐‐9.9.
Hauer KE, Hauer KE, TeheraniTeherani A, Kerr KM, O'Sullivan PS, A, Kerr KM, O'Sullivan PS, IrbyIrby DM. DM. Impact Impact of the United States medical licensing examination step 2 of the United States medical licensing examination step 2 clinical skills exam on medical school clinical skills assessmenclinical skills exam on medical school clinical skills assessment. t. AcadAcad Med. 2006;81:S13Med. 2006;81:S13‐‐6.6.
Hauer KE, Hauer KE, TeheraniTeherani A, Kerr KM, OA, Kerr KM, O’’Sullivan PS, Sullivan PS, IrbyIrby DM. DM. Student Student performance problems in medical school clinical skills performance problems in medical school clinical skills assessments. assessments. AcadAcad Med. 2007;82:S69Med. 2007;82:S69‐‐72.72.
Hauer KE, Hauer KE, TeheraniTeherani A, A, IrbyIrby DM, Kerr KM, ODM, Kerr KM, O’’Sullivan PS. Sullivan PS. Approaches to medical student remediation after a Approaches to medical student remediation after a comprehensive clinical skills exam. Med Educ. 2008; 42;104comprehensive clinical skills exam. Med Educ. 2008; 42;104‐‐110110
Hauer KE, Hauer KE, TeheraniTeherani A, Kerr KM, A, Kerr KM, IrbyIrby DM, ODM, O’’Sullivan PS. Sullivan PS. Consequences for students of poor performance on a medical Consequences for students of poor performance on a medical school standardized patient comprehensive assessment. school standardized patient comprehensive assessment. AcadAcadMed. 2009;84:663Med. 2009;84:663‐‐8.8.
Hauer KE, Hauer KE, CicconeCiccone A, A, HenzelHenzel TR, TR, KatsufrakisKatsufrakis, P, Miller SH, , P, Miller SH, Norcross WA, Norcross WA, PapadakisPapadakis MA, Irby DM. Remediation of physicians MA, Irby DM. Remediation of physicians across the continuum from medical school to practice: a thematicacross the continuum from medical school to practice: a thematicreview of the literature. review of the literature. AcadAcad Med 2009; 84:1822Med 2009; 84:1822‐‐32.32.
HojatHojat M, M, NascaNasca TJ, Erdmann JB, TJ, Erdmann JB, FrisbyFrisby AJ, AJ, VeloskiVeloski JJ, JJ, GonnellaGonnellaJS. An operational measure of physician lifelong learning: its JS. An operational measure of physician lifelong learning: its development, components and preliminary psychometric data. development, components and preliminary psychometric data. Med.TeachMed.Teach 2003;;25:4332003;;25:433‐‐7. 7.
MazottiMazotti L, OL, O’’Brien BC, Tong L, Hauer KE. Brien BC, Tong L, Hauer KE. Faculty and student Faculty and student perceptions of evaluation in two clerkship models. In press, Medperceptions of evaluation in two clerkship models. In press, MedEduc. Educ.
McGaghieMcGaghie WC, Richards BF, WC, Richards BF, PetrusaPetrusa ER, et al. Development of a ER, et al. Development of a measure of medical faculty attitudes toward clinical evaluation measure of medical faculty attitudes toward clinical evaluation of students. of students. AcadAcad Med 1995;70:47Med 1995;70:47‐‐51.51.
SargeantSargeant J et al. The processes and dimensions of informed selfJ et al. The processes and dimensions of informed self‐‐assessment: a conceptual model. assessment: a conceptual model. AcadAcad Med 2010;85:1212Med 2010;85:1212‐‐20.20.
SaxenaSaxena V, OV, O’’Sullivan PS, Sullivan PS, TeheraniTeherani A, A, IrbyIrby DM, Hauer KE. DM, Hauer KE. Remediation techniques for student performance problems after Remediation techniques for student performance problems after a comprehensive clinical skills exam. a comprehensive clinical skills exam. AcadAcad Med. 2009;84:669Med. 2009;84:669‐‐76.76.
Shute VJ. Focus on formative feedback. Review Shute VJ. Focus on formative feedback. Review EducEduc RschRsch 2008.2008. SrinivasanSrinivasan M, Hauer KE, M, Hauer KE, DerDer‐‐MartirosianMartirosian C, Wilkes M, C, Wilkes M,
GesundheitGesundheit N. Does feedback matter in practiceN. Does feedback matter in practice‐‐based learning? based learning? Impacting the accuracy of medical student selfImpacting the accuracy of medical student self‐‐assessment after assessment after a multia multi‐‐institutional clinical performance examination. Med institutional clinical performance examination. Med Educ. 2007;41:857Educ. 2007;41:857‐‐65 65