shifting paradigms: competency-based medical education and the quality of care problem reynolds...
TRANSCRIPT
Shifting Paradigms: Competency-based Medical Education
and the Quality of Care Problem
Reynolds Meeting2012
Disclosures
Employed by the American Board of Internal Medicine
I receive royalties from Mosby-Elsevier for a textbook on assessment
I am a member of the board of NBME and Medbiquitous
I serve on committees at the AAMC, ABMS, ACGME and NBME
The Quality of Care Problem
Teaching Vs. Non-Teaching Hospital Quality
Performance Ind. COTH Teaching
Non-COTH Teaching
Non-Teaching
30-day Mortality
• AMI 15.1% 15.9% 16.3%
• Pneumonia 10.8% 11.1% 11.7%
30-day Readmission
• AMI 20.3% 19.7% 19.6%
HCAHPS
• Nurse communicated well
70.5% 70.9% 74.9%
• MD communicated well
76.2% 77.0% 81%
• Help when wanted 55.0% 57.0% 64.1%
Shahian DM, Nordberg P, Meyer GS, et al. Contemporary performance of U.S. teaching and nonteaching hospitals. Acad Med. 2012; 87: online.
Care of the Vulnerable Elderly Study Performance on Geriatric Process of Care
ResidentClinics
Mean %
PracticingPhysicians
Mean %
UnivariateF
Structurecoefficients
Documentation of:
Gait evaluation 28.4% 74.2%
77.53** .90
Balance evaluation 21.6% 66.4%
65.51** .82
Medical surrogate 28.0% 54.4%
24.00** .65
End-of-life preferences 29.5% 49.3%
12.85** .55
Vision testing done 40.0% 64.7%
19.09** .55
Hearing assessment 23.3% 40.3%
8.06* .41
Screens for:
Falls risk 18.6% 60.8%
49.60** .67
Cognitive impairment 18.3% 52.0%
29.02** .60
Depression 33.7% 62.6%
24.09** .57
Lynn LA, et al. Acad Med. 2009.
Hospital Comparisons on Quality and Resource Use (Higher scores represent better performance)
Non-teaching (N= 997) Teaching (N=186)
Quality Composite Score
Res
ou
rce
use
Co
mp
osi
te S
core Exemplary
Teaching Hospitals
Source: L. Binder, CEO of Leapfrog Group, email communication, March 2010
“Every system is perfectly designed to achieve the results it gets.”
Paul Batalden
Medical Education: Restraining Forces on Change
The Current “Miracle” of Medical Education
Dwell Time
Medical Education Architecture1
1Holmboe E, Ginsburg S, Bernabeo E. The rotational approach to medical education: time to confront our assumptions. Med Educ. 2011; 45(1):69-80.
Thomas Kuhn and “Normal Science”
“Normal science, the activity in which most scientists inevitably spend almost all of their time, is predicated on the assumption that the scientific community knows what the world is like. Much of the success of the enterprise derives from the community’s willingness to defend that assumption, if necessary at considerable cost”
Thomas S. Kuhn. The Structure of Scientific Revolutions. University of Chicago Press. Chicago. 1962. Pg. 5.
Could the Same be True of UME and GME?
“Normal education, the activity in which most educators inevitably spend almost all of their time, is predicated on the assumption that the educational community knows what the world is like. Much of the success of the enterprise derives from the community’s willingness to defend that assumption, if necessary at considerable cost”
Thomas S. Kuhn. The Structure of Scientific Revolutions. University of Chicago Press. Chicago. 1962. Pg. 5.
The “Big Assumption as Truth”
We operate on many assumptions that over time become “truth” without our testing or questioning the veracity of those assumptions– Test assumptions as assumptions
Immunity to change– Preservation of status quo through fear
– More comfortable to stay with familiar even when status quo isn’t effective
Kegan and Lahey. The Way We Talk Can Change the Way We Work; Immunity to Change.
Competency-based Medical Education:
A Way Forward?
Effective Systems: Where Education Must Occur
Nelson EC, et al. Quality by Design. 2007
Early Principles: CBME
World Health Organization (1978):– “The intended output of a competency-
based programme is a health professional who can practise medicine at a defined level of proficiency, in accord with local conditions, to meet local needs.”
McGaghie WC, Miller GE, Sajid AW, Telder TV. Competency-based Curriculum Development in Medical Education. World Health Organization, Switzerland, 1978.
Traditional versus CBME: Start with System Needs
17
Frenk J. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet. 2010
Competency-Based Medical Education
…is an outcomes-based approach to the design, implementation, assessment and evaluation of a medical education program using an organizing framework of competencies1
1Frank, JR, Snell LS, ten Cate O, et. al. Competency-based medical education: theory to practice. Med Teach. 2010; 32: 638–645
Outcomes-based Education: General Principles
Patient outcomes ≈ Educational outcomes
Experience ≠ Expertise– Exposure and dwell time are not sufficient
proxies for competence• You can do something a 100 times wrong and
develop experience, but it’s still wrong!
Must engage in effective experiences• Critical role for work-based assessments
Need for System Approach:Assessment Perspective
Structured PortfolioITE (formative only)Monthly EvaluationsMiniCEXMedical record audit/QI projectClinical question logMultisource feedbackTrainee contributions (personal portfolio)
o Research project
TraineeReview portfolio Reflect on contentsContribute to portfolio
Program LeadersReview portfolio periodically and systematicallyDevelop early warning systemEncourage reflection and self-assessment
Clinical Competency CommitteePeriodic review – professional growth opportunities for allEarly warning systems
Program Summative Assessment Process
Licensing and Certification USMLEAmerican Boards of Medical Specialties
Assessment During Training: Components
Advisor
Structured Portfolio
Medical record audit
andQI project
MSF: Directed per protocolTwice/year
Practice-based learning and improvement
Systems-based prac
Mini-CEX:10/year
Interpersonal skills and Communication
ITE:1/year
Patient care
Faculty Evaluations
EBM/Question Log
Medical knowledge
Professionalism
Multi-faceted Evaluation
■ Trainee-directed ■ Direct observation
Time
AssessmentActivities
TrainingActivities
SupportingActivities
v v v v v v
= learning task
= learning artifact
= single assessment data-point
= single certification data point for mastery tasks
= learner reflection and planning= social interaction around reflection (supervision)
= learning task being an assessment task also
Model For Programmatic Assessment(With permission from CPM van der Vleuten)
Committee
Structured PortfolioITE (formative only)Monthly EvaluationsMiniCEXMedical record audit/QI projectClinical question logMultisource feedbackTrainee contributions (personal portfolio)
o Research project
TraineeReview portfolio Reflect on contentsContribute to portfolio
Program LeadersReview portfolio periodically and systematicallyDevelop early warning systemEncourage reflection and self-assessment
Clinical Competency CommitteePeriodic review – professional growth opportunities for allEarly warning systems
Program Summative Assessment Process
Licensing and Certification USMLEAmerican Boards of Medical Specialties
Assessment During Training: Components
Advisor
“Wisdom of the Crowd”
• Williams, Teach. Learn. Med. (2005)– No evidence that individuals in groups
dominate discussions.• No evidence of ganging up/piling on
• Thomas (2011) – Group assessment improved inter-rater reliability and reduced range restriction in multiple domains in an internal medicine residency
Narratives and Judgments
• Pangaro (1999) – matching students to a “synthetic” descriptive framework (RIME) reliable and valid across multiple clerkships
• Regehr (2007) – Matching students to a standardized set of holistic, realistic vignettes improved discrimination of student performance
• Regehr (2012) – Faculty created narrative “profiles” (16 in all) found to produce consistent rankings of excellent, competent and problematic performance.
Strategy toestablish trustworthiness Criteria
Potential Assessment Strategy (sample)
Credibility Prolonged engagement Training of examiners
Triangulation Tailored volume of expert judgment based on certainty of information
Peer examination Benchmarking examiners
Member checking Incorporate learner view
Structural coherence Scrutiny of committee inconsistencies
Transferability Time sampling Judgment based on broad sample of data points
Thick description Justify decisions
Dependability Stepwise replication Use multiple assessors who have credibility
Confirmability Audit Give learners the possibility to appeal to the assessment decision*With permission CPM van der Vleuten
The Road Forward: Kelly Caverzagie