![Page 1: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/1.jpg)
Shifting Paradigms: Competency-based Medical Education
and the Quality of Care Problem
Reynolds Meeting2012
![Page 2: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/2.jpg)
Disclosures
Employed by the American Board of Internal Medicine
I receive royalties from Mosby-Elsevier for a textbook on assessment
I am a member of the board of NBME and Medbiquitous
I serve on committees at the AAMC, ABMS, ACGME and NBME
![Page 3: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/3.jpg)
The Quality of Care Problem
![Page 4: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/4.jpg)
Teaching Vs. Non-Teaching Hospital Quality
Performance Ind. COTH Teaching
Non-COTH Teaching
Non-Teaching
30-day Mortality
• AMI 15.1% 15.9% 16.3%
• Pneumonia 10.8% 11.1% 11.7%
30-day Readmission
• AMI 20.3% 19.7% 19.6%
HCAHPS
• Nurse communicated well
70.5% 70.9% 74.9%
• MD communicated well
76.2% 77.0% 81%
• Help when wanted 55.0% 57.0% 64.1%
Shahian DM, Nordberg P, Meyer GS, et al. Contemporary performance of U.S. teaching and nonteaching hospitals. Acad Med. 2012; 87: online.
![Page 5: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/5.jpg)
Care of the Vulnerable Elderly Study Performance on Geriatric Process of Care
ResidentClinics
Mean %
PracticingPhysicians
Mean %
UnivariateF
Structurecoefficients
Documentation of:
Gait evaluation 28.4% 74.2%
77.53** .90
Balance evaluation 21.6% 66.4%
65.51** .82
Medical surrogate 28.0% 54.4%
24.00** .65
End-of-life preferences 29.5% 49.3%
12.85** .55
Vision testing done 40.0% 64.7%
19.09** .55
Hearing assessment 23.3% 40.3%
8.06* .41
Screens for:
Falls risk 18.6% 60.8%
49.60** .67
Cognitive impairment 18.3% 52.0%
29.02** .60
Depression 33.7% 62.6%
24.09** .57
Lynn LA, et al. Acad Med. 2009.
![Page 6: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/6.jpg)
Hospital Comparisons on Quality and Resource Use (Higher scores represent better performance)
Non-teaching (N= 997) Teaching (N=186)
Quality Composite Score
Res
ou
rce
use
Co
mp
osi
te S
core Exemplary
Teaching Hospitals
Source: L. Binder, CEO of Leapfrog Group, email communication, March 2010
![Page 7: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/7.jpg)
“Every system is perfectly designed to achieve the results it gets.”
Paul Batalden
![Page 8: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/8.jpg)
Medical Education: Restraining Forces on Change
![Page 9: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/9.jpg)
The Current “Miracle” of Medical Education
Dwell Time
![Page 10: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/10.jpg)
Medical Education Architecture1
1Holmboe E, Ginsburg S, Bernabeo E. The rotational approach to medical education: time to confront our assumptions. Med Educ. 2011; 45(1):69-80.
![Page 11: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/11.jpg)
Thomas Kuhn and “Normal Science”
“Normal science, the activity in which most scientists inevitably spend almost all of their time, is predicated on the assumption that the scientific community knows what the world is like. Much of the success of the enterprise derives from the community’s willingness to defend that assumption, if necessary at considerable cost”
Thomas S. Kuhn. The Structure of Scientific Revolutions. University of Chicago Press. Chicago. 1962. Pg. 5.
![Page 12: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/12.jpg)
Could the Same be True of UME and GME?
“Normal education, the activity in which most educators inevitably spend almost all of their time, is predicated on the assumption that the educational community knows what the world is like. Much of the success of the enterprise derives from the community’s willingness to defend that assumption, if necessary at considerable cost”
Thomas S. Kuhn. The Structure of Scientific Revolutions. University of Chicago Press. Chicago. 1962. Pg. 5.
![Page 13: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/13.jpg)
The “Big Assumption as Truth”
We operate on many assumptions that over time become “truth” without our testing or questioning the veracity of those assumptions– Test assumptions as assumptions
Immunity to change– Preservation of status quo through fear
– More comfortable to stay with familiar even when status quo isn’t effective
Kegan and Lahey. The Way We Talk Can Change the Way We Work; Immunity to Change.
![Page 14: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/14.jpg)
Competency-based Medical Education:
A Way Forward?
![Page 15: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/15.jpg)
Effective Systems: Where Education Must Occur
Nelson EC, et al. Quality by Design. 2007
![Page 16: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/16.jpg)
Early Principles: CBME
World Health Organization (1978):– “The intended output of a competency-
based programme is a health professional who can practise medicine at a defined level of proficiency, in accord with local conditions, to meet local needs.”
McGaghie WC, Miller GE, Sajid AW, Telder TV. Competency-based Curriculum Development in Medical Education. World Health Organization, Switzerland, 1978.
![Page 17: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/17.jpg)
Traditional versus CBME: Start with System Needs
17
Frenk J. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet. 2010
![Page 18: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/18.jpg)
Competency-Based Medical Education
…is an outcomes-based approach to the design, implementation, assessment and evaluation of a medical education program using an organizing framework of competencies1
1Frank, JR, Snell LS, ten Cate O, et. al. Competency-based medical education: theory to practice. Med Teach. 2010; 32: 638–645
![Page 19: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/19.jpg)
Outcomes-based Education: General Principles
Patient outcomes ≈ Educational outcomes
Experience ≠ Expertise– Exposure and dwell time are not sufficient
proxies for competence• You can do something a 100 times wrong and
develop experience, but it’s still wrong!
Must engage in effective experiences• Critical role for work-based assessments
![Page 20: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/20.jpg)
Need for System Approach:Assessment Perspective
![Page 21: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/21.jpg)
Structured PortfolioITE (formative only)Monthly EvaluationsMiniCEXMedical record audit/QI projectClinical question logMultisource feedbackTrainee contributions (personal portfolio)
o Research project
TraineeReview portfolio Reflect on contentsContribute to portfolio
Program LeadersReview portfolio periodically and systematicallyDevelop early warning systemEncourage reflection and self-assessment
Clinical Competency CommitteePeriodic review – professional growth opportunities for allEarly warning systems
Program Summative Assessment Process
Licensing and Certification USMLEAmerican Boards of Medical Specialties
Assessment During Training: Components
Advisor
![Page 22: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/22.jpg)
Structured Portfolio
Medical record audit
andQI project
MSF: Directed per protocolTwice/year
Practice-based learning and improvement
Systems-based prac
Mini-CEX:10/year
Interpersonal skills and Communication
ITE:1/year
Patient care
Faculty Evaluations
EBM/Question Log
Medical knowledge
Professionalism
Multi-faceted Evaluation
■ Trainee-directed ■ Direct observation
![Page 23: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/23.jpg)
Time
AssessmentActivities
TrainingActivities
SupportingActivities
v v v v v v
= learning task
= learning artifact
= single assessment data-point
= single certification data point for mastery tasks
= learner reflection and planning= social interaction around reflection (supervision)
= learning task being an assessment task also
Model For Programmatic Assessment(With permission from CPM van der Vleuten)
Committee
![Page 24: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/24.jpg)
Structured PortfolioITE (formative only)Monthly EvaluationsMiniCEXMedical record audit/QI projectClinical question logMultisource feedbackTrainee contributions (personal portfolio)
o Research project
TraineeReview portfolio Reflect on contentsContribute to portfolio
Program LeadersReview portfolio periodically and systematicallyDevelop early warning systemEncourage reflection and self-assessment
Clinical Competency CommitteePeriodic review – professional growth opportunities for allEarly warning systems
Program Summative Assessment Process
Licensing and Certification USMLEAmerican Boards of Medical Specialties
Assessment During Training: Components
Advisor
![Page 25: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/25.jpg)
“Wisdom of the Crowd”
• Williams, Teach. Learn. Med. (2005)– No evidence that individuals in groups
dominate discussions.• No evidence of ganging up/piling on
• Thomas (2011) – Group assessment improved inter-rater reliability and reduced range restriction in multiple domains in an internal medicine residency
![Page 26: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/26.jpg)
Narratives and Judgments
• Pangaro (1999) – matching students to a “synthetic” descriptive framework (RIME) reliable and valid across multiple clerkships
• Regehr (2007) – Matching students to a standardized set of holistic, realistic vignettes improved discrimination of student performance
• Regehr (2012) – Faculty created narrative “profiles” (16 in all) found to produce consistent rankings of excellent, competent and problematic performance.
![Page 27: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/27.jpg)
Strategy toestablish trustworthiness Criteria
Potential Assessment Strategy (sample)
Credibility Prolonged engagement Training of examiners
Triangulation Tailored volume of expert judgment based on certainty of information
Peer examination Benchmarking examiners
Member checking Incorporate learner view
Structural coherence Scrutiny of committee inconsistencies
Transferability Time sampling Judgment based on broad sample of data points
Thick description Justify decisions
Dependability Stepwise replication Use multiple assessors who have credibility
Confirmability Audit Give learners the possibility to appeal to the assessment decision*With permission CPM van der Vleuten
![Page 28: Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012](https://reader036.vdocument.in/reader036/viewer/2022062410/5697bff01a28abf838cba9ed/html5/thumbnails/28.jpg)
The Road Forward: Kelly Caverzagie