the minicex and the evaluation of clinical skills national health service foundation training...
TRANSCRIPT
![Page 1: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/1.jpg)
The MiniCEX and the The MiniCEX and the Evaluation of Clinical SkillsEvaluation of Clinical Skills
National Health Service
Foundation Training Program
![Page 2: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/2.jpg)
Patient Care CompetenciesPatient Care Competencies
Gather accurate, essential information from all sources, including medical interviews, physical examinations, medical records and diagnostic/therapeutic procedures
Make informed recommendations about preventive, diagnostic and therapeutic options and interventions that are based on clinical judgment, scientific evidence, and patient preference.
![Page 3: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/3.jpg)
Patient Care: ThemesPatient Care: Themes
Clinical skills essential to patient careCannot make “good” decisions unless you
work with good and accurate information– GIGO principle
Evaluation of clinical skills requires direct observation
![Page 4: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/4.jpg)
Workshop ObjectivesWorkshop Objectives
Review current state of – Physician clinical skills– Faculty evaluation skills
Understand the importance of direct observation by faculty for assessment of clinical skills
![Page 5: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/5.jpg)
Workshop ObjectivesWorkshop Objectives
Discuss practical strategies for focused direct observation
Review rater training methods
– Direct Observation of Competence (DOC) training
![Page 6: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/6.jpg)
Workshop ElementsWorkshop Elements
Mini-Lectures: – State of clinical skills– Quality of faculty ratings
Direct observation exercises– Performance dimension exercise– Frame of reference training
![Page 7: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/7.jpg)
Video ExerciseVideo Exercise
Situation: A foundation trainee performs a medical interview in the outpatient setting. Using the MiniCEX form provided, please rate the performance of this trainee.
![Page 8: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/8.jpg)
Key “Basic” Clinical SkillsKey “Basic” Clinical Skills
Medical interviewingPhysical examinationsCounseling/patient educationClinical judgment/reasoningReflective practice
– Self-directed learning– Professional growth and improvement– Medical errors
![Page 9: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/9.jpg)
Are Clinical Skills Important?Are Clinical Skills Important?
Where do clinical skills fall into the hierarchy of physician competencies and mastery in an era of advanced technology?
![Page 10: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/10.jpg)
Diagnosis and Medical InterviewDiagnosis and Medical Interview
Hampton (BMJ, 1975):– Medical interview: 82%– Physical exam: 9%– Laboratory: 9%
Kirch (Medicine, 1996)– Medical interview (+PE): 70%– Imaging: 35%
![Page 11: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/11.jpg)
Importance of Sound Clinical SkillsImportance of Sound Clinical Skills
Diagnostic errors– Inaccurate/ incomplete medical interview
one of leading causes (Bordage)• Wrong information leads to wrong decisions
Patient satisfaction– Higher with better communication skills
Patient self-care– Better adherence and outcomes associated with
better physician communication skills
![Page 12: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/12.jpg)
Clinical Reasoning: A PrimerClinical Reasoning: A Primer
Patient/situation
characteristicsPrior knowledge
Problem Representation*
Information GatheringContext
Evaluation Action
Gruppen and Frohna, International Handbook on Research, 2002
![Page 13: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/13.jpg)
Clinical Skills: U.K. TraineesClinical Skills: U.K. Trainees
Fox (2000)– Voluntary study of 22 PRHOs using OSCE– Only 45% with passing score on drug advice
communication station– 0% passed locomotor system examination
Evans (2004)– 26 new PHROs
– All had passed 22 station OSCE in medical school
– Majority failed skill stations in blood pressure measurement and cannulation
![Page 14: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/14.jpg)
Clinical Skills: U.S. TraineesClinical Skills: U.S. Trainees
Stillman (1990)– OSCE: wide variability in graduating
medical student clinical skills Mangione (1997)
– Deficient cardiac and pulmonary auscultatory skills
– Medical students, FP and IM residents– Replicated findings in Canada and U.K.
![Page 15: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/15.jpg)
Clinical Skills: Practicing MDsClinical Skills: Practicing MDs
Ramsey (1998)– Incomplete history-taking / preventive health
screening among Primary care physicians in Northwest Unites States
Braddock (1999)– Study of informed decision making (IDM) and
counseling– Simple analysis of presence or absence of 7 key
elements– 1058 outpatient visits: only 9% of visits met minimal
criteria for IDM
![Page 16: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/16.jpg)
Importance of Faculty: U.K. StudiesImportance of Faculty: U.K. Studies
Grant (Med Educ, 2003)– Inadequate coverage and frequency of
supervision activities– Discordance between specialist registrars and
attendings Kilminster (Med Educ, 2000)
– Systematic review of supervision– Better supervision associated with improved
patient safety and quality of care.
![Page 17: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/17.jpg)
Importance of Faculty: U.S. StudiesImportance of Faculty: U.S. Studies
Inpatient Study (Lancet, 2003)– Reviewed 100 consecutive admissions to
teaching service in U.S.– Faculty detected 26 PExam findings missed by
house officer that impacted patient’s care Outpatient Studies
– Two separate studies showed that faculty assessment disagreed with that of house officer in up to 30% of patients
![Page 18: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/18.jpg)
Clinical Skills: Themes Clinical Skills: Themes
Deficiencies exist across continuumSpecific skills more “error-prone”
– Eg: musculoskeletal and neuro exams
Not detected by other evaluation methods
– Performance of basic clinic skills does not correlate with performance in other dimensions of competence
![Page 19: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/19.jpg)
Clinical Skills: ThemesClinical Skills: Themes
House officers:
– Aware of importance
– Recognize under-emphasisWithout detection deficiencies in
clinical skills cannot be corrected
![Page 20: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/20.jpg)
Miller’s PyramidMiller’s Pyramid
KNOWS
KNOWS HOW
SHOWS
HOW
DOES
MCQ EXAM
Extended matching / CRQ
OSCE
Portfolios
Faculty ObservationFaculty Observation
![Page 21: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/21.jpg)
Faculty Observation / Rating SkillsFaculty Observation / Rating Skills
Patient care settings– Ratings based mostly on perceived
knowledge and personality– Little evidence of direct observation– Significant “Halo” effect
Gray, Thompson, Haber, Grant, etc.
![Page 22: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/22.jpg)
Faculty Observation / Rating SkillsFaculty Observation / Rating Skills
Research settings– Poor inter-rater reliability– Brief rater training methods ineffective
• Didactic instructions
• Demonstration videos without practice
– Accuracy: structured > open-ended forms– Increased accuracy discriminative ability
Kalet, Herbers, Noel, Kroboth
![Page 23: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/23.jpg)
Faculty as Raters – Key IssuesFaculty as Raters – Key Issues
Faculty do not observe actual performance
Faculty ratings lack:– Reliability– Accuracy/validity
Content specificity– How comfortable are you with own skills?
![Page 24: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/24.jpg)
Improving Faculty Ratings: SolutionsImproving Faculty Ratings: Solutions
Step 1: Getting faculty to observe– Required a part of Foundation Program– Focused observations are logistically possible– 5 to 10 minute observations are valuable– Build on faculty “epiphany”
• The “you will not believe what I saw today” experience
– Provide “usable tool”
![Page 25: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/25.jpg)
Foundation Mini - CEX ToolFoundation Mini - CEX Tool
Simple rating scale using 6 dimensions and overall rating
“Structured” approach to direct observationDirect assessment of actual patient careIncorporation of CEX into daily activitiesEvidenced-based
![Page 26: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/26.jpg)
Research: Mini - CEX ToolResearch: Mini - CEX Tool
Two large scale U.S. studies involving 36 total residency programs
Logistically feasible to incorporate miniCEX into daily activities
High satisfaction among house staff Good to excellent reliability characteristics Overall scores and interpersonal scores
correlated with trainee’s ECFMG OSCE scores
![Page 27: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/27.jpg)
Logistics: Outpatient ClinicLogistics: Outpatient Clinic
One mini-CEX per trainee per day per week– One attending observes portion of first visit
of the day– Minimizes disruption of clinic– Perform over course of academic year– Easy to obtain 6-8 Mini-CEX’s per year per
trainee in single setting
![Page 28: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/28.jpg)
The Patient EncounterThe Patient Encounter
Sampling “parts” of the encounter:
INTERVIEWPHYSICAL
EXAMCOUNSELING
![Page 29: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/29.jpg)
Solutions: Step 2Solutions: Step 2
Improve reliability– Multiple brief observations– Perform over time: outpatient setting allows
for longitudinal observation– Involve multiple faculty– MiniCEX: sufficient reliability for pass/fail
determinations after just 4 observations
![Page 30: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/30.jpg)
Solutions: Step 3Solutions: Step 3
Improve accuracy and validity
– Most difficult step
– Use structured rating forms
– Rater training (faculty development)
• Caveat: brief “one time” interventions do not work
![Page 31: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/31.jpg)
Does Faculty Training Work?Does Faculty Training Work?
Performance Appraisal Literature:Can reduce rating errorsCan improve discriminative abilityCan improve accuracy
![Page 32: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/32.jpg)
Approaches to Rater TrainingApproaches to Rater Training
Behavioral Observation TrainingPerformance Dimension TrainingFrame of Reference TrainingDirect Observation of Competence
Training
![Page 33: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/33.jpg)
Videotape Exercise: BOTVideotape Exercise: BOT
Situation: An attending is performing a miniCEX of a house officer performing a physical exam.
Questions:– How well did this attending evaluate the
house officer?– How was the house officer-patient
interaction affected?
![Page 34: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/34.jpg)
Basic Faculty Observation SkillsBasic Faculty Observation Skills
Prepare for the observation– Faculty: Know what you’re looking for
– Resident: Let them know what to expect
– Patient: Let them know why you are there
Minimize intrusiveness – correct positioning Minimize interference with the house officer-
patient interaction Avoid distractions
![Page 35: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/35.jpg)
TriangulationTriangulation
DESK
Resident
Patient
Attending
![Page 36: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/36.jpg)
Basic Observation StrategiesBasic Observation StrategiesIncrease the amount of “sampling”
– More observations lead to more accurate evaluations (“practice makes perfect”)
Use of observational “aides”
– Behavioral diary to record observed performance.
– U.S. study: simple 3X5 card diary lead to increased comments on forms
![Page 37: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/37.jpg)
Performance Dimension TrainingPerformance Dimension Training
Group exercises designed to familiarize faculty with the specific elements of a competency
Should involve discussion of the criteria required for each element
Use defined, agreed upon elements of a competency to calibrate faculty– Playing from the “same sheet of music”
![Page 38: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/38.jpg)
PDT ExercisePDT Exercise
In your small group, discuss what should be the basic components of an effective medical interview for a foundation trainee performing an outpatient consultation
![Page 39: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/39.jpg)
Frame of Reference TrainingFrame of Reference Training
Goal is to improve “judgment” and accuracySteps in FOR training:1. Group performance dimension training (PDT)
exercise
2. Review clinical vignettes that describe critical incidents of performance: unsatisfactory to average to superior
![Page 40: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/40.jpg)
Frame of Reference TrainingFrame of Reference Training
3. Faculty, using framework developed in PDT exercise, provide ratings on a behaviorally anchored rating scale (BARS)
4. Session trainer provides feedback on what “true” ratings should be for each vignette along with rationale
5. Group finishes by discussing discrepancies between trainer’s ratings and the participants’ ratings
![Page 41: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/41.jpg)
Frame of Reference TrainingFrame of Reference Training
Most difficult aspect of FOR:
– Setting the actual standards that distinguish between levels of performance
– Reaching agreement and/or consensus among teaching faculty
![Page 42: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/42.jpg)
DOC TrainingDOC Training
Combination of:– Behavioral observation training– Performance dimension training– Frame of reference training– “Live” practice in observation with
standardized residents/patients• Individual evaluation and feedback• Group debrief with Eval and FB
![Page 43: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/43.jpg)
DOC Training TrialDOC Training Trial
Randomized controlled trial of 40 faculty from 16 residency programs
DOC training:– High satisfaction (favorite aspect of course)– Increased comfort in observation– Changed rating behavior at 8 months– Increased accuracy in identifying
unsatisfactory performance
![Page 44: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/44.jpg)
Direct Observation: ChallengesDirect Observation: Challenges
Like all skills, requires training and practice
Faculty “calibration” important– Agreeing on “metrics” of performance– Faculty comfort with own skills
Faculty training– Brief interventions mostly ineffective
![Page 45: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/45.jpg)
Observation: Helpful HintsObservation: Helpful Hints
Sample “parts” of the visit:– History-taking– Physical examination– Counseling
Perform longitudinally– No need to do it all at once
Agree on performance metrics with faculty
![Page 46: The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program](https://reader036.vdocument.in/reader036/viewer/2022062717/56649e415503460f94b32947/html5/thumbnails/46.jpg)
SummarySummary
Basic clinical skills are important: so is the need to observe them!
Observation is a complex skill that requires training and practice
Direct observation by educators will remain a critical component of both evaluation and feedback