assessing knowledge and performance john littlefield, phd university of texas at san antonio
TRANSCRIPT
![Page 1: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/1.jpg)
Assessing Knowledge and
Performance
John Littlefield, PhD
University of Texas
at San Antonio
![Page 2: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/2.jpg)
Goals: Assessment of Knowledge and Performance
1. Clarify 2 distinct uses for assessments of knowledge and performance
2. Define 2 aspects of validity for all knowledge and performance assessment methods
3. Compare and contrast 3 techniques for assessing clinical knowledge and performance
4. Identify poorly written multiple choice test items and write a key features test item
5. Describe 3 options for scoring OSCE performance
6. Describe 3 elements of a clinical performance assessment system
7. Critique a clinical performance assessment system that you use
8. Recognize legal issues related to evaluating and dismissing students and residents
![Page 3: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/3.jpg)
Agenda: Assessment of Knowledge and Performance
1. Exercise: Warm-up for assessing clinical knowledge and performance
2. Presentation: Quality assurance when assessing clinical knowledge and performance
3. Exercise: Take then critique a multiple choice test4. Presentation: Key features test items5. Exercise: Write several key features test items6. Presentation: Widening the lens on SP assessment7. Exercise: Strengths & weaknesses of a clinical
performance assessment system that you use
8. Presentation: Improving clinical performance assessment systems
9. Exercise: Critique your clinical performance assessment system
10. Discussion: Legal context for evaluations and dismissals
![Page 4: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/4.jpg)
Recall a student/resident whose clinical performance made you uneasy
1. What action did you take?a. Talk with faculty colleagues
about your concerns: Yes Nob. Write a candid performance
assessment and send it to clerkship/residency director: Yes No
2. Did any administrative action occur related to your concern? Yes No
3. Do you think the performance assessments in your clerkship/residency files reflect facultys’ candid performance appraisals? Yes No
4. Why do you think this is true?
![Page 5: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/5.jpg)
Making Evaluation Decisions
Student/Res.
IS
Competent
Student/Res.
IS Not
Competent
Pass
(promote, graduate, etc.)
Correct
Decision
False
Positive
Error
Fail
(remediate, dismiss, etc.)
False
Negative
Error
Correct
Decision
Complete Knowledge (God)
Pa
rtia
l Kn
ow
led
ge
(U
s)
![Page 6: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/6.jpg)
What concerns do you have about clinical knowledge and performance assessment?
![Page 7: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/7.jpg)
Uses for Assessment: Formative vs. Summative
Purpose Feedback for Certification/
Learning Grading
Breadth of Narrow Focus on Broad Focus on
Scope Specific Objectives General Goals
Scoring Explicit Feedback Overall
Performance
Learner Affective Little Anxiety Moderate to High
Response Anxiety
Target Audience Learner Society
![Page 8: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/8.jpg)
Validity of Knowledge and Performance Assessments
1. Content - Does the assessment method measure a representative cross-section of student/resident competencies?
2. Internal structure – Do content and scoring focus on a specific clinical competency (e.g., patient care)?
3. Relation to other assessments - Do scores from this assessment correlate highly with other measures of same student competency?
4. Consequences - Do various subgroups of students (e.g., different ethnic groups) score equally well on the assessment?
5. Generalizability– Does the student perform at about the same level across 5 to 7
different patients / case problems?– Does the student receive a similar rating from different faculty?
6. Cognitive process – the context surrounding the assessment evokes the domain of cognitive processing used by a physician
1. Standards for Educ. & Psych. Testing, AERA, APA, NCME, 1999, p 11-16.
![Page 9: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/9.jpg)
Generalizability
Content
Cognitive process
ConsequencesRelation to other assessments
Internal structure
Six Aspects of Assessment Validity Viewed as a Cube
![Page 10: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/10.jpg)
Generalizability of Physician Performance Scores on Multiple Patients
0
20
40
60
80
100
120
Patient 1 Patient 2 Patient 3 Patient 4 Patient 5
IdealActual
![Page 11: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/11.jpg)
Validity of Knowledge and Performance Assessments
Content - Does the assessment method measure a representative cross-section of student/resident competencies?
Internal structure – Do content and scoring focus on a specific clinical competency (e.g., patient care)?
Relation to other assessments - Do scores from this assessment correlate highly with other measures of same student competency?
Consequences - Do various subgroups of students (e.g., different ethnic groups) score equally well on the assessment?
Generalizability– Does the student perform at about the same level across 5 to 7
different patients / case problems?– Does the student receive a similar rating from different faculty?
Cognitive process – the context surrounding the assessment evokes the domain of cognitive processing used by a physician
1. Standards for Educ. & Psych. Testing, AERA, APA, NCME, 1999, p 11-16.
![Page 12: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/12.jpg)
Cognitive Process Aspect of Validity: Four Levels of Performance Assessment 1
Knows
(Examination – Multiple-choice)
Knows How
(Examination – Oral)
Shows How
( OSCE)
Does
(Global Rating)
1. Miller, GE. Assessment of clinical skills/competence/performance, Academic Medicine, 65(9), supplement, 1990, S63-7
![Page 13: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/13.jpg)
Compare and Contrast Three Assessment Techniques(Multiple choice exam, OSCE, Global ratings)
M.C.E. OSCE Global rtgs.
Content +++ ++ + Internal structure +++ ++ + Rel. to other assessments + + + Consequences + ++ + Generalizability
– 5 to 7 case problems +++ ++ +– agreement among raters +++ ++ +
Cognitive process + ++ +++
+ = adequate ++ = good +++ = excellent
![Page 14: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/14.jpg)
Interim Summary of Session
Session thus far– Two uses of knowledge and performance
assessments: Formative and Summative– Validity of all knowledge and
performance assessment techniques– Compare and contrast 3 assessment
techniques Coming up
– Take and critique a 14 item multiple choice exam
– Presentation on Key Features items
![Page 15: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/15.jpg)
How are Multiple Choice Items Selected for an Exam?
![Page 16: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/16.jpg)
Sample Exam Blueprint based on Clinical Problems
Patient Life Span % Total Patients
% Men/Women Number of Problems
Pregnancy/Infant 5 50/50 4
Pediatrics 16 53/47 6
Adolescence 16 31/69 6
Adults 47 34/66 18
Geriatrics 16 45/46 6
Total 100 39/61 40
Page G, Bordage G, Allen T. Developing key-feature problems and examinations to assess clinical decision-making skills, Acad. Med. 70(3), 1995.
![Page 17: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/17.jpg)
Key Features of a Clinical Problem 1
Definition: Critical steps that must be taken to identify and manage a patient’s problem– focuses on a step in which examinees are likely to make an
error
– is a difficult aspect in identifying and managing the problem Example: For a pregnant woman experiencing third-
trimester bleeding with no abdominal pain, the physician should:– generate placenta previa as the leading diagnosis
– avoid performing a pelvic examination (may cause bleeding)
– avoid discharging from clinic or emergency room
– order coagulation tests and cross-match
1. Page G, Bordage G, Allen T. Developing key-feature problems and examinations to assess clinical decision-making skills, Acad. Med. 70(3), 1995.
![Page 18: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/18.jpg)
Test Items based on a Clinical Problem and its Key Features
Jennifer, a 24-year-old woman, G3, P2, 31 weeks pregnant, comes to the emergency room at 8:00 PM complaining of bright red vaginal bleeding for the past two hours. The three sanitary napkins that she used were completely soaked. Her pregnancy has been uneventful, as were the previous ones. She has not had any contractions or abdominal pain. The fetus is moving as usual.
Her BP is 110/70 mm Hg, and her pulse is 92/min. The examination of the abdomen reveals a uterine height of 31 cm. with a soft and nontender uterus. The fetus is in a breech position and has a heart rate of 150/min. No active bleeding has occurred since she arrived 25 minutes ago.
1. What is your leading diagnosis at this time? List only one diagnosis. Write “normal” if you judge Jennifer’s situation to be within normal limits.
2. What steps would you include in your immediate assessment and management of this patient? List as many as are appropriate.
![Page 19: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/19.jpg)
Scoring the Placenta Previa Clinical Problem
Key Feature 1: To receive one point, must list placenta previa or one of the following synonyms: marginal placenta or low placental insertion
Key Features 2-4: Receive 1/3 point for listing each of the following: 1. Avoid performing a pelvic exam, 2. Avoid discharging from clinic, 3. Order coagulation tests and cross match
Total Score for Problem: Add scores for items 1 and 2 and divide by 2 (range: 0 - 1)
![Page 20: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/20.jpg)
Steps to Develop a Clinical-Problem Based Exam
1. Define the domain of clinical problems to be sampled by the exam
2. Develop an exam blueprint to guide selection of clinical problems
3. Develop a key-feature problem for each clinical problem selected– define clinical situation for the problem (e.g. single typical
problem, life-threatening situation etc.)– define key features of the problem– select a clinical case to represent the problem and write
scenario– write exam items for case; in general one item for each key
feature– select suitable format for each item (e.g., write-in or mcq)– develop scoring key for each item– pilot test items for item analysis data to guide refinement
![Page 21: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/21.jpg)
Interim Summary of Session
Session thus far– Two uses of knowledge and
performance assessments: Formative and Summative
– Validity of all assessment techniques
– Compare and contrast three assessment techniques
– Take and critique a 14 item multiple choice exam
– Write a Key Features item Coming up
– Scoring performance on an SP exam
![Page 22: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/22.jpg)
Schematic Diagram of a 9 Station OSCE
Start
End
1 2 3 4
5
678 9
![Page 23: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/23.jpg)
Scoring OSCE Performance
Traditional scoring of SP assessment focuses on numerical data typically from checklists
Checklist scoring may not accurately assess clinical performance quality of residents and expert clinicians 1
Dimensions of the SP exam 2
– basic science knowledge (organize the information)– physical exam skills (memory of routines)– establishing a human connection– role of the student (appear knowledgeable)– existential dimension of the human encounter (balance one’s
own beliefs with the patient’s) Clinical competence – mixture of knowledge and feeling,
information processing and intuition
1. Hodges et. al. OSCE checklists do not capture increasing levels of expertise, Acad. Med. 74(10), 1999, 1129-34. 2. Rose & Wilkerson. Widening the lens on SP assessment: What the encounter can reveal about development of clinical competence, Acad. Med. 76(8), 2001, 856-59.
![Page 24: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/24.jpg)
Interim Summary of Session
Session thus far– Two uses of knowledge and performance
assessments: Formative and Summative
– Validity of all assessment techniques
– Compare and contrast three assessment techniques
– Take and critique a 14 item multiple choice exam
– Write a Key Features test item
– Use global ratings and narrative comments when scoring OSCE performance
Coming up– Improving clinical performance assessment
systems
![Page 25: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/25.jpg)
Bubble Diagram of a Resident Performance Assessment System
![Page 26: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/26.jpg)
Diagnostic Checklist for Clinical Performance Assessment System
Identify Problems
Recommend Improvements
A. Organizational Infrastructure B. Individual Evaluators’ Role 1. Communicate Expectations 2. Observe Performance 3. Interpret and Judge Performance 4. Communicate Performance Info 5. Coach Resident 6. Complete PA Form C. Program Director’s Role 1. Monitor & Interpret Appraisals 2. Committee Decision 3. Formally Inform Resident
![Page 27: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/27.jpg)
Three Year Study to Improve the Quality of Resident Performance Assessment Data
1. What median percentage of each resident’s rotations returned one or more completed forms?
2. How precise were the scores marked on the returned forms?
3. What median percentage of each resident’s rotations returned one or more forms with behaviorally-specific
written comments?
Littlefield, DaRosa, Paukert et. al. Improving Resident Performance Assessment Data: Numeric Precision & Narrative Specificity, Acad. Med. 80(5), 489-95, 2005
![Page 28: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/28.jpg)
Results of the Study
Research Question Baseline Year
Intervene Year 1
Intervene Year 2
Stat Sig.
% Rot. Returned Form
Site A
Site B
M = 71%
M = 75%
M = 100%
M = 100%
M = 96%
M = 100%
Χ²
<.01
<.01
Gen. Coef. (95% Cnf Int)
Site A
Site B
.65(± .78)
.58(± .55)
.85 (± .58)
.79 (± .55)
.84 (± .61)
.77 (± .56)
N/A
N/A
% Rot. Beh. Spec. Cmnt
Site A
Site B
M = 50%
M = 50%
M = 57%
M = 60%
M = 82%
M = 67%
Χ²
<.01
<.01
![Page 29: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/29.jpg)
Legal Issues Related to Evaluation and Dismissal 1
Fair and equitable treatment Due process for academic
dismissals Documentation, libel, and
defamation
1. Irby & Milam. The legal context for evaluating and dismissing medical students and residents. Acad Med, 64(11), 1989, 639-43
![Page 30: Assessing Knowledge and Performance John Littlefield, PhD University of Texas at San Antonio](https://reader030.vdocument.in/reader030/viewer/2022032722/56649f4e5503460f94c70301/html5/thumbnails/30.jpg)
Goals: Assessment of Knowledge & Performance
1. Clarify 2 distinct uses for assessments of knowledge and performance
2. Define 2 aspects of validity for all knowledge and assessment methods
3. Compare and contrast 3 techniques for assessing clinical knowledge and performance
4. Identify poorly written multiple choice test items and write a key features test item
5. Describe 3 options for scoring OSCE performance
6. Describe three elements of a clinical performance assessment system
7. Critique a clinical performance assessment system that you use
8. Recognize legal issues related to evaluating and dismissing students and residents