Establishing Content Validity of a Novel Written Examination to Assess Medical

Students on the Surgery ClerkshipA Berlin MD, A Reinert BA, A Swan-Sein MEd PhD,

R Nowygrod MD, A Fingeret MD

Agenda

• Needs assessment• Exam Development• Exam Design• Methodology• Results• Discussion

Needs Assessment• Historical assessment of students– Clinical evaluations– Unstructured oral exam – NBME surgery subject exam

• Goals of new evaluation tool– Objective and reliable– Congruent with educational objectives– Based on ACGME core competencies

Exam Design

• Novel Exam: Clinical Skills Exam (CSE)– 20+ clinical scenarios• 5 randomly selected per exam

– Short answer or essay response– Administered over 3 hours– ExamSoft computer software

• Grading – Rubric– Senior residents

Example 1

• Identify operation

• Obtain surgical consent

• Determine order

• Describe steps shown

Example 2

• Generate differential based on history

• Describe risk factors• Interpret radiographic

study• Identify next step in

management

Methodology• July, 2011 - February, 2013– n = 262

• Analysis– Descriptive statistics– Multivariate analysis

• Comparison data for validity assessment – USMLE Step 1 and 2CK– NBME Subject Exams in Medicine and Surgery– Clerkship clinical evaluations – Clerkship order

Results: Descriptive Statistics – CSE Content and Score Distribution

Results: Trend Analysis – CSE and NBME Surgery Subject Exam

Results: Multivariate Statistics - Validity Analysis

CSE NBME

CorrelationLinear

Regression CorrelationLinear

Regressionr p value r p value

NBME Surgery 0.36* 0.82 NBME Medicine 0.01 0.54 0.08 0.20Step 1 0.34* 0.82 0.55* 0.02Step 2 0.44* 0.007 0.56* 0.004Clerk - Clinical Reasoning 0.48* 0.001 0.43* 0.09

Clerk - Fund of Knowledge 0.56* 0.004 0.63* < 0.001

* = p < 0.001

Results: Multivariate Statistics – Validity Analysis

CSE NBMEBivariate Analysis

Bivariate Analysis

Logistic Regression

p value p value p value

Clerkship order ANOVA 0.30 < 0.001 0.01

OB/Gyn prior X2 0.34 < 0.001 0.02

Medicine prior X2 0.69 < 0.001 0.03

Results: Student Feedback on CSE

Results: Student Feedback - Word Cloud

Limitations

–Study•Generalizability• Statistical analysis

Future Directions

• Future directions– New content– Open access– Computer algorithm based grading– Use as formative assessment mid-clerkship

Conclusions

• CSE is a useful adjunct to NBME surgery subject exam– Independent of clerkship order– Course goals and objectives– ACGME core competencies

•Questions / Comments