2005 clear usmle cs hawkins handout

“USMLE Step 2 Clinical Skills”

Richard E. Hawkins, MDNational Board of Medical Examiners

Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona


Presentation Structure

• Overview of USMLE Step 2 CS

• Measures to ensure fairness and consistency


Overview of USMLE Step 2 CS

Purpose and Delivery Model


USMLE: Purpose• Three step examination system primarily

designed to support the medical licensing process

• Jointly sponsored by

– Federation of State Medical Boards (FSMB)

– National Board of Medical Examiners (NBME)

• Step 2 CS: collaboration with ECFMG


USMLE: Purpose

• Overall purpose

– Assess physician's ability to apply knowledge, concepts, and principles, and to demonstrate fundamental patient-centered skills that constitute the basis of safe and effective patient care


USMLE: Purpose

• Patient-centered Skills– History taking / Physical examination– Communication and Interpersonal Skills– Medical Record Documentation

• Important for safe and effective patient care– History/PE Diagnosis and Management– Communication Patient Health Outcomes– Medical Record Errors and Patient Safety


USMLE Step 2 CS

• Enhancement to USMLE Step 2

• USMLE Step 2 – components:

– Clinical knowledge (CK)

– Clinical skills (CS)

• Standard for Step 2 – appropriate for entry into postgraduate training


USMLE Step 2 CS: Logistics• Regional delivery model

– Optimal combination of convenience, cost-efficiency and standardization

• Five regional test centers across US• Projected examine volume – 30,000+ / year• Individual center capacity

– 3 examinations / day (33 examinees); up to 7 days/week


Requirements of a High StakesPerformance-based

ExaminationEnsuring Fairness and

Consistency


Threats to Validity and Reliability• Content and/or tasks not relevant or realistic• Individual test forms vary in content

coverage• Scoring methods not appropriate for skills

tested• Inconsistency in SP portrayal and scoring

– Between cases, across sites, over time• Level of difficulty of cases / exams

inconsistent• Standard setting approach appropriate


USMLE Step 2 CS: Content• Step 2 CS Blueprint

– Defines content categories

– In meeting blueprint specifications, each test form provides:

• Adequate sampling of content domain

• Comparable content between test forms


USMLE Step 2 CS: Content

Blueprint Content Categories:– Common and important medical

problems / patient presentations– Acuity– Age– Gender– Race / ethnicity


Case Content

CardiovascularRespiratory GastrointestinalMusculoskeletalConstitutional

Neurological Psychiatric Genitourinary Women’s health

Unclassified / multi-system

Test Form

Patient age

Age less than 18Age 18 – 44Age 45 – 64Age 65 +

Patient Gender

Male Female

Case Acuity

Acute Subacute / Chronic


Step 2 CS: Structure & Tasks• 12 patient encounters• 15 min. for encounter / 10 min. for patient note• Each encounter:

– Elicit pertinent history, Perform appropriate physical examination, Communicate effectively – Document:

• Findings from the history and physical • Diagnostic impression / Further work-up


Step 2 CS: Score Components• Integrated Clinical Encounter (ICE)

– Data gathering: history and physical exam – Patient note

• Communication / Interpersonal Skills (CIS)– Gathering information; sharing information;

manner & rapport• Spoken English Proficiency (SEP)

– Listener effort, examinee pronunciation / word choice


Scoring Approach: ICE

• Data gathering (Hx / PE)– Dichotomous checklists completed by SPs

+• Patient note

– Physician raters using holistic methods=

• Clinical Process + Clinical Outcome Measure


Case and Checklist Development• Iterative Process

– Involves test committees– Focus on clinical presentation – Checklists “limited”– Encounters with SPs– Appropriateness of content and

difficulty• Stepwise progression through

pilot and calibration stages– Validation / refinement via

review of examinee performance data


Patient Note

• Holistic scoring• Rater training

– General policies– Case specific:

• Consensus development on key features

– Calibration phase– Quality Assurance


Derivation of Communication / Interpersonal Skills (CIS) Scale

• Initial instrument – validated ECFMG scale

• Reviewed against national consensus recommendations and commonly used scales


CIS Subscales• Data gathering skills

– Open-ended questions, transitional statements, not interrupting the patient

• Information sharing skills– Responsiveness to patient questions/concerns,

provision of counseling when appropriate, avoidance of jargon

• Personal manner and rapport– Expression of interest in the impact of the illness,

concern for patient comfort and modesty


Step 2 CS: Scoring

• Reporting is Pass or Fail only• Examinees must pass all three

subcomponents • Feedback to examinees

– Performance report – overall and subcomponent outcomes

– Failing examinees only – graphical profiles• Intended to show relative strengths

and weaknesses


USMLE Step 2 Clinical Skills

P/F Outcome

Total Test FAIL-----------------------------------------------------------------------------------------

Examination Subcomponents

Integrated Clinical Encounter Pass

Communication/Interpersonal Skills Fail

Spoken English Proficiency Pass

Sample Performance Report


Sample Performance Profile

Integrated Clinical Encounter

Data Gathering

Patient Note

Communication / Interpersonal Skiills

Spoken English Proficiency

Lower Performance

BorderlinePerformance

Higher Performance

XXXXXXXXXXXXX

XXXXXXXXXXXXXXXX

XXXXXXXXXXXXXX

XXXXXXXXXXXXX

XXXXXXXXXXXXXXXX


Threats to Validity and Reliability• Content and/or tasks not relevant or realistic• Individual test forms vary in content

coverage• Scoring methods not appropriate for skills

tested• Inconsistency in SP portrayal and scoring

– Between cases, across sites, over time• Level of difficulty of cases / exams

inconsistent• Standard setting approach appropriate


Ensuring Fairness and Consistency

• Rigorous SP (and SP trainer) training• Meticulous attention to quality assurance• Application of equating procedures


SP and SP Trainer TrainingStandardized Patients • Rigorous training –

generic and case specific

• Electronic delivery (“Ecase”) of case materials

• Sign off process for SPs – Criteria : # portrayals, tests

SP Trainers• Adherence to training

protocols• “Training academy” for

SP trainers


Quality Assurance

• Begins with the SP sign-off process

• Monitoring procedures and analyses:– Qualitative (portrayal and scoring

accuracy):• Live and video review of SP

performances – Quantitative

• Score-based analyses• Case level and item level comparisons


Equating Procedures

• Within site– SP-case combination

• Between site – Central video review

Data gathering Communication and Interpersonal Skills

• Patient note rater – case combination


Standard Setting• USMLE Standard Setting System

– Committee assigned responsibility to establish and monitor standards

• Decision-making process– Survey of constituent opinion – Standard-setting exercises by

independent groups – Examinee performance data and score

reliability


Conclusions• Numerous logistical and psychometric

challenges were identified and responses described

• In order to ensure exam fairness and consistency:– Sound test and case development practices– Intensive SP and SP trainer training– Rigorous quality assurance– Well-considered equating and standard setting

procedures• Supported by ongoing research on reliability

and validity


Speaker Contact Information

Richard E. Hawkins, MDNational Board of Medical Examiners3750 Market StreetPhiladelphia, PA 19104215-590-9204 / Fax 215-590-9440Email: [email protected]:

http://www.usmle.orghttp://www.nbme.org

mailto:[email protected]

http://www.usmle.org/

http://www.nbme.org/

http://www.nbme.org/

2005 clear usmle cs hawkins handout

Documents