Impact of Case-Vignette Impact of Case-Vignette Test Items on National Board Test Items on National Board

ScoresScores

Michael Oglesby, Ph.D.Michael Oglesby, Ph.D.

UNTHSC-FWUNTHSC-FW

Assessing Clinical CompetenceAssessing Clinical Competence

Until recently, Licensing Board examinations assessed acquisition of medical knowledge (primarily via recall of isolated facts, definitions, etc)

However, Licensing Boards have been under increasing pressure to produce examinations that:

1. Assess clinically relevant concepts and abilities

2. Document competence

Assessing Clinical CompetenceAssessing Clinical Competence

In response to that challenge, Licensing Boards adopted changes in the design of test item formats – most notably the introduction of case-vignette-based test questions, which now serve as the stem for the vast majority of licensure test items.

Assessing Clinical CompetenceAssessing Clinical Competence

Both the NBME and NBOME intend to increase the number of test items beginning with a case vignette to approximately 80% – 90% (a dramatic increase from an estimated 30% just a few years ago)

AAMC and Step 2AAMC and Step 2Impact of Complicated VignettesImpact of Complicated Vignettes

• Case et al. Verbosity, window dressing, and red herrings: do they make a better test item? Acad. Med. 71: 528-530, 1996.

• 3 variants of vignette formatting studied• Point biserial analysis

• High/Low analysis (upper & lower 20% of exam takers)

First-Order Recall First-Order Recall (Case (Case et alet al., 1996)., 1996)

The most likely renal abnormality in children with nephrotic syndrome and normal renal function is

A. Acute poststreptococcal glomerulonephritis

B. Hemolytic-uremic syndrome

C. Minimal change nephrotic syndrome

D. Nephrotic syndrome due to focal and segmental glomerulosclerosis

E. Shönlein-Henoch purpura with nephritis

A B C D E

High 1 0 99 0 0

Low 8 1 90 1 0

Focused Vignette Focused Vignette (Case (Case et alet al., 1996)., 1996)

A 2-year-old boy has a 1-week history of edema. His blood pressure is 100/60 mm Hg, and there is a generalized edema and ascites. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood. The most likely diagnosis is (same option list).

A B C D E

High 0 0 98 2 0

Low 5 2 82 8 0

Extraneous Information Extraneous Information (Case (Case et alet al., 1996)., 1996)

A 2-year-old African American child developed swelling of his eyes and ankles over the past week. Blood pressure is 100/60 mm Hg, pulse 110/min, and respirations 28/min. In addition to swelling of his eyes and 2+ pitting edema of his ankles, he has abdominal distention with a positive fluid wave. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood.

(Same lead-in and option list followed)

A B C D E

High 0 1 98 1 0

Low 10 9 66 10 5

Summary of Case et al et al.: .: Impact of VignettesImpact of Vignettes

A B C D E

High 0 1 98 1 0

Low 10 9 66 10 5

A B C D E

High 0 0 98 2 0

Low 5 2 82 8 0

A B C D E

High 1 0 99 0 0

Low 8 1 90 1 0

No Vignette

Brief Vignette

Full Case Vignette

Addition of Noise Complicates Addition of Noise Complicates Pattern RecognitionPattern Recognition

Assessing Knowledge-Base Assessing Knowledge-Base CompetenceCompetence

1. Licensing Board examinations now utilize case vignettes extensively

2. Case vignettes are designed to assess comprehension and application skills - these are higher level cognitive skills than information acquisition (previously referred to as recall or memorization capabilities)

3. Case Vignettes are frequently “Second-Order” questions, i.e. they assess higher level cognitive skills (comprehension and application capabilities)

22ndnd Order Questions Order Questions

• 2nd Order Questions have two components.1. A patient case-vignette that first requires a

diagnosis2. Then, a basic science (Level 1 Board) or

management (Level 2 Board) component of the question that can only be answered if a. A correct diagnosis is made initially, andb. The candidate also knows the additional

basic or clinical sciences concepts relevant to the disease represented by the case vignette

Typical 2Typical 2ndnd Order Question Order QuestionA 53-year-old male reports recurrent chest pain, typically of 10-20 minutes duration. He was not physically exerting during onset of the pain; indeed, the attacks most often started at near morning while sleeping. History is otherwise unremarkable. ECG reveals elevated ST segments during these attacks. If pharmacologic prophylaxis is ultimately adopted for this condition, which agent would be preferred for chronic treatment?

A. AtenololB. HydralazineC. IsradipineD. NitroglycerinE. Propranolol

Typical 2Typical 2ndnd Order Question Order Question

• This question requires that the candidate• Recognize the diagnosis (Prinzmetal’s Angina)• Recall that calcium channel blockers are appropriate

prophylactic therapy for this condition (and select the correct medication from the list)

• TCOM students in 2005 estimated that more than 40% of COMLEX Level 1 questions were 2nd-order (this number will increase dramatically in the near future)

22ndnd Order Questions Order Questions

• Probability of getting this item correct is a function of two knowledge-bases

1. Diagnostic knowledge

2. Treatment (pharmacology) knowledge

• In a well-written 2nd order question, if an incorrect diagnosis is made, the probability of getting the question correct is at best chance

22ndnd Order Questions Order QuestionsSuppose a candidate knows both diagnosis and treatment with 0.8 probability. Because these probabilities are independent, the probability of getting the question correct is: 0.8* 0.8 = 0.64The following graph assumes equal capability for each of these variables (diagnostic knowledge and scientific knowledge)

0.9 * 0.9 = 0.720.8 * 0.8 = 0.640.7 * 0.7 = 0.49etc

Impact of 2Impact of 2ndnd Order Questions Order Questions

Science Knowledge Base, 1st-Order Question

= 80% + 10%

DDX Knowledge Base

X

Science Knowledge Base

Assuming Science and DDX Knowledge Bases are Equal Probability

0

5

10

15

20

20 30 40 50 60 70 80 90 99Student Percentile Score

% of Students

0

5

10

15

20

20 30 40 50 60 70 80 90 99

% of Students

Student Percentile Score

Impact of 2Impact of 2ndnd Order Questions Order Questions

• Flattens the distribution

• Broadens the distribution• Generates much lower scores for the low end

of test takers

Impact of 2Impact of 2ndnd Order Questions Order Questions

• Note what happens if DDX improves

DDX Science Score

0.8 0.8 = 0.64

0.9 0.8 = 0.72

Impact of 2Impact of 2ndnd Order Questions Order Questions• Students must be able to recognize (diagnose)

the disease represented by the case vignette to answer the question associated with the case vignette

• The better the DDX capabilities, the better the score on Boards; the less skilled in DDX, the poorer the score on Boards

COMLEX Level 1 AveragesCOMLEX Level 1 AveragesTCOM ’01-’04TCOM ’01-’04

Science DDX1 2

0

20

40

60

80

Percentile Score

Curricular Change for Class Taking Curricular Change for Class Taking COMLEX in 2005COMLEX in 2005

• More emphasis on case-based teaching

• Active training of DDX• Students were required to diagnose the

common causes of patient problems utilizing a computer-based program (KBIT) that presented cases and corrected diagnostic errors

COMLEX Level 1 AveragesCOMLEX Level 1 AveragesTCOM ’01-’04 vs ‘05TCOM ’01-’04 vs ‘05

1 20

20

40

60

80

Percentile Score

Science DDX