running an exam from beginning to end - university of bristol

Running an Exam From Beginning to End

Andrew Blythe

Autumn 2011

BLUEPRINTING

What are we assessing?

“The outcomes in Tomorrows Doctors (2009) provide a structure against which everything else can be planned & evaluated. Some outcomes cover more ground than others so will require more extensive testing.”

Paragraph 29, Assessment in Undergraduate Medical Education – Advice

Supplementary to Tomorrow’s Doctors (2009)

Hierarchy of Learning Outcomes

Unit Intended Learning Outcome Approved by MEC & Programme Approval

MB ChB Programme Specification University of Bristol Under revision

GMC 16 high level outcomes 32 practical procedures

Learning Outcomes for Preparing for Professional Practice (Year 5)

Examples Tomorrows

Doctors Brief description topic/learning objective

Method of Assessment

14a Practitioner

Synthesises history, examination, mental state & patient factors

Formative & summative long cases Portfolios Case-based discussion within tutorial

17a 17b Practitioner

Takes a drug history including OTC medications & complementary therapies & plan appropriate therapy for common indications, including pain & distress

Examples of appropriate drug histories in clerking portfolio with copies of original drug charts Student script countersigned Oncology/palliative care prescribing tutorial signed off

Community Orientated Medical Practice 2 (COMP2)

Blueprint for Primary Care

ROLE OF EXTERNAL EXAMINER

External Examiner from Report Received

Human Basis of Medicine Oxford Yes

Molecular & Cellular Basis of Med

Cardiff, Kings College + 2 others

Yes (2 out of 4)

Systems of the Body 1 & 2 Kings College Yes

Junior Medicine & Surgery

Cardiff + 1 other Yes (1 out of 2)

Psychiatry & Ethics Southampton & Hull-York Yes (both)

MDEMO 1 No

COMP1 Highlands + 2 Yes (2 out of 3)

Applied Clinical Sciences 2 No

Reproductive Health Care Swansea & Penninsula Yes (both)

COMP2 Kings College & Sheffield Yes (both)

Senior Surgery Oxford No

Oncology & Palliative Care No one in place

Senior Medicine Birmingham Yes

SSCs (years 3 & 5) 1 No (2nd year running)

University Guidelines

“The role of the external examiner is not confined to consideration of examination results and attendance at examination boards”.

“External examiners may, and are encouraged, to comment and advise on the content, balance and structure of the programmes and units, the development and review of programmes and/or units and on assessment processes.”

The External Examiner’s Annual Cycle

Review content & assessment of unit. Help set priorities for coming year

Comment on proposed

examination questions

Contribute to standard setting

exercise

Attend examination & scrutinize

student performance

Submit report

Report viewed by Unit Assessment Group APR Response sent to External Examiner

Planning Changes

Reviewing Questions

Standard Setting

Attends Exam

Reviews marks of failing students

HBoM

MCBoM

Systems 1

Systems 2

Junior Med & Surg OSCE

Psych & Ethics Presentations

MDEMO

COMP1 Prize viva

ACS

RHCN SSC Pres’n

COMP2 OSCEs & SSC presentation

Senior Surgery

Senior Medicine OSCE & resit long cases

Problems

Long delays in

• getting some reports

• reviewing & responding to some reports

Not involving external examiners enough in

• reviewing some units

• looking at performance of students

Examination Format: written

• Replace essays with more short answers

• Replace true/false with best-of-5

• Standardise Extended Matching Questions

• Base majority of questions on clinical topics

Examination Format: clinical

• More OSCE stations where student has to examine the patient/actor

• Standardise assessments

• Set standard for portfolios

Communication with External Examiner

• Discuss course details with him/her

• Discuss student performance with him/her

• Inform him/her of student progress

Other comments

• Concern about method of scaling marks

• Some unit leads need more support

TRAINING EXAMINERS

Doves & Hawks

STANDARD SETTING

Written Clinical

Year 1 HBoM: Team of trained examiners mark essays. Check-lists created

MCBoM: Trial standard setting group spanning all 5 academic departments in Faculty MVS

Year 2 Ebel method used for written papers Mock standard setting exercise for spot anatomy

Year 3 Ebel for Junior Medicine & Surgery Borderline Group Method for OSCE in Junior Med & Surgery

Year 4 Full implementation of Ebel in all 4 units Borderline Group Method for OSCEs.

Year 5 Full implementation of Angoff in EMQs for Finals

Angoff for DOSCE stations Borderline Regression Method for OSCE stations

SSCs

Competent

Pass

Incompetent

Pass

Competent

Fail

Incompetent

Fail

4 components to COMP2 exam

• OSCE 10 stations (each out of 10)

• MCQ (negatively marked)

• Student selected component (essay & group presentation)

• Satisfactory professional behaviour

4 components

• OSCE 10 stations (each out of 10)

• MCQ (negatively marked)

• Student selected component (essay & group presentation)

• Satisfactory professional behaviour

50%

50%

45%

Pass/fail

Marks in an OSCE

10 20 30 40 50 60 70 80 90 100

No. of students

Pass mark

Total mark in OSCE (%)

The marks of a poorly performing student

Station

A B C D E F G H I J

4 4 6 5 5 4 4.5 7 6 5 50.5

New rule:

“Students should get more than 50% overall & should not fail more than 2 stations”

Station

A B C D E F G H I J

4 4 6 5 5 4 4.5 7 6 5 50.5

FAIL

EXAMINER FORM STICK STUDENT ID LABEL HERE

Check that you have correct

student’s name

and number on the marking sheet.

This station is testing the ability of the student to

Manage a patient with mechanical low back pain Elicit the patient’s expectation of a consultation

COMPETENCE TASK SCORE

Competence task (Maximum mark) Mark ( ½ marks allowed) Comment (only if required)

Student takes a good history including the effect of the

symptoms on the patient’s work (2)

Student checks for red flag symptoms: bladder, bowels & saddle

numbness (1 ½)

Student explores patients concerns & finds out that patient was

expecting an X-ray; student explains why this is not a good

idea (2 )

Student explains natural history of low back pain (1)

Student negotiates management plan with patient. This should

using a stronger analgesic drug, encouraging patient to mobilise

and possibility of self certification/ Med3 (2)

Student gives advice on safe lifting (1 ½ )

TOTAL (out of 10)

OVERALL IMPRESSION How do YOU feel the student has performed? Please circle the description that best describes the performance of this student:

FAIL BORDERLINE

PASS

CLEAR PASS

EXAMINER FORM STICK STUDENT ID LABEL HERE

Check that you have correct

student’s name

and number on the marking sheet.

This station is testing the ability of the student to

Manage a patient with mechanical low back pain Elicit the patient’s expectation of a consultation

COMPETENCE TASK SCORE

Competence task (Maximum mark) Mark ( ½ marks allowed) Comment (only if required)

Student takes a good history including the effect of the

symptoms on the patient’s work (2) 1

Student checks for red flag symptoms: bladder, bowels & saddle

numbness (1 ½) 1

Student explores patients concerns & finds out that patient was

expecting an X-ray; student explains why this is not a good

idea (2 )

½

Student explains natural history of low back pain (1) ½

Student negotiates management plan with patient. This should

using a stronger analgesic drug, encouraging patient to mobilise

and possibility of self certification/ Med3 (2)

1½

Student gives advice on safe lifting (1 ½ ) ½

TOTAL (out of 10) 5

OVERALL IMPRESSION How do YOU feel the student has performed? Please circle the description that best describes the performance of this student:

FAIL BORDERLINE

PASS

CLEAR PASS

Borderline Group Method C

om

pet

en

ce

task

sco

re

10

6.1

0

Clear Borderline Clear Very Outstanding

Fail Pass Good

EXAMINER FORM STICK STUDENT ID LABEL HERE

Check that you have correct

student’s name

and number on the marking sheet.

This station is testing the ability of the student to

Manage a patient with mechanical low back pain Elicit the patient’s expectation of a consultation

COMPETENCE TASK SCORE

Competence task (Maximum mark) Mark ( ½ marks allowed) Comment (only if required)

Student takes a good history including the effect of the

symptoms on the patient’s work (2)

Student checks for red flag symptoms: bladder, bowels & saddle

numbness (1 ½)

Student explores patients concerns & finds out that patient was

expecting an X-ray; student explains why this is not a good

idea (2 )

Student explains natural history of low back pain (1)

Student negotiates management plan with patient. This should

using a stronger analgesic drug, encouraging patient to mobilise

and possibility of self certification/ Med3 (2)

Student gives advice on safe lifting (1 ½ )

TOTAL (out of 10)

OVERALL IMPRESSION How do YOU feel the student has performed? Please circle the description that best describes the performance of this student:

FAIL BORDERLINE

PASS

CLEAR PASS VERY GOOD OUTSTANDING

EXAMINER FORM STICK STUDENT ID LABEL HERE

Check that you have correct

student’s name

and number on the marking sheet.

This station is testing the ability of the student to

Manage a patient with mechanical low back pain Elicit the patient’s expectation of a consultation

COMPETENCE TASK SCORE

Competence task (Maximum mark) Mark ( ½ marks allowed) Comment (only if required)

Student takes a good history including the effect of the

symptoms on the patient’s work (2) 1

Student checks for red flag symptoms: bladder, bowels & saddle

numbness (1 ½) 1

Student explores patients concerns & finds out that patient was

expecting an X-ray; student explains why this is not a good

idea (2 )

½

Student explains natural history of low back pain (1) ½

Student negotiates management plan with patient. This should

using a stronger analgesic drug, encouraging patient to mobilise

and possibility of self certification/ Med3 (2)

1½

Student gives advice on safe lifting (1 ½ ) ½

TOTAL (out of 10) 5

OVERALL IMPRESSION How do YOU feel the student has performed? Please circle the description that best describes the performance of this student:

FAIL BORDERLINE

PASS

CLEAR PASS VERY GOOD OUTSTANDING

Borderline Regression Method

Clear Borderline Clear Very Outstanding

Fail Pass Good

Co

mp

ete

nce

ta

sk s

core

10

6.1

0

EXAMINER FORM STICK STUDENT ID LABEL HERE

Check that you have correct

student’s name

and number on the marking sheet.

This station is testing the ability of the student to

Manage a patient with mechanical low back pain Elicit the patient’s expectation of a consultation

COMPETENCE TASK SCORE

Competence task (Maximum mark) Mark ( ½ marks allowed) Comment (only if required)

Student takes a good history including the effect of the

symptoms on the patient’s work (2) 1

Student checks for red flag symptoms: bladder, bowels & saddle

numbness (1 ½) 1

Student explores patients concerns & finds out that patient was

expecting an X-ray; student explains why this is not a good

idea (2 )

½

Student explains natural history of low back pain (1) ½

Student negotiates management plan with patient. This should

using a stronger analgesic drug, encouraging patient to mobilise

and possibility of self certification/ Med3 (2)

1½

Student gives advice on safe lifting (1 ½ ) ½

TOTAL (out of 10) 5

OVERALL IMPRESSION How do YOU feel the student has performed? Please circle the description that best describes the performance of this student:

FAIL BORDERLINE

PASS

CLEAR PASS VERY GOOD OUTSTANDING

Competence Task Score of borderline students at one station

5½ 6 6½ 7

Competence Score out of 10

No. of students

7

6

5

4

3

2

1

Mean = 6.14

Extract of OSCE Spreadsheet showing 10 students’ marks & the pass mark calculated by Borderline

Group Method

StudentStation

A

Station

B

Station

C

Station

D

Station

E

Station

F

Station

G

Station

H

Station

I

Station

J

TOTAL

(out of 100)

a 9.5 5 7 6.5 5.5 6 8 7.5 8.5 7 70.5

b 5 6 8 7.5 2.5 6 6.5 5.5 6.5 9 62.5

c 7.5 7.5 8 6.5 4 7.5 5.5 6.5 6 8 67

d 5.5 6 4.5 5.5 1 6 8 5.5 7 9 58

e 5.5 5 5.5 6.5 5 5 6.5 8 4.5 6 57.5

f 9.5 7 7 5.5 6 6 8 7 6.5 4 66.5

g 9.5 6 7 7.5 8 8 6 6.5 8 8.5 75

h 7 6.5 7 7 6 5.5 5 8 6.5 5.5 64

I 10 5.5 8 8.5 4 9.5 7 6.5 7 7.5 73.5

j 9 7 6 6 5.5 7 8 7 6.5 7 69

Pass

Mark 5.68 6.4 5.81 7 5 6.14 5.82 6.13 6 5.82 59.8

Training package for OSCE examiners

A DVD showing students perform to different levels at same station

Marks from a written exam

10 20 30 40 50 60 70 80 90 100

No. of students

50% Pass mark

“Moderating” (lowering the pass mark to 45%)

10 20 30 40 50 60 70 80 90 100

No. of students

45% Pass mark

Ebel Table

Easy Medium Hard

Essential

Important

Acceptable

Ebel Table

Easy Medium Hard

Essential 90% 80% 70%

Important 60% 50% 40%

Acceptable 30% 20% 10%

Ebel Table

Easy Medium Hard

Essential 1a, 1b 1c 1d

Important 1e

Acceptable

Ebel Table for Extended Matching Questions that have 10 options

Easy Medium Hard

Essential 75% 55% 45%

Important 60% 50% 30%

Acceptable 40% 25% 10%

Ebel Method

Category Number of

Questions in

this category

Proportion of time

minimally competent

student will get this

question correct (%)

Score expected

from minimally

competent

student

Essential Easy 4 75 3.00

Medium 3 55 1.65

Hard 1 45 0.45

Important Easy 4 60 2.40

Medium 5 50 2.50

Hard 2 30 0.6

Acceptable Easy 1 40 0.40

Medium 35 0

Hard 10 0

Total 20 11

Ebel Method

Category Number of

Questions in

this category

Proportion of time

minimally competent

student will get this

question correct (%)

Score expected

from minimally

competent student

Essential Easy 26 75 19.5

Medium 31 55 17.05

Hard 2 45 0.90

Important Easy 32 60 19.20

Medium 33 50 16.5

Hard 11 30 3.30

Acceptable Easy 1 40 0.40

Medium 3 25 0.75

Hard 1 10 0.10

Total 140 77.4

Divided by number of judges (7)

Total 20 11.1

(55.5%)

Summary

Borderline Group Method for OSCE

Ebel Method for written (EMQ or best of 5)

Both

Built on concept of minimally competent student

Use panel of judges to assess each question

The final marks S

tud

en

t

SS

C /20

MC

Q /130

EM

Q/5

0

EM

Q /100

Written

/230

Written a

s

% Wri

tten

/35

OS

CE

as

% OS

CE

/45

To

tal

a 14 79 40 80 159 69.1 24.2 60.7 27.3 65.5

b 17 82 32 64 146 63.5 22.2 52.7 23.7 62.9

c 15.4 69 32 64 133 57.8 20.2 57.2 25.7 61.4

d 12 54 31 62 116 50.4 17.7 46.7 21.0 50.7 Fail

e 14 68 32 64 132 57.4 20.1 47.7 21.5 55.6 Fail

f 18.4 87 41 82 169 73.5 25.7 56.7 25.5 69.6 Merit

g 14.2 91 40 80 171 74.3 26.0 65.2 29.3 69.6 Merit

h 15.6 69 35 70 139 60.4 21.2 54.2 24.4 61.1

I 18.6 79 36 72 151 65.7 23.0 63.7 28.7 70.2 Merit

j 14.2 80 37 74 154 67.0 23.4 59.2 26.6 64.3

HANDLING MARKS

Spreadsheets & Scaling

EXAM BOARDS

GIVING FEEDBACK TO STUDENTS ON THEIR PERFORMANCE