wbas: the northern ireland school of surgery experience richard mayes 1 robert gilliland 1, 2...

WBAs: the Northern Ireland School of Surgery

ExperienceRichard Mayes1 Robert Gilliland1, 2

Jeffrey Campbell1, 2 Helen Holscher3

Department of Surgery, Ulster Hospital Dundonald, Northern Ireland1

School of Surgery, Northern Ireland Medical & Dental Training Agency2,Royal College of Surgeons of England3..

The Royal College of Surgeons of England

Regional Representatives Meeting

Wednesday 10th November 2010

Introduction of a Curriculum

• Web based curricula • Workplace Based Assessments (WBAs)

• Valid and reliable

• Formative and summative

• Clear indicators of professional progress

GMC April 2010

AimsTo determine if WBAs:

• Differentiate between trainees of different abilities (summative)

• Demonstrate progression towards competence (summative)

• Are completed at regular intervals through training (formative)

Methods

• Core Surgical trainees appointed Aug 2008

• Trainees not completing 24 months excluded

• Trainees were ranked according to interview

score (Q1, Q2, Q3 and Q4)

• 4 x 6 month placements (P1, P2, P3 and P4)

• Median (range): ANOVA

Methods

• CBDs, DOPS, Mini-CEXs – score 1-6• 1-2 - Below expectations• 3 - Borderline• 4 - Meets expectations• 5-6 - Above expectations

• PBAs – score 0-4• Level 0 – Insufficient evidence to support judgement• Level 1 – Unable to perform procedure• Level 2 – Able to perform procedure, or part under

supervision• Level 3 – Minimum supervision• Level 4 – Unsupervised, able to deal with complications

Results

• 39 trainees • 12 excluded• n=27 trainees

• 1,256 assessments• 381 CBDs• 408 DOPS• 346 Mini-CEXs• 121 PBAs

Number of assessments

0

50

100

150

200

250

300

350

P1 P2 P3 P4

Placement

Nu

mb

er PBA

CEX

DOPS

CBD

p=0.39

Can WBAs differentiate between trainees of different abilities?

Can WBAs differentiate between trainees of different abilities?

Trainees from Q1 would out perform trainees from Q4

• number of assessments performed • scores obtained

Median (range) Scores in CBD and Mini-CEXs

0

2

4

6

Quartile

p=0.11

Med

ian

Sco

re

Median (range) Number of WBAs Completed per Trainee

0

20

40

60

80

Quartile

p=0.92

Num

ber

of A

sses

smen

ts

Can WBAs demonstrate progression towards competence?

Can WBAs demonstrate progression towards competence?

Scores should show improvement over

4 placements

CBD, DOPS and Mini-CEX Median (range) Scores

P1 P2 P3 P40

2

4

6

DOPS

Mini-CEX

CBD

Placement

Sco

re

CBD p=0.47, DOPS p=0.19, Mini-CEX p=0.96

PBA Median (range) Scores

P1* P2 P3

P4*0

1

2

3

4

I----NS----I I----NS----I I----NS----I

I-------------*p=0.0007-------------I

Placements

Sco

re

Are WBAs being used in formative assessment?

Are WBAs being used in formative assessment?

WBAs should be completed at regular intervals through training placements

Total Number of Assessments per Month

A S O N D J F M A M J J A S O N D J F M A M J J0

50

100

150

200

P1 P2 P3 P4

*

*

*

*

Months of Placement

Nu

mb

er o

f A

sses

smen

ts

Summary

• No inter-quartile differences with regards to the numbers of WBAs performed or scores obtained

• CBD, DOPS, Mini-CEX scores do not improve with time

• PBAs scores improve with time

• WBAs are clustered immediately prior to appraisal

Conclusions

• WBAs do not fulfil GMC requirements

• Need modification

• Assessors require to be more robust in their evaluations

Is one assessor type more robust than the other?

Median (range) Scores in WBAswith HST/Consultant assessors

HSTsCons

HSTsCons

HSTsCons

0

2

4

6

P1 P2 P3

Placements

Sco

re

ANOVA p=0.42

Distribution of assessment scores

1 2 3 4 5 60

200

400

600

Score

Nu

mb

er