“Back to the future: An online OSCE management Information system for nursing OSCEs"

Dr Thomas JB Kropmans

Back to the future, the movie

Developed by Ronald Harden in 1975 Medicine, nursing, dentistry, veterinary medicine, law, health sciences Student competency being assessed in consecutive series of stations Paper burden

Objective Structured Clinical Examinations e.g. OSCE

Admission interviews (Students) Selection interviews (HR) Objective Structured Clinical Examinations (Medicine, Nursing, Health Care, Law) Workplace Based Assessment

Where to be used

100 students x 15 stations = 1500 paper forms Not cost effective A 6-station OSCE report 327.5 hours of staff and faculty

time for each rotation of students (Ratanawongsa et al., 2008; Walsh & Smith, 2006)

US$ 6.90, equivalent to €4.70 per student per station (Cusimano et al., 1994)

We administer 11, 7-12 station OSCEs for a cohort of 670 students, produces 9380 assessment forms. To produce final OSCE results, the administrative cost of this procedure is €29,500, which is €2.80 per paper form.

Due to high error rates 30% (unpublished data)

Disadvantages

Only student ability to pass an examination is being assessed Difficulty of the assessment is not taken into account Heterogeneity of measurement instruments (stations) limits comparability Comparison of student competence across institutions and assessment (OSCE) settings is required Universal adoption of a standardized instrument is recommended (Clinical Teacher September 2014)

Without standard setting

Objective Structured Clinical Examinations e.g. OSCE

Of 1000 forms 300 contains errors Low Cronbach’s Alpha (< 0.8 poor) Pass mark of 50%? Introduction Online Marking Tool in December 2008

30% based upon error

Assess clinical skills or non-cognitive skills Determine between ‘good’ and ‘bad’ performance To provide adequate and timely feedback Reliable decision on ‘pass’ or ‘fail’

Purpose of a (formative or summative) clinical skills exam?

Real time solution

Real Time Data

Cronbach’s Alpha Internal consistency

Issues with examiners

Skew distribution • Median = 74%; min score = 4 (20%) and max 20 (100%)

Distribution of scores

74% 4

100%

Nr o

f stu

dent

s

20 20%

50%

Professional impression examiners

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

9.0

10.0

0 1 2 3 4

Stat

ion

scor

es

Method 1: Global score (Fail=0, Borderline=1, Clear Pass=2, Good Pass=3, Excellent=4)

Method 1 Single borderline score

Borderline Regression Analysis

Improved pass/fail decisions

User Acceptance Test n=18

Satisfaction using the E-OSCE package 88.40% (min- 74; max = 100)

22 prestigious universities & professional bodies >200 OSCEs successfully administered Student and Examiner results being analysed Cost reduction of 70% Error reduction by 30%

19% MORE NUIG students fail after introducing BRA Internal consistency varies from 0.45 – 0.85 Cut-off score varies between stations from 40 – 75% Generalisability Kappa between departments 0.4 – 0.9

References