self-assessment of surgical skill acquisition with computer-based video training and the impact on...

Self-Assessment of Surgical Skill Acquisition with Computer-Based Video Training and the

Impact on Self-Directed Training

Nathan JowettVicki LeBlanc

George XeroulisHelen MacRae

Adam Dubrowski

Surgical Skills CentreThe Wilson Centre

University of Toronto, Toronto, Ontario, Canada

Rogers et al (1998): CBVT vs. didactic lectureEqual cognitive performance but lower technical performance for CBVT group

Summers et al (1999): CBVT with expert vs. didactic instructionCBVT superior on performance retention testing

Rogers et al (2000): CBVT vs. Concurrent feedback with CBVTImproved performance by external feedback group

Xeroulis et al (2005): CBVT vs. expert feedbackNo difference in performance between groups

Is CBVT as effective as expert feedback in basic surgical skill acquisition?

Research Objective

Are trainees able to adequately acquire basic technical skills on their own practice schedules using CBVT?

Research Question:

Is technical skill acquisition by means of CBVT compromised when trainees practice in unstructured as opposed to structured learning settings?

Hypothesis: Yes

CBVT Module

Practice block (3 minutes)

Structured Practice (SP)

Unstructured Practice (UP)

Video Captured Trials with Hand Motion AnalysisSelf-assessment GRS + Proficiency, Expert GRSSelf-assessment GRS + ProficiencyExpert GRS

Open practice period (variable)Additional practice period (fixed)

(1 week delay)

Pre-Test Post-Test 1

Self-assessed proficiency

Post-Test 2 Retention Test

Experimental Paradigm

N = 30 Mean open practice time = 22.7 minutes

Unstructured

Retention

Structured

Tot

al t

ime

(s)

Normalized time to self-assessed proficiency (%)

50100150200250300350

00 25 50 75 100

Trials after proficiency

1 2 3 4

Learning Curve: Time

Practice Group

StructuredUnstructured

Tim

e (

s)Time

Pre-test Post-test 1 Retention (Self-assessed proficiency)

Practice Group

StructuredUnstructured

Nu

mb

er

of

Ha

nd

Mo

vem

en

tsHand Movements

Pre-test Post-test 1 Retention (Self-assessed proficiency)

Practice Group

StructuredUnstructured

Pa

th L

en

gth

(d

eci

me

tre

s)

Path Length

Pre-test Post-test 1 Retention (Self-assessed proficiency)

Practice Group

StructuredUnstructured

Ex

pe

rt G

lob

al R

ati

ng

Sc

ale

Expert Global Rating Scale

Pre-test Post-test 1 Retention (Self-assessed proficiency)

Did significant performance improvement occur between pre and retention tests for both groups?

Yes (p < 0.05)

Did SP and UP groups differ on retention test performance?

No (p > 0.05)

Did improvement occur between first and second post-tests among SP participants?

No (p > 0.05)

Summary

Trainees learning basic knot tying skills using CBVT in an unstructured setting performed equally as well as those in a structured setting

This suggests that trainees were able to accurately self-assess the progression of their technical performance and thus determine when further practice in a single session would no longer be beneficial

Conclusions

Our results support the initiation of self-directed online or CD-ROM CBVT for basic technical skills as a plausible solution to the economic and staffing constraints associated with teaching technical skills outside of the O.R.

Implications

Supported by:

Dean’s Excellence Fund

Faculty of Medicine, University of Toronto

Medical Student Summer Scholarship

Faculty of Medicine, University of Toronto

End

Global Rating Scale and Proficiency

Based on your current knowledge and/or performance of the double one-hand square knot, how do you expect to perform on this task a week from now with no additional practice?

Time and Motion

1Many Unnecessary

moves

2 3Efficient

time/motion but some unnecessary

moves

4 5Clear economy of

movement and maximum efficiency

Instrument Handling

1Repeatedly makes

tentative or awkward moves with

hands/fingers through inappropriate use

2 3Competent use of hands/fingers but

occasionally appeared stiff or

awkward

4 5Fluid movements with hands/fingers and no

stiffness or awkwardness

Flow of Operation

1Frequently stopped

operating and seemed unsure of next move

2 3Demonstrated some forward planning with

reasonable progression of

procedure

4 5Obviously planned

course of operation with effortless flow from one

move to the next

Overall Performance

1Very poor

2 3Competent

4 5Clearly superior

Do you believe you have attained sufficient proficiency at this task and would therefore like to quit practice at this point?

Yes No

Pre-intervention variable Structured Unstructured P Value*

Years of medical school training, mean 1.40 (0.70) 1.55 (0.51) 0.51

Sex, No (%) Female 6 (60) 7 (35)

Male 4 (40) 13 (65)

0.26

Experience: two -handed square knot, No (%)

Yes 1 (10) 1 (5) No 9 (90) 19 (95)

1.00

Experience: instrument tie, No (%)

Yes 6 (60) 10 (50) No 4 (40) 10 (50)

0.71

General self efficacy score†, mean (SD) 4.18 (0.46) 3.97 (0.39) 0.21

Practice time to self(min), (SD)

-assessed proficiency

18.30 (6.40) 24.90 (13.67) 0.16

*Independent t test for continuous data (2-tailed), Pearson χ2 test for nominal data (2-tailed, exact) †Sherer general self-efficacy scale

Randomization Efficacy

Table 6. Paired-samples t test for performance variables between first and second post tests (structured practice group).

Performance Variable t P Value*

Mean expert global rating scale assessment

-1.830.10 (NS)

Hand-motion analysis

Time 1.75 0.12 (NS)

Number of hand movements

Right hand 1.66 0.13 (NS)

Left hand 1.61 0.14 (NS)

*paired-samples t-test, 2-tailedNS = non-significant

Post tests 1 and 2 comparisons (SP group)

Performance variable Pre-test Post-test Retention-test

Mean global rating scale score (/20) 4.5 11.6* 10.4*

4.5 11.0* 8.7*

4.5 11.2* 9.2*

Hand-motion analysis

Time (seconds)

158.5 36.7* 37.5* 183.0 41.9* 62.2*

174.6 40.2* 54.0*

Number of hand movements

96.6 35.3* 37.3* 135.8 38.5* 49.3*

122.3 37.4* 45.4*

StructuredUnstructuredAverage

StructuredUnstructuredAverage

StructuredUnstructuredAverage

P>.05

P>.05

P>.05

Mean Performance: Pre, Post-1, Retention