self-assessment of surgical skill acquisition with computer-based video training and the impact on...
TRANSCRIPT
Self-Assessment of Surgical Skill Acquisition with Computer-Based Video Training and the
Impact on Self-Directed Training
Nathan JowettVicki LeBlanc
George XeroulisHelen MacRae
Adam Dubrowski
Surgical Skills CentreThe Wilson Centre
University of Toronto, Toronto, Ontario, Canada
Rogers et al (1998): CBVT vs. didactic lectureEqual cognitive performance but lower technical performance for CBVT group
Summers et al (1999): CBVT with expert vs. didactic instructionCBVT superior on performance retention testing
Rogers et al (2000): CBVT vs. Concurrent feedback with CBVTImproved performance by external feedback group
Xeroulis et al (2005): CBVT vs. expert feedbackNo difference in performance between groups
Is CBVT as effective as expert feedback in basic surgical skill acquisition?
Research Objective
Are trainees able to adequately acquire basic technical skills on their own practice schedules using CBVT?
Research Question:
Is technical skill acquisition by means of CBVT compromised when trainees practice in unstructured as opposed to structured learning settings?
Hypothesis: Yes
CBVT Module
Practice block (3 minutes)
Structured Practice (SP)
Unstructured Practice (UP)
Video Captured Trials with Hand Motion AnalysisSelf-assessment GRS + Proficiency, Expert GRSSelf-assessment GRS + ProficiencyExpert GRS
Open practice period (variable)Additional practice period (fixed)
(1 week delay)
Pre-Test Post-Test 1
Self-assessed proficiency
Post-Test 2 Retention Test
Experimental Paradigm
N = 30 Mean open practice time = 22.7 minutes
Unstructured
Retention
Structured
Tot
al t
ime
(s)
Normalized time to self-assessed proficiency (%)
50100150200250300350
00 25 50 75 100
Trials after proficiency
1 2 3 4
Learning Curve: Time
Practice Group
StructuredUnstructured
Tim
e (
s)Time
Pre-test Post-test 1 Retention (Self-assessed proficiency)
Practice Group
StructuredUnstructured
Nu
mb
er
of
Ha
nd
Mo
vem
en
tsHand Movements
Pre-test Post-test 1 Retention (Self-assessed proficiency)
Practice Group
StructuredUnstructured
Pa
th L
en
gth
(d
eci
me
tre
s)
Path Length
Pre-test Post-test 1 Retention (Self-assessed proficiency)
Practice Group
StructuredUnstructured
Ex
pe
rt G
lob
al R
ati
ng
Sc
ale
Expert Global Rating Scale
Pre-test Post-test 1 Retention (Self-assessed proficiency)
Did significant performance improvement occur between pre and retention tests for both groups?
Yes (p < 0.05)
Did SP and UP groups differ on retention test performance?
No (p > 0.05)
Did improvement occur between first and second post-tests among SP participants?
No (p > 0.05)
Summary
Trainees learning basic knot tying skills using CBVT in an unstructured setting performed equally as well as those in a structured setting
This suggests that trainees were able to accurately self-assess the progression of their technical performance and thus determine when further practice in a single session would no longer be beneficial
Conclusions
Our results support the initiation of self-directed online or CD-ROM CBVT for basic technical skills as a plausible solution to the economic and staffing constraints associated with teaching technical skills outside of the O.R.
Implications
Supported by:
Dean’s Excellence Fund
Faculty of Medicine, University of Toronto
Medical Student Summer Scholarship
Faculty of Medicine, University of Toronto
End
Global Rating Scale and Proficiency
Based on your current knowledge and/or performance of the double one-hand square knot, how do you expect to perform on this task a week from now with no additional practice?
Time and Motion
1Many Unnecessary
moves
2 3Efficient
time/motion but some unnecessary
moves
4 5Clear economy of
movement and maximum efficiency
Instrument Handling
1Repeatedly makes
tentative or awkward moves with
hands/fingers through inappropriate use
2 3Competent use of hands/fingers but
occasionally appeared stiff or
awkward
4 5Fluid movements with hands/fingers and no
stiffness or awkwardness
Flow of Operation
1Frequently stopped
operating and seemed unsure of next move
2 3Demonstrated some forward planning with
reasonable progression of
procedure
4 5Obviously planned
course of operation with effortless flow from one
move to the next
Overall Performance
1Very poor
2 3Competent
4 5Clearly superior
Do you believe you have attained sufficient proficiency at this task and would therefore like to quit practice at this point?
Yes No
Pre-intervention variable Structured Unstructured P Value*
Years of medical school training, mean 1.40 (0.70) 1.55 (0.51) 0.51
Sex, No (%) Female 6 (60) 7 (35)
Male 4 (40) 13 (65)
0.26
Experience: two -handed square knot, No (%)
Yes 1 (10) 1 (5) No 9 (90) 19 (95)
1.00
Experience: instrument tie, No (%)
Yes 6 (60) 10 (50) No 4 (40) 10 (50)
0.71
General self efficacy score†, mean (SD) 4.18 (0.46) 3.97 (0.39) 0.21
Practice time to self(min), (SD)
-assessed proficiency
18.30 (6.40) 24.90 (13.67) 0.16
*Independent t test for continuous data (2-tailed), Pearson χ2 test for nominal data (2-tailed, exact) †Sherer general self-efficacy scale
Randomization Efficacy
Table 6. Paired-samples t test for performance variables between first and second post tests (structured practice group).
Performance Variable t P Value*
Mean expert global rating scale assessment
-1.830.10 (NS)
Hand-motion analysis
Time 1.75 0.12 (NS)
Number of hand movements
Right hand 1.66 0.13 (NS)
Left hand 1.61 0.14 (NS)
*paired-samples t-test, 2-tailedNS = non-significant
Post tests 1 and 2 comparisons (SP group)
Performance variable Pre-test Post-test Retention-test
Mean global rating scale score (/20) 4.5 11.6* 10.4*
4.5 11.0* 8.7*
4.5 11.2* 9.2*
Hand-motion analysis
Time (seconds)
158.5 36.7* 37.5* 183.0 41.9* 62.2*
174.6 40.2* 54.0*
Number of hand movements
96.6 35.3* 37.3* 135.8 38.5* 49.3*
122.3 37.4* 45.4*
StructuredUnstructuredAverage
StructuredUnstructuredAverage
StructuredUnstructuredAverage
P>.05
P>.05
P>.05
Mean Performance: Pre, Post-1, Retention