foundation.cap.org slidetutor project – using virtual slides to understand how pathologists think...

Foundation.cap.org

SlideTutor Project – Using Virtual Slides to Understand How Pathologists Think and Learn

Rebecca Crowley, MD, MSSaturday, April 16, 2011

• Microscopic diagnosis is one of the most complex visual classification tasks that humans perform

• In domains outside of medicine, intelligent computer-based training is common

−Aviation, nuclear and power industries, and the military

Background - I

© 2011 College of American Pathologists. All rights reserved. 2

• Research has shown that these systems can approach benefits of one-on-one teaching

• Advances in virtual slide and educational technologies may help us to rethink the platform we use to educate pathologists

Background - II


• Build the most advanced medical training system possible with current technology

• Use the system to more efficiently train pathology residents and fellows

• Understand how we learn the complex task of microscopic diagnosis

• Predict future performance (including errors

Goals of the SlideTutor Project


Q1. How do pathologists develop expertise in microscopic diagnosis?

Early studies of expertise


7

Overall accuracy

Accuracy

0

20

40

60

80

100

120

Novice Intermediate ExpertSpecific

Category

Crowley et al. JAMIA, 2003

© 2011 College of American Pathologists. All rights reserved.

Resident looking at DCIS


Intermediate I5:

20 and some of the ducts that are expanded with small cells 21 with focal, possibly central, area of necrosis. 22 So just scan this slide around and try to determine some

focal areas that I want to concentrate and focus on. 23 Now I’m looking at some of the ducts that are expanded. 25 And some of these ducts, they also have holes, 26 and these are sort of punched-out holes, 27 very uniform, which… 29 So at this magnification, I think it is a DCIS.

identify-histologic-cue


identify-histologic-cueidentify-histologic-cueidentify-histologic-cue

statement-of-hypothesis

Attending looking at DCIS


Expert E7:

• Okay let’s look at low power• I think the tissue is breast• I recognize some normal• Here is in situ carcinoma• I have to find out if there is any

invasion


identify-anatomic-location

statement-of-hypothesisset-goal-identify-feature

10

Process Differences Measure Novice Intermediate Expert ANOVA Mean SD Mean SD Mean SD F

Value P

Value Pairwise

Comparison (Tukey HSD)

P Value

Data examination Identification 14.1 5.4 20.5 8.2 8.0 2.8 9.4 .001 N, I

N, E E, I

.065

.110

.001 Comparison 2.2 1.2 3.3 1.5 0.93 0.66 8.7 .001 N, I

N, E E, I

.132

.077

.001 History 1.4 0.5 1.7 0.7 1.2 0.8 1.7 .210 N, I

N, E E, I

.511

.741

.191 Data exploration and explanation

2.6 1.6 4.5 2.4 2.0 1.6 4.4 .023 N, I N, E E, I

.081

.794

.027 Data interpretation 4.9 1.6 11.4 4.1 8.4 3.6 9.8 .001 N, I

N, E E, I

.000

.084

.153 Operational processes

2.6 2.1 4.4 1.8 1.5 1.6 5.4 .011 N, I N, E E, I

.119

.408

.009 Goal-setting 0.4 0.4 1.7 1.3 1.8 2.0 3.3 .052 N, I

N, E E, I

.094

.083

.981 Unique hypotheses 2.2 0.6 4.0 0.9 3.3 1.2 11.3 <.001 N, I

N, E E, I

<.001 .031 .210

Crowley et al. JAMIA, 2003© 2011 College of American Pathologists. All rights reserved.

Task Errors Errors Description Novice Intermediate Expert Statistics Case Level

Errors coded as present or absent in each case

Number of cases / Total (%)



Chi-Square

P Value

Pairwise Comparison

P Value

1 Lesion never brought under objective

8/30 (26.7 %)

1/28 (3.6 %)

0/23 (0 %)

13.25

.0021

N, I N, E E, I

.0059

.0006

.3189 2 Lesion

traversed without recognition

7/30 (23.3 %)

0/28 (0 %)

0/23 (0 %)

8.11

.0209

N, I N, E E, I

.0056

.0058 -

3 Error in identifying anatomic location

14/40 (35.0 %)

2/38 (5.3 %)

0/32 (0 %)

22.76

<.0001

N, I N, E E, I

.0008 <.0001 .1612

Segment Level

Errors counted for each case

Mean/ case

SD Mean/ case

SD Mean/ case

SD F Value

P Value

Pairwise Comparison (TukeyHSD)

P Value

4 Incorrectly names normal structure

0.35

0.74

0.11

0.31

0

0

5.28

.012

N, I N, E E, I

.068

.013

.656 5 Incorrectly

names histopathologic cue

0.93

1.4

0.76

1.08

.003

.18

7.05

.004

N, I N, E E, I

.669

.024

.004

6 Error in assigning significance,

0.48

0.82

0.32

0.57

.003

.18

3.17

.059

N, I N, E E, I

.552

.048

.298

Crowley et al. JAMIA, 200311© 2011 College of American Pathologists. All rights

reserved.

12

Task Analysis Latencies


Q2. What makes our system different than other virtual-slide educational systems?

Student Model

Pedagogic Knowledge

Interface

Expert Module

• Allow correct steps• Correct errors• Give hints on next step

• Collect data on what student does• Make predictions on what student knows• Provide data for pedagogic decision making

• Case sequence• When to intervene• How much to intervene• How to intervene

14/47


Student draws another link. Object is red and bug message explains

error

Student draws between

evidence and hypothesis

Selects hypothesis

Student specifies feature further to make it more meaningful evidence

Tutor permits correct answer

3. Selects finding

1. Student indicates new finding

2. Clicks on finding, creating ‘x’

System moves viewer or moves and annotates the area

for students

Hints are hierarchically

structured providing increasing help

Student has already identified some evidence

Diagnostic Reasoning

Reporting with CAP Protocols


Calibrating Confidence



Movie of SlideTutor

Q3. How effective is the system?


Errors decline


22/47

Test performance improves

Multiple Choice Test Scores

0.0

20.0

40.0

60.0

80.0

100.0

Pretest Posttest Retention

Test

Sc

ore

*** ***


23/47

Diagnostic accuracy improves

Case Diagnosis: Tutored Patterns vs Untutored Patterns

0

20

40

60

80

100

Pretest Posttest Retention

Test

Sc

ore

TutoredPatternsUntutoredPatterns

*** ***


Measures of calibration

24/47

Saadawi et al, Adv in Health Sci Ed, 2010


25/47

Calibration improves

Test 1 Test 2 Test 1 to 2

Delta F p-value

G Correlation 0.57 0.74 0.17 5.04 0.04Bias 0.13 0.13 0.0 0.07 0.79Discrimination 0.30 0.42 0.12 5.92 0.02

Saadawi et al, Adv in Health Sci Ed, 2010

26/47

Community Pathologists

Crowley et al Proceedings of ITS 2010

27/47

Adherence to CAP improves

Crowley et al Proceedings of ITS 2010

Q4. Can we predict subsequent diagnostic performance and competency?


Modeling and Prediction

• Store every action and it’s meaning

• Create learning curves for skills and knowledge

• Predict whether residents will be able to identify a particular finding or make a particular diagnosis based on previous performance


30/47

Bayesian Models of Skill Acquisition

Yudelson, Medvedeva, and Crowley. User Modeling and User Adapted Interaction, 2008


Modeling and Prediction


Accuracy of our models

• Models that we have developed are reasonably accurate

• Current work is focused on predicting diagnostic accuracy using both visual and symbolic information- Where and how the resident ‘looks’

at a slide can help us predict the kinds of mistakes they may make

32/47© 2011 College of American Pathologists. All rights reserved.

Conclusions

• Pathologists learn the extraordinarily complex task of microscopic diagnosis in a rather predictable fashion that can be studied

• We can use this information to develop highly advanced medical training systems in our domain

33/47

Conclusions

• These systems could significantly enhancement to training future pathologists

• Information we can obtain from the system can be used to make much more detailed assessments of competency than are currently available

34/47

Future landscape

• Electronic books tied directly into systems that provide practice

• Improved competency based training and assessment

• Diagnostic decision support that builds on ‘awareness’ of the pathologist as a unique individual

35/47

http://slidetutor.upmc.edu


37

Acknowledgementso Collaborators:

− Drazen Jukic, MD, PhD, Alka Palakar, MD− Gilan Saadawi, MD, PhD, Claudia Mello-Thoms, PhD− Roger Azevedo, PhD (McGill University)− George Xu, MD, PhD and Michael Feldman MD, PhD (University of Pennsylvania)− Dana Grzybicki, MD, PhD, Steve Raab, MD (University of Colorado)

o Programmers:− Olga Medvedeva− Eugene Tseytlin− Girish Chavan

o Research Associates:− Elizabeth Legowski, Kayse Gearhart

o Knowledge Engineer− Melissa Castine

o Students:− Velma Payne

o Communications:− Karma Edwards

o Funding:− National Library of Medicine 1R01 LM007891 and 2R01 LM007891 (ARRA)− National Cancer Institute 1R25 CA101959 and 2R25 CA101959− Agency for Health Care Research and Quality I UL1 RR024153-01(Grzybicki, PI)− National Library of Medicine Training Grant 5-T15-LM07059− University of Pittsburgh Competitive Medical Research Fund


foundation.cap.org slidetutor project – using virtual slides to understand how pathologists think...

Documents

unique hypothesesisdh

virtual slide

focal areas

early studies of expertise

humans performin domains

current technologyuse

slidetutor project

low poweri