discuscrowds (image '14, dresden, selected slides).pdf

8/19/2019 DiscusCrowds (IMAGE '14, Dresden, selected slides).pdf

1/29

Silk Purses, Sow’s Ears

IMAGE meeting

12-14 March 2014, Schloss Eckberg, Dresden

Paul H Artes, PhDOphthalmology and Visual Sciences, Dalhousie University


2/29

How damaged are these discs?

What is the likelihood of damage? (0-100%)What is the amount of damage (neural tissue, vision)?


3/29

How good are these observers?

...distinguish between discs of healthy people & patients

...match decision criteria to prevalence.

you?


4/29

Discus, GONE, others

www.gone-project.com

www.discusproject.blogspot.com


5/29

V RI BILITY

OF EXPERT OBSERVERS IN

EV LU TING T

OPTIC

DISC

BYPaul R.

Lichter MD

INTRODUCTION

CORRECT

EVALUATION

OF

THE

OPTIC

DISC

IS FUND MENT L TO THE

DIAGNOSIS

and

management

of

g l au co ma . Y e t

this is sometimes

an extremely

difficult

t sk

Numerous attempts have been

made to

describe what is normal

or

abnormal about an optic disc

yet

no

precise

means is

available

to

deter

mine the

status

of an optic

disc

with

reference to

glaucoma.

Descriptions

and measurements

used

by

some

individuals

have

no t been utilized

by

others. With this in

mind a

study of

optic disc evaluation

was

under

taken. The

study was

particularly aimed at testing the

validity

of

disc

evaluation utilizing nonstereo

and

stereo

photographs.

The

purpose

of

this

report

is

to

analyze the observer

variability

of

16 glaucoma

experts

in

evaluating the

optic disc. This

analysis includes

the

determination ofcup/disc ratio as well as

the

estimate of

the

probability

of pathologic

glaucomatous change

in

a

series

of optic

discs.

B CKGROUND

Ever

since the invention

of

the ophthalmoscope in

1851

the

optic

disc

and

its

relationship

to

glaucoma

has

intrigued

many

an

ophthalmologist.

In

1855 Weber

stated

that the optic disc

was

cupped in

glaucoma

rather

than

swollen or bulging as previously

believed.

Through the next

several

decades little

was added to

knowledge

about th e disc

in

glaucoma.

At

the

turn

of the century Elschnig2 noted th e

variable

appearance

of

normal discs.

He described five

types

of discs and

this

description

h as g en er al ly s to od

the test of

time.

However, this classification

of

normal discs

is

not widely used

today.

From the Department

of

Ophthalmology

University

of

Michigan Medical School

nn

Arbor Michigan.

TR.

AM. OPHTH. Soc. vol.

LXXIV, 1976


6/29


7/29

positive rate in VF!negative group

p o s i t i v e

r a t e

i n

V F ! p o s i t i v e

g r o u p

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

A

BC

D

AUC=0.8

83%

r=0.75

mean.Diff=0.5

criterion=!0.56

A

0

5

10

15

20

25

30

response

r e s p o n s e

a

e n c y

s

dh ph ns pd dd

0

5

10

15

20

25

30

40 24 2 20 14


8/29

response time - ceiling at 20 secs.1026 submissions,674 observers*403 IP addresses* A single person could have generated several IDs. If several attempts were made under the sameID, those responses were averaged (per image).


9/29


p o s i t i v e

r a t e

i n

V F

! p o s i t i v e

g r o u p

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

A

BC

D

AUC=0.5

!1%

r=0.3

mean.Diff=0.96

criterion=!0.04

A

0

5

10

15

20

25

30

response

r e s p o n s e l a

t e n c y ( s )

dh ph ns pd dd

0

5

10

15

20

25

30

7 46 1 3 6 10


p o s i t i v e

r a t e

i n


g r o u p

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

A

B

C

D

AUC=0.85

96%

r=0.7

mean.Diff=0.58

criterion=!0.61

A

0

5

10

15

20

25

30

response

r e s p o n s e

l a t e n c y ( s )

dh ph ns pd dd

0

5

10

15

20

25

30

37 23 15 14 11

removed ~5% of data,2.5% from either extremes.

Area under ROC curve


10/29


p

o s i t i v e

r a t e

i n


g r o u p

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

A

B

C

D

AUC=0.77

73%

r=0.55

mean.Diff=0.5

criterion=!1.2

A

11


p

o s i t i v e r a t e i n


g r o u p

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

A

B

C

D

AUC=0.8

82%

r=0.66

mean.Diff=0.69

criterion=!0.04

A

11 1

o b

s e r v e r s

Two observers with similar “skill” (ie, discriminatoryperformance, but vastly different decision criteria…


11/29


12/29

average (0.69)

“voting” (0.82) o b s e r v e r s


13/29

i m a g e s

C E N

S O R E D


14/29

i m a g e s

C E N

S O R E D


15/29

i m a g e s

C E N

S O R E D


16/29

i m a g e s

C E N

S O R E D


17/29

i m a g e s

C E N

S O R E D


18/29

i m a g e s

C E N

S O R E D


19/29

i m a g e s

C E N

S O R E D


20/29

i m a g e s

C E N

S O R E D


21/29

i m a g e s

C E N

S O R E D


22/29

i m a g e s

C E N

S O R E D


23/29

C E N

S O R

E D


24/29

C E N S

O R E D


25/29

C E N S

O R E D


26/29

C E N S O

R E D


27/29

C E N

S O R E D


28/29

C E N

S O R E D


29/29

C E N

S O R E D

discuscrowds (image '14, dresden, selected slides).pdf

Documents