talk

1

Spatially Constrained Segmentation of Dermoscopy Images

Howard Zhou1, Mei Chen2, Le Zou2, Richard Gass2,

Laura Ferris3, Laura Drogowski3, James M. Rehg1

1School of Interactive Computing, Georgia Tech2Intel Research Pittsburgh

3Department of Dermatology, University of Pittsburgh

2

Skin cancer and melanoma Skin cancer : most common of all cancers

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

3

Skin cancer and melanoma Skin cancer : most common of all cancers Melanoma : leading cause of mortality (75%)

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

4

Skin cancer and melanoma Skin cancer : most common of all cancers Melanoma : leading cause of mortality (75%) Early detection significantly reduces mortality

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

5

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Clinical ViewDermoscopy view

6

Dermoscopy Improve diagnostic accuracy by 30% in the hands

of trained physicians May require as much as 5 year experience to have

the necessary training Motivation for Computer-aided diagnosis (CAD) in

this area

Clinical view Dermoscopy view

7

First step of analysis:Segmentation Separating lesions from surrounding skin Resulting border

Gives lesion size and border irregularity Crucial to the extraction of dermoscopic features for

diagnosis

Previous Work : PDE approach – Erkol et al. 2005, … Histogram thresholding – Hintz-Madsen et al. 2001, … Clustering – Schmid 1999, Melli et al. 2006… Statistical region merging – Celebi et al. 2007, …

8

Domain specific constraints Spatial constraints

Four corners are skin (Melli et al.2006, Celebi et al. 2007) Implicitly enforcing Local neighborhood constraints on image

Cartesian coordinates (Meanshift)

9

Domain specific constraints Spatial constraints

Four corners are skin (Melli et al.2006, Celebi et al. 2007) Implicitly enforcing Local neighborhood constraints on image

Cartesian coordinates (Meanshift)

Meanshift (c = 32, s = 8)

10

We explore … Spatial constraints arise from the growth

pattern of pigmented skin lesions

Meanshift (c = 32, s = 8)

11

We explore … Spatial constraints arise from the growth

pattern of pigmented skin lesions –

radiating pattern

Meanshift (c = 32, s = 8)

12

Embedding constraints

Meanshift (c = 32, s = 8) Polar (k = 6)

Radiating pattern from lesion growth Embedding constraints as polar coords

improves segmentation performance

13

Polar (k = 6)Meanshift Polar

Embedding constraints Radiating pattern from lesion growth Embedding constraints as polar coords

improves segmentation performance

14

Meanshift Polar

Comparison to the Doctors Radiating pattern from lesion growth Embedding constraints as polar coords

improves segmentation performance White: Dr. FerrisRed : Dr. ZhangBlue : computer

15

Dermoscopy images Common radiating appearance

16

Growth pattern of pigmented skin lesions lesions grow in both radial and vertical direction Skin absorbs and scatters light. Appearance of pigmented cells varies with depth

Dark brown tan blue-gray

Common radiating appearance pattern on skin surface

[ Image courtesy of “Dermoscopy : An Atlas of Surface Microscopy of Pigmented Skin Lesions]

17

Radiating growth pattern on skin surface Difference in appearance: more significant

along the radial direction than any other direction.

18

Radiating growth pattern on skin surface Difference in appearance: more significant

along the radial direction than any other direction.

19

Each pixel feature vector in R4 3D: R,G,B or L, a, b in the color space 1D: polar radius measured from the center of

the image (normalized by w)

Embedding spatial constraintsFeature vectors

original

r

{R, G, B}

20

Each pixel feature vector in R4 Clustering pixels in the feature space Replace pixels with mean for compact

representation

Embedding spatial constraintsGrouping features

filteredoriginal

r

{R, G, B}

21

Radiating pattern Dermoscopy vs. natural images

Derm dataset (216)

… …

BSD dataset (300)

22

Mean per-pixel residue: average per-pixel color difference of each pair

Embedding spatial constraintsGrouping features

original

{Ro, Go, Bo}

polar

{Rp, Gp, Bp}

Cartesian

{Rc, Gc, Bc}

23

Dermoscopy vs. natural images Polar vs. Cartesion

BSD dataset (300)

Residue (polar)

Residue (Cartesian)

Derm dataset (216)

Residue (Cartesian)

Residue (polar)

Mean per-pixel residue (k-means++, k = 30)

24

Dermoscopy vs. natural images Polar vs. Cartesion Mean per-pixel residue (k-means++, k = 30)

25

Polar vs. Cartesian The regions appear more blocky in the

Cartesian case

Polar (k = 30) Cartesian (k = 30)

26

Six super-regions 30 clusters 6 super clusters (K-means++)

Polar (k = 6) Cartesian (k = 6)

27

Final segmentation

Polar Cartesian

28

Polar vs. Meanshift The regions appear more blocky in the

Meanshift case

Polar (k = 6) Meanshift (c = 32, s = 8)

29

Final segmentation

Polar Meanshift

30

Given a dermoscopy image

Algorithm overview

31

Given a dermoscopy image

Algorithm overview

original

32

1. First round clustering: K-means++ (k = 30)

Algorithm overview

original 30 clusters

33

2. Second round: clusters(30) super-regions(6)

Algorithm overview

original 30 clusters 6 Super-regions

34

3. Apply texture gradient filter (Martin, et al. 2004)

Algorithm overview

original 30 clusters 6 Super-regions

Texture edge map

35

4. Find optimal boundary (color+texture)

Algorithm overview

original 30 clusters 6 Super-regions

Texture edge map Final segmentation

36

First round clustering: K-means++ (k = 30) Reduce noise Groups pixels into homogenous regions – a

more compact representation of the image Artuhur and Vassilvitskii, 2007

R4 : {L*a*b* (3D), w * polar radius (1D)}

1. First round clustering

original

37

First round clustering: K-means++ (k = 30) Reduce noise Groups pixels into homogenous regions – a

more compact representation of the image Artuhur and Vassilvitskii, 2007

R4 : {L*a*b* (3D), w * polar radius (1D)}

1. First round clustering

original 30 clusters

38

K = 6 : clusters(30) super-regions(6) Account for intra-skin and intra-lesion variations Avoid a large k

Super-regions correspond to meaningful regions such as skin, skin-lesion transition, and inner lesion, etc.

2. Second round clustering

original 30 clusters

39

K = 6 : clusters(30) super-regions(6) Account for intra-skin and intra-lesion variations Avoid a large k

Super-regions correspond to meaningful regions such as skin, skin-lesion transition, and inner lesion, etc.

2. Second round clustering

original 30 clusters 6 super-regions

40

3. Color-texture integration Incorporating texture information can

improve segmentation performance. Severely sun damaged skin; texture variations

at boundaries in addition to color variations

original

41

3. Color-texture integration Incorporating texture information can

improve segmentation performance. Severely sun damaged skin; texture variations

at boundaries in addition to color variations Apply texture gradient filter (Martin, et al. 2004)

original

42

3. Color-texture integration Incorporating texture information can

improve segmentation performance. Severely sun damaged skin; texture variations

at boundaries in addition to color variations Apply texture gradient filter (Martin, et al. 2004)

Texture edge map: pseudo-likelihood

original Texture edge map

43

Optimal skin-lesion boundary Color: Earth Mover’s Distance (EMD) between every

pair of super-regions

4. Optimal boundary

6 super-regions

44

Optimal skin-lesion boundary Color: Earth Mover’s Distance (EMD) between every pair

of super-regions Texture: Texture edge map

4. Optimal boundary

Texture edge map6 super-regions

45

Optimal skin-lesion boundary Color: Earth Mover’s Distance (EMD) between every pair

of super-regions Texture: Texture edge map Minimizing the integrated color-texture measure

4. Optimal boundary

Texture edge map6 super-regions

46

Our collaborating dermatologist Dr. Ferris manually outline the lesions in 67 dermoscopy images

The border error is given by

Computer : binary image obtained by filling the automatic detected border

ground-truth : obtained by filling in the boundaries outlined by Dr. Ferris

Validation and results

47

Typical segmentation result

Error = 12.96%

White: Dr. FerrisRed : Dr. ZhangBlue : computer

48

ComparisonCompared to ground-truth outlined by Dr. Ferris

11.32

20.64 21.41

16.9219.49 20.13

15.91 14.93

0

5

10

15

20

25

30

35

none Cartesian polar Dr. Zhang

Spatial constraints

Per

ce

nta

ge

err

or Dr. Zhang

RGB

CIELAB

Color + texture

To account for inter-operator variation, we also asked Dr. Alex Zhang to manually outline boundaries on the same dataset

49

Additional results

Error = 5.80%

White: Dr. FerrisRed : Dr. ZhangBlue : computer

50

Additional results

Error = 13.61%

White: Dr. FerrisRed : Dr. ZhangBlue : computer

51

Additional results

Error = 16.60%

White: Dr. FerrisRed : Dr. ZhangBlue : computer

52

Additional results

Error = 34.09%

White: Dr. FerrisRed : Dr. ZhangBlue : computer

53

Limitation

Assumption that lesions appear relatively near the center may not hold

Fairly low number of super regions (6) may limit the algorithm to perform well on lesions with more colors

54

Conclusion

Growth pattern of pigmented skin lesions can be used to improve lesion segmentation accuracy in dermoscopy images.

An unsupervised segmentation algorithm incorporating these spatial constraints

We demonstrate its efficacy by comparing the segmentation results to ground-truth segmentations determined by an expert.

55

Future work

Extend to meanshift?

56

Comparison to other methods

Compared to ground-truth outlined by Dr. Ferris

26.74

20.43 20.77 20.13

14.93

11.32

0

5

10

15

20

25

30

Meanshift JSEG (Celebi2006)

SRM (Celebi2007)

SCS Cartesian SCS polar Dr. Zhang

Segmentation methods

Pe

rce

nta

ge

err

or

57

Color and texture cue integration

Apply texture gradient filter (Martin, et al. 2004)

Pseudo-likelihood map - edge caused by texture variation is present at a certain location