talk
DESCRIPTION
TRANSCRIPT
1
Spatially Constrained Segmentation of Dermoscopy Images
Howard Zhou1, Mei Chen2, Le Zou2, Richard Gass2,
Laura Ferris3, Laura Drogowski3, James M. Rehg1
1School of Interactive Computing, Georgia Tech2Intel Research Pittsburgh
3Department of Dermatology, University of Pittsburgh
2
Skin cancer and melanoma Skin cancer : most common of all cancers
[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]
3
Skin cancer and melanoma Skin cancer : most common of all cancers Melanoma : leading cause of mortality (75%)
[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]
4
Skin cancer and melanoma Skin cancer : most common of all cancers Melanoma : leading cause of mortality (75%) Early detection significantly reduces mortality
[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]
5
[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]
Clinical ViewDermoscopy view
6
Dermoscopy Improve diagnostic accuracy by 30% in the hands
of trained physicians May require as much as 5 year experience to have
the necessary training Motivation for Computer-aided diagnosis (CAD) in
this area
Clinical view Dermoscopy view
7
First step of analysis:Segmentation Separating lesions from surrounding skin Resulting border
Gives lesion size and border irregularity Crucial to the extraction of dermoscopic features for
diagnosis
Previous Work : PDE approach – Erkol et al. 2005, … Histogram thresholding – Hintz-Madsen et al. 2001, … Clustering – Schmid 1999, Melli et al. 2006… Statistical region merging – Celebi et al. 2007, …
8
Domain specific constraints Spatial constraints
Four corners are skin (Melli et al.2006, Celebi et al. 2007) Implicitly enforcing Local neighborhood constraints on image
Cartesian coordinates (Meanshift)
9
Domain specific constraints Spatial constraints
Four corners are skin (Melli et al.2006, Celebi et al. 2007) Implicitly enforcing Local neighborhood constraints on image
Cartesian coordinates (Meanshift)
Meanshift (c = 32, s = 8)
10
We explore … Spatial constraints arise from the growth
pattern of pigmented skin lesions
Meanshift (c = 32, s = 8)
11
We explore … Spatial constraints arise from the growth
pattern of pigmented skin lesions –
radiating pattern
Meanshift (c = 32, s = 8)
12
Embedding constraints
Meanshift (c = 32, s = 8) Polar (k = 6)
Radiating pattern from lesion growth Embedding constraints as polar coords
improves segmentation performance
13
Polar (k = 6)Meanshift Polar
Embedding constraints Radiating pattern from lesion growth Embedding constraints as polar coords
improves segmentation performance
14
Meanshift Polar
Comparison to the Doctors Radiating pattern from lesion growth Embedding constraints as polar coords
improves segmentation performance White: Dr. FerrisRed : Dr. ZhangBlue : computer
15
Dermoscopy images Common radiating appearance
16
Growth pattern of pigmented skin lesions lesions grow in both radial and vertical direction Skin absorbs and scatters light. Appearance of pigmented cells varies with depth
Dark brown tan blue-gray
Common radiating appearance pattern on skin surface
[ Image courtesy of “Dermoscopy : An Atlas of Surface Microscopy of Pigmented Skin Lesions]
17
Radiating growth pattern on skin surface Difference in appearance: more significant
along the radial direction than any other direction.
18
Radiating growth pattern on skin surface Difference in appearance: more significant
along the radial direction than any other direction.
19
Each pixel feature vector in R4 3D: R,G,B or L, a, b in the color space 1D: polar radius measured from the center of
the image (normalized by w)
Embedding spatial constraintsFeature vectors
original
r
{R, G, B}
20
Each pixel feature vector in R4 Clustering pixels in the feature space Replace pixels with mean for compact
representation
Embedding spatial constraintsGrouping features
filteredoriginal
r
{R, G, B}
21
Radiating pattern Dermoscopy vs. natural images
Derm dataset (216)
… …
BSD dataset (300)
22
Mean per-pixel residue: average per-pixel color difference of each pair
Embedding spatial constraintsGrouping features
original
{Ro, Go, Bo}
polar
{Rp, Gp, Bp}
Cartesian
{Rc, Gc, Bc}
23
Dermoscopy vs. natural images Polar vs. Cartesion
BSD dataset (300)
Residue (polar)
Residue (Cartesian)
Derm dataset (216)
Residue (Cartesian)
Residue (polar)
Mean per-pixel residue (k-means++, k = 30)
24
Dermoscopy vs. natural images Polar vs. Cartesion Mean per-pixel residue (k-means++, k = 30)
25
Polar vs. Cartesian The regions appear more blocky in the
Cartesian case
Polar (k = 30) Cartesian (k = 30)
26
Six super-regions 30 clusters 6 super clusters (K-means++)
Polar (k = 6) Cartesian (k = 6)
27
Final segmentation
Polar Cartesian
28
Polar vs. Meanshift The regions appear more blocky in the
Meanshift case
Polar (k = 6) Meanshift (c = 32, s = 8)
29
Final segmentation
Polar Meanshift
30
Given a dermoscopy image
Algorithm overview
31
Given a dermoscopy image
Algorithm overview
original
32
1. First round clustering: K-means++ (k = 30)
Algorithm overview
original 30 clusters
33
2. Second round: clusters(30) super-regions(6)
Algorithm overview
original 30 clusters 6 Super-regions
34
3. Apply texture gradient filter (Martin, et al. 2004)
Algorithm overview
original 30 clusters 6 Super-regions
Texture edge map
35
4. Find optimal boundary (color+texture)
Algorithm overview
original 30 clusters 6 Super-regions
Texture edge map Final segmentation
36
First round clustering: K-means++ (k = 30) Reduce noise Groups pixels into homogenous regions – a
more compact representation of the image Artuhur and Vassilvitskii, 2007
R4 : {L*a*b* (3D), w * polar radius (1D)}
1. First round clustering
original
37
First round clustering: K-means++ (k = 30) Reduce noise Groups pixels into homogenous regions – a
more compact representation of the image Artuhur and Vassilvitskii, 2007
R4 : {L*a*b* (3D), w * polar radius (1D)}
1. First round clustering
original 30 clusters
38
K = 6 : clusters(30) super-regions(6) Account for intra-skin and intra-lesion variations Avoid a large k
Super-regions correspond to meaningful regions such as skin, skin-lesion transition, and inner lesion, etc.
2. Second round clustering
original 30 clusters
39
K = 6 : clusters(30) super-regions(6) Account for intra-skin and intra-lesion variations Avoid a large k
Super-regions correspond to meaningful regions such as skin, skin-lesion transition, and inner lesion, etc.
2. Second round clustering
original 30 clusters 6 super-regions
40
3. Color-texture integration Incorporating texture information can
improve segmentation performance. Severely sun damaged skin; texture variations
at boundaries in addition to color variations
original
41
3. Color-texture integration Incorporating texture information can
improve segmentation performance. Severely sun damaged skin; texture variations
at boundaries in addition to color variations Apply texture gradient filter (Martin, et al. 2004)
original
42
3. Color-texture integration Incorporating texture information can
improve segmentation performance. Severely sun damaged skin; texture variations
at boundaries in addition to color variations Apply texture gradient filter (Martin, et al. 2004)
Texture edge map: pseudo-likelihood
original Texture edge map
43
Optimal skin-lesion boundary Color: Earth Mover’s Distance (EMD) between every
pair of super-regions
4. Optimal boundary
6 super-regions
44
Optimal skin-lesion boundary Color: Earth Mover’s Distance (EMD) between every pair
of super-regions Texture: Texture edge map
4. Optimal boundary
Texture edge map6 super-regions
45
Optimal skin-lesion boundary Color: Earth Mover’s Distance (EMD) between every pair
of super-regions Texture: Texture edge map Minimizing the integrated color-texture measure
4. Optimal boundary
Texture edge map6 super-regions
46
Our collaborating dermatologist Dr. Ferris manually outline the lesions in 67 dermoscopy images
The border error is given by
Computer : binary image obtained by filling the automatic detected border
ground-truth : obtained by filling in the boundaries outlined by Dr. Ferris
Validation and results
47
Typical segmentation result
Error = 12.96%
White: Dr. FerrisRed : Dr. ZhangBlue : computer
48
ComparisonCompared to ground-truth outlined by Dr. Ferris
11.32
20.64 21.41
16.9219.49 20.13
15.91 14.93
0
5
10
15
20
25
30
35
none Cartesian polar Dr. Zhang
Spatial constraints
Per
ce
nta
ge
err
or Dr. Zhang
RGB
CIELAB
Color + texture
To account for inter-operator variation, we also asked Dr. Alex Zhang to manually outline boundaries on the same dataset
49
Additional results
Error = 5.80%
White: Dr. FerrisRed : Dr. ZhangBlue : computer
50
Additional results
Error = 13.61%
White: Dr. FerrisRed : Dr. ZhangBlue : computer
51
Additional results
Error = 16.60%
White: Dr. FerrisRed : Dr. ZhangBlue : computer
52
Additional results
Error = 34.09%
White: Dr. FerrisRed : Dr. ZhangBlue : computer
53
Limitation
Assumption that lesions appear relatively near the center may not hold
Fairly low number of super regions (6) may limit the algorithm to perform well on lesions with more colors
54
Conclusion
Growth pattern of pigmented skin lesions can be used to improve lesion segmentation accuracy in dermoscopy images.
An unsupervised segmentation algorithm incorporating these spatial constraints
We demonstrate its efficacy by comparing the segmentation results to ground-truth segmentations determined by an expert.
55
Future work
Extend to meanshift?
56
Comparison to other methods
Compared to ground-truth outlined by Dr. Ferris
26.74
20.43 20.77 20.13
14.93
11.32
0
5
10
15
20
25
30
Meanshift JSEG (Celebi2006)
SRM (Celebi2007)
SCS Cartesian SCS polar Dr. Zhang
Segmentation methods
Pe
rce
nta
ge
err
or
57
Color and texture cue integration
Apply texture gradient filter (Martin, et al. 2004)
Pseudo-likelihood map - edge caused by texture variation is present at a certain location