dermatologist-level classification of skin cancer with ...€¦ · dermatologist 1 65.6%...
TRANSCRIPT
Dermatologist-level classification of skin cancer with deep neural networks
Enhancing the Expert
Andre EstevaPI: Sebastian ThrunStanford University
1
How can technology assist a human?
2
3
4
5
How can AI assist a dermatologist?
6
Skin Cancer
7
● 5.4M cases of non-melanoma skin cancer each year in US
Skin Cancer
8
● 5.4M cases of non-melanoma skin cancer each year in US● 20% of Americans will get skin cancer
Skin Cancer
9
● 5.4M cases of non-melanoma skin cancer each year in US● 20% of Americans will get skin cancer● Actinic Keratosis (pre-cancer) affects 58 million Americans
Skin Cancer
10
● 5.4M cases of non-melanoma skin cancer each year in US● 20% of Americans will get skin cancer● Actinic Keratosis (pre-cancer) affects 58 million Americans● 76,000 melanomas each year - 10,000 deaths
Skin Cancer
11
● 5.4M cases of non-melanoma skin cancer each year in US● 20% of Americans will get skin cancer● Actinic Keratosis (pre-cancer) affects 58 million Americans● 76,000 melanomas each year - 10,000 deaths● $8.1B in US annual costs for skin cancer
Skin Cancer
12
● 5.4M cases of non-melanoma skin cancer each year in US● 20% of Americans will get skin cancer● Actinic Keratosis (pre-cancer) affects 58 million Americans● 76,000 melanomas each year - 10,000 deaths● $8.1B in US annual costs for skin cancer
Skin Cancer
13Years
100
75
50
25
0
Sur
viva
l Pro
babi
lity
1 5 10 15
Stage I
Stage II
Stage III
Stage IV
● 5.4M cases of non-melanoma skin cancer each year in US● 20% of Americans will get skin cancer● Actinic Keratosis (pre-cancer) affects 58 million Americans● 76,000 melanomas each year - 10,000 deaths● $8.1B in US annual costs for skin cancer
Skin Cancer
Years
100
75
50
25
0
Sur
viva
l Pro
babi
lity
1 5 10 15
Stage I
Stage II
Stage III
Stage IV
14
0 1 2 3 4
15
16
Early detection is critical
17
6.3 billion smartphones
18
Skin Cancer Classification
19
~130,000 images of skin
2000 diseases
Skin Cancer Classification
20
~130,000 images of skin
2000 diseases
Skin Cancer Classification
21
Epidermal Lesions Melanocytic Lesions Melanocytic Lesions(Dermoscopy)
Benign
Malignant
Skin Cancer Classification
22
Deep Convolutional Neural Network (Inception-v3)
Skin Cancer Classification
23
Acral-lent. melanomaAmelanotic melanomaLentigo melanoma...
Blue nevusHalo nevusMongolian spot…
Training Classes (757)
Deep Convolutional Neural Network (Inception-v3)
Skin Lesion Image
Skin Cancer Classification
Partitioning Algorithm 24
Acral-lent. melanomaAmelanotic melanomaLentigo melanoma...
Blue nevusHalo nevusMongolian spot…
Training Classes (757)
Deep Convolutional Neural Network (Inception-v3)
Inference Classes (varies by task)
92% Malignant
8% Benign
Skin Lesion Image
Skin Cancer Classification
Partitioning Algorithm 25
Skin Cancer Classification
P = 0.1
P = 0.05
P = 0.05
P = 0.1
P = 0.02 P = 0.03
P = 0.05Training Classes
Inference Classes
P = 0.4
26
Dermatologist-level performance
27
Validation set
Skin Cancer Classification
28
Validation set Classifier Three-way accuracy
Dermatologist 1 65.6%
Dermatologist 2 66.0%
CNN 69.5%
CNN - PA 72.0%
Disease classes: three-way classification
0. Benign single lesions1. Malignant single lesions2. Non-neoplastic lesions
Skin Cancer Classification
29
Validation set Classifier Three-way accuracy
Dermatologist 1 65.6%
Dermatologist 2 66.0%
CNN 69.5%
CNN - PA 72.0%
Classifier Nine-way accuracy
Dermatologist 1 53.3%
Dermatologist 2 55.0%
CNN 48.9%
CNN - PA 55.3%
Disease classes: nine-way classification
0. Cutaneous lymphoma and lymphoid infiltrates 1. Benign dermal tumors, cysts, sinuses2. Malignant dermal tumor3. Benign epidermal tumors, hamartomas, milia, and
growths4. Malignant and premalignant epidermal tumors5. Genodermatoses and supernumerary growths6. Inflammatory conditions7. Benign melanocytic lesions8. Malignant Melanoma
Disease classes: three-way classification
0. Benign single lesions1. Malignant single lesions2. Non-neoplastic lesions
Skin Cancer Classification
30
Skin Cancer Classification
Test set
31
Skin Cancer Classification
Test set: Dermatologist Comparison (376 images)
32
Sensitivity
Carcinoma: 135 images
Spe
cific
ity
Sensitivity
Skin Cancer Classification
Algorithm: AUC = 0.96Dermatologists (25)Average Dermatologist
Test set: Dermatologist Comparison (376 images)
33
Sensitivity Sensitivity Sensitivity
Carcinoma: 135 images Melanoma: 130 images Melanoma: 111 dermoscopy images
Spe
cific
ity
Spe
cific
ity
Spe
cific
ity
Sensitivity Sensitivity Sensitivity
Skin Cancer Classification
Algorithm: AUC = 0.96Dermatologists (25)Average Dermatologist
Algorithm: AUC = 0.94Dermatologists (22)Average Dermatologist
Algorithm: AUC = 0.91Dermatologists (21)Average Dermatologist
Test set: Dermatologist Comparison (376 images)
34
Sensitivity Sensitivity Sensitivity
Carcinoma: 707 images Melanoma: 225 images Melanoma: 1010 dermoscopy images
Spe
cific
ity
Spe
cific
ity
Spe
cific
ity
Sensitivity Sensitivity Sensitivity
Skin Cancer Classification
Test set: Total (1942 images)
Algorithm: AUC = 0.96 Algorithm: AUC = 0.96 Algorithm: AUC = 0.94
35
How does the algorithm work?
36
37
T-SNE Visualization
Van der Maaten & Hinton, 2008
Epidermal BenignEpidermal MalignantMelanocytic BenignMelanocytic Malignant
38
Basal Cell Carcinomas
Squamous Cell Carcinomas
Melanomas
Nevi
Seborrheic Keratoses
T-SNE Visualization
Epidermal BenignEpidermal MalignantMelanocytic BenignMelanocytic Malignant
Basal Cell Carcinomas
Squamous Cell Carcinomas
Melanomas
Nevi
Seborrheic Keratoses
39
T-SNE Visualization
40
What is the network fixating on?
Malignant Melanocytic Lesion
41
What is the network fixating on?
Simonyan, Zisserman, 2014
Malignant Melanocytic Lesion
Malignant Epidermal Lesion
Malignant Dermal Lesion
Benign Melanocytic Lesion
Benign Epidermal Lesion
Benign Dermal Lesion
Inflammatory Condition
Genodermatosis
Cutaneous Lymphoma
42
What is the network fixating on?
43
What does the network misclassify?
CNN Dermatologist 1 Dermatologist 2
True
Lab
el
Predicted Label Predicted Label Predicted Label
012345678
0 1 2 3 4 5 6 7 8 0 1 2 3 4 5 6 7 8 0 1 2 3 4 5 6 7 8 0
1
44
What does the network misclassify?
Dermatologist-level Classification of Skin Cancer with Deep Neural Networks
Andre Esteva*, Brett Kuprel*, Rob Novoa, Justin Ko, Susan Swetter, Helen Blau, Sebastian ThrunNature, 2017(Equal contribution authors*)
45
How can AI assist a dermatologist?
46
Community
47