explainable machine learning - 國立臺灣大學
TRANSCRIPT
![Page 1: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/1.jpg)
EXPLAINABLE MACHINE LEARNINGHung-yi Lee 李宏毅
This is a “cat”.
Because …
![Page 2: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/2.jpg)
Why we need Explainable ML?
• Correct answers ≠ Intelligent
![Page 3: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/3.jpg)
Why we need Explainable ML?
• Loan issuers are required by law to explain their models.
• Medical diagnosis model is responsible for human life. Can it be a black box?
• If a model is used at the court, we must make sure the model behaves in a nondiscriminatory manner.
• If a self-driving car suddenly acts abnormally, we need to explain why.
![Page 4: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/4.jpg)
https://www.explainxkcd.com/wiki/index.php/1838:_Machine_Learning
I know why the answers are wrong, so I can fix it.
We can improve ML model based on explanation.
With explainable ML
![Page 5: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/5.jpg)
Interpretable v.s. Powerful
• Some models are intrinsically interpretable.
• For example, linear model (from weights, you know the importance of features)
• But not very powerful.
• Deep network is difficult to interpretable. Deep networks are black boxes … but powerful than a linear model.
We don’t want to use a more powerful model because it is a black box.
This is “cut the feet to fit the shoes.” (削足適履)
![Page 6: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/6.jpg)
Simple, but interpretable model.
Source of image: https://kknews.cc/news/pnynzgp.html
Let’s make deep network explainable.
Powerful model
![Page 7: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/7.jpg)
Interpretable v.s. Powerful
• Are there some models interpretable and powerful at the same time?
• How about decision tree?
Source of image: https://mropengate.blogspot.com/2015/06/ai-ch13-2-decision-tree.html
![Page 8: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/8.jpg)
Decision tree is all you need!?
![Page 9: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/9.jpg)
Interpretable v.s. Powerful
• A tree can still be terrible!
• We use a forest!
https://stats.stackexchange.com/questions/230581/decision-tree-too-large-to-interpret
![Page 10: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/10.jpg)
Goal of Explainable ML
• Completely know how an ML model works?
• We do not completely know how brains work!
• But we trust the decision of humans!
https://jamesclear.com/wp-content/uploads/2015/03/copy-machine-study-ellen-langer.pdf
The Copy Machine Study
“Excuse me, I have 5 pages. May I use the Xerox machine?”60% accept
“Excuse me, I have 5 pages. May I use the Xerox machine, because I’m in a rush?” 94% accept
(Ellen Langer, Harvard University)
“Excuse me, I have 5 pages. May I use the Xerox machine, because I have to make copies?” 93% accept
![Page 11: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/11.jpg)
Make people (your customers, your boss, yourself) comfortable. (my two cents)
![Page 12: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/12.jpg)
Explainable ML
Classifier
This is a “cat”.
Local Explanation
Global Explanation
What does a “cat” look like?
Why do you think this image is a cat?
Because …
(not referred to a specific image)
![Page 13: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/13.jpg)
Local Explanation:
Explain the Decision
Questions: Why do you think this
image is a cat?
![Page 14: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/14.jpg)
Which component is critical?
Object 𝑥
𝑥1, ⋯ , 𝑥𝑛, ⋯ , 𝑥𝑁
Image: pixel, segment, etc.Text: a word
Components:
Which component is critical for making decision?
• Removing or modifying the components
Important component
• Large decision change
Image, text, etc.
![Page 15: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/15.jpg)
Reference: Zeiler, M. D., & Fergus, R. (2014). Visualizing and understanding
convolutional networks. In Computer Vision–ECCV 2014 (pp. 818-833)
![Page 16: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/16.jpg)
Karen Simonyan, Andrea Vedaldi, Andrew Zisserman, “Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps”, ICLR, 2014
𝑥1, ⋯ , 𝑥𝑛, ⋯ , 𝑥𝑁 𝑥1, ⋯ , 𝑥𝑛 + ∆𝑥,⋯ , 𝑥𝑁
loss of an example (the difference between model output and ground truth)
𝑒 + ∆𝑒𝑒|∆𝑒
∆𝑥| |
𝜕𝑒
𝜕𝑥𝑛|
Saliency Map
pixels
![Page 17: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/17.jpg)
Case Study: Pokémon v.s. Digimon
https://medium.com/@tyreeostevenson/teaching-a-computer-to-classify-anime-8c77bc89b881
![Page 18: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/18.jpg)
Task Pokémon images: https://www.Kaggle.com/kvpratama/pokemon-images-dataset/data
Digimon images:https://github.com/DeathReaper0965/Digimon-Generator-GAN
Pokémon Digimon
Testing Images:
![Page 19: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/19.jpg)
Experimental Results
Training Accuracy: 98.9%
Testing Accuracy: 98.4%Amazing!!!!!!
![Page 20: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/20.jpg)
Saliency Map
![Page 21: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/21.jpg)
Saliency Map
![Page 22: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/22.jpg)
What Happened?
• All the images of Pokémon are PNG, while most images of Digimon are JPEG.
Machine discriminates Pokémon and Digimon based on the background colors.
png files have transparent background
transparent background becomes black
loading the files
![Page 23: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/23.jpg)
More Examples …
This slide is from: GCPR 2017 Tutorial — W. Samek & K.-R. Müller
• PASCAL VOC 2007 data set
![Page 24: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/24.jpg)
Limitation: Noisy Gradient
Gazelle(瞪羚)
Typical SmoothGrad
SmoothGrad: Randomly add noises to the input image, get saliency maps of the noisy images, and average them.
https://arxiv.org/abs/1706.03825
![Page 25: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/25.jpg)
Limitation: Gradient Saturation
Is it an elephant?
𝜕 elephant
𝜕 length≈ 0?
Alternative: Integrated gradient (IG)
trunk's length
Gradient cannot always reflect importance
https://arxiv.org/abs/1611.02639
![Page 26: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/26.jpg)
How a network processes the input data?• Visualization
phoneme
...
Layer 1
Layer N
Layer 2
100 neurons
100 neurons
100 dims
100 dims
2 dims
2 dims
Plot on figure
PCA or t-SNE
![Page 27: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/27.jpg)
How a network processes the input data?• Visualization
Input Acoustic Feature (MFCC)
A. Mohamed, G. Hinton, and G. Penn, “Understanding how Deep Belief Networks Perform Acoustic Modelling,” in ICASSP, 2012.
Colors: speakers
8-th Hidden Layer
![Page 28: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/28.jpg)
How a network processes the input data?• Visualization
Attention
Attention is not not Explanationhttps://arxiv.org/abs/1908.04626
Attention is not Explanationhttps://arxiv.org/abs/1902.10186
![Page 29: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/29.jpg)
How a network processes the input data?• Probing
𝒙𝟒𝒙𝟑𝒙𝟐𝒙𝟏
Block
Block
Block
… … … …
BERT
POSClassifier
NERClassifier
Be careful about the classifiers you use …
Noun?Verb?
Place?Name?
![Page 30: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/30.jpg)
How a network processes the input data?• Probing
text
...
Layer 1
Layer N
Layer 2
Hi (John)
Text-to-speech (TTS)
no speaker info
Hi (unknown)
reconstruction
![Page 31: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/31.jpg)
What does a network layer hear? Analyzing hidden
representations of end-to-end ASR through speech
synthesis
https://arxiv.org/abs/1911.01102
https://youtu.be/6gtn7H-pWr8
![Page 32: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/32.jpg)
GLOBAL EXPLANATION:EXPLAIN THE WHOLE MODEL
Question: What does a “cat” look like?
![Page 33: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/33.jpg)
Convolution
Max Pooling
input
Convolution
Max Pooling
flatten
What does a filter detect?
filters
filters
3 -1 -3
-3 1 0
-3 -3 0
image 𝑋
output of filter 1Large values
Image 𝑋 contains the patterns filter 1 can detect.
Let’s create an image including the patterns.
output of filter 2
unknown
![Page 34: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/34.jpg)
Convolution
Max Pooling
input
Convolution
Max Pooling
flatten
What does a filter detect?
filters
filters
3 -1 -3
-3 1 0
-3 -3 0
image 𝑋
output of filter 1Large values
output of filter 2
𝑋∗ = 𝑎𝑟𝑔max𝑋
𝑖
𝑗
𝑎𝑖𝑗
𝑎𝑖𝑗
The image contains the patterns filter 1 can detect.
(gradient ascent)
unknown
![Page 35: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/35.jpg)
Convolution
Max Pooling
input
Convolution
Max Pooling
flatten
What does a filter detect?
filters
filters
image 𝑋
𝑋∗ for each filter
E.g., Digit classifier
![Page 36: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/36.jpg)
Convolution
Max Pooling
input
Convolution
Max Pooling
flatten
filters
filters
image 𝑋
E.g., Digit classifier
What does a digit look like for CNN?
𝑦𝑖
𝑋∗ = 𝑎𝑟𝑔max𝑋
𝑦𝑖 Can we see digits?
0 1 2
3 4 5
6 7 8
Surprise? Consider adversarial attack!
![Page 37: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/37.jpg)
The image should looks like a digit. Find the image that maximizes class probability
𝑋∗ = 𝑎𝑟𝑔max𝑋
𝑦𝑖𝑅 𝑋 = −
𝑖,𝑗
𝑋𝑖𝑗
0 1 2
3 4 5
6 7 8
𝑋∗ = 𝑎𝑟𝑔max𝑋
𝑦𝑖 + 𝑅 𝑋
How likely 𝑋 is a digit
What does a digit look like for CNN?
![Page 38: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/38.jpg)
https://arxiv.org/abs/1506.06579
With several regularization terms, and hyperparameter tuning …..
![Page 39: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/39.jpg)
Constraint from Generator
• Training a generator
Image Generator
(by GAN, VAE, etc.)
(Simplified Version)
𝑧Image𝑋
Training Examples
low-dim vector
𝑋∗ = 𝑎𝑟𝑔max𝑋
𝑦𝑖
Image Generator
𝑧Image𝑋
Image Classifier
𝑦
𝑧∗ = 𝑎𝑟𝑔max𝑧
𝑦𝑖
𝐺 𝑋 = 𝐺 𝑧
𝑋∗ = 𝐺 𝑧∗Show image:
![Page 40: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/40.jpg)
https://arxiv.org/abs/1612.00005
![Page 41: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/41.jpg)
Concluding Remarks
Classifier
This is a “cat”.
Local Explanation
Global Explanation
What does a “cat” look like?
Why do you think this image is a cat?
Because …
(not referred to a specific image)
![Page 42: EXPLAINABLE MACHINE LEARNING - 國立臺灣大學](https://reader031.vdocument.in/reader031/viewer/2022020620/61e4477c401ec1189364be95/html5/thumbnails/42.jpg)
Outlook
BlackBox
(e.g. Neural Network)
𝒙𝟏, 𝒙𝟐, ⋯ , 𝒙𝑵 𝑦1, 𝑦2, ⋯ , 𝑦𝑁
𝒙𝟏, 𝒙𝟐, ⋯ , 𝒙𝑵 𝑦1, 𝑦2, ⋯ , 𝑦𝑁
as close as possible⋯⋯
LinearModel
Using an interpretable model to mimic the behavior of an uninterpretable model.
Local Interpretable Model-Agnostic Explanations (LIME)https://youtu.be/K1mWgthGS-Ahttps://youtu.be/OjqIVSwly4k