![Page 1: Tutorial on Interpreting and Explaining Deep Models in ...heatmapping.org/slides/2018_CVPR_1.pdf · Tutorial on Interpreting and Explaining Deep Models in Computer Vision Wojciech](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec461ba17d06d7cdf35bb6f/html5/thumbnails/1.jpg)
Tutorial on Interpreting and Explaining Deep Models in Computer Vision
Wojciech Samek(Fraunhofer HHI)
Grégoire Montavon(TU Berlin)
Klaus-Robert Müller(TU Berlin)
08:30 - 09:15 Introduction KRM09:15 - 10:00 Techniques for Interpretability GM10:00 - 10:30 Coffee Break ALL10:30 - 11:15 Applications of Interpretability WS11:15 - 12:00 Further Applications and Wrap-Up KRM
![Page 2: Tutorial on Interpreting and Explaining Deep Models in ...heatmapping.org/slides/2018_CVPR_1.pdf · Tutorial on Interpreting and Explaining Deep Models in Computer Vision Wojciech](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec461ba17d06d7cdf35bb6f/html5/thumbnails/2.jpg)
Why interpretability?
![Page 3: Tutorial on Interpreting and Explaining Deep Models in ...heatmapping.org/slides/2018_CVPR_1.pdf · Tutorial on Interpreting and Explaining Deep Models in Computer Vision Wojciech](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec461ba17d06d7cdf35bb6f/html5/thumbnails/3.jpg)
Why interpretability?
![Page 4: Tutorial on Interpreting and Explaining Deep Models in ...heatmapping.org/slides/2018_CVPR_1.pdf · Tutorial on Interpreting and Explaining Deep Models in Computer Vision Wojciech](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec461ba17d06d7cdf35bb6f/html5/thumbnails/4.jpg)
Why interpretability?
![Page 5: Tutorial on Interpreting and Explaining Deep Models in ...heatmapping.org/slides/2018_CVPR_1.pdf · Tutorial on Interpreting and Explaining Deep Models in Computer Vision Wojciech](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec461ba17d06d7cdf35bb6f/html5/thumbnails/5.jpg)
Why interpretability?
![Page 6: Tutorial on Interpreting and Explaining Deep Models in ...heatmapping.org/slides/2018_CVPR_1.pdf · Tutorial on Interpreting and Explaining Deep Models in Computer Vision Wojciech](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec461ba17d06d7cdf35bb6f/html5/thumbnails/6.jpg)
Why interpretability? Insights!
![Page 7: Tutorial on Interpreting and Explaining Deep Models in ...heatmapping.org/slides/2018_CVPR_1.pdf · Tutorial on Interpreting and Explaining Deep Models in Computer Vision Wojciech](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec461ba17d06d7cdf35bb6f/html5/thumbnails/7.jpg)
Why interpretability?
![Page 8: Tutorial on Interpreting and Explaining Deep Models in ...heatmapping.org/slides/2018_CVPR_1.pdf · Tutorial on Interpreting and Explaining Deep Models in Computer Vision Wojciech](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec461ba17d06d7cdf35bb6f/html5/thumbnails/8.jpg)
Overview and Intuition for different Techniques: sensitivity, deconvolution, LRP and friends.
![Page 9: Tutorial on Interpreting and Explaining Deep Models in ...heatmapping.org/slides/2018_CVPR_1.pdf · Tutorial on Interpreting and Explaining Deep Models in Computer Vision Wojciech](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec461ba17d06d7cdf35bb6f/html5/thumbnails/9.jpg)
Understanding Deep Nets: Two Views
Understanding what mechanism the network uses to solve a problem or implement a function.
Understanding how the network relates the input to the output variables.
![Page 10: Tutorial on Interpreting and Explaining Deep Models in ...heatmapping.org/slides/2018_CVPR_1.pdf · Tutorial on Interpreting and Explaining Deep Models in Computer Vision Wojciech](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec461ba17d06d7cdf35bb6f/html5/thumbnails/10.jpg)
![Page 11: Tutorial on Interpreting and Explaining Deep Models in ...heatmapping.org/slides/2018_CVPR_1.pdf · Tutorial on Interpreting and Explaining Deep Models in Computer Vision Wojciech](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec461ba17d06d7cdf35bb6f/html5/thumbnails/11.jpg)
Approach 1: Class Prototypes
Image from Symonian’13
“How does a goose typically look like according to the neural network?”
goose
non-goose
Class prototypes
![Page 12: Tutorial on Interpreting and Explaining Deep Models in ...heatmapping.org/slides/2018_CVPR_1.pdf · Tutorial on Interpreting and Explaining Deep Models in Computer Vision Wojciech](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec461ba17d06d7cdf35bb6f/html5/thumbnails/12.jpg)
Approach 2: Individual Explanations
Images from Lapuschkin’16
“Why is a given image classified as a sheep?”
sheep
non-sheep
![Page 13: Tutorial on Interpreting and Explaining Deep Models in ...heatmapping.org/slides/2018_CVPR_1.pdf · Tutorial on Interpreting and Explaining Deep Models in Computer Vision Wojciech](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec461ba17d06d7cdf35bb6f/html5/thumbnails/13.jpg)
3. Sensitivity analysis
Sensitivity analysis: The relevance of input feature i is given by the squared partial derivative:
evidence for “car”
DNN
input
![Page 14: Tutorial on Interpreting and Explaining Deep Models in ...heatmapping.org/slides/2018_CVPR_1.pdf · Tutorial on Interpreting and Explaining Deep Models in Computer Vision Wojciech](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec461ba17d06d7cdf35bb6f/html5/thumbnails/14.jpg)
Understanding Sensitivity Analysis
Problem: sensitivity analysis does not highlight cars
Sensitivity analysis explains a variation of the function, not the function value itself.
Observation:
Sensitivity analysis:
![Page 15: Tutorial on Interpreting and Explaining Deep Models in ...heatmapping.org/slides/2018_CVPR_1.pdf · Tutorial on Interpreting and Explaining Deep Models in Computer Vision Wojciech](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec461ba17d06d7cdf35bb6f/html5/thumbnails/15.jpg)
Sensitivity Analysis Problem: Shattered Gradients
[Montufar’14, Balduzzi’17]
Input gradient (on which sensitivity analysis is based), becomes increasingly highly varying and unreliable with neural network depth.
![Page 16: Tutorial on Interpreting and Explaining Deep Models in ...heatmapping.org/slides/2018_CVPR_1.pdf · Tutorial on Interpreting and Explaining Deep Models in Computer Vision Wojciech](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec461ba17d06d7cdf35bb6f/html5/thumbnails/16.jpg)
Shattered Gradients II [Montufar’14, Balduzzi’17]
Example in [0,1]:
Input gradient (on which sensitivity analysis is based), becomes increasingly highly varying and unreliable with neural network depth.
![Page 17: Tutorial on Interpreting and Explaining Deep Models in ...heatmapping.org/slides/2018_CVPR_1.pdf · Tutorial on Interpreting and Explaining Deep Models in Computer Vision Wojciech](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec461ba17d06d7cdf35bb6f/html5/thumbnails/17.jpg)
LPR is not sensitive to gradient shattering
![Page 18: Tutorial on Interpreting and Explaining Deep Models in ...heatmapping.org/slides/2018_CVPR_1.pdf · Tutorial on Interpreting and Explaining Deep Models in Computer Vision Wojciech](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec461ba17d06d7cdf35bb6f/html5/thumbnails/18.jpg)
Explaining Neural Network Predictions
Layer-wise relevance Propagation (LRP, Bach et al 15) first method to explain nonlinear classifiers - based on generic theory (related to Taylor decomposition – deep taylor decomposition M et al 16)
- applicable to any NN with monotonous activation, BoW models, Fisher Vectors, SVMs etc.
Explanation: “Which pixels contribute how much to the classification” (Bach et al 2015) (what makes this image to be classified as a car)
Sensitivity / Saliency: “Which pixels lead to increase/decrease of prediction score when changed” (what makes this image to be classified more/less as a car) (Baehrens et al 10, Simonyan et al 14)
Cf. Deconvolution: “Matching input pattern for the classified object in the image” (Zeiler & Fergus 2014) (relation to f(x) not specified)
Each method solves a different problem!!!
![Page 19: Tutorial on Interpreting and Explaining Deep Models in ...heatmapping.org/slides/2018_CVPR_1.pdf · Tutorial on Interpreting and Explaining Deep Models in Computer Vision Wojciech](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec461ba17d06d7cdf35bb6f/html5/thumbnails/19.jpg)
Classification
cat
ladybug
dog
large activation
Explaining Neural Network Predictions
![Page 20: Tutorial on Interpreting and Explaining Deep Models in ...heatmapping.org/slides/2018_CVPR_1.pdf · Tutorial on Interpreting and Explaining Deep Models in Computer Vision Wojciech](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec461ba17d06d7cdf35bb6f/html5/thumbnails/20.jpg)
Explanation
cat
ladybug
dog
=
Initialization
Explaining Neural Network Predictions
![Page 21: Tutorial on Interpreting and Explaining Deep Models in ...heatmapping.org/slides/2018_CVPR_1.pdf · Tutorial on Interpreting and Explaining Deep Models in Computer Vision Wojciech](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec461ba17d06d7cdf35bb6f/html5/thumbnails/21.jpg)
Explanation
cat
ladybug
dog
Theoretical interpretation Deep Taylor Decomposition
?
Explaining Neural Network Predictions
depends on the activations and the weights: LRP naive z-rule
![Page 22: Tutorial on Interpreting and Explaining Deep Models in ...heatmapping.org/slides/2018_CVPR_1.pdf · Tutorial on Interpreting and Explaining Deep Models in Computer Vision Wojciech](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec461ba17d06d7cdf35bb6f/html5/thumbnails/22.jpg)
Explanation
cat
ladybug
dog
Relevance Conservation Property
Explaining Neural Network Predictions
large relevance
![Page 23: Tutorial on Interpreting and Explaining Deep Models in ...heatmapping.org/slides/2018_CVPR_1.pdf · Tutorial on Interpreting and Explaining Deep Models in Computer Vision Wojciech](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec461ba17d06d7cdf35bb6f/html5/thumbnails/23.jpg)
Gradients
LRP(Bach&et&al.,&2015)
Deep/Taylor/Decomposition(Montavon&et&al.,&2017&(arXiv&2015))
LRP/for/LSTM(Arras&et&al.,&2017)
Probabilistic/Diff(Zintgraf&et&al.,&2016)
Sensitivity(Baehrens&et&al.&2010)
Sensitivity(Simonyan&et&al.&2014)
Deconvolution(Zeiler&&&Fergus&2014)
Meaningful/Perturbations(Fong&&&Vedaldi 2017)
DeepLIFT(Shrikumar&et&al.,&2016)
Decomposition
Sensitivity(Morch&et&al.,&1995)
Gradient/vs./Decomposition(Montavon&et&al.,&2018)
Optimization
Guided/Backprop(Springenberg&et&al.&2015)
Integrated/Gradient/(Sundararajan&et&al.,&2017)
Gradient/times/input/(Shrikumar&et&al.,&2016)
PatternLRP(Kindermans&et&al.,&2017)
LIME(Ribeiro&et&al.,&2016)
Deconvolution
Understanding/the/Model
Network/Dissection(Zhou&et&al.&2017)Inverting/CNNs
(Mahendran&&&Vedaldi,&2015)
Deep/Visualization(Yosinski&et&al.,&2015)
Feature/visualization(Erhan&et&al.&2009)
Synthesis/of/preferred/inputs(Nguyen&et&al.&2016)Inverting/CNNs
(Dosovitskiy&&&Brox,&2015)
GradKCAM(Selvaraju&et&al.,&2016)
Excitation/Backprop(Zhang&et&al.,&2016)
RNN/cell/state/analysis(Karpathy&et&al.,&2015)
Historical remarks on Explaining Predictors
TCAV(Kim&et&al.&2018)