machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4...
TRANSCRIPT
![Page 2: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/2.jpg)
2
![Page 3: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/3.jpg)
3
class
probabilities
Image credit: http://farm1.static.flickr.com/1/3375151_bf4e904a5d.jpg
weights
feature maps
What is inside the black box (filters and feature maps)?
#classes
conv1
22
4
224
55
96256
384 384
4096 4096
27
13 13
conv2
conv3 conv4
conv5
fc6 fc7
fc8
6
512
![Page 4: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/4.jpg)
4
![Page 5: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/5.jpg)
5
![Page 6: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/6.jpg)
6
conv1:
96 filters 11x11x3
22
4
224
55
96256
384 384
4096 4096
#classes
27
13 13
conv1conv2
conv3 conv4
conv5
fc6 fc7
fc8
6512
![Page 7: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/7.jpg)
7
conv1:
96 filters 11x11x3
22
4
224
55
96256
384 384
4096 4096
#classes
27
13 13
conv1conv2
conv3 conv4
conv5
fc6 fc7
fc8
6512
![Page 8: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/8.jpg)
8
conv1:
96 filters 11x11x3
22
4
224
55
96256
384 384
4096 4096
#classes
27
13 13
conv1conv2
conv3 conv4
conv5
fc6 fc7
fc8
![Page 9: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/9.jpg)
9
GOOD BADBAD BAD
too noisy too correlated
lack structure
Slide credits: Ranzatto & Lecun
![Page 10: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/10.jpg)
1. Pool5
10Girshick et al. Rich feature hierarchies for accurate object detection and semantic segmentation, CVPR’14
1. Pick a neuron at a layer
2. Record it for multiple images
3. Show the images with highest activation
value
4. See whether the images correspond to a
common concept
![Page 11: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/11.jpg)
11
🙂 Easy to implement
😞 Only qualitative analysis
Girshick et al. Rich feature hierarchies for accurate object detection and semantic segmentation, CVPR’14
![Page 12: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/12.jpg)
12Bau et al., Network Dissection: Quantifying Interpretability of Deep Visual Representations, CVPR’17
![Page 13: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/13.jpg)
13
Forward pass
Bau et al., Network Dissection: Quantifying Interpretability of Deep Visual Representations, CVPR’17
![Page 14: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/14.jpg)
14
Forward pass
A
Intersection over union
IoU = A∩BA∪B
A ∩ 𝐵 B
Bau et al., Network Dissection: Quantifying Interpretability of Deep Visual Representations, CVPR’17
![Page 15: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/15.jpg)
conv5 unit 79 car (object) IoU=0.13
conv5 unit 107 road (object) IoU=0.15
Bau et al., Network Dissection: Quantifying Interpretability of Deep Visual Representations, CVPR’17
![Page 16: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/16.jpg)
16Bau et al., Network Dissection: Quantifying Interpretability of Deep Visual Representations, CVPR’17
![Page 17: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/17.jpg)
17
22
4
3
55
96256
27
conv1
conv2
conv3
Zeiler & Fergus, Visualizing and Understanding Convolutional Networks, ECCV‘14
So far, finding correlations between a set of images and activations
![Page 18: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/18.jpg)
18
Convnet
Zeiler & Fergus, Visualizing and Understanding Convolutional Networks, ECCV‘14
![Page 19: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/19.jpg)
19
Deconvnet Convnet
Zeiler & Fergus, Visualizing and Understanding Convolutional Networks, ECCV‘14
![Page 20: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/20.jpg)
20
![Page 21: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/21.jpg)
21Zeiler & Fergus, Visualizing and Understanding Convolutional Networks, ECCV‘14
![Page 22: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/22.jpg)
22
![Page 23: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/23.jpg)
23
ReLU𝐻𝑙−1 𝐻𝑙
𝐻𝑙 = max(𝐻𝑙−1, 0)
𝑅𝑙−1 𝑅𝑙
𝑅𝑙−1 = max(𝑅𝑙 , 0)
Un-
ReLU
![Page 24: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/24.jpg)
24
𝐻𝑙−1 𝐻𝑙
Convolution
𝐻𝑙 = 𝑐𝑜𝑛𝑣(𝐻𝑙−1,𝑊𝑙)
𝑡.𝑐𝑜𝑛𝑣
𝑅𝑙−1 𝑅𝑙
Transpose
convolution
𝑅𝑙−1 = 𝑐𝑜𝑛𝑣(𝑅𝑙 ,𝑊𝑙𝑇)
𝑐𝑜𝑛𝑣
![Page 25: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/25.jpg)
25Zeiler & Fergus, Visualizing and Understanding Convolutional Networks, ECCV‘14
![Page 26: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/26.jpg)
26Zeiler & Fergus, Visualizing and Understanding Convolutional Networks, ECCV‘14
![Page 27: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/27.jpg)
27Zeiler & Fergus, Visualizing and Understanding Convolutional Networks, ECCV‘14
![Page 28: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/28.jpg)
28
𝑆𝑐 𝐼
0.9
(dog)
55
96256
384 384
4096 4096
27
13 13
conv2
conv3 conv4
conv5
fc6 fc7
fc8
conv1
Simonyan, Vedaldi and Zisserman, Deep Inside Convolutional Networks, NIPS’14
![Page 29: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/29.jpg)
29
𝑆𝑐 𝐼
0.9
(dog)
55
96256
384 384
4096 4096
27
13 13
conv2
conv3 conv4
conv5
fc6 fc7
fc8
conv1
Simonyan, Vedaldi and Zisserman, Deep Inside Convolutional Networks, NIPS’14
6
512
Question
Can we calculate influence of each pixel on the class probability?
![Page 30: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/30.jpg)
30
𝛿𝑆𝑐𝛿𝐼
𝑆𝑐 𝐼
𝛿𝑆𝑐/𝛿𝐼
0.9
(dog)
55
96256
384 384
4096 4096
27
13 13
conv2
conv3 conv4
conv5
fc6 fc7
fc8
conv1
Simonyan, Vedaldi and Zisserman, Deep Inside Convolutional Networks, NIPS’14
6
512
![Page 31: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/31.jpg)
31Simonyan, Vedaldi and Zisserman, Deep Inside Convolutional Networks, NIPS’14
![Page 32: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/32.jpg)
32
𝑅𝑙−1 = 𝑀 ⋅ 𝑅𝑙𝑅𝑙−1 = max(𝑅𝑙,0)
![Page 33: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/33.jpg)
33
𝑆𝑐 𝐼
image
? dog55
96256
384 384
4096 4096
27
13 13
conv2
conv3 conv4
conv5
fc6 fc7
fc8
conv1
Simonyan, Vedaldi and Zisserman, Deep Inside Convolutional Networks, NIPS’14
![Page 34: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/34.jpg)
34
𝑆𝑐 𝐼
init with
random
noise
𝛿𝑆𝑐/𝛿𝐼
dog55
96256
384 384
4096 4096
27
13 13
conv2
conv3 conv4
conv5
fc6 fc7
fc8
conv1
Simonyan, Vedaldi and Zisserman, Deep Inside Convolutional Networks, NIPS’14
![Page 35: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/35.jpg)
35Simonyan, Vedaldi and Zisserman, Deep Inside Convolutional Networks, NIPS’14
![Page 36: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/36.jpg)
36Mordvintsev et al, Inceptionism: Going Deeper into Neural Networks
![Page 37: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/37.jpg)
37
𝑆𝑐 𝐼
𝛿𝑆𝑐/𝛿𝐼
dog55
96256
384 384
4096 4096
27
13 13
conv2
conv3 conv4
conv5
fc6 fc7
fc8
conv1
Simonyan, Vedaldi and Zisserman, Deep Inside Convolutional Networks, NIPS’14
cat saliency
![Page 38: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/38.jpg)
Szegedy et al., Intriguing properties of neural networks, ICLR’14
Problem common to any discriminative method!
38
![Page 39: Machine learning practical · conv1 224 224 55 96 256 384 384 4096 4096 27 13 13 conv2 conv3 conv4 conv5 fc6 fc7 fc8 6 512. 4. 5. 6 conv1: 96 filters 11x11x3 224 224 55 96 256 384](https://reader035.vdocument.in/reader035/viewer/2022081410/60a17c289c285f3f6c1eeeda/html5/thumbnails/39.jpg)
39