a critical analysis of self-supervision, or what we can ... · unsupervised representation learning...
TRANSCRIPT
![Page 1: A critical analysis of self-supervision, or what we can ... · Unsupervised representation learning by predicting image rotations S. Gidaris, P. Singh, and N. Komodakis ICLR, 2018](https://reader036.vdocument.in/reader036/viewer/2022070710/5ec5aaf7bd278d405c141f31/html5/thumbnails/1.jpg)
A critical analysis of self-supervision, or what we can learn from a single image
Yuki M. Asano
CDT Annual Meeting Oct 2019
work with Christian Rupprecht and Andrea Vedaldi at VGG
![Page 2: A critical analysis of self-supervision, or what we can ... · Unsupervised representation learning by predicting image rotations S. Gidaris, P. Singh, and N. Komodakis ICLR, 2018](https://reader036.vdocument.in/reader036/viewer/2022070710/5ec5aaf7bd278d405c141f31/html5/thumbnails/2.jpg)
Outline
• Self-supervised learning saga
•Or is it?
![Page 3: A critical analysis of self-supervision, or what we can ... · Unsupervised representation learning by predicting image rotations S. Gidaris, P. Singh, and N. Komodakis ICLR, 2018](https://reader036.vdocument.in/reader036/viewer/2022070710/5ec5aaf7bd278d405c141f31/html5/thumbnails/3.jpg)
Self-supervised learning like we do
1. Unlabelled, large collection of images
2. Train your network without labels
3. Use the image representations (vectors) for new tasks
![Page 4: A critical analysis of self-supervision, or what we can ... · Unsupervised representation learning by predicting image rotations S. Gidaris, P. Singh, and N. Komodakis ICLR, 2018](https://reader036.vdocument.in/reader036/viewer/2022070710/5ec5aaf7bd278d405c141f31/html5/thumbnails/4.jpg)
Self-supervised learning like we do?
CNN
Unlabelled data
+ transformations
Proxy
task/Loss
e.g. DeepCluster
• Run k-means on features
• Train classifier on k classes
• Repeat for 200 epochs
Deepclustering for unsupervised learning of visual featuresM. Caron, P. Bojanowski, A. Joulin, and M. Douze
ECCV, 2018
e.g. RotNet
• Create 4 classes based on rotations
• Exploits photographer bias
• Simple but works
Unsupervised representation learning by predicting image rotations S. Gidaris, P. Singh, and N. Komodakis
ICLR, 2018
![Page 5: A critical analysis of self-supervision, or what we can ... · Unsupervised representation learning by predicting image rotations S. Gidaris, P. Singh, and N. Komodakis ICLR, 2018](https://reader036.vdocument.in/reader036/viewer/2022070710/5ec5aaf7bd278d405c141f31/html5/thumbnails/5.jpg)
Or colorizing images
Zhang, Isola, Efros.Colorful Image Colorization.
In ECCV, 2016
![Page 6: A critical analysis of self-supervision, or what we can ... · Unsupervised representation learning by predicting image rotations S. Gidaris, P. Singh, and N. Komodakis ICLR, 2018](https://reader036.vdocument.in/reader036/viewer/2022070710/5ec5aaf7bd278d405c141f31/html5/thumbnails/6.jpg)
Hypothesis
What/how humans learn“Priors”
Transformations ?
![Page 7: A critical analysis of self-supervision, or what we can ... · Unsupervised representation learning by predicting image rotations S. Gidaris, P. Singh, and N. Komodakis ICLR, 2018](https://reader036.vdocument.in/reader036/viewer/2022070710/5ec5aaf7bd278d405c141f31/html5/thumbnails/7.jpg)
Getting there, but not quite yet
![Page 8: A critical analysis of self-supervision, or what we can ... · Unsupervised representation learning by predicting image rotations S. Gidaris, P. Singh, and N. Komodakis ICLR, 2018](https://reader036.vdocument.in/reader036/viewer/2022070710/5ec5aaf7bd278d405c141f31/html5/thumbnails/8.jpg)
Where are we?
What/how humans learn“Priors”
Transformations ?
![Page 9: A critical analysis of self-supervision, or what we can ... · Unsupervised representation learning by predicting image rotations S. Gidaris, P. Singh, and N. Komodakis ICLR, 2018](https://reader036.vdocument.in/reader036/viewer/2022070710/5ec5aaf7bd278d405c141f31/html5/thumbnails/9.jpg)
“Learn” from one image… using multiple transformations
A critical analysis of self-supervision, or what we can learn from a single imageYM Asano, C Rupprecht, A Vedaldi
arXiv 1904.13132
![Page 10: A critical analysis of self-supervision, or what we can ... · Unsupervised representation learning by predicting image rotations S. Gidaris, P. Singh, and N. Komodakis ICLR, 2018](https://reader036.vdocument.in/reader036/viewer/2022070710/5ec5aaf7bd278d405c141f31/html5/thumbnails/10.jpg)
Learned first convolutional layer – from one image
![Page 11: A critical analysis of self-supervision, or what we can ... · Unsupervised representation learning by predicting image rotations S. Gidaris, P. Singh, and N. Komodakis ICLR, 2018](https://reader036.vdocument.in/reader036/viewer/2022070710/5ec5aaf7bd278d405c141f31/html5/thumbnails/11.jpg)
Performance
11.6
17.1 16.9 16.314.1
20.7
31.5 32.5
28.5
21
18
32.5
39.237.2
30.6
19.3
36.3
44.2
48.350.5
Conv1 Conv2 Conv3 Conv4 Conv5
Comparison of random, DeepCluster (1 & 1M images) and supervised
Random 1-image 1M images Supervised
![Page 12: A critical analysis of self-supervision, or what we can ... · Unsupervised representation learning by predicting image rotations S. Gidaris, P. Singh, and N. Komodakis ICLR, 2018](https://reader036.vdocument.in/reader036/viewer/2022070710/5ec5aaf7bd278d405c141f31/html5/thumbnails/12.jpg)
Conclusion
1. Early layers of deep networks contain limited information about natural images
2. These can be learned through self-supervision or supervised learning
3. Notably, only one image + transformations are necessary for this
4. Much space to go the right direction in self-supervised learning
![Page 14: A critical analysis of self-supervision, or what we can ... · Unsupervised representation learning by predicting image rotations S. Gidaris, P. Singh, and N. Komodakis ICLR, 2018](https://reader036.vdocument.in/reader036/viewer/2022070710/5ec5aaf7bd278d405c141f31/html5/thumbnails/14.jpg)
Appendix
![Page 15: A critical analysis of self-supervision, or what we can ... · Unsupervised representation learning by predicting image rotations S. Gidaris, P. Singh, and N. Komodakis ICLR, 2018](https://reader036.vdocument.in/reader036/viewer/2022070710/5ec5aaf7bd278d405c141f31/html5/thumbnails/15.jpg)