5 semi-supervised learning - github pages€¦ · 5 semi-supervised learning bvm tutorial: advanced...
TRANSCRIPT
5 Semi-Supervised LearningBVM Tutorial: Advanced Deep Learning Methods
David Zimmerer, Division of Medical Image Computing
Page202.11.16 |
AuthorDivision
Semi-Supervised Learning
2 | David Zimmerer, Division of Medical Image Computing
Page302.11.16 |
AuthorDivision
Semi-Supervised Learning
3 | David Zimmerer, Division of Medical Image Computing
Page402.11.16 |
AuthorDivision
Semi-Supervised Learning
4 | David Zimmerer, Division of Medical Image Computing
Page502.11.16 |
AuthorDivision
Semi-Supervised Learning
5 | David Zimmerer, Division of Medical Image Computing
Page602.11.16 |
AuthorDivision
Semi-Supervised Learning
6 | David Zimmerer, Division of Medical Image Computing
Page702.11.16 |
AuthorDivision
Semi-Supervised Learning
7 | David Zimmerer, Division of Medical Image Computing
Page802.11.16 |
AuthorDivision
Semi-Supervised Learning
8 | David Zimmerer, Division of Medical Image Computing
Page902.11.16 |
AuthorDivision
Semi-Supervised Learning
9 | David Zimmerer, Division of Medical Image Computing
Page1002.11.16 |
AuthorDivision
Semi-Supervised Learning
10 | David Zimmerer, Division of Medical Image Computing
Page1102.11.16 |
AuthorDivision
Semi-Supervised Learning
11 | David Zimmerer, Division of Medical Image Computing
Page1202.11.16 |
AuthorDivision
Semi-Supervised Learning
12 | David Zimmerer, Division of Medical Image Computing
Page1302.11.16 |
AuthorDivision
Semi-Supervised Learning
13 | David Zimmerer, Division of Medical Image Computing
Page1402.11.16 |
AuthorDivision
Semi-Supervised Learning
14 | David Zimmerer, Division of Medical Image Computing
Page1502.11.16 |
AuthorDivision
Semi-Supervised Learning
15 | David Zimmerer, Division of Medical Image Computing
Page1602.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
16 | David Zimmerer, Division of Medical Image Computing
Page1702.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
17 | David Zimmerer, Division of Medical Image Computing
x
z
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page1802.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
18 | David Zimmerer, Division of Medical Image Computing
x
z
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page1902.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
19 | David Zimmerer, Division of Medical Image Computing
x
z
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page2002.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
20 | David Zimmerer, Division of Medical Image Computing
x
z
x
μ
σ
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page2102.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
21 | David Zimmerer, Division of Medical Image Computing
x
z
x
μ
σz
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page2202.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
22 | David Zimmerer, Division of Medical Image Computing
x
z
x
μ
σ
x’
z
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page2302.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
23 | David Zimmerer, Division of Medical Image Computing
x
z
x
μ
σ
x’
z
MSE
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page2402.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
24 | David Zimmerer, Division of Medical Image Computing
x
z
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page2502.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
25 | David Zimmerer, Division of Medical Image Computing
x
z
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page2602.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
26 | David Zimmerer, Division of Medical Image Computing
x
zy
National Park
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page2702.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
27 | David Zimmerer, Division of Medical Image Computing
x
zy
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page2802.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
28 | David Zimmerer, Division of Medical Image Computing
x
zy
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page2902.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
29 | David Zimmerer, Division of Medical Image Computing
x
zy
→ Unlabeled Data
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page3002.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
30 | David Zimmerer, Division of Medical Image Computing
x
zy
x
y’
→ Unlabeled Data
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page3102.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
31 | David Zimmerer, Division of Medical Image Computing
x
zy
x
μ
σ
y’
→ Unlabeled Data
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page3202.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
32 | David Zimmerer, Division of Medical Image Computing
x
zy
x
μ
σz
y’
→ Unlabeled Data
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page3302.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
33 | David Zimmerer, Division of Medical Image Computing
x
zy
x
μ
σ
x’
z
y’
→ Unlabeled Data
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page3402.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
34 | David Zimmerer, Division of Medical Image Computing
x
zy
x
μ
σ
x’
z
MSE
y’
→ Unlabeled Data
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page3502.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
35 | David Zimmerer, Division of Medical Image Computing
x
zy
→ Labeled Data
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page3602.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
36 | David Zimmerer, Division of Medical Image Computing
x
zy
x
y
→ Labeled Data
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page3702.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
37 | David Zimmerer, Division of Medical Image Computing
x
zy
x
yy’
→ Labeled Data
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page3802.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
38 | David Zimmerer, Division of Medical Image Computing
x
zy
x
μ
σ
yy’
→ Labeled Data
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page3902.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
39 | David Zimmerer, Division of Medical Image Computing
x
zy
x
μ
σz
yy’
→ Labeled Data
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page4002.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
40 | David Zimmerer, Division of Medical Image Computing
x
zy
x
μ
σ
x’
z
yy’
→ Labeled Data
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page4102.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
41 | David Zimmerer, Division of Medical Image Computing
x
zy
x
μ
σ
x’
z
yy’
MSE
CE
→ Labeled Data
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page4202.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
42 | David Zimmerer, Division of Medical Image Computing
x
zy
→ Labeled Data
x
μ
σ
x’
z
yy’
x
μ
σ
x’
z
y’
→ Unlabeled Data
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page4302.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
43 | David Zimmerer, Division of Medical Image Computing
y
z
KNN VAE + KNN Semi-Sup. VAE
22.07 34.37 63.98
Classification Accuracy on the SVHN dataset with 1000 labels Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page4402.11.16 |
AuthorDivision
Semi-Supervised Variational Autoencoder
44 | David Zimmerer, Division of Medical Image Computing
y
z
KNN VAE + KNN Semi-Sup. VAE
22.07 34.37 63.98
Classification Accuracy on the SVHN dataset with 1000 labels Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Page4502.11.16 |
AuthorDivision
Self-Supervised Learning
45 | David Zimmerer, Division of Medical Image Computing
Zhang, Richard, et al.. "Colorful image colorization." European Conference on Computer Vision. Springer, Cham, 2016.Ross, Tobias, et al. "Exploiting the potential of unlabeled endoscopic video data with self-supervised learning." arXiv preprint arXiv:1711.09726 (2017).
Page4602.11.16 |
AuthorDivision
Self-Supervised Learning
46 | David Zimmerer, Division of Medical Image Computing
Zhang, Richard, et al.. "Colorful image colorization." European Conference on Computer Vision. Springer, Cham, 2016.Ross, Tobias, et al. "Exploiting the potential of unlabeled endoscopic video data with self-supervised learning." arXiv preprint arXiv:1711.09726 (2017).
Page4702.11.16 |
AuthorDivision
Self-Supervised Learning
47 | David Zimmerer, Division of Medical Image Computing
Labels ???
Zhang, Richard, et al.. "Colorful image colorization." European Conference on Computer Vision. Springer, Cham, 2016.Ross, Tobias, et al. "Exploiting the potential of unlabeled endoscopic video data with self-supervised learning." arXiv preprint arXiv:1711.09726 (2017).
Lin, Tsung-Yi, et al. "Microsoft coco: Common objects in context." European conference on computer vision. Springer, Cham, 2014.
Page4802.11.16 |
AuthorDivision
Self-Supervised Learning
→ Use an auxiliary task for unlabeled data
48 | David Zimmerer, Division of Medical Image Computing
Page4902.11.16 |
AuthorDivision
Self-Supervised Learning
→ Use an auxiliary task for unlabeled data
Semi-Supervised Variational Autoencoder ?
Problem: Reconstruction is a “weak” task
49 | David Zimmerer, Division of Medical Image Computing
Page5002.11.16 |
AuthorDivision
Self-Supervised Learning
→ Use an auxiliary task for unlabeled data
Semi-Supervised Variational Autoencoder ?
Problem: Reconstruction is a “weak” task
→ What is a good auxiliary task ?
50 | David Zimmerer, Division of Medical Image Computing
Page5102.11.16 |
AuthorDivision
Self-Supervised Learning - Context Encoders
→ Use inpainting as auxiliary task
51 | David Zimmerer, Division of Medical Image Computing
Pathak, Deepak, et al. "Context encoders: Feature learning by inpainting." Proceedings of the IEEE CVPR. 2016.
Page5202.11.16 |
AuthorDivision
Self-Supervised Learning - Context Encoders
→ Use inpainting as auxiliary task
52 | David Zimmerer, Division of Medical Image Computing
Pathak, Deepak, et al. "Context encoders: Feature learning by inpainting." Proceedings of the IEEE CVPR. 2016.
Page5302.11.16 |
AuthorDivision
Self-Supervised Learning - Context Encoders
→ Use inpainting as auxiliary task
53 | David Zimmerer, Division of Medical Image Computing
Pathak, Deepak, et al. "Context encoders: Feature learning by inpainting." Proceedings of the IEEE CVPR. 2016.
Page5402.11.16 |
AuthorDivision
Self-Supervised Learning - Context Encoders
54 | David Zimmerer, Division of Medical Image Computing
Pathak, Deepak, et al. "Context encoders: Feature learning by inpainting." Proceedings of the IEEE CVPR. 2016.
Page5502.11.16 |
AuthorDivision
Self-Supervised Learning - Context Encoders
55 | David Zimmerer, Division of Medical Image Computing
ImageNet Pretrained Random Initialization Autoencoder Context Encoders
78.2 53.3 53.8 56.5
Classification Accuracy on the Pascal VOC dataset with different pretraining methods
Pathak, Deepak, et al. "Context encoders: Feature learning by inpainting." Proceedings of the IEEE CVPR. 2016.
Page5602.11.16 |
AuthorDivision
Self-Supervised Learning - Image Recolorization
→ Use image recoloriation as auxiliary task
56 | David Zimmerer, Division of Medical Image Computing
Zhang, Richard, et al.. "Colorful image colorization." European Conference on Computer Vision. Springer, Cham, 2016.
Page5702.11.16 |
AuthorDivision
Self-Supervised Learning - Image Recolorization
→ Use image recoloriation as auxiliary task
57 | David Zimmerer, Division of Medical Image Computing
Zhang, Richard, et al.. "Colorful image colorization." European Conference on Computer Vision. Springer, Cham, 2016.
Page5802.11.16 |
AuthorDivision
Self-Supervised Learning - Image Recolorization
58 | David Zimmerer, Division of Medical Image Computing
Zhang, Richard, et al.. "Colorful image colorization." European Conference on Computer Vision. Springer, Cham, 2016.
Page5902.11.16 |
AuthorDivision
Self-Supervised Learning - Image Recolorization
59 | David Zimmerer, Division of Medical Image Computing
ImageNet Pretrained Random Initialization Autoencoder Recolorization
79.9 53.3 53.8 67.1
Classification Accuracy on the Pascal VOC dataset with different pretraining methods
Zhang, Richard, et al.. "Colorful image colorization." European Conference on Computer Vision. Springer, Cham, 2016.Zhang, Richard, et al.. "Split-brain autoencoders: Unsupervised learning by cross-channel prediction." CVPR. Vol. 1. No. 2. 2017.
Page6002.11.16 |
AuthorDivision
Self-Supervised Learning - Image Recolorization
→ Application to medical
60 | David Zimmerer, Division of Medical Image Computing
Adversarial Discriminator
EncoderRoss, Tobias, et al. "Exploiting the potential of unlabeled endoscopic video data with self-supervised learning." arXiv preprint arXiv:1711.09726 (2017).
Page6102.11.16 |
AuthorDivision
Self-Supervised Learning - Image Recolorization
→ Application to medical
61 | David Zimmerer, Division of Medical Image Computing
Adversarial Discriminator
EncoderRoss, Tobias, et al. "Exploiting the potential of unlabeled endoscopic video data with self-supervised learning." arXiv preprint arXiv:1711.09726 (2017).
Page6202.11.16 |
AuthorDivision
Self-Supervised Learning - Image Recolorization
→ Application to medical
62 | David Zimmerer, Division of Medical Image Computing
Adversarial Discriminator
EncoderRoss, Tobias, et al. "Exploiting the potential of unlabeled endoscopic video data with self-supervised learning." arXiv preprint arXiv:1711.09726 (2017).
Page6302.11.16 |
AuthorDivision
Self-Supervised Learning - Image Recolorization
63 | David Zimmerer, Division of Medical Image Computing
Ross, Tobias, et al. "Exploiting the potential of unlabeled endoscopic video data with self-supervised learning." arXiv preprint arXiv:1711.09726 (2017).
Page6402.11.16 |
AuthorDivision
Self-Supervised Learning - Image Recolorization
64 | David Zimmerer, Division of Medical Image Computing
Dice
Sco
re
Fraction of data
better
Ross, Tobias, et al. "Exploiting the potential of unlabeled endoscopic video data with self-supervised learning." arXiv preprint arXiv:1711.09726 (2017).
Page6502.11.16 |
AuthorDivision
Self-Supervised Learning - Medical Applications
→ Medical Images
65 | David Zimmerer, Division of Medical Image Computing
Cai, Yunliang, et al."Multi-modality vertebra recognition in arbitrary views using 3D deformable hierarchical model." IEEE transactions on medical imaging 2015
Zhang, Pengyue, et al. "Self supervised deep representation learning for fine-grained body part recognition." Biomedical Imaging (ISBI 2017), 2017 IEEE 14th International Symposium on. IEEE, 2017.
Page6602.11.16 |
AuthorDivision
Self-Supervised Learning - Medical Applications
→ Sorting body part (from top to toe)
66 | David Zimmerer, Division of Medical Image Computing
Zhang, Pengyue, et al. "Self supervised deep representation learning for fine-grained body part recognition." Biomedical Imaging (ISBI 2017), 2017 IEEE 14th International Symposium on. IEEE, 2017.
Page6702.11.16 |
AuthorDivision
Self-Supervised Learning - Medical Applications
→ Sorting body part (from top to toe)
67 | David Zimmerer, Division of Medical Image Computing
Zhang, Pengyue, et al. "Self supervised deep representation learning for fine-grained body part recognition." Biomedical Imaging (ISBI 2017), 2017 IEEE 14th International Symposium on. IEEE, 2017.
Page6802.11.16 |
AuthorDivision
Self-Supervised Learning - Medical Applications
68 | David Zimmerer, Division of Medical Image Computing
Recognition error (in mm) of body part recognition on CT and MR data
Image 1 Image 2
proposed ordering
Zhang, Pengyue, et al. "Self supervised deep representation learning for fine-grained body part recognition." Biomedical Imaging (ISBI 2017), 2017 IEEE 14th International
Symposium on. IEEE, 2017.
Page6902.11.16 |
AuthorDivision
Self-Supervised Learning - Medical Applications
69 | David Zimmerer, Division of Medical Image Computing
Recognition error (in mm) of body part recognition on CT and MR data(lower is better)
Zhang, Pengyue, et al. "Self supervised deep representation learning for fine-grained body part recognition." Biomedical Imaging (ISBI 2017), 2017 IEEE 14th International
Symposium on. IEEE, 2017.
Page7002.11.16 |
AuthorDivision
Self-Supervised Learning - Medical Applications
70 | David Zimmerer, Division of Medical Image Computing
Recognition error (in mm) of body part recognition on CT and MR data(lower is better)
Zhang, Pengyue, et al. "Self supervised deep representation learning for fine-grained body part recognition." Biomedical Imaging (ISBI 2017), 2017 IEEE 14th International
Symposium on. IEEE, 2017.
Page7102.11.16 |
AuthorDivision
Self-Supervised Learning - Medical Applications
71 | David Zimmerer, Division of Medical Image Computing
Recognition error (in mm) of body part recognition on CT and MR data(lower is better)
Zhang, Pengyue, et al. "Self supervised deep representation learning for fine-grained body part recognition." Biomedical Imaging (ISBI 2017), 2017 IEEE 14th International
Symposium on. IEEE, 2017.
Page7202.11.16 |
AuthorDivision
Self-Supervised Learning - Medical Applications
→ Differentiate between subjects
72 | David Zimmerer, Division of Medical Image Computing
Jamaludin, Amir, et al.. "Self-Supervised Learning for Spinal MRIs." Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. Springer, Cham, 2017. 294-302.
Page7302.11.16 |
AuthorDivision
Self-Supervised Learning - Medical Applications
→ Differentiate between subjects
73 | David Zimmerer, Division of Medical Image Computing
Jamaludin, Amir, et al.. "Self-Supervised Learning for Spinal MRIs." Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. Springer, Cham, 2017. 294-302.
Page7402.11.16 |
AuthorDivision
Self-Supervised Learning - Medical Applications
→ Differentiate between subjects
74 | David Zimmerer, Division of Medical Image Computing
Jamaludin, Amir, et al.. "Self-Supervised Learning for Spinal MRIs." Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. Springer, Cham, 2017. 294-302.
Page7502.11.16 |
AuthorDivision
Self-Supervised Learning - Medical Applications
75 | David Zimmerer, Division of Medical Image Computing
Clas
sific
atio
n Ac
cura
cy
Number of Scans
better
Jamaludin, Amir, et al.. "Self-Supervised Learning for Spinal MRIs." Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. Springer, Cham, 2017. 294-302.
Page7602.11.16 |
AuthorDivision
Semi-Supervised Learning -
“Mean teachers are better role models”
76 | David Zimmerer, Division of Medical Image Computing
Tarvainen, Antti, and Harri Valpola. "Weight-averaged consistency targets improve semi-supervised deep learning results."arXiv:1703.01780 (2017).
Page7702.11.16 |
AuthorDivision
Semi-Supervised Learning -
“Mean teachers are better role models”
77 | David Zimmerer, Division of Medical Image Computing
Student Teacher
Tarvainen, Antti, and Harri Valpola. "Weight-averaged consistency targets improve semi-supervised deep learning results."arXiv:1703.01780 (2017).
Page7802.11.16 |
AuthorDivision
Semi-Supervised Learning -
“Mean teachers are better role models”
78 | David Zimmerer, Division of Medical Image Computing
Student Teacher
Input
Tarvainen, Antti, and Harri Valpola. "Weight-averaged consistency targets improve semi-supervised deep learning results."arXiv:1703.01780 (2017).
Page7902.11.16 |
AuthorDivision
Semi-Supervised Learning -
“Mean teachers are better role models”
79 | David Zimmerer, Division of Medical Image Computing
Student Teacher
Input
cat dog
Tarvainen, Antti, and Harri Valpola. "Weight-averaged consistency targets improve semi-supervised deep learning results."arXiv:1703.01780 (2017).
Page8002.11.16 |
AuthorDivision
Semi-Supervised Learning -
“Mean teachers are better role models”
80 | David Zimmerer, Division of Medical Image Computing
Student Teacher
Input
cat dogMSE
Tarvainen, Antti, and Harri Valpola. "Weight-averaged consistency targets improve semi-supervised deep learning results."arXiv:1703.01780 (2017).
Page8102.11.16 |
AuthorDivision
Semi-Supervised Learning -
“Mean teachers are better role models”
81 | David Zimmerer, Division of Medical Image Computing
Student Teacher
Input
cat dogdog
LabelMSE
Tarvainen, Antti, and Harri Valpola. "Weight-averaged consistency targets improve semi-supervised deep learning results."arXiv:1703.01780 (2017).
Page8202.11.16 |
AuthorDivision
Semi-Supervised Learning -
“Mean teachers are better role models”
82 | David Zimmerer, Division of Medical Image Computing
Student Teacher
Input
cat dogdog
Label
CE MSE
Tarvainen, Antti, and Harri Valpola. "Weight-averaged consistency targets improve semi-supervised deep learning results."arXiv:1703.01780 (2017).
Page8302.11.16 |
AuthorDivision
Semi-Supervised Learning -
“Mean teachers are better role models”
83 | David Zimmerer, Division of Medical Image Computing
Old StudentNew Student Error Gradient
+=
Tarvainen, Antti, and Harri Valpola. "Weight-averaged consistency targets improve semi-supervised deep learning results."arXiv:1703.01780 (2017).
Page8402.11.16 |
AuthorDivision
Semi-Supervised Learning -
“Mean teachers are better role models”
84 | David Zimmerer, Division of Medical Image Computing
Teacher
Old Students
=
Weighted Sum
Tarvainen, Antti, and Harri Valpola. "Weight-averaged consistency targets improve semi-supervised deep learning results."arXiv:1703.01780 (2017).
Page8502.11.16 |
AuthorDivision
85 | David Zimmerer, Division of Medical Image Computing
Semi-Supervised Learning -
“Mean teachers are better role models”
Top-5 validation error on Imagenet with 10% of the labels (lower is better)
Tarvainen, Antti, and Harri Valpola. "Weight-averaged consistency targets improve semi-supervised deep learning results."arXiv:1703.01780 (2017).
Page8602.11.16 |
AuthorDivision
Thanks !
86 | David Zimmerer, Division of Medical Image Computing
Page8702.11.16 |
AuthorDivision
References
Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.
Pathak, Deepak, et al. "Context encoders: Feature learning by inpainting." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016.
Zhang, Richard, et al.. "Colorful image colorization." European Conference on Computer Vision. Springer, Cham, 2016.
Zhang, Richard, et al.. "Split-brain autoencoders: Unsupervised learning by cross-channel prediction." CVPR. Vol. 1. No. 2. 2017.
Ross, Tobias, et al. "Exploiting the potential of unlabeled endoscopic video data with self-supervised learning." arXiv preprint arXiv:1711.09726 (2017).
Jamaludin, Amir, et al.. "Self-Supervised Learning for Spinal MRIs." Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. Springer, Cham, 2017. 294-302.
Zhang, Pengyue, et al. "Self supervised deep representation learning for fine-grained body part recognition." Biomedical Imaging (ISBI 2017), 2017 IEEE 14th International Symposium on. IEEE, 2017.
Tarvainen, Antti, and Harri Valpola. "Weight-averaged consistency targets improve semi-supervised deep learning results." arXiv preprint arXiv:1703.01780 (2017).
Cai, Yunliang, et al. "Multi-modality vertebra recognition in arbitrary views using 3D deformable hierarchical model." IEEE transactions on medical imaging 34.8 (2015): 1676-1693.
Lin, Tsung-Yi, et al. "Microsoft coco: Common objects in context." European conference on computer vision. Springer, Cham, 2014.
87 | David Zimmerer, Division of Medical Image Computing