lecture 10: convolutional neural networks · cs231n: convolutional neural networks k. simonyan and...
TRANSCRIPT
Lecture 10: Convolutional Neural Networks
Zerrin YumakUtrecht University
Announcements
• Project proposal deadline is extended!
• New project proposal deadline: 23 Dec Monday 23:59pm
• We will look at your proposals and give feedback by email.
• If you have any questions, you can ask for an appointment afterChristmas holidays (meetings will happen during lecture hours)
• Not mandatory, please ask for an appointment 3-4 days in advance.
In this lecture
• A bit of history• CNN basics
• Convolutions, strides, pooling• CNN architectures
• AlexNext• ZFNet• VGGNet• GoogLeNet• Residual Networks
• Applications of CNN• Object detection• Object segmentation• …
A bit of history
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
A bit of history
CS231n: Convolutional Neural Networks
A bit of history
CS231n: Convolutional Neural Networks
A bit of history
CS231n: Convolutional Neural Networks
CNNs are widely used now..
CS231n: Convolutional Neural Networks
Images are numbers
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
Fully Connected Neural Network
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
Using Spatial Structure
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
Using Spatial Structure
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
Feature Extraction with Convolution
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
Feature Extraction and Convolution
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
Features of X
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
Filters to Detect X Features
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
The Convolution Operation
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
The Convolution Operation
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
The Convolution Operation
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
The Convolution Operation
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
The Convolution Operation
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
Producing Feature Maps
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
CNNs for Classification
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
Convolutional Layers: Local Connectivity
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
CNNs: Spatial Arrangement of Output Volume
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
Introducing Non-Linearity
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
Pooling
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
Pooling
CS231n: Convolutional Neural Networks
Convolution Layer
CS231n: Convolutional Neural Networks
Convolution Layer
CS231n: Convolutional Neural Networks
Convolution Layer
CS231n: Convolutional Neural Networks
Convolution Layer
CS231n: Convolutional Neural Networks
Convolution Layer
CS231n: Convolutional Neural Networks
Convolution Layer
CS231n: Convolutional Neural Networks
Stride
CS231n: Convolutional Neural Networks
Stride
CS231n: Convolutional Neural Networks
Stride
CS231n: Convolutional Neural Networks
Stride
CS231n: Convolutional Neural Networks
Stride
CS231n: Convolutional Neural Networks
Stride
CS231n: Convolutional Neural Networks
Stride
CS231n: Convolutional Neural Networks
Stride
CS231n: Convolutional Neural Networks
In practice: Common to zero pad the border
CS231n: Convolutional Neural Networks
In practice: Common to zero pad the border
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
[ConvNetJS demo: training on CIFAR-10]
CS231n: Convolutional Neural Networks
CNNs for Classification: Feature Learning
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
CNNs for Classification: Class Probabilities
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
CNNs: Training with Backpropagation
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
ImageNet Dataset
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
ImageNet Challenge
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
ImageNet Challenge: Classification Task
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
ImageNet Challenge: Classification Task
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2017. ImageNet classification with deep convolutional neural networks. Commun. ACM 60, 6 (May 2017), 84-90
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
ImageNet Large Scale Visual Recognition Challenge (ILSVRC) winners
CS231n: Convolutional Neural Networks
ImageNet Large Scale Visual Recognition Challenge (ILSVRC) winners
M. D. Zeiler and R. Fergus, Visualizing and Understanding Convolutional Networks, European conference on computer vision, 818-833, 2013CS231n: Convolutional Neural Networks
ImageNet Large Scale Visual Recognition Challenge (ILSVRC) winners
CS231n: Convolutional Neural Networks
K. Simonyan and A. Zisserman, Very Deep Convolutional Networks for Large-Scale Image Recognition, Computer Vision and Pattern Recognition, 2014
CS231n: Convolutional Neural Networks
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelv, D. Erhan, V. Vanhoucke, A. Rabinovich, Going deeper with convolutions, IEEE Conference on Computer Vision and PatternRecognition, 2015
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
ImageNet Large Scale Visual Recognition Challenge (ILSVRC) winners
CS231n: Convolutional Neural Networks
K. He, X. Zhang, S. Ren, J. Sun, Deep Residual Learning for Image Recognition, IEEE Conference on Computer Vision and Pattern Recognition, 2016
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
Comparing complexity
CS231n: Convolutional Neural Networks
DataDo not use MNIST in your projects! Use it only for practicing with Tensorflow.
© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
So far: Image Classification
CS231n: Convolutional Neural Networks
Other Computer Vision Tasks
CS231n: Convolutional Neural Networks
Semantic Segmentation
CS231n: Convolutional Neural Networks
Semantic Segmentation: FCNs
Long, Shelhamer, and Darrell, “Fully Convolutional Networks for Semantic Segmentation”, CVPR 2015 Noh et al, “Learning Deconvolution Network for Semantic Segmentation”, ICCV 2015 © MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
Classification + Localization
CS231n: Convolutional Neural Networks
Classification + Localization
CS231n: Convolutional Neural Networks
Human Pose Estimation
Johnson and Everingham, "Clustered Pose and Nonlinear Appearance Models for Human Pose Estimation", BMVC 2010Toshev and Szegedy, “DeepPose: Human Pose Estimation via Deep Neural Networks”, CVPR 2014
CS231n: Convolutional Neural Networks
Human Pose Estimation
CS231n: Convolutional Neural Networks
Object Detection/Instance Segmentation
Girshick et al, “Rich feature hierarchies for accurate object detection and semantic segmentation”, CVPR 2014Redmon et al, “You Only Look Once: Unified, Real-Time Object Detection”, CVPR 2016
He et al, “Mask R-CNN”, arXiv 2017© MIT 6.S191: Introduction to Deep LearningIntroToDeepLearning.com
Image Captioning using RNNs
CS231n: Convolutional Neural Networks
Image Captioning using RNNs
A. Karpathy, L. Fei-Fei, Deep Visual-Semantic Alignments for Generating Image Descriptions, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017https://cs.stanford.edu/people/karpathy/deepimagesent/
CS231n: Convolutional Neural Networks
Visualizing Convolutional Networks
http://www.cs.cmu.edu/~aharley/vis/For more info: http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture12.pdf © MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
For your projects..• Use research papers mentioned in the slides and other papers you found
• There are links in the slides and on the project website • An interesting source is https://paperswithcode.com
• Do your research and investigate what is possible and what you like to do
• Choose a problem• e.g. object classification, object detection, segmentation, face detection/recognition, pose estimation, image/video
captioning• You can also choose more advanced problems such as GANs
• Choose a dataset• e.g. CIFAR, ImageNet, MSCoco (Do not use MNIST)
• Modify/reimplement different architectures• e.g. AlexNet, GoogLeNet, ResNet etc.
• Try different hyperparameters
• More challenging, state-of-the-art problems and in depth analysis result in higher grade!
Supplementary material and references
• Deep Learning book, Chapter 9
• CS231N: Convolutional Neural Networks, Stanford University • http://cs231n.stanford.edu/2017/syllabus
• 6.S191: Introduction to Deep Learning, MIT• http://introtodeeplearning.com/2018/#schedule
• Coursera Deeplearning.ai on YouTube: https://www.youtube.com/channel/UCcIXc5mJsHVYTZR1maL5l9w/videos
• (Slides are mainly adopted from the above courses)