cs376 computer vision lecture 20: deep learning basicshuangqx/cs376_lecture_20.pdf · cs376...
TRANSCRIPT
![Page 1: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/1.jpg)
CS376 Computer Vision Lecture 20: Deep Learning Basics
Qixing Huang
April 10th 2019
Slide material from: http://cs231n.stanford.edu/slides/2018/cs231n_2018_lecture04.pdf
http://cs231n.stanford.edu/slides/2018/cs231n_2018_lecture05.pdf
![Page 2: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/2.jpg)
Today’s Topic
• Neural Networks
– Activation functions
– Fully connected
– Convolutional neural networks
• Optimization
• Back-propagation
![Page 3: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/3.jpg)
Neural Networks
![Page 4: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/4.jpg)
From linear to multi-layer
![Page 5: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/5.jpg)
From linear to multi-layer
![Page 6: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/6.jpg)
Activation functions
![Page 7: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/7.jpg)
Neural Networks: Architectures
![Page 8: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/8.jpg)
Summary
• We arrange neurons into fully-connected layers
• The abstraction of a layer has the nice property that it allows us to use efficient vectorized code (e.g. matrix multiplies)
• Neural networks are not really neural
![Page 9: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/9.jpg)
Convolutional Neural Network
![Page 10: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/10.jpg)
Convolutional Neural Networks
• A bit of history
![Page 11: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/11.jpg)
Convolutional Neural Networks
• A bit of history
![Page 12: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/12.jpg)
Recall fully connected layers
![Page 13: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/13.jpg)
Recall fully connected layers
![Page 14: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/14.jpg)
Convolution layer
![Page 15: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/15.jpg)
Convolution layer
![Page 16: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/16.jpg)
Convolution layer
![Page 17: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/17.jpg)
Convolution layer
![Page 18: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/18.jpg)
Convolution layer
![Page 19: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/19.jpg)
ConvNet
• ConvNet is a sequence of Convolution Layers, interspersed with activation functions
![Page 20: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/20.jpg)
A typical ConvNet
![Page 21: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/21.jpg)
A closer look at spatial dimensions
![Page 22: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/22.jpg)
Activations
![Page 23: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/23.jpg)
Output size
Padding is necessary
![Page 24: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/24.jpg)
Padding
![Page 25: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/25.jpg)
Note that
![Page 26: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/26.jpg)
An example
![Page 27: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/27.jpg)
General form of a Conv layer
![Page 28: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/28.jpg)
An extreme case
![Page 29: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/29.jpg)
The brain/neuron view of CONV layer
![Page 30: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/30.jpg)
The brain/neuron view of CONV layer
![Page 31: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/31.jpg)
Reminder: Fully Connected Layer
![Page 32: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/32.jpg)
Pooling layer
• makes the representations smaller and more manageable
• operates over each activation map independently:
![Page 33: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/33.jpg)
Max pooling
![Page 34: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/34.jpg)
General form
![Page 35: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/35.jpg)
Fully Connected Layer (FC Layer)
• Contains neurons that connect to the entire input volume, as in ordinary Neural Networks
![Page 36: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/36.jpg)
Summary
• ConvNets stack CONV,POOL,FC layers
• Trend towards smaller filters and deeper architectures
• Trend towards getting rid of POOL/FC layers (just CONV)
• Typical architectures look like
• [(CONV-RELU)*N-POOL?]*M-(FC-RELU)*K,SOFTMAX where N is usually up to ~5, M is large, 0 <= K <= 2.
– but recent advances such as ResNet/GoogLeNet challenge this paradigm
![Page 37: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/37.jpg)
Optimization
![Page 38: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/38.jpg)
Let us go back to the linear case
![Page 39: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/39.jpg)
Optimization
![Page 40: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/40.jpg)
Gradient descent
![Page 41: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/41.jpg)
Compuational graphs
![Page 42: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/42.jpg)
Back-propagation
![Page 43: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/43.jpg)
A toy example
![Page 44: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/44.jpg)
Back-propagation
![Page 45: CS376 Computer Vision Lecture 20: Deep Learning Basicshuangqx/CS376_Lecture_20.pdf · CS376 Computer Vision Lecture 20: Deep Learning Basics Qixing Huang April 10th 2019 Slide material](https://reader033.vdocument.in/reader033/viewer/2022060600/6053ba3a2cef7b1e9d02b7ec/html5/thumbnails/45.jpg)
Back-propagation
• recursive application of the chain rule along a computational graph to compute the gradients of all inputs /parameters / intermediates
• implementations maintain a graph structure, where the nodes implement the forward() / backward() API
• forward: compute result of an operation and save any intermediates needed for gradient computation in memory
• backward: apply the chain rule to compute the gradient of the loss function with respect to the inputs