fundamentals of deep (artificial) neural networks …hy590-21/pdfs/fundamentals_of...transferable...
TRANSCRIPT
![Page 1: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/1.jpg)
Fundamentals of Deep (Artificial) Neural Networks (DNN)Greg Tsagkatakis
CSD - UOC
ICS - FORTH
![Page 2: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/2.jpg)
The Big Data era
FUNDAMENTALS OF DEEP NEURAL NETWORKS 2
![Page 3: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/3.jpg)
Brief history of DL
3FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 4: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/4.jpg)
Why Today?Lots of Data
4FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 5: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/5.jpg)
Why Today?Lots of Data
Deeper Learning
5FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 6: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/6.jpg)
Why Today?Lots of Data
Deep Learning
More Power
https://blogs.nvidia.com/blog/2016/01/12/accelerating-ai-artificial-intelligence-gpus/https://www.slothparadise.com/what-is-cloud-computing/
6FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 7: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/7.jpg)
Machine learning is everywhere
FUNDAMENTALS OF DEEP NEURAL NETWORKS 7
![Page 8: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/8.jpg)
Deep LearningInspiration from brain
86Billion neurons
1014-1015 synapses
FUNDAMENTALS OF DEEP NEURAL NETWORKS 8
![Page 9: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/9.jpg)
Key components of ANN Architecture (input/hidden/output layers)
9FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 10: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/10.jpg)
Key components of ANN Architecture (input/hidden/output layers)
Weights
10FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 11: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/11.jpg)
Key components of ANN Architecture (input/hidden/output layers)
Weights
Activations
11
LINEAR RECTIFIED
LINEAR (ReLU)
LOGISTIC /
SIGMOIDAL / TANH
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 12: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/12.jpg)
Perceptron: an early attempt
Activation function
Need to tune and
σ
x1
x2
…
b
w1
w2
1
12FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 13: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/13.jpg)
Multilayer perceptron
w1
w2
w3
A
B
C
D
E
A neuron is of the form
σ(w.x + b) where σ is
an activation function
We just added a neuron layer!
We just introduced non-linearity!
w1A
w2B
w1D
wAE
wDE
13FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 14: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/14.jpg)
14
Sasen Cain (@spectralradius)
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 15: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/15.jpg)
Training & TestingTraining: determine weights◦ Supervised: labeled training examples
◦ Unsupervised: no labels available
◦ Reinforcement: examples associated with rewards
Testing (Inference): apply weights to new examples
15FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 16: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/16.jpg)
Training DNN1. Get batch of data
2. Forward through the network -> estimate loss
3. Backpropagate error
4. Update weights based on gradient
Errors
16FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 17: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/17.jpg)
BackpropagationChain Rule in Gradient Descent: Invented in 1969 by Bryson and Ho
Defining a loss/cost function
Assume a function
and associated predictions
Examples of Loss function
•Hinge
•Exponential
•Logistic
17FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 18: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/18.jpg)
Gradient DescentMinimize function J w.r.t. parameters θ
Gradient
Stochastic Gradient Descend
18
New weights Gradient
Old weights Learning rate
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 19: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/19.jpg)
Backpropagation
19
Chain Rule:
Given:
…
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 20: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/20.jpg)
Backpropagation
Chain rule:
◦ Single variable
◦ Multiple variables
20
g fxy=g(x)
z=f(y)=f(g(x))
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 21: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/21.jpg)
Online exampleshttps://google-developers.appspot.com/machine-learning/crash-course/backprop-scroll/
21FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 22: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/22.jpg)
Optimization algorithms
22FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 23: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/23.jpg)
Visualization
23FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 24: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/24.jpg)
Training Characteristics
24
Under-fitting
Over-fitting
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 25: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/25.jpg)
Supervised Learning
25FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 26: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/26.jpg)
Supervised Learning
Exploiting prior knowledge
Expert users
Crowdsourcing
Other instruments
Spiral
Elliptical
?
26
ModelPrediction
Data Labels
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 27: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/27.jpg)
State-of-the-art (before Deep Learning)Support Vector Machines Binary classification
27FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 28: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/28.jpg)
State-of-the-art (before Deep Learning)Support Vector Machines Binary classification
Kernels <-> non-linearities
28FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 29: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/29.jpg)
State-of-the-art (before Deep Learning)Support Vector Machines Binary classification
Kernels <-> non-linearities
Random Forests Multi-class classification
29FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 30: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/30.jpg)
State-of-the-art (before Deep Learning)Support Vector Machines Binary classification
Kernels <-> non-linearities
Random Forests Multi-class classification
Markov Chains/Fields Temporal data
30FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 31: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/31.jpg)
State-of-the-art (since 2015)Deep Learning (DL)
Convolutional Neural Networks (CNN) <-> Images
Recurrent Neural Networks (RNN) <-> Audio
31FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 32: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/32.jpg)
Convolutional Neural Networks
(Convolution + Subsampling) + () … + Fully Connected
32FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 33: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/33.jpg)
Convolution operator
33FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 34: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/34.jpg)
Convolutional Layers
channels
hei
ght
32x32x1 Image
5x5x1 filter
hei
ght
28x28xK activation map
K filters
34FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 35: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/35.jpg)
Convolutional LayersCharacteristics
Hierarchical features
Location invariance
Parameters
Number of filters (32,64…)
Filter size (3x3, 5x5)
Stride (1)
Padding (2,4)
“Machine Learning and AI for Brain Simulations” –Andrew Ng Talk, UCLA, 2012
35FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 36: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/36.jpg)
Activation LayerIntroduction of non-linearity◦ Brain: thresholding -> spike trains
36FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 37: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/37.jpg)
Activation LayerReLU: x=max(0,x)
Simplifies backprop
Makes learning faster
Avoids saturation issues
~ non-negativity constraint
(Note: The brain)
No saturated gradients
37FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 38: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/38.jpg)
Subsampling (pooling) Layers
38
<-> downsampling
Scale invariance
Parameters
• Type
• Filter Size
• Stride
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 39: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/39.jpg)
Fully Connected LayersFull connections to all activations in previous layer
Typically at the end
Can be replaced by conv
39
Feat
ure
s
Cla
sses
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 40: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/40.jpg)
LeNet [1998]
40FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 41: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/41.jpg)
AlexNet [2012]
Alex Krizhevsky, Ilya Sutskever and Geoff Hinton, ImageNet ILSVRC challenge in 2012http://vision03.csail.mit.edu/cnn_art/data/single_layer.png
41FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 42: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/42.jpg)
K. Simonyan, A. Zisserman Very Deep Convolutional Networks for Large-Scale Image Recognition, arXiv technical report, 2014
VGGnet [2014]
42FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 43: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/43.jpg)
VGGnet
D: VGG16E: VGG19All filters are 3x3
More layers smaller filters
43FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 44: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/44.jpg)
Inception (GoogLeNet, 2014)
Inception moduleInception module with dimensionality reduction
44FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 45: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/45.jpg)
Residuals
45FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 46: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/46.jpg)
ResNet, 2015
46
He, Kaiming, et al. "Deep residual learning for image recognition." IEEE CVPR. 2016.
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 47: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/47.jpg)
DenseNet
47
Densely Connected Convolutional Networks, 2016
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 48: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/48.jpg)
Training protocolsFully Supervised• Random initialization of weights
• Train in supervised mode (example + label)
Unsupervised pre-training + standard classifier• Train each layer unsupervised
• Train a supervised classifier (SVM) on top
Unsupervised pre-training + supervised fine-tuning• Train each layer unsupervised
• Add a supervised layer
48FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 49: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/49.jpg)
Dropout
49
Srivastava, Nitish, et al. "Dropout: a simple way to prevent neural networks from overfitting." Journal of machine learning research15.1 (2014): 1929-1958.
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 50: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/50.jpg)
Batch Normalization
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift [Ioffe and Szegedy 2015] 50FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 51: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/51.jpg)
Transfer LearningLayer 1 Layer 2 Layer L
1x
2x
……
……
……
……
……
elephant
……Nx
Pixels
……
……
……
FUNDAMENTALS OF DEEP NEURAL NETWORKS 51
![Page 52: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/52.jpg)
Transfer Learning
52
Layer 1 Layer 2 Layer L
1x
2x
……
……
……
……
……
elephant
……Nx
Pixels
……
……
……
Layer 1 Layer 2 Layer L
1x
2x
……
……
……
……
Healthy
Malignancy
Nx
Pixels
……
……
……
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 53: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/53.jpg)
Layer Transfer - Image
Jason Yosinski, Jeff Clune, Yoshua Bengio, Hod Lipson, “How transferable are features in deep neural networks?”, NIPS, 2014
Only train the rest layers
fine-tune the whole network
Source: 500 classes from ImageNet
Target: another 500 classes from ImageNet
FUNDAMENTALS OF DEEP NEURAL NETWORKS 53
![Page 54: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/54.jpg)
ImageNET
Validation classification
Validation classification
Validation classification
• ~14 million labeled images, 20k classes
• Images gathered from Internet
• Human labels via Amazon MTurk
• ImageNet Large-Scale Visual Recognition Challenge (ILSVRC): 1.2 million training images, 1000 classes
www.image-net.org/challenges/LSVRC/
54FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 55: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/55.jpg)
Summary: ILSVRC 2012-2015
Team Year Place Error (top-5) External data
(AlexNet, 7 layers) 2012 - 16.4% no
SuperVision 2012 1st 15.3% ImageNet 22k
Clarifai – NYU (7 layers) 2013 - 11.7% no
Clarifai 2013 1st 11.2% ImageNet 22k
VGG – Oxford (16 layers) 2014 2nd 7.32% no
GoogLeNet (19 layers) 2014 1st 6.67% no
ResNet (152 layers) 2015 1st 3.57%
Human expert* 5.1%
http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/
55FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 56: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/56.jpg)
Esteva, Andre, et al. "Dermatologist-level classification of skin cancer with deep neural networks." Nature 542.7639 (2017): 115-118.
Skin cancer detection
56FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 57: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/57.jpg)
CNN & FMRI
57FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 58: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/58.jpg)
Structured deep models
58
Material from Thomas Kipf, CompBio Seminar, University of Cambridge, 5/2018
BiologicalNeural Network
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 59: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/59.jpg)
Graph CNNs
59FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 60: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/60.jpg)
60FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 61: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/61.jpg)
Graph CNN
61FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 62: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/62.jpg)
Different types of mapping
Image classification
Image captioning
Sentiment analysis
Machine translation
Synced sequence(video classification)
62
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 63: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/63.jpg)
Recurrent Neural NetworksMotivation
Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized vector as output
fixed amount of computational steps
recurrent nets allow us to operate over sequences of vectors
Use cases
Video: sequence understanding
Audio: speech transcription
Text: natural language processing
63FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 64: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/64.jpg)
Recurrent neuron
64
next time step
● xt: Input at time t
● ht-1: State at time t-1
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 65: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/65.jpg)
Recurrent Neural Networks
65FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 66: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/66.jpg)
Unfolding RNNsEach node represents a layer of network units at a single time step.
The same weights are reused at every time step.
67FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 67: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/67.jpg)
Long Short-Term Memory networks (LSTMs)
68FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 68: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/68.jpg)
Multitask LearningThe multi-layer structure makes NN suitable for multitask learning
Input feature
Task A Task B
Task A Task B
Input feature for task A
Input feature for task B
FUNDAMENTALS OF DEEP NEURAL NETWORKS 69
![Page 69: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/69.jpg)
app-level mobile data analysis
70FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 70: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/70.jpg)
Unsupervised Learning
71FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 71: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/71.jpg)
Why unsupervised?Challenges
• Massive volumes of observations
• No user-introduced annotations
Applications
72
Clustering Dimensionality reduction
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 72: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/72.jpg)
K-means
73FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 73: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/73.jpg)
K-means
74FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 74: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/74.jpg)
TopicsSparse coding
Autoencoders
Generative Adversarial Networks
75FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 75: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/75.jpg)
Sparse Coding
50 100 150 200 250 300 350 400 450 500
50
100
150
200
250
300
350
400
450
500
50 100 150 200 250 300 350 400 450 500
50
100
150
200
250
300
350
400
450
500
50 100 150 200 250 300 350 400 450 500
50
100
150
200
250
300
350
400
450
500
0.8 * + 0.3 * + 0.5 *
2 1min s.t. x Ds s K
0.8 * + 0.3 * + 0.5 *
FUNDAMENTALS OF DEEP NEURAL NETWORKS 76
![Page 76: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/76.jpg)
Deep ModelsApplications of SC
Shortcoming◦ Shallow model <-> single level of representation
◦ Complexity <-> dictionary size
◦ Task specific
77FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 77: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/77.jpg)
AutoEncoders (AE)
NNEncoder
NNDecoder
codeCompact
representation of the input object
codeCan reconstruct
the original object
Learn together28 X 28 = 784
Usually <784
FUNDAMENTALS OF DEEP NEURAL NETWORKS 78
![Page 78: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/78.jpg)
AutoEncoders (AE)
Unsupervised feature learning
Network is trained to output the input (learn identify function).
Encoder
Decoder
79
x4
x5
x6
Input Layer Hidden
x1
x2
x3
x4
x5
x6
x1
x2
x3
+1
Output
h1
h2
h3
h4
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 79: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/79.jpg)
Regularized Autoencoders
Sparse neuron activation
Denoising auto-encoders
Convolutional AE
80FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 80: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/80.jpg)
Stacked AutoEncoders (SAE)
Extended AE with multiple layers of hidden units
Challenges of Backpropagation
Efficient training◦ Normalization of input
Unsupervised pre-training◦ Greedy layer-wise training
◦ Fine-tune w.r.t criterion
81
Bengio, Learning deep architectures for AI, Foundations and Trends in Machine Learning ,2009
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 81: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/81.jpg)
x4
x5
x6
x1
x2
x3
x4
x5
x6
x1
x2
x3
a1
a2
a3
SAE
82
a4
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 82: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/82.jpg)
x4
x5
x6
x1
x2
x3
x4
x5
x6
x1
x2
x3
a1
a2
a3
SAE
83
a4
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 83: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/83.jpg)
x4
x5
x6
x1
x2
x3
a1
a2
a3
SAE
84
a4
b1
b2
b3
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 84: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/84.jpg)
x4
x5
x6
x1
x2
x3
a1
a2
a3
SAE
85
a4
b1
b2
b3
a1
a2
a3
a4
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 85: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/85.jpg)
x4
x5
x6
x1
x2
x3
a1
a2
a3
SAE
86
a4
b1
b2
b3
a1
a2
a3
a4
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 86: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/86.jpg)
x4
x5
x6
x1
x2
x3
a1
a2
a3
SAE
87
a4
b1
b2
b3
a1
a2
a3
a4
x4
x5
x6
x1
x2
x3
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 87: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/87.jpg)
Deep Auto-encoder
Original Image
PCA
DeepAuto-encoder
78
4
78
4
78
4
10
00
50
0
25
0
30
25
0
50
0
10
00
78
4
30
FUNDAMENTALS OF DEEP NEURAL NETWORKS 88
![Page 88: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/88.jpg)
AWGN channel modeling
89FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 89: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/89.jpg)
Application in modulationtraining a learned physical layer, where a channel autoencoder learns how to optimize BER for two bits of information over a simple Additive White Gaussian Noise (AWGN) effect.
90FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 90: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/90.jpg)
91FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 91: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/91.jpg)
Generators?
NNDecoder
code
FUNDAMENTALS OF DEEP NEURAL NETWORKS 92
![Page 92: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/92.jpg)
Generative Adversarial Networks
93
Goodfellow, Ian, et al. "Generative adversarial nets." NIPS 2014
Yann LeCun,
“adversarial
training is the
coolest thing
since sliced bread.”
Co
de
Generator
Discriminator
True
Fake
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 93: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/93.jpg)
GANs
94FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 94: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/94.jpg)
GANs
95FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 95: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/95.jpg)
GANs
96FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 96: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/96.jpg)
GANs
97FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 97: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/97.jpg)
Training Procedure: Basic Idea
G tries to fool D
D tries not to be fooled
Models are trained simultaneously◦ As G gets better, D has a more challenging task
◦ As D gets better, G has a more challenging task
Ultimately, we don’t care about the D◦ Its role is to force G to work harder
DiscriminativeModel
real or fake?
GenerativeModel
noise (𝒛)
Real world
FUNDAMENTALS OF DEEP NEURAL NETWORKS 98
![Page 98: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/98.jpg)
DCGANsDeep Convolutional Generative Adversarial Networks
99
Radford et. al. Unsupervised Representation Learning with Deep
Convolutional Generative Adversarial Networks. ICLR 2016
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 99: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/99.jpg)
100FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 100: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/100.jpg)
101FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 101: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/101.jpg)
102FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 102: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/102.jpg)
Image arithmetic
103FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 103: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/103.jpg)
104FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 104: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/104.jpg)
GAN based Style Transfer
105FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 105: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/105.jpg)
GANs for Super Resolution
106
Ledig, Christian, et al. "Photo-realistic single image super-resolution using a generative adversarial network." arXiv preprint arXiv:1609.04802 (2016).
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 106: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/106.jpg)
GANs for Super Resolution
107FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 107: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/107.jpg)
Fake newshttps://youtu.be/8siezzLXbNo
https://www.youtube.com/watch?v=DIZf7eRlD4w&t=108s
108FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 108: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/108.jpg)
Reinforcement learningReinforcement learning: system interacts with environmentand must perform a certain goal without explicitly telling itwhether it has come close to its goal or not.
109FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 109: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/109.jpg)
Types of Reinforcement LearningSearch-based: evolution directly on a policy◦ E.g. genetic algorithms
Model-based: build a model of the environment◦ Then you can use dynamic programming
◦ Memory-intensive learning method
Model-free: learn a policy without any model◦ Temporal difference methods (TD)
◦ Requires limited episodic memory (though more helps)
Q-learning◦ The TD version of Value Iteration
◦ This is the most widely used RL algorithm
110FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 110: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/110.jpg)
Q-Learning
111
Define Q*(s,a): “Total reward if an agent in state s takes action a, then acts optimally at all subsequent time steps”
Optimal policy: π*(s)=argmaxaQ*(s,a)
Q(s,a) is an estimate of Q*(s,a)
Q-learning motion policy: π(s)=argmaxaQ(s,a)
Update Q recursively:
)','(max),('
asQrasQa
10
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 111: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/111.jpg)
Deep Q-Learning
112
Mnih et al. Human-level control through deep reinforcement learning, Nature 2015
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 112: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/112.jpg)
Deep Q learning in AtariEnd-to-end learning of Q(s,a) from pixels s
Output is Q(s,a) for 18 joystick/button configurations
Reward is change in score for that step
Q(s,a1)Q(s,a2)Q(s,a3)...........Q(s,a18)
FUNDAMENTALS OF DEEP NEURAL NETWORKS 113
![Page 113: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/113.jpg)
Breakout demo
FUNDAMENTALS OF DEEP NEURAL NETWORKS 114
![Page 114: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/114.jpg)
Multi-agent case
115FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 115: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/115.jpg)
116FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 116: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/116.jpg)
TensorFlowDeep learning library, open-sourced by Google (11/2015)
TensorFlow provides primitives for ◦ defining functions on tensors
◦ automatically computing their derivatives
What is a tensor
What is a computational graph
117
Material from lecture by Bharath Ramsundar, March 2018, Stanford
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 117: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/117.jpg)
Introduction to KerasOfficial high-level API of TensorFlow
◦ Python
◦ 250K developers
Same front-end <-> Different back-ends◦ TensorFlow (Google)
◦ CNTK (Microsoft)
◦ MXNet (Apache)
◦ Theano (RIP)
Hardware◦ GPU (Nvidia)
◦ CPU (Intel/AMD)
◦ TPU (Google)
Companies: Netflix, Uber, Google, Nvidia…
118
Material from lecture by Francois Chollet, 2018, Stanford
FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 118: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/118.jpg)
Keras modelsInstallation ◦ Anaconda -> Tensorflow -> Keras
Build-in◦ Conv1D, Conv2D, Conv3D…
◦ MaxPooling1D, MaxPooling2D, MaxPooling3D…
◦ Dense, Activation, RNN…
The Sequential Model◦ Very simple
◦ Single-input, Single-output, sequential layer stacks
The functional API◦ Mix & Match
◦ Multi-input, multi-output, arbitrary static graph topologies
119FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 119: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/119.jpg)
Sequential>>from keras.models import Sequential
>>model = Sequential()
>> from keras.layers import Dense
>> model.add(Dense(units=64, activation='relu', input_dim=100))
>> model.add(Dense(units=10, activation='softmax'))
>> model.compile(loss='categorical_crossentropy', optimizer='sgd', metrics=['accuracy'])
>> model.fit(x_train, y_train, epochs=5, batch_size=32)
>> loss_and_metrics = model.evaluate(x_test, y_test, batch_size=128)
>> classes = model.predict(x_test)
120FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 120: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/120.jpg)
Functional >> from keras.layers import Input, Dense
>> from keras.models import Model
>> inputs = Input(shape=(784,))
>> x = Dense(64, activation='relu')(inputs)
>> x = Dense(64, activation='relu')(x)
>> predictions = Dense(10, activation='softmax')(x)
>> model = Model(inputs=inputs, outputs=predictions)
>> model.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])
>> model.fit(data, labels)
121FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 121: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/121.jpg)
ReferencesStephens, Zachary D., et al. "Big data: astronomical or genomical?." PLoS biology 13.7 (2015): e1002195.
LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. "Deep learning.“ Nature 521.7553 (2015): 436-444.
Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Kietzmann, Tim Christian, Patrick McClure, and NikolausKriegeskorte. "Deep Neural Networks In Computational Neuroscience. " bioRxiv (2017): 133504.
122FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 122: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/122.jpg)
https://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=circle®Dataset=reg-plane&learningRate=0.03®ularizationRate=0&noise=0&networkShape=3,2&seed=0.57693&showTestData=false&discretize=true&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=true&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false
123FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 123: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/123.jpg)
Model vs Data parallelism
124FUNDAMENTALS OF DEEP NEURAL NETWORKS
![Page 124: Fundamentals of Deep (Artificial) Neural Networks …hy590-21/pdfs/Fundamentals_of...transferable are features in deep neural networks?”, NIPS, 2014 Only train the rest layers fine-tune](https://reader030.vdocument.in/reader030/viewer/2022011903/5f1863182746db3c8f3b6e96/html5/thumbnails/124.jpg)
Parameter server approach
125FUNDAMENTALS OF DEEP NEURAL NETWORKS