feed-forward neural networks (part 1)
TRANSCRIPT
![Page 1: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/1.jpg)
Feed-forward Neural Networks (Part 1)
![Page 2: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/2.jpg)
Outline (part 1)‣ Feed-forward neural networks ‣ The power of hidden layers ‣ Learning feed-forward networks
- SGD and back-propagation
![Page 3: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/3.jpg)
Motivation‣ So far our classifiers rely on pre-compiled features
y = sign�✓ · �(x)
�
![Page 4: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/4.jpg)
Neural Networks
1. (c) Michael DeBellis
![Page 5: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/5.jpg)
(Artificial) Neural Networks
…
x1
x2
xd
(e.g., a linear classifier)
f
2. Image on Wikimedia by Users: Ramón Santiago y Cajal.
(c) Michael DeBellis
![Page 6: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/6.jpg)
A unit in a neural network…
x1
x2
xd
f
Image on Wikimedia by Users: Ramón Santiago y Cajal.
![Page 7: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/7.jpg)
A unit in a neural network…
x1
x2
xd
f
Image on Wikimedia by Users: Ramón Santiago y Cajal.
![Page 8: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/8.jpg)
Deep Neural Networks
‣ Deep neural networks- loosely motivated by biological neurons, networks - adjustable processing units (~ linear classifiers) - highly parallel, typically organized in layers - deep = many transformations (layers) before output
e.g., edges -> simple parts-> parts -> objects -> scenes
(c) Michael DeBellis
![Page 9: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/9.jpg)
Deep Learning‣ Deep learning has overtaken a number of academic
disciplines in just a few years - computer vision (e.g., image, scene analysis) - natural language processing (e.g., machine translation) - speech recognition - computational biology, etc.
![Page 10: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/10.jpg)
Deep Learning‣ Deep learning has overtaken a number of academic
disciplines in just a few years - computer vision (e.g., image, scene analysis) - natural language processing (e.g., machine translation) - speech recognition - computational biology, etc.
‣ Key role in recent successes - self driving vehicles - speech interfaces - conversational agents - superhuman game playing
![Page 11: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/11.jpg)
Deep Learning‣ Deep learning has overtaken a number of academic
disciplines in just a few years - computer vision (e.g., image, scene analysis) - natural language processing (e.g., machine translation) - speech recognition - computational biology, etc.
‣ Key role in recent successes - self driving vehicles - speech recognition - conversational agents - superhuman game playing
‣ Many more underway - personalized/automated medicine - chemistry, robotics, materials science, etc.
![Page 12: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/12.jpg)
‣ Reason #1: lots of data - many significant problems can only be solved at scale
‣ Reason #2: computational resources (esp. GPUs) - platforms/systems that support running deep (machine)
learning algorithms at scale
‣ Reason #3: large models are easier to train - large models can be successfully estimated with simple
gradient based learning algorithms
‣ Reason #4: flexible neural “lego pieces” - common representations, diversity of architectural choices
Deep learning … why now?
![Page 13: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/13.jpg)
One hidden layer model
x1
x2z2
z1 f1
f2
zf
layer 0 layer 1 (tanh)
layer 2 (linear)
W11
W12
W21
W22
![Page 14: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/14.jpg)
One hidden layer model
x1
x2z2
z1 f1
f2
zf
layer 0 layer 1 (tanh)
layer 2 (linear)
W11
W12
W21
W22
![Page 15: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/15.jpg)
One hidden layer model‣ Neural signal transformation
x1
x2z2
z1 f1
f2
layer 0 layer 1 (tanh)
W11
W12
W21
W22
![Page 16: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/16.jpg)
Example Problem
![Page 17: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/17.jpg)
Hidden layer representation
(1)
(2)
Hidden layer units
![Page 18: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/18.jpg)
(1)
(2)
(1)
(2)
Linear activationHidden layer units
Hidden layer representation
![Page 19: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/19.jpg)
(1)
(2)
(1)
(2)
Linear activationHidden layer units
Hidden layer representation
![Page 20: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/20.jpg)
(1)
(2)
(1)
(2)
Linear activationHidden layer units
Hidden layer representation
![Page 21: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/21.jpg)
(1)
(2)
(1)
(2)
Linear activationHidden layer units
Hidden layer representation
![Page 22: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/22.jpg)
(1)
(2)
(1)
(2)
tanh activationHidden layer units
Hidden layer representation
![Page 23: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/23.jpg)
(1)
(2)
(1)
(2)
ReLU activationHidden layer units
Hidden layer representation
![Page 24: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/24.jpg)
Does orientation matter?
(1)
(2)
Hidden layer units
![Page 25: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/25.jpg)
(1)
(2)
Hidden layer units
(1)
(2)
tanh activation
Does orientation matter?
![Page 26: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/26.jpg)
(1)
(2)
Hidden layer units
(1)
(2)
ReLU activation
Does orientation matter?
![Page 27: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/27.jpg)
Random hidden units
(1)(2)
Hidden layer units
![Page 28: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/28.jpg)
(1)(2)
(1)
(2)
tanh activationHidden layer units
Random hidden units
![Page 29: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/29.jpg)
(10 randomly chosen units)
Hidden layer units
Random hidden units
![Page 30: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/30.jpg)
(10 randomly chosen units)
Are the points linearly separablein the resulting
10 dimensional space?
Hidden layer units
Random hidden units
![Page 31: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/31.jpg)
(10 randomly chosen units)
Are the points linearly separablein the resulting
10 dimensional space?
YES!
Random hidden units
Hidden layer units
![Page 32: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/32.jpg)
(10 randomly chosen units)
Random hidden units
Hidden layer units
what are the coordinates??
![Page 33: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/33.jpg)
Summary ‣ Units in neural networks are linear classifiers, just with
different output non-linearity‣ The units in feed-forward neural networks are arranged
in layers (input, hidden,…, output)‣ By learning the parameters associated with the hidden
layer units, we learn how to represent examples (ashidden layer activations)
‣ The representations in neural networks are learneddirectly to facilitate the end-to-end task
‣ A simple classifier (output unit) suffices to solve complexclassification tasks if it operates on the hidden layerrepresentations
![Page 34: Feed-forward Neural Networks (Part 1)](https://reader030.vdocument.in/reader030/viewer/2022012811/61c17ac57148751558230cd6/html5/thumbnails/34.jpg)
Attribution List - Machine Learning - 6.86x
1.Unit 1 Lecture 8: Introduction to Machine LearningStructure of a neuron with the soma (cell body), dendrites and axonSlides: #4, #5, #8Object Source / URL: http://www.neuropsychologysketches.com/Citation/Attribution: (c) Michael DeBellis
2.Unit 1 Lecture 8: Introduction to Machine LearningIllustration of the neuronal morphologies in the auditory cortexSlides: #5, #6, #7, #8Object Source / URL: https://commons.wikimedia.org/wiki/File:Cajal_actx_inter.jpgCitation/Attribution: Image on Wikimedia by Users: Ramón Santiago y Cajal.