deep learning models › 49f9 › 8dc084e65e19...brain-like cognitive computing & deep learning...
TRANSCRIPT
![Page 1: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/1.jpg)
Deep Learning Models
2012-05-03
Byoung-Hee Kim
Biointelligence Lab, CSE,
Seoul National University
NOTE: most slides are from talks of Geoffrey Hinton, Andrew Ng, and Yoshua Bengio.
![Page 2: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/2.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 2
![Page 3: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/3.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 3
![Page 4: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/4.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 4
Input output target
Two!
![Page 5: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/5.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 5
![Page 6: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/6.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 6
![Page 7: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/7.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 7
![Page 8: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/8.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 8
![Page 9: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/9.jpg)
Artificial Neural Networks
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 9
![Page 10: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/10.jpg)
Historical background:
First generation neural networks
Perceptrons (~1960) used a layer of hand-coded features and tried to recognize objects by learning how to weight these features.
There was a neat learning algorithm for adjusting the weights.
But perceptrons are fundamentally limited in what they can learn to do.
non-adaptive
hand-coded
features
output units e.g.
class labels
input units
e.g. pixels
Sketch of a typical
perceptron from the 1960’s
Bomb Toy
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 10
![Page 11: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/11.jpg)
Second generation neural networks (~1985)
input vector
hidden
layers
outputs
Back-propagate
error signal to
get derivatives
for learning
Compare outputs with
correct answer to get
error signal
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 11
![Page 12: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/12.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 12
But, finding any model with deep architecture was not successful till 2006
![Page 13: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/13.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 13
http://www.iro.umontreal.ca/~pift6266/H10/notes/deepintro.html
![Page 14: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/14.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 14
![Page 15: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/15.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 15
![Page 16: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/16.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 16
![Page 17: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/17.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 17
![Page 18: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/18.jpg)
Agenda
Computer Perception
Unsupervised feature learning
Various deep learning models
Application cases of deep learning models
Written digit recognition/generation (MNIST dataset)
Image classification
Audio recognition
Language modeling
Motion generation
References
Appendix
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 18
![Page 19: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/19.jpg)
Brain-like Cognitive Computing & Deep Learning
It is well know that the brain has a hierarchical structure
Researchers try to build models that simulate and/or act like the brain
Learning deep structures from data, or the deep learning is a new frontier in Artificial Intelligence research
Researchers try to find analogies between the characteristics of the brain and their deep models
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 19
![Page 20: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/20.jpg)
Feature Learning
Input
Input space Motorbikes
“Non”-Motorbikes
Learning algorithm
pixel 1
pix
el 2
pixel 1
pixel 2
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 20
![Page 21: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/21.jpg)
Feature Learning
Input
Input space Feature space Motorbikes
“Non”-Motorbikes
Feature Extractor
Learning algorithm
pixel 1
pix
el 2
“wheel”
“han
dle
”
handle
wheel
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 21
![Page 22: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/22.jpg)
How is computer perception done?
Image Low-level
vision features Recognition
Low-level state
features Action Helicopter
Audio Low-level
audio features
Speaker
identification
Object
detection
Audio
classification
Helicopter
control
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 22
![Page 23: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/23.jpg)
Learning representations
Sensor Learning algorithm
Feature
Representation
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 23
![Page 24: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/24.jpg)
Computer vision features
SIFT Spin image
HoG RIFT
Textons GLOH (C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 24
![Page 25: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/25.jpg)
Audio features
ZCR
Spectrogram MFCC
Rolloff Flux
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 25
![Page 26: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/26.jpg)
Problems of hand-tuned features
Needs expert knowledge
Sub-optimal
Time-consuming and expensive
Does not generalize to other domains
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 26
Can we automatically learn good feature representations?
![Page 27: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/27.jpg)
Sensor representation in the brain
[BrainPort; Martinez et al; Roe et al.]
Seeing with your tongue Human echolocation (sonar)
Auditory cortex
learns to see.
Auditory
Cortex
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 27
![Page 28: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/28.jpg)
Unsupervised Feature Learning
Find a better way to represent images than pixels
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 28
![Page 29: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/29.jpg)
The goal of Unsupervised Feature Learning
Unlabeled images
Learning algorithm
Feature representation
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 29
![Page 30: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/30.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 30
![Page 31: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/31.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 31
![Page 32: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/32.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 32
![Page 33: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/33.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 33
![Page 34: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/34.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 34
![Page 35: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/35.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 35
![Page 36: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/36.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 36
![Page 37: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/37.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 37
![Page 38: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/38.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 38
![Page 39: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/39.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 39
![Page 40: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/40.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 40
![Page 41: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/41.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 41
![Page 42: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/42.jpg)
Stochastic binary units (Bernoulli variables)
These have a state of 1 or 0.
The probability of turning on is determined by the weighted input from other units (plus a bias)
0
0
1
j
jijii
wsbsp
)exp(1)(
11
j
jiji wsb
)( 1isp
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 42
![Page 43: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/43.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 43
Binary
Stochastic
Neuron
![Page 44: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/44.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 44
![Page 45: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/45.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 45
![Page 46: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/46.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 46
![Page 47: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/47.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 47
![Page 48: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/48.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 48
![Page 49: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/49.jpg)
A model of digit recognition
2000 top-level neurons
500 neurons
500 neurons
28 x 28
pixel
image
10 label
neurons
The model learns to generate
combinations of labels and images.
To perform recognition we start with a
neutral state of the label units and do
an up-pass from the image followed
by a few iterations of the top-level
associative memory.
The top two layers form an
associative memory whose
energy landscape models the low
dimensional manifolds of the
digits.
The energy valleys have names
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 49
![Page 50: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/50.jpg)
Generation & Recognition of Digits by DBN
Deep belief network that learns to generate handwritten digits
http://www.cs.toronto.edu/~hinton/digits.html
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 50
![Page 51: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/51.jpg)
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 51
![Page 52: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/52.jpg)
First stage of visual processing in brain: V1
Schematic of simple cell Actual simple cell
[Images from DeAngelis, Ohzawa & Freeman, 1995]
“Gabor functions.”
The first stage of visual processing in the brain (V1) does
“edge detection.”
![Page 53: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/53.jpg)
Sparse coding illustration
Natural Images Learned bases (f1 , …, f64): “Edges”
50 100 150 200 250 300 350 400 450 500
50
100
150
200
250
300
350
400
450
500
50 100 150 200 250 300 350 400 450 500
50
100
150
200
250
300
350
400
450
500
50 100 150 200 250 300 350 400 450 500
50
100
150
200
250
300
350
400
450
500
0.8 * + 0.3 * + 0.5 *
x 0.8 * f36 + 0.3 * f42
+ 0.5 * f63
[0, 0, …, 0, 0.8, 0, …, 0, 0.3, 0, …, 0, 0.5, …] = [a1, …, a64] (feature representation)
Test example
Compact & easily interpretable
![Page 54: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/54.jpg)
Supervised learning
Testing:
What is this?
Cars Motorcycles
![Page 55: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/55.jpg)
Semi-supervised learning
Unlabeled images (all cars/motorcycles)
Testing:
What is this?
Car Motorcycle
![Page 56: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/56.jpg)
Self-taught learning
Testing:
What is this?
Car Motorcycle
Unlabeled images (random internet images)
![Page 57: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/57.jpg)
Self-taught learning
Sparse coding, LCC, etc.
f1, f2, …, fk
Car Motorcycle
Use learned f1, f2, …, fk to represent training/test sets.
Using f1, f2, …, fk a1, a2, …, ak
![Page 58: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/58.jpg)
Convolutional DBN for Images
![Page 59: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/59.jpg)
Convolutional DBN on face images
pixels
edges
object parts
(combination
of edges)
object models
![Page 60: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/60.jpg)
Examples of learned object parts from object categories
Learning of object parts
Faces Cars Elephants Chairs
![Page 61: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/61.jpg)
Training on multiple objects
Plot of H(class|neuron active)
Trained on 4 classes (cars, faces, motorbikes, airplanes).
Second layer: Shared-features and object-specific features.
Third layer: More specific features.
![Page 62: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/62.jpg)
Input images
Samples from feedforward Inference
(control)
Samples from Full posterior inference
Hierarchical probabilistic inference
Generating posterior samples from faces by “filling in” experiments
(cf. Lee and Mumford, 2003). Combine bottom-up and top-down inference.
![Page 63: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/63.jpg)
An application to modeling motion capture data (Taylor, Roweis & Hinton, 2007)
Human motion can be captured by placing reflective markers on the joints and then using lots of infrared cameras to track the 3-D positions of the markers.
Given a skeletal model, the 3-D positions of the markers can be converted into the joint angles plus 6 parameters that describe the 3-D position and the roll, pitch and yaw of the pelvis. We only represent changes in yaw because physics
doesn’t care about its value and we want to avoid circular variables.
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 63
Video lecture: http://videolectures.net/gesturerecognition2011_taylor_tutorial/
![Page 64: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/64.jpg)
Motion Generation by Conditional RBM
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 64
![Page 65: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/65.jpg)
Motion Generation by Conditional RBM
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 65
![Page 66: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/66.jpg)
Motion Generation by Conditional RBM
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 66
![Page 67: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/67.jpg)
Motion Generation by Conditional RBM
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 67
![Page 68: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/68.jpg)
Hinton’s Talk in Google:
http://www.youtube.com/watch?v=VdIURAu1-aU
Andrew Ng’s Talk in Bay Area Vision Meeting: Unsupervised Feature Learning and Deep Learning
http://www.youtube.com/watch?v=ZmNOAtZIgIk&feature=relmfu
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 68
![Page 69: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/69.jpg)
References
General Info on Deep Learning
http://deeplearning.net/
Review
Y. Bengio, Learning deep architectures for AI, Foundations and Trends in Machine Learning, 2(1):1-127, 2009.
I. Arel, D.C. Rose, and T.P. Karnowski, Deep machine learning – A new frontier in Artificial Intelligence Research, Computational Intelligence Magazine, 14:12-18, 2010.
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 69
![Page 70: Deep Learning Models › 49f9 › 8dc084e65e19...Brain-like Cognitive Computing & Deep Learning It is well know that the brain has a hierarchical structure Researchers try to build](https://reader034.vdocument.in/reader034/viewer/2022042402/5f11cc524f81e24eb70d59af/html5/thumbnails/70.jpg)
References
Tutorials & Workshops
Deep Learning and Unsupervised Feature Learning workshop – NIPS 2010: http://deeplearningworkshopnips2010.wordpress.com/schedule/acceptedpapers/
Workshop on Learning Feature Hierarchies - ICML 2009: http://www.cs.toronto.edu/~rsalakhu/deeplearning/index.html
(C) 2012, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 70