RECURRENT NEURAL NETWORKSA QUICK OVERVIEW
Filippo Maria Bianchi – [email protected]
Machine Learning Group – Department of Physics and Technology
Universitet i Tromsø
Geilo 2017 – Winter School
1
PRESENTATION OUTLINE
1. Introduction
2. The Recurrent Neural Network
3. RNN applications
4. The RNN model
5. Gated RNN (LSTM and GRU)
6. Echo State Network
2
1. INTRODUCTION
3
THE RECURRENT NEURAL NETWORK
A recurrent neural network (RNN) is a universal approximator of dynamical systems.
It can be trained to reproduce any target dynamics, up to a given degree of precision.
4Dynamical
system
Reccurrent
Hidden LayerOutput
RNN input
System observation
Target output
Evolution of
system state
Evolution of RNN
internal state
5
An RNN generalizes naturally to new inputs with any lengths.
An RNN make use of sequential information, by modelling a temporal dependencies in the inputs.
Example: if you want to predict the next word in a sentence you need to know which words came before it
The output of the network depends on the current input and on the value of the previous internal state.
The internal state maintains a (vanishing) memory about history of all past inputs.
RNNs can make use of information coming from arbitrarily long sequences, but in practice they are limited to look
back only a few time steps.
6
RNN can be trained to predict a future value, of the driving input.
A side-effect we get a generative model, which allows us to generate new elements by sampling from the output
probabilities.
RNNinput output
teacher
forcing
-
training
RNN
output
feedback
generative
mode
DIFFERENCES WITH CNN
Convolution in space (CNN) VS convolution in time (RNN) .
CNN: models relationships in space. Filter slides along 𝑥 and 𝑦 dimensions.
RNN: models relationships in time. ‘‘Filter’’ slides along time dimension.
7
Filter
𝑥
𝑦
Spatial data𝑡𝑖𝑚𝑒
CNN RNN
Temporal data
2. RNN APPLICATIONS
8
APPLICATION 1: NATURAL LANGUAGE PROCESSING
Given a sequence of words,
RNN predicts the probability
of next word given the
previous ones.
Input/output words are
encoded as one-hot vector.
We must provide the RNN all
the dictionary of interest
(usually, just the alphabet).
In the output layer, we want
the green numbers to be high
and red numbers to be low.
9
[Image: Andrej Karpathy]
10
Once trained, the RNN can work in generative mode.
In NLP context, a generative RNN can be used in Natural Language Generation.
Applications:
Generate text (human readable data) from databse of numbers and log files, not readable by human.
What you see is what you meant. Allows users to see and manipulate the continuously rendered view (NLG output) of an
underlying formal language document (NLG input), thereby editing the formal language without learning it.
NATURAL LANGUAGE
GENERATION: SHAKESPEARE
Dataset: all the works of
Shakespeare, concatenated
them into a single (4.4MB) file.
3-layer RNN with 512 hidden
nodes on each layer.
Few hours of training.
11[Source: Andrej Karpathy]
TEXT GENERATION:
WIKIPEDIA
Hutter Prize 100MB dataset of
raw Wikipedia.
LSTM
The link does not exist
12
[Source: Andrej Karpathy]
TEXT GENERATION:
SCIENTIFIC PAPER
RNN trained on a book
(LaTeX source code of 16MB).
Multilayer LSTM
13[Source: Andrej Karpathy]
APPLICATION II: MACHINE TRANSLATION
Similar to language modeling.
Train 2 different RNNs.
Input RNN: trained on a source language (e.g.
German).
Output RNN: trained on a target language (e.g.
English).
The second RNN computes the output from the
hidden layer of the first RNN.
Google translator.
14
[Image: Richard Socher]
APPLICATION III: SPEECH RECOGNITION
Input: input sequence of acoustic
signals.
Output phonetic segments.
Necessity of encoder/decoder to
transit from digital/analogic
domain.
Graves, Alex, and Navdeep Jaitly.
"Towards End-To-End Speech
Recognition with Recurrent Neural
Networks.“, 2014.
15
APPLICATION IV: IMAGE TAGGING
RNN + CNN jointly trained.
CNN generates features (hidden state representation).
RNN reads CNN features and produces output (end-to-end training).
Aligns the generated words with features found in the images
Karpathy, Andrej, and Li Fei-Fei. "Deep visual-semantic alignments for generating image descriptions.", 2015. 16
APPLICATION V: TIME SERIES PREDICTION
Forecast of future values in a time series, from past seen values.
Many applications:
Weather forcast.
Load forecast.
Financial time series.
17
Telephonic traffic
Electricity Load
APPLICATION VI: MUSIC INFORMATION RETRIEVAL
MIR: identification of songs/music
Automatic categorization.
Recommender systems.
Track separation and instrument recognition.
Music generation.
Automatic music transcription.
18
[Source: Meinard Müller]
Music transcription example
Automatic categorization software Software with recommender system
3. DESCRIPTION OF RNN MODEL
19
ARCHITECTURE COMPONENTS
𝑥: input
𝑦: output
ℎ: internal state (memory of the network)
𝑊𝑖ℎ: input weights
𝑊ℎℎ: recurrent layer weights
𝑊ℎ𝑜: output weights
𝑧−1: time-delay unit
: neuron transfer function
20
STATE UPDATE AND OUTPUT GENERATION
An RNN selectively summarize an input sequence in a fixed-size state vector via a
recursive update.
Discrete, time-independent difference equations of RNN state and output:
ℎ 𝑡 + 1 = 𝑓 𝑊ℎℎℎ 𝑡 +𝑊𝑖
ℎ𝑥 𝑡 + 1 + 𝑏ℎ ,
𝑦 𝑡 + 1 = 𝑔(𝑊ℎ𝑜ℎ 𝑡 + 1 + 𝑏𝑜).
𝑓() is the transfer function implemented by each neuron (usually the same non-linear
function for all neurons).
𝑔() is the readout of the RNN. Usually is the identity function - all the non-linearity is
provided by the internal processing units (neurons) – or the softmax function.
21
NEURON TRANSFER FUNCTION
The activation function in a RNN is traditionally
implemented by a sigmoid.
Saturation causes vanishing gradient.
Non-zero centering produces only positive outputs, which lead to
zig-zagging dynamics in the gradient updates.
Another common choice is the tanh.
Saturation causes vanishing gradient.
ReLU (not very much used in RNN).
Greatly accelerate gradient convergence and it has low
demanding computational time.
No vanishing gradient.
Large gradient flowing through a ReLU neuron could cause the its
“death”.22
TRAINING
Model's parameters are trained with gradient descent.
A loss function is evaluated on the error performed by the network on the training set and, usually, also a regularization term.
𝐿 = 𝐸 𝑦, ො𝑦 + λ𝑅
Where 𝐸() is the error function, 𝑦 and ො𝑦 are target and estimated outputs, λ is the regularization parameter, 𝑅 is the regularization term.
The derivative of the loss function, with respect to the model parameters, is backpropagated through the network.
Weights are adjusted until a stop criterion is met:
Maximum number of epochs is reached.
Loss function stop decreasing.
23
REGULARIZATION
Introduce a bias, necessary to prevent the RNN to overfit on training data.
In order to generalize well to unseen data, the variance (complexity) of the model should be limited.
Common regularization terms:
1. 𝐿1 regularization of the weights: 𝑊 1. Enforce sparsity in the weights.
2. 𝐿2 regularization of the weights: 𝑊 2. Enforce small values for the weights.
3. 𝐿1 + 𝐿2 (elastic net penalty). Combines the two previous regularizations.
4. Dropout. Done usually only on the output weights. Dropout on recurrent layer is more complicated (the weights are
constrained to be the same in each time step by the BPPT) → requires workaround.
24
RNN UNFOLDING
In order to train the network with
gradient descent, the RNN must be
unfolded.
Each replica of the network is
relative to a different time interval.
Now, the architecture of the
network become very deep, even
starting from a shallow RNN.
The weights are constrained to be
the same.
Less parameters than in other deep
architectures. 25
BACK PROPAGATION THROUGH TIME
In the example, we need to backpropagate the
gradient 𝜕𝐸3
𝜕𝑊from current time (𝑡3) to initial
time (𝑡0) → chain rule (eq. on the right).
We sum up the contributions of each time step to the gradient.
With of very long sequence (possibly infinite)we have untreatable depth.
Repeate the procedure only up to a given time (truncate BPPT).
Why it works? Because each state carries a little bit of information on each previous input.
Once the network is unfolded, the procedure is analogue to standard backpropagation used in deep Feedforward Neural Networks.
26
𝜕𝐸3𝜕𝑊
=
𝑘=0
3𝜕𝐸3𝜕 ො𝑦3
𝜕 ො𝑦3𝜕ℎ3
𝜕ℎ3𝜕ℎ𝑘
𝜕ℎ𝑘𝜕𝑊
VANISHING GRADIENT: SIMPLE EXPERIMENT
Bengio, 1991.
A simple RNN is trained to
keep 1 bit of information for 𝑇time steps.
𝑃(𝑠𝑢𝑐𝑐𝑒𝑠𝑠|𝑇) decreases
exponentially as 𝑇 increases.
27
[Image: Yoshua Bengio]
VANISHING GRADIENT: TOO MANY PRODUCTS!
In order to have (local) stability, the spectral radius of the matrix 𝑊ℎℎ must be lower than 1.
Consider state update equation ℎ 𝑡 + 1 = 𝑓 ℎ 𝑡 , 𝑢 𝑡 + 1 . We can see it is a recursive equation.
When input sequence is given, the previous equation can be rewritten explicitly as:
ℎ 𝑡 + 1 = 𝑓𝑡 ℎ 𝑡 = 𝑓𝑡 𝑓𝑡−1(…𝑓0(ℎ 0 )). (1)
The resulting gradient, relative to the loss at time 𝑡 will be:
𝜕𝐿𝑡𝜕𝑊
=
𝜏
𝜕𝐿𝑡𝜕ℎ𝑡
𝜕ℎ𝑡𝜕ℎ𝜏
𝜕ℎ𝜏𝜕𝑊
. (2)
The Jacobian of matrix derivatives𝜕ℎ𝑡
𝜕ℎ𝜏can be factorized as follows
𝜕ℎ𝑡𝜕ℎ𝜏
=𝜕ℎ𝑡𝜕ℎ𝑡−1
𝜕ℎ𝑡−1𝜕ℎ𝑡−2
…𝜕ℎ𝜏+1𝜕ℎ𝜏
= 𝑓𝑡′𝑓𝑡−1
′ …𝑓𝜏+1′ (3)
28
29
In order to reliably “store” information in the state of the network ℎ𝑡, RNN dynamics must remain close to a
stable attractor.
According to local stability analysis, the latter condition is met when 𝑓𝑡′ < 1
However, the previous product 𝜕ℎ𝑡
𝜕ℎ𝜏, expanded in (3) rapidly (exponentially) converges to 0 when 𝑡 − 𝜏 increases.
Consequently, the sum in (2) is dominated by terms corresponding to short-term dependencies.
This effects is called “vanishing gradient”.
As an effect, weights are less and less updates, as the gradient flows backward through the architecture.
On the other hand, when 𝑓𝑡′ > 1 we obtain an opposite effect called “exploding gradient”, which leads to
instability in the network.
HOW TO LIMIT VANISHING GRADIENT ISSUE?
Use ReLU activations (in RNN however, they cause the “dying neurons” problem).
Use LSTM or GRU architectures (discussed later).
Use a proper initialization of the weights in 𝑊.
30
WEIGHTS INITIALIZATION
This has to be repeated for each layer. The value n is the number of neurons in each layer.
This ensures that all neurons in the network initially have approximately the same output distribution.
The biases instead should be initialized to 0.
Random initialization is important to break symmetry (prevents coupling of neurons).31
A suitable initialization of the weights permits the gradient to flow quicker through the layers.
A smoother flow ensures faster convergence of the training procedure (faster reach of the minimum).
It also helps to reduce the issue of vanishing gradient.
When using sigmoids or hyperbolic tangent neurons, use the following weight initialization:
LEARNING RATE
The standard weights-update procedure is called Stochastic Gradient Descent (SGD).
SGD depends on a learning rate hyperparameter 𝛾:
𝑊𝑖ℎ = 𝑊𝑖
ℎ − 𝛾𝜕𝐿
𝜕𝑊𝑖ℎ , 𝑊ℎ
ℎ = 𝑊ℎℎ − 𝛾
𝜕𝐿
𝜕𝑊ℎℎ , 𝑊ℎ
𝑜 = 𝑊ℎ𝑜 − 𝛾
𝜕𝐿
𝜕𝑊ℎ𝑜 .
𝛾 is a critical parameter. How to set it?
Use cross-validationn to find optimal value.
Several strategies can be used to improve learning:
1. Annealing of the learning rate.
2. Second order methods.
3. Add momentum to SGD
4. Adaptive learning rate methods.32
ANNEALING OF THE LEARNING RATE
With a high learning rate, the system has too much kinetic energy and the parameter vector bounces around
chaotically, unable to settle down into deeper, but narrower parts of the loss function.
Possible solution: anneal the learning rate over time.
Decay it slowly and you’ll be wasting computation bouncing around chaotically with little improvement for a long
time.
Decay it too aggressively and the system will cool too quickly, unable to reach the best position it can.
3 common ways to decay learning rate:
1. Step decay: half 𝛾 every few epochs.
2. Exponential decay: 𝛾 = 𝛾0𝑒−𝑘𝑡, where 𝛾0 and 𝑘 are hyperparameters and 𝑡 is epoch number.
3. Multiplicative decay: 𝛾 =𝛾0
1+𝑘𝑡, where 𝛾0 and 𝑘 are hyperparameters and 𝑡 is epoch number.
33
SECOND ORDER METHODS
Weights are updated as follows
ℎ ← ℎ − 𝐻𝑓 ℎ −1𝛻𝑓(ℎ)
Where 𝐻𝑓 ℎ is the Hessian matrix, containing second-order partial derivatives of 𝑓.
Hessian describes the local curvature of the loss function.
More aggressive steps in directions of shallow curvature and shorter steps in directions of steep curvature.
Perform a more efficient update.
Impractical for most deep learning applications: computing (and inverting) the Hessian is a very costly process in both space and time.
Martens, James. "Deep learning via Hessian-free optimization.“, 2010: reduces cost, but still much slower than first-order methods.
34
SGD WITH MOMENTUM
In SGD update, gradient directly integrates the position.
With momentum, the gradient only directly influences the velocity, which in turn has an effect on the position.
Momentum update: 𝑣 ← 𝜇 ∙ 𝑣 − 𝛾𝜕𝑓 ℎ
𝜕ℎ, ℎ ← ℎ + 𝑣
Nesterov momentum (NAG): ℎ∗ ← ℎ + 𝜇 ∙ 𝑣, 𝑣 ← 𝜇 ∙ 𝑣 − 𝛾𝜕𝑓 ℎ∗
𝜕ℎ∗, ℎ ← ℎ + 𝑣
35
[Image: Andrej Karpathy]
ADAPTIVE LEARNING RATE
Adapts the learning rate to the parameters, performing larger updates for infrequent parameters and smaller
updates for frequent parameters.
Adagrad: 𝑔𝑖,𝑡 = 𝛻𝑊𝐽 𝑤𝑖 , 𝑤𝑖,𝑡+1 = 𝑤𝑖,𝑡 −𝛾
𝐺𝑡,𝑖𝑖+𝜖∙ 𝑔𝑖,𝑡, where 𝐺𝑡 is a diagonal matrix whose elements are the
sum of the squares of the gradients w.r.t. individual weights 𝑤𝑖 at time step 𝑡 (they might be updated differently)
and 𝜖 is a smoothing term that avoids division by zero.
Other adaptive learning rate methods are:
1. Adadelta and Rmsprop: reduce the aggressive, monotonically decreasing learning rate of Adagrad, by restricting the window
of accumulated past gradients to some fixed size.
2. Adam: adaptive learning rate + momentum.
3. Nadam: adaptive learning rate + Nesterov momentum.36
COMPARISON OF LEARNING PROCESS DYNAMICS (1/2)
Contours of a loss surface and
time evolution of different
optimization algorithms.
37
[Image: Alec Radford]
COMPARISON OF LEARNING PROCESS DYNAMICS (2/2)
A visualization of a saddle point in the optimization landscape.
The curvature along different dimension has different signs (one dimension curves up and another down – very common in deep learning!).
SGD has a very hard time breaking symmetry and gets stuck on the top.
RMSprop will see very low gradients in the saddle direction
38
[Image: Alec Radford]
4. RNN EXTENSIONS
39
DEEP RNN (1/2)
Increase the depth of RNN to increase expressive power.
N.B. here we add depth in SPACE (like FFNN), not in TIME.
Pascanu, R., Gulcehre, C., Cho, K., & Bengio, Y, “How to
construct deep recurrent neural networks”, 2013.
40
Standard RNN
Stacked RNN
(learns different time-scales at each layer –
from fast to slow dynamics)
DEEP RNN (2/2)
41
Deep input-to-hidden +
Deep hidden-to-hidden +
Deep hidden-to-output
Deep input-to-hidden +
Deep hidden-to-hidden +
Shortcut connections (useful for letting the
gradient flow faster during backpropagation).
BIDIRECTIONAL RNN
The output at time 𝑡 may not only depend on the
previous elements in the sequence, but also future
elements.
Example: to predict a missing word in a sequence
you want to look at both the left and the right
context.
Two RNNs stacked on top of each other.
Output computed based on the hidden state of
both RNNs.
Huang Zhiheng, Xu Wei, Yu Kai. Bidirectional LSTM
Conditional Random Field Models for Sequence Tagging.
42
DEEP (BIDIRECTIONAL) RNN
Similar to Bidirectional RNNs, but with multiple hidden layers per time
step.
Higher learning capacity.
Needs a lot of training data (the deeper the architecture the harder is
the training).
Graves Alex, Navdeep Jaitly, and Abdel-rahman Mohamed. "Hybrid speech
recognition with deep bidirectional LSTM.", 2013.
43
5. GATED RNNS
44
Ispired by: http://colah.github.io/posts/2015-08-Understanding-LSTMs/
LONG-TERM DEPENDENCIES
Due to vanishing gradient, RNN are uncapable of learning long-term dependencies.
Some applications require both short and long term dependencies.
Example: Natural Language Processing (NLP).
In some cases, short-term dependencies are sufficient to make predictions.
45
Consider to train the RNN to make 1-step ahead prediction.
To predict the word ‘sea’ it is sufficient to look only 3 step back in time.
In this case, it is sufficient to backpropagate the gradient 3 step back to succesfully learn this task.
46
Let’s stick to 1-step ahead prdiction.
Consider the sentence: I am from Rome. I was born 30 years ago. I like eating good food and riding my bike. My native
language is Italian.
When we want to predict the word Italian, we have to look back several time steps, up to the word Rome.
In this case, the short-term memory of the RNN would not do the trick.
As the gap in time grows, the harder for an RNN become to handle the problem.
Let’s see how Long-Short Term Memory (LSTM) can handle this difficulty.
LSTM OVERVIEW
Let’s recall the unfolded version of
a RNN.
A very simple processing unit is
repeated each time.
47
Introduced by Hochreiter & Schmidhuber (1997).
Work very well on many different problems and are widely used nowadays.
Like RNN, they must be unfolded in time to be trained and understood.
48
The processing unit of the LSTM is more complex and is called cell.
An LSTM cell is composed of 4 layers, interacting with each other in a special way.
CELL STATE AND GATES
The state of a cell at time 𝑡 is 𝐶𝑡.
The LSTM modify the state only through linear interactions:
information flows smoothly across time.
LSTM protect and control the information in the cell through 3
gates.
Gates are implemented by a sigmoid and a pointwise
multiplication.
49
Cell state
Gate
FORGET GATE
Decide what information should be discarded from the cell state.
𝑓𝑡 = 𝜎 𝑊𝑓 ∙ ℎ𝑡−1, 𝑥𝑡 + 𝑏𝑓 = ቊ0 → 𝑐𝑜𝑚𝑝𝑙𝑒𝑡𝑒𝑙𝑦 𝑔𝑒𝑡 𝑟𝑖𝑑 𝑜𝑓 𝑐𝑜𝑛𝑡𝑒𝑛𝑡 𝑖𝑛 𝐶𝑡1 → 𝑐𝑜𝑚𝑝𝑙𝑒𝑡𝑒𝑙𝑦 𝑘𝑒𝑒𝑝 𝑡ℎ𝑒 𝑐𝑜𝑛𝑡𝑒𝑛𝑡 𝑖𝑛 𝐶𝑡
50
Gate controlled by current input 𝑥𝑡 and
past cell output ℎ𝑡−1.
NLP example: cell state keep the gender
of the present subject to use correct
pronouns.
When sees a new subject, forget the
gender of the old subject.
UPDATE GATE
With forget gate we decided wheter or not to forget cell content.
After, with update gate we decide how much to update the old state 𝐶𝑡−1 with a new candidate ሚ𝐶𝑡.
Update gate: 𝑖𝑡 = 𝜎 𝑊𝑖 ∙ ℎ𝑡−1, 𝑥𝑡 + 𝑏𝑖 = ቊ0 → 𝑛𝑜 𝑢𝑝𝑑𝑎𝑡𝑒
1 → 𝑐𝑜𝑚𝑝𝑙𝑒𝑡𝑒𝑙𝑦 𝑢𝑝𝑑𝑎𝑡𝑒
51
New candidate: ሚ𝐶𝑡 = tanh 𝑊𝑐 ∙ ℎ𝑡−1, 𝑥𝑡 + 𝑏𝐶 .
New state: 𝐶𝑡 = 𝐶𝑡−1 + 𝑖𝑡 ∗ ሚ𝐶𝑡.
In the NLP example, we update the cell state as we see a new subject.
Note that the new candidate is computed exactly like the state in traditional RNN (same difference equation).
OUTPUT GATE
The output is a filtered version of the cell state.
Cell state is fed into a tanh, which squashed its values between -1 and 1.
Then, the gate select the part to be returned as output.
Output gate: 𝑜𝑡 = 𝜎 𝑊𝑜 ∙ ℎ𝑡−1, 𝑥𝑡 + 𝑏𝑜 = ቊ0 → 𝑛𝑜 𝑜𝑢𝑡𝑝𝑢𝑡
1 → 𝑟𝑒𝑡𝑢𝑟𝑛 𝑐𝑜𝑚𝑝𝑙𝑒𝑡𝑒 𝑐𝑒𝑙𝑙 𝑠𝑡𝑎𝑡𝑒
Cell output: ℎ𝑡 = 𝑜𝑡 ∗ tanh(𝐶𝑡)
52
In NLP example, after having seen a
subject, if a verb comes next, the cell
outputs information about being singluar
or plural.
COMPLETE FORWARD PROPAGATION STEP
𝑓𝑡 = 𝜎 𝑊𝑓 ∙ ℎ𝑡−1, 𝑥𝑡 + 𝑏𝑓 - forget gate
𝑖𝑡 = 𝜎 𝑊𝑖 ∙ ℎ𝑡−1, 𝑥𝑡 + 𝑏𝑖 - input gate
𝑜𝑡 = 𝜎 𝑊𝑜 ∙ ℎ𝑡−1, 𝑥𝑡 + 𝑏𝑜 - output gate
ሚ𝐶𝑡 = tanh 𝑊𝑐 ∙ ℎ𝑡−1, 𝑥𝑡 + 𝑏𝐶 - new state candidate
𝐶𝑡 = 𝐶𝑡−1 + 𝑖𝑡 ∗ ሚ𝐶𝑡 - new cell state
ℎ𝑡 = 𝑜𝑡 ∗ tanh(𝐶𝑡) – cell output
Parameters of the model: 𝑊𝑖 ,𝑊𝑐 ,𝑊𝑜 , 𝑏𝑖 , 𝑏𝑐 , 𝑏𝑜
Note that the weight matrix have a larger size than in normal RNN (they multiply the concatentation of 𝑥𝑡and ℎ𝑡). 53
LSTM DOWNSIDES
Gates are never really 1 or 0. The content of the cell is inevitably corrupted after long time.
Even if LSTM provides a huge improvement w.r.t. RNN, it still struggles with very long time dependencies.
Number of parameters: 4 ∙ 𝑁𝑖 + 1 ∙ 𝑁𝑜 +𝑁𝑜2.
Example: input = time series of 100 elements, LSTM units = 256 → 168960 parameters.
Scales up quickly!! Lot of parameters = lot of training data.
Rule of thumb: data elements must always be one order of magnitude greater than the number of parameters.
Memory problems when dealing with lot of data.
Long training time (use GPU computing).54
GATED RECURRENT UNITS
Several LSTM variants exists.
GRU is one of the most famous alternative architectures (Cho, et al. (2014).
It combines the forget and input gates into a single update gate.
It also merges the cell state and hidden state, and makes some other changes.
The cell is characterize by fewer parameters.
55
GRU FORWARD PROPAGATION STEP
𝑟𝑡 = 𝜎 𝑊𝑟 ∙ ℎ𝑡−1, 𝑥𝑡 - reset gate (merge of input and forget gate).
𝑧𝑡 = 𝜎 𝑊𝑧 ∙ ℎ𝑡−1, 𝑥𝑡 - output gate.
෨ℎ𝑡 = tanh 𝑊 ∙ 𝑟𝑡 ∗ ℎ𝑡−1, 𝑥𝑡 - new candidate output (merge internal state and output).
ℎ𝑡 = 1 − 𝑧𝑡 ∗ ℎ𝑡−1 + 𝑧𝑡 ∗ ෨ℎ𝑡 - cell output
56
Performs better LSTM or GRU?
Depends on the problem at hand: Chung, Junyoung, et al.
"Empirical evaluation of gated recurrent neural networks on
sequence modeling.“, 2014.
‘No free lunch’ theorem
6. ECHO STATE NETWORKS
57
ESN OVERVIEW
Introduced by Jaeger (2001).
With Liquid State Machine, they form the family of Reservoir Computing approaches.
Large, untrained recurrent layer (reservoir).
It has to be sparse and as much heterogeneous as possible to generate a large variety of internal dynamics.
Linear, memoryless readout trained with linear regression, picks the dynamics useful to solve the task at hand.
State of the art in real-valued time series prediction.
Pros:
Fast and easy to train (no backpropagation).
Relatively small, they are used in embedded systems.
Cons:
High sensitivity to hyperparameters (model parameters, usually set by hand).
Random initialization add stochasticity to the results. 58
59
Dashed boxes are randomly initialized weights, left untrained.
Solid boxes are readout weights, usually trained with linear
regression.
Outputs of the network are forced to match target values.
𝐿2 regularization is used in the linear regression to prevent
overfitting.
ESN state and output update:
ℎ 𝑡 + 1 = 𝑓(𝑊𝑟𝑟ℎ 𝑡 +𝑊𝑖
𝑟𝑥 𝑡 + 1 +𝑊𝑜𝑟𝑦[𝑡])
𝑦 𝑡 + 1 = 𝑔 𝑊𝑟𝑜ℎ 𝑡 + 1 +𝑊𝑖
𝑜𝑥 𝑡 + 1
𝑓() is usually implemented as a tanh.
g() is usually the identity function (more complicated readouts
are possible though).
PRINCIPAL ESN HYPERPARAMETERS
Reservoir size:
Should be large enough to provide a great variety of dynamics.
If too large, there could be overfit (the readout just learn a simple mapping 1-to-1 from network state to output (effect is
damped by regularization).
Reservoir spectral radius:
Is the largest eigenvalue of the reservoir matrix.
Controls the dynamics of the system. Should be tuned to provide a stable dynamics, yet sufficient rich (edge of chaos).
Input scaling:
Controls the amount of nonlinearity introduce by the neurons.
High value → high amount of nonlinearity.
60
61
COOL PEOPLE IN DEEP LEARNINGPeople whose work (partially) inspired this presentation.
Y. Bengio G. Hinton J. Schmidthuber A. Graves
C. Olah A. Karpathy
THE ENDTHANKS FOR THE ATTENTION
62