neural network and deep learning - welcome to iitpai-nlp-ml/resources/talks/deeplearning... ·...
TRANSCRIPT
![Page 1: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/1.jpg)
Neural Network
and
Deep Learning
Md Shad Akhtar
Research Scholar
IIT Patna
![Page 2: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/2.jpg)
Neural Network
• Mimics the functionality of a brain.
• A neural network is a graph with neurons
(nodes, units etc.) connected by links.
![Page 3: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/3.jpg)
Neural Network: Neuron
![Page 4: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/4.jpg)
Neural Network: Perceptron
• Network with only single layer.
• No hidden layers
![Page 5: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/5.jpg)
Neural Network: Perceptron
W2 = ?
t = ?
W1 = ?
AND Gate
X1
X2
a
W2 = ?
t = ?
W1 = ?
OR Gate
X1
X2
a
t = ? W1 = ?
NOT GateX1
a
![Page 6: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/6.jpg)
Neural Network: Perceptron
W2
= 1
t = 1.5
W1
= 1
AND Gate
X1
X2
a
W2
= 1
t = 0.5
W1
= 1
OR Gate
X1
X2
a
t = -0.5
W1
= -1
NOT GateX1
a
![Page 7: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/7.jpg)
Neural Network: Multi Layer Perceptron
(MLP) or Feed-Forward Network (FNN)
• Network with n+1 layers
• One output and n hidden layers.
• Typically n = 1
![Page 8: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/8.jpg)
Training: Back propagation algorithm
• Gradient decent algorithm
![Page 9: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/9.jpg)
Training: Back propagation algorithm
![Page 10: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/10.jpg)
Training: Back propagation algorithm
![Page 11: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/11.jpg)
Training: Back propagation algorithm
![Page 12: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/12.jpg)
Training: Back propagation algorithm
![Page 13: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/13.jpg)
Training: Back propagation algorithm
1. Initialize network with random weights
2. For all training cases (called examples):
a. Present training inputs to network and calculate output
b. For all layers (starting with output layer, back to input layer):
i. Compare network output with correct output (error function)
ii. Adapt weights in current layer
![Page 14: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/14.jpg)
Deep Learning
![Page 15: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/15.jpg)
What is Deep Learning?
• A family of methods that uses deep architectures to learn high-level feature representations
![Page 16: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/16.jpg)
Example 1
![Page 17: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/17.jpg)
Example 2
MAN
![Page 18: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/18.jpg)
Why are Deep Architectures hard to train?
• Vanishing gradient problem in Back Propagation
![Page 19: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/19.jpg)
Layer-wise Pre-training
• First, train one layer
at a time, optimizing
data-likelihood
objective P(x)
![Page 20: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/20.jpg)
Layer-wise Pre-training
• Then, train second
layer next, optimizing
data-likelihood
objective P(h)
![Page 21: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/21.jpg)
Layer-wise Pre-training
• Finally, fine-tune labelled objective P(y|x) by
Backpropagation
![Page 22: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/22.jpg)
Deep Belief Nets
• Uses Restricted Boltzmann Machines (RBMs)
• Hinton et al. (2006), A fast learning algorithm
for deep belief nets.
![Page 23: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/23.jpg)
Restricted Boltzmann Machine (RBM)
• RBM is a simple energy-based model:
where
Example:
• Let weights (h1; x1), (h1; x3) be positive, others be
zero, b = d = 0.
• Calculate p(x,h) ?
• Ans: p(x1 = 1; x2 = 0; x3 = 1; h1 = 1; h2 = 0; h3 = 0)
![Page 24: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/24.jpg)
Restricted Boltzmann Machine (RBM)
• P(x, h) = P(h|x) P(x)
• P(h|x): easy to compute
• P(x): hard if datasets are large.
Contrastive Divergence:
![Page 25: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/25.jpg)
Deep Belief Nets (DBN) = Stacked RBM
![Page 26: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/26.jpg)
Auto-Encoders: Simpler alternative to
RBMs
![Page 27: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/27.jpg)
Deep Learning - Architecture
• Recurrent Neural Network (RNN)
• Convolution Neural Network (CNN)
![Page 28: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/28.jpg)
Recurrent Neural Network (RNN)
![Page 29: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/29.jpg)
Recurrent Neural Network (RNN)
• Enable networks to do temporal processing
and learn sequences
![Page 30: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/30.jpg)
Character level language model Vocabulary: [h,e,l,o]
![Page 31: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/31.jpg)
Training of RNN: BPTT
: Actual
: Predicted
U
W
V
.
![Page 32: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/32.jpg)
Training of RNN: BPTT
.
![Page 33: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/33.jpg)
One to many:
Sequence output (e.g. image captioning takes an image and outputs a sentence of
words)
Many to one:
Sequence input (e.g. sentiment analysis where a given sentence is classified as
expressing positive or negative sentiment)
Many to many:
Sequence input and sequence output (e.g. Machine Translation: an RNN reads a
sentence in English and then outputs a sentence in French)
Many to many:
Synced sequence input and output (e.g. Language modelling where we wish to
predict next words.
![Page 34: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/34.jpg)
RNN Extension
• Bidirectional RNN
• Deep (Bidirectional) RNNs
![Page 35: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/35.jpg)
RNN (Cont..)
• “the clouds are in the sky”
the clouds are in the
W1theclouds are in
![Page 36: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/36.jpg)
RNN (Cont..)
• “India is my home country. I can speak fluent Hindi.”
India is my speak fluent
fluentmyis home W2
Ideally: It should
Practically: It is very hard for RNN to learn “Long Term Dependency”.
![Page 37: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/37.jpg)
LSTM
• Capable of learning long-term dependencies.
Simple RNN
LSTM
![Page 38: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/38.jpg)
LSTM
• LSTM remove or add information to the cell
state, carefully regulated by structures called
gates.
• Cell state: Conveyer belt of the cell
![Page 39: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/39.jpg)
LSTM
• Gates
– Forget Gate
– Input Gate
– Output Gate
![Page 40: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/40.jpg)
LSTM
• Gates
– Forget Gate
– Input Gate
– Output Gate
![Page 41: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/41.jpg)
LSTM
• Gates
– Forget Gate
– Input Gate
– Output Gate
![Page 42: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/42.jpg)
LSTM
• Gates
– Forget Gate
– Input Gate
– Output Gate
![Page 43: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/43.jpg)
LSTM- Variants
![Page 44: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/44.jpg)
Convolutional Neural Network (CNN)
![Page 45: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/45.jpg)
Convolutional Neural Network (CNN)
• A special kind of multi-layer neural networks.
• Implicitly extract relevant features.
• Fully-connected network architecture does not
take into account the spatial structure.
• In contrast, CNN tries to take advantage of the s
patial structure.
![Page 46: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/46.jpg)
Convolutional Neural Network (CNN)
1. Convolutional layer
2. Pooling layer
3. Fully connected layer
![Page 47: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/47.jpg)
Convolutional Neural Network (CNN)
1. Convolutional layer1 0 1
0 1 0
1 0 1
1 1 1 0 0
0 1 1 1 0
0 0 1 1 1
0 0 1 1 0
0 1 1 0 0
Image
Convolution Filter
![Page 48: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/48.jpg)
Convolutional Neural Network (CNN)
1. Convolutional layer1 0 1
0 1 0
1 0 1
![Page 49: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/49.jpg)
Convolutional Neural Network (CNN)
1 0 1
0 1 0
1 0 1
1. Convolutional layer• Local receptive field
• Shared weights
![Page 50: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/50.jpg)
Convolutional Neural Network (CNN)
2. Pooling layer
![Page 51: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/51.jpg)
Convolutional Neural Network (CNN)
3. Fully connected layer
![Page 52: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/52.jpg)
Convolutional Neural Network (CNN)
Putting it all together
![Page 53: Neural Network and Deep Learning - Welcome to IITPai-nlp-ml/resources/talks/DeepLearning... · 2017-02-10 · Neural Network and Deep Learning Md Shad Akhtar Research Scholar IIT](https://reader031.vdocument.in/reader031/viewer/2022040308/5f07e8e07e708231d41f5e28/html5/thumbnails/53.jpg)
References
• http://www.wildml.com/2015/09/recurrent-
neural-networks-tutorial-part-1-introduction-
to-rnns/
• http://karpathy.github.io/2015/05/21/rnn-
effectiveness/
• http://colah.github.io/posts/2015-08-
Understanding-LSTMs/
• http://cl.naist.jp/~kevinduh/a/deep2014/