neural networks - iima · neural networks prof. ankur sinha indian institute of management...
TRANSCRIPT
![Page 1: Neural Networks - IIMA · Neural Networks Prof. Ankur Sinha Indian Institute of Management Ahmedabad Gujarat India. A typical Neuron Information Flow Releases neurotransmitters to](https://reader035.vdocument.in/reader035/viewer/2022081408/60612c6702bc3625116a3434/html5/thumbnails/1.jpg)
Neural Networks
Prof. Ankur SinhaIndian Institute of Management Ahmedabad
Gujarat India
![Page 2: Neural Networks - IIMA · Neural Networks Prof. Ankur Sinha Indian Institute of Management Ahmedabad Gujarat India. A typical Neuron Information Flow Releases neurotransmitters to](https://reader035.vdocument.in/reader035/viewer/2022081408/60612c6702bc3625116a3434/html5/thumbnails/2.jpg)
A typical Neuron
Information Flow
Releases neurotransmitters to other neurons
![Page 3: Neural Networks - IIMA · Neural Networks Prof. Ankur Sinha Indian Institute of Management Ahmedabad Gujarat India. A typical Neuron Information Flow Releases neurotransmitters to](https://reader035.vdocument.in/reader035/viewer/2022081408/60612c6702bc3625116a3434/html5/thumbnails/3.jpg)
Applications
• Speech recognition
• Handwriting recognition
• Driverless Cars
• Products: Google translate, Alexa
![Page 4: Neural Networks - IIMA · Neural Networks Prof. Ankur Sinha Indian Institute of Management Ahmedabad Gujarat India. A typical Neuron Information Flow Releases neurotransmitters to](https://reader035.vdocument.in/reader035/viewer/2022081408/60612c6702bc3625116a3434/html5/thumbnails/4.jpg)
![Page 5: Neural Networks - IIMA · Neural Networks Prof. Ankur Sinha Indian Institute of Management Ahmedabad Gujarat India. A typical Neuron Information Flow Releases neurotransmitters to](https://reader035.vdocument.in/reader035/viewer/2022081408/60612c6702bc3625116a3434/html5/thumbnails/5.jpg)
![Page 6: Neural Networks - IIMA · Neural Networks Prof. Ankur Sinha Indian Institute of Management Ahmedabad Gujarat India. A typical Neuron Information Flow Releases neurotransmitters to](https://reader035.vdocument.in/reader035/viewer/2022081408/60612c6702bc3625116a3434/html5/thumbnails/6.jpg)
MLP Architecture
Inputlayer
Outputlayer
Hidden Layers
![Page 7: Neural Networks - IIMA · Neural Networks Prof. Ankur Sinha Indian Institute of Management Ahmedabad Gujarat India. A typical Neuron Information Flow Releases neurotransmitters to](https://reader035.vdocument.in/reader035/viewer/2022081408/60612c6702bc3625116a3434/html5/thumbnails/7.jpg)
A Simple Architecture
x1
x2
xn
w1
w2
wn
a=i wi xi
y
1 if a qy=
0 if a < q{
inputs
activationoutput
q
A Threshold Logic Unit
![Page 8: Neural Networks - IIMA · Neural Networks Prof. Ankur Sinha Indian Institute of Management Ahmedabad Gujarat India. A typical Neuron Information Flow Releases neurotransmitters to](https://reader035.vdocument.in/reader035/viewer/2022081408/60612c6702bc3625116a3434/html5/thumbnails/8.jpg)
Decision Surface of a TLU
x1
x2
Decision line
w1 x1 + w2 x2 = q
1
1 1
0
0
00
0
1
A TLU works as a linear classifier
Similar to SVM?
How do you identify the weights and threshold?
![Page 9: Neural Networks - IIMA · Neural Networks Prof. Ankur Sinha Indian Institute of Management Ahmedabad Gujarat India. A typical Neuron Information Flow Releases neurotransmitters to](https://reader035.vdocument.in/reader035/viewer/2022081408/60612c6702bc3625116a3434/html5/thumbnails/9.jpg)
Types of Activation Functions
a
y
a
y
a
y
a
y
linear
piece-wise linear sigmoid
threshold
![Page 10: Neural Networks - IIMA · Neural Networks Prof. Ankur Sinha Indian Institute of Management Ahmedabad Gujarat India. A typical Neuron Information Flow Releases neurotransmitters to](https://reader035.vdocument.in/reader035/viewer/2022081408/60612c6702bc3625116a3434/html5/thumbnails/10.jpg)
Types of Activation Functions
![Page 11: Neural Networks - IIMA · Neural Networks Prof. Ankur Sinha Indian Institute of Management Ahmedabad Gujarat India. A typical Neuron Information Flow Releases neurotransmitters to](https://reader035.vdocument.in/reader035/viewer/2022081408/60612c6702bc3625116a3434/html5/thumbnails/11.jpg)
Training Neural Network
• A training set S of examples {x,t} is required
– x is an input vector
– t is the desired target vector
• Finding acceptable values of w and q
– Assume some values for w and q
– For the training example x, compute the network output y
– Compare output y with targets t, a difference denotes error
– Adjust w and q so that the error can be reduced
– Accept w and q that leads to minimum error
![Page 12: Neural Networks - IIMA · Neural Networks Prof. Ankur Sinha Indian Institute of Management Ahmedabad Gujarat India. A typical Neuron Information Flow Releases neurotransmitters to](https://reader035.vdocument.in/reader035/viewer/2022081408/60612c6702bc3625116a3434/html5/thumbnails/12.jpg)
A Linear Unit
x1
x2
xn
w1
w2
wna=i wi xi
inputs
output
y
y=a
Tries to give the best linear relationship between input and outputSimilar to regression?
![Page 13: Neural Networks - IIMA · Neural Networks Prof. Ankur Sinha Indian Institute of Management Ahmedabad Gujarat India. A typical Neuron Information Flow Releases neurotransmitters to](https://reader035.vdocument.in/reader035/viewer/2022081408/60612c6702bc3625116a3434/html5/thumbnails/13.jpg)
Neuron with Sigmoid Function
x1
x2
xn
w1
w2
wn
a=i wi xi
y
inputs
activationoutput
A Threshold Logic Unit
y=s(a) =1/(1+e-a)
Gradient descent rules are used to learn the parameters of the NN
![Page 14: Neural Networks - IIMA · Neural Networks Prof. Ankur Sinha Indian Institute of Management Ahmedabad Gujarat India. A typical Neuron Information Flow Releases neurotransmitters to](https://reader035.vdocument.in/reader035/viewer/2022081408/60612c6702bc3625116a3434/html5/thumbnails/14.jpg)
Multiple Layers
input layer
hidden layer
output layer
Backpropagation approach is used to train the neural network
![Page 15: Neural Networks - IIMA · Neural Networks Prof. Ankur Sinha Indian Institute of Management Ahmedabad Gujarat India. A typical Neuron Information Flow Releases neurotransmitters to](https://reader035.vdocument.in/reader035/viewer/2022081408/60612c6702bc3625116a3434/html5/thumbnails/15.jpg)
More about NN Parameters
• The weights of the neural network are determined by training data
• As more training data is obtained the weights should be updated
![Page 16: Neural Networks - IIMA · Neural Networks Prof. Ankur Sinha Indian Institute of Management Ahmedabad Gujarat India. A typical Neuron Information Flow Releases neurotransmitters to](https://reader035.vdocument.in/reader035/viewer/2022081408/60612c6702bc3625116a3434/html5/thumbnails/16.jpg)
Neural Networks are Universal
• Any boolean function can be learnt by a neural network with single hidden layer– It might require a large number of hidden units
• Any mathematical function that is continuous and bounded can be approximated to an arbitrarily small accuracy using a neural network with one hidden layer– A large number of hidden units might be required if the error of
approximation is very small
![Page 17: Neural Networks - IIMA · Neural Networks Prof. Ankur Sinha Indian Institute of Management Ahmedabad Gujarat India. A typical Neuron Information Flow Releases neurotransmitters to](https://reader035.vdocument.in/reader035/viewer/2022081408/60612c6702bc3625116a3434/html5/thumbnails/17.jpg)
Be Careful!
• Neural network can easily lead to overfitting
• Try to minimize the generalization error than the training error
![Page 18: Neural Networks - IIMA · Neural Networks Prof. Ankur Sinha Indian Institute of Management Ahmedabad Gujarat India. A typical Neuron Information Flow Releases neurotransmitters to](https://reader035.vdocument.in/reader035/viewer/2022081408/60612c6702bc3625116a3434/html5/thumbnails/18.jpg)
Be Careful!
• Neural network can easily lead to overfitting
• Try to minimize the generalization error than the training error
Evaluation Point 1
![Page 19: Neural Networks - IIMA · Neural Networks Prof. Ankur Sinha Indian Institute of Management Ahmedabad Gujarat India. A typical Neuron Information Flow Releases neurotransmitters to](https://reader035.vdocument.in/reader035/viewer/2022081408/60612c6702bc3625116a3434/html5/thumbnails/19.jpg)
Be Careful!
• Neural network can easily lead to overfitting
• Try to minimize the generalization error than the training error
Evaluation Point 1
Evaluation Point 2
![Page 20: Neural Networks - IIMA · Neural Networks Prof. Ankur Sinha Indian Institute of Management Ahmedabad Gujarat India. A typical Neuron Information Flow Releases neurotransmitters to](https://reader035.vdocument.in/reader035/viewer/2022081408/60612c6702bc3625116a3434/html5/thumbnails/20.jpg)
MNIST Database
The MNIST database contains 60,000 training images and 10,000 testing images.
MNIST: Modified National Institute of Standards and Technology database
![Page 21: Neural Networks - IIMA · Neural Networks Prof. Ankur Sinha Indian Institute of Management Ahmedabad Gujarat India. A typical Neuron Information Flow Releases neurotransmitters to](https://reader035.vdocument.in/reader035/viewer/2022081408/60612c6702bc3625116a3434/html5/thumbnails/21.jpg)
MNIST Database
![Page 22: Neural Networks - IIMA · Neural Networks Prof. Ankur Sinha Indian Institute of Management Ahmedabad Gujarat India. A typical Neuron Information Flow Releases neurotransmitters to](https://reader035.vdocument.in/reader035/viewer/2022081408/60612c6702bc3625116a3434/html5/thumbnails/22.jpg)
MNIST Database
Input
Output
Hidden Layers
784 Pixels
Each of the output nodes fires a 0 or 1(or the probability)