neural networks: backpropagation algorithm data mining and semantic web university of belgrade...
TRANSCRIPT
![Page 1: Neural Networks: Backpropagation algorithm Data Mining and Semantic Web University of Belgrade School of Electrical Engineering Chair of Computer Engineering](https://reader035.vdocument.in/reader035/viewer/2022081516/56649c815503460f94938e09/html5/thumbnails/1.jpg)
Neural Networks: Backpropagation algorithm
Data Mining and Semantic Web
University of BelgradeSchool of Electrical Engineering Chair of Computer Engineering and Information Theory
Miroslav Tiš[email protected]
![Page 2: Neural Networks: Backpropagation algorithm Data Mining and Semantic Web University of Belgrade School of Electrical Engineering Chair of Computer Engineering](https://reader035.vdocument.in/reader035/viewer/2022081516/56649c815503460f94938e09/html5/thumbnails/2.jpg)
You see this:
But the camera sees this:
What is this?
23.12.2011. Miroslav Tišma 2/21
![Page 3: Neural Networks: Backpropagation algorithm Data Mining and Semantic Web University of Belgrade School of Electrical Engineering Chair of Computer Engineering](https://reader035.vdocument.in/reader035/viewer/2022081516/56649c815503460f94938e09/html5/thumbnails/3.jpg)
Computer Vision: Car detection
Testing:
What is this?
Not a carCars
23.12.2011. Miroslav Tišma 3/21
![Page 4: Neural Networks: Backpropagation algorithm Data Mining and Semantic Web University of Belgrade School of Electrical Engineering Chair of Computer Engineering](https://reader035.vdocument.in/reader035/viewer/2022081516/56649c815503460f94938e09/html5/thumbnails/4.jpg)
pixel 1
pixel 2
Raw image
Cars“Non”-Cars
50 x 50 pixel images→ 2500 pixels (7500 if RGB)
pixel 1 intensity
pixel 2 intensity
pixel 2500 intensity
Quadratic features ( ): ≈3 million features
Learning Algorithm
pixel 1
pixel 2
23.12.2011. Miroslav Tišma 4/21
![Page 5: Neural Networks: Backpropagation algorithm Data Mining and Semantic Web University of Belgrade School of Electrical Engineering Chair of Computer Engineering](https://reader035.vdocument.in/reader035/viewer/2022081516/56649c815503460f94938e09/html5/thumbnails/5.jpg)
Neural Networks
• Origins: Algorithms that try to mimic the brain
• Was very widely used in 80s and early 90s; popularity diminished in late 90s.
• Recent resurgence: State-of-the-art technique for many applications
23.12.2011. Miroslav Tišma 5/21
![Page 6: Neural Networks: Backpropagation algorithm Data Mining and Semantic Web University of Belgrade School of Electrical Engineering Chair of Computer Engineering](https://reader035.vdocument.in/reader035/viewer/2022081516/56649c815503460f94938e09/html5/thumbnails/6.jpg)
Neurons in the brain
Dendr(I)tes
Ax(O)n
23.12.2011. Miroslav Tišma 6/21
![Page 7: Neural Networks: Backpropagation algorithm Data Mining and Semantic Web University of Belgrade School of Electrical Engineering Chair of Computer Engineering](https://reader035.vdocument.in/reader035/viewer/2022081516/56649c815503460f94938e09/html5/thumbnails/7.jpg)
Neuron model: Logistic unit
Sigmoid (logistic) activation function.
hΘ (𝑥 )= 1
1+𝑒−Θ𝑇 𝑥
𝑔 (𝑧 )= 1
1+𝑒− 𝑧
“bias unit”
“output”
“input wires”
“weights” - parameters
23.12.2011. Miroslav Tišma 7/21
![Page 8: Neural Networks: Backpropagation algorithm Data Mining and Semantic Web University of Belgrade School of Electrical Engineering Chair of Computer Engineering](https://reader035.vdocument.in/reader035/viewer/2022081516/56649c815503460f94938e09/html5/thumbnails/8.jpg)
Neural Network
Layer 3Layer 1 Layer 2
“bias unit”
“output layer”“hidden layer”“input layer”
23.12.2011. Miroslav Tišma 8/21
![Page 9: Neural Networks: Backpropagation algorithm Data Mining and Semantic Web University of Belgrade School of Electrical Engineering Chair of Computer Engineering](https://reader035.vdocument.in/reader035/viewer/2022081516/56649c815503460f94938e09/html5/thumbnails/9.jpg)
Neural Network“activation” of unit in layer
matrix of weights controlling function mapping from layer to layer
If network has units in layer , units in layer , then will be of dimension .
23.12.2011. Miroslav Tišma 9/21
![Page 10: Neural Networks: Backpropagation algorithm Data Mining and Semantic Web University of Belgrade School of Electrical Engineering Chair of Computer Engineering](https://reader035.vdocument.in/reader035/viewer/2022081516/56649c815503460f94938e09/html5/thumbnails/10.jpg)
Simple example: AND
0 00 11 01 1
-30
+20
+20
hΘ (𝑥 )=𝑔(−30+20 𝑥1+20 𝑥2)
hΘ (𝑥 )≈ 𝑥1𝐴𝑁𝐷 𝑥223.12.2011. Miroslav Tišma 10/21
![Page 11: Neural Networks: Backpropagation algorithm Data Mining and Semantic Web University of Belgrade School of Electrical Engineering Chair of Computer Engineering](https://reader035.vdocument.in/reader035/viewer/2022081516/56649c815503460f94938e09/html5/thumbnails/11.jpg)
Example: OR function
0 00 11 01 1
-10
+20
+20
hΘ (𝑥 )=𝑔(−10+20 𝑥1+20𝑥2)
hΘ (𝑥 )≈ 𝑥1𝑂𝑅𝑥223.12.2011. Miroslav Tišma 11/21
![Page 12: Neural Networks: Backpropagation algorithm Data Mining and Semantic Web University of Belgrade School of Electrical Engineering Chair of Computer Engineering](https://reader035.vdocument.in/reader035/viewer/2022081516/56649c815503460f94938e09/html5/thumbnails/12.jpg)
Multiple output units: One-vs-all.
Pedestrian Car Motorcycle Truck
Want ,
when pedestrian 23.12.2011. Miroslav Tišma 12/21
when car when motorcycle
, etc.,
![Page 13: Neural Networks: Backpropagation algorithm Data Mining and Semantic Web University of Belgrade School of Electrical Engineering Chair of Computer Engineering](https://reader035.vdocument.in/reader035/viewer/2022081516/56649c815503460f94938e09/html5/thumbnails/13.jpg)
Neural Network (Classification)
Binary classification
1 output unit
Layer 1 Layer 2 Layer 3 Layer 4
Multi-class classification (K classes)
K output units
total no. of layers in network
no. of units (not counting bias unit) in layer
pedestrian car motorcycle truck
E.g. , , ,
23.12.2011. Miroslav Tišma 13/21
![Page 14: Neural Networks: Backpropagation algorithm Data Mining and Semantic Web University of Belgrade School of Electrical Engineering Chair of Computer Engineering](https://reader035.vdocument.in/reader035/viewer/2022081516/56649c815503460f94938e09/html5/thumbnails/14.jpg)
Cost function
Logistic regression:
23.12.2011. Miroslav Tišma 14/21
Neural network:
![Page 15: Neural Networks: Backpropagation algorithm Data Mining and Semantic Web University of Belgrade School of Electrical Engineering Chair of Computer Engineering](https://reader035.vdocument.in/reader035/viewer/2022081516/56649c815503460f94938e09/html5/thumbnails/15.jpg)
Gradient computation
Need code to compute:- -
23.12.2011. Miroslav Tišma 15/21
Our goal is to minimize the cost function
![Page 16: Neural Networks: Backpropagation algorithm Data Mining and Semantic Web University of Belgrade School of Electrical Engineering Chair of Computer Engineering](https://reader035.vdocument.in/reader035/viewer/2022081516/56649c815503460f94938e09/html5/thumbnails/16.jpg)
Given one training example ( , ):Forward propagation:
Layer 1 Layer 2 Layer 3 Layer 4
𝑎 (1 ) 𝑎 (2) 𝑎 (3) 𝑎 (4 )
23.12.2011. Miroslav Tišma 16/21
Backpropagation algorithm
![Page 17: Neural Networks: Backpropagation algorithm Data Mining and Semantic Web University of Belgrade School of Electrical Engineering Chair of Computer Engineering](https://reader035.vdocument.in/reader035/viewer/2022081516/56649c815503460f94938e09/html5/thumbnails/17.jpg)
Backpropagation algorithm
Intuition: “error” of node in layer .
Layer 1 Layer 2 Layer 3 Layer 4
For each output unit (layer L = 4) 𝛿(4 )𝛿(3 )𝛿(2 )
(h𝜃 (𝑥 ) ) 𝑗
the derivate of activation function can be written as
𝜕𝜕Θ𝑖𝑗
❑ 𝐽 (𝜃 )=𝑎(𝑙)𝛿❑(𝑙+1)
element-wise multiplication operator
23.12.2011. Miroslav Tišma 17/21
![Page 18: Neural Networks: Backpropagation algorithm Data Mining and Semantic Web University of Belgrade School of Electrical Engineering Chair of Computer Engineering](https://reader035.vdocument.in/reader035/viewer/2022081516/56649c815503460f94938e09/html5/thumbnails/18.jpg)
Backpropagation algorithmTraining set
Set (for all ).
ForSetPerform forward propagation to compute for Using , computeCompute
used to compute
23.12.2011. Miroslav Tišma 18/21
![Page 19: Neural Networks: Backpropagation algorithm Data Mining and Semantic Web University of Belgrade School of Electrical Engineering Chair of Computer Engineering](https://reader035.vdocument.in/reader035/viewer/2022081516/56649c815503460f94938e09/html5/thumbnails/19.jpg)
Advantages:- Relatively simple implementation- Standard method and generally wokrs well- Many practical applications: * handwriting recognition, autonomous driving car
Disadvantages:- Slow and inefficient- Can get stuck in local minima resulting in sub-optimal solutions
23.12.2011. Miroslav Tišma 19/21
![Page 20: Neural Networks: Backpropagation algorithm Data Mining and Semantic Web University of Belgrade School of Electrical Engineering Chair of Computer Engineering](https://reader035.vdocument.in/reader035/viewer/2022081516/56649c815503460f94938e09/html5/thumbnails/20.jpg)
Literature:
- http://en.wikipedia.org/wiki/Backpropagation
- http://www.ml-class.org
- http://home.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html
23.12.2011. Miroslav Tišma 20/21
![Page 21: Neural Networks: Backpropagation algorithm Data Mining and Semantic Web University of Belgrade School of Electrical Engineering Chair of Computer Engineering](https://reader035.vdocument.in/reader035/viewer/2022081516/56649c815503460f94938e09/html5/thumbnails/21.jpg)
23.12.2011. Miroslav Tišma 21/21
Thank you for your attention!