![Page 1: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/1.jpg)
1
Pattern Recognition:Statistical and Neural
Lonnie C. Ludeman
Lecture 20
Oct 26, 2005
Nanjing University of Science & Technology
![Page 2: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/2.jpg)
2
Lecture 20 Topics
1. Perceptron Algorithm Revisited
2. Local Delta Training Algorithm for ANE
3. General Definition of Neural Networks
4. Basic Neural Network Structures-Examples
5. Analysis and Synthesis of Neural Networks
![Page 3: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/3.jpg)
3
Signum Function Activation Training Algorithm(Perceptron)
Weight Update Algorithm
y = +1 if input vector x is from C1
y = -1 if input vector x is from C2
Review
![Page 4: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/4.jpg)
4
How do we train an Artificial Neural Element(ANE) to do classification ???
Question
Answer
Use the Delta Training Algorithm !!!
![Page 5: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/5.jpg)
5
Given an Artificial Neural Element as follows
Wish to find weight vector such that training patterns are correctly classified
![Page 6: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/6.jpg)
6
x(p) ε { x1, x2, … , xK }
d( x(p) ) = { d(x1), d(x2), … , d(xK) }
Define a performance measure Ep for sample x(p) and decision d[ x(p) ] as
Given:
![Page 7: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/7.jpg)
7
Use the gradient method to minimize EpNew Weight wk+1 in terms of
previous weight wk
where the Gradient is
Derivation of Delta weight update Equation
![Page 8: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/8.jpg)
8
Substituting the gradient vector into the weight update gives the General Local Delta Algorithm
or rewriting gives
w(p+1) = w(p) + {d[x(p)] – f(net)} f /(net)) x(p)where net = wT(p)x(p)
General Local Delta Algorithm Weight Update Equation
![Page 9: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/9.jpg)
9
Continuous Perceptron Training Algorithm
Sometimes called the
![Page 10: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/10.jpg)
10
Case 1: Local Delta Algorithm for Training an ANE with Logistic Activation Function
Given:
Solution:
![Page 11: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/11.jpg)
11
Substituting the derivative gives the Local algorithm for the Logistic Activation function as
Local Weight Update Equation for Logistic Activation Function
![Page 12: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/12.jpg)
12
Case 2: Local Delta Algorithm for for Training an ANE - Hyperbolic Tangent Activation Function
Given:
Solution; Taking derivative of the nonlinearity and substituting into the general update equation yields the following
Local Weight Update Equation for Hyperbolic Activation Function
![Page 13: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/13.jpg)
13
Scale Factors for Case 2: Tanh Activation Function SF = ( d[x(p) ] –f(net) )(1 – f 2(net) )
d[x(p)]= 1 SF1 = ( 1 – f(net) )(1 – f 2(net) )
d[x(p)] = -1 SF-1 = ( -1 – f(net) )(1 – f 2(net) )
d[x(p)]= 1 d[x(p)] = -1
![Page 14: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/14.jpg)
14
Scale Factors for Case 2: Tanh Activation Function (desired values = +0.9 and -0.9 )
![Page 15: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/15.jpg)
15
Case 3: Local Delta Algorithm for Training an ANE - Linear Activation Function
Given:
Solution:Taking derivative and substituting in general update equation gives
Local Weight Update Equation for Linear Activation Function
( Widrow-Hoff Training Rule )
![Page 16: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/16.jpg)
16
General Global Delta Algorithm
Define a performance measure ETOT for all samples xk and decisions d[ xk) ] as
Using Gradient technique gives the Global Delta Algorithm as
Global Weight Update Equation
![Page 17: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/17.jpg)
17
Definitions
A Neural Network is defined as any connection of Neural Elements.
An Artificial Neural Network is defined as any connection of Artificial Neural Elements.
![Page 18: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/18.jpg)
18
Examples of Artificial Neural Networks
(a) Two Layer neural Network
(b) Special Three Layer Form: Hyperplane-AND-OR structure
(c) General 3-Layer Feedforward structure and nomenclature
Feedback Artificial Neural Networks
(d) One Layer Hopfield Net
(e) Two Layer Feedback
Feed Forward Artificial Neural Networks
![Page 19: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/19.jpg)
19
(a) Example - Two Layer Neural Network Using Signum Nonlinearity
![Page 20: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/20.jpg)
20
(b) Special Hyperplane-AND-OR structure
x
Hyperplanes Logical AND
Logical OR
yinput outputLayer
1Layer 2
Layer 3
![Page 21: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/21.jpg)
21
Building Block- Hyperplane
μ
![Page 22: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/22.jpg)
22
Building Block- AND
μ
-(n-½)
![Page 23: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/23.jpg)
23
Building Block- OR
½
μ
![Page 24: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/24.jpg)
24
AND Layer
OR Layer
Hyperplanes Layerall f(·) = u(·) unit step
(b) Example- Hyperplanes-AND-OR Structure
![Page 25: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/25.jpg)
25
(c) General Feedforward Structure
![Page 26: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/26.jpg)
26
(d) Example: Feedback Structure one Layer
![Page 27: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/27.jpg)
27
(e) Example: Feedback Structure Two Layer
/
![Page 28: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/28.jpg)
28
Definitions:
Analysis of Neural Networks-
Synthesis of Neural Networks-
Given a Neural Network describe the output for all inputs ( Mathematical or computer generated)
Given a list of properties and requirements build a Neural Network to satisfy the requirements ( Mathematical or computer generated)
![Page 29: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/29.jpg)
29
Example: Analyze the following Neural Network
-1
1
-1
110
00
1
Determine the output y1(2)
for all (x1,x2).
Solution:
(Next Lecture)
![Page 30: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/30.jpg)
30
Example: Synthesize a Neural Network
Given the following decision regions build a neural network to perform the classification process
Solution: Use Hyperplane-AND-OR Structure (Next Lecture)
![Page 31: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/31.jpg)
31
Summary Lecture 20
1. Perceptron Algorithm Revisited
2. Local Delta Training Algorithms for ANE
3. General Definition of Neural Networks
4. Basic Neural Network Structures-Examples
5. Analysis and Synthesis of Neural Networks
![Page 32: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/32.jpg)
32
Question
How do we train an Artificial Neural Network to perform the classification problem???
Answer
Not a simple answer but we will look at one way that uses the backpropagation algorithm to do the Training.
Not Today, we have to wait until Friday.
☺☻☺☻☺☻☺☻☺
![Page 33: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 20 Oct 26, 2005 Nanjing University of Science & Technology](https://reader031.vdocument.in/reader031/viewer/2022032606/56649eb35503460f94bba52b/html5/thumbnails/33.jpg)
33
End of Lecture 20