introduction to neural networks (under graduate course) lecture 5 of 9

Neural Networks

Dr. Randa Elanwar

Lecture 5

Lecture Content

• Linearly separable functions: 2-class problem implementation

– Learning laws: Perceptron learning rule

– Pattern mode solution method

– Batch mode solution method

2Neural Networks Dr. Randa Elanwar

Linearly Separable Functions

• Example: logical OR, with initial weights 0.5, 0.3 with bias = 0.5 and activation step function at t=0.5. The learning rate = 1


x2

w1= 0.5

w2 = 0.3

x1

yin = x1w1 + x2w2

y

Activation Function:Binary Step Functiont = 0.5,

(y-in) = 1 if y-in >= totherwise (y-in) = 0

Solving Linearly Separable Functions (Pattern mode)

• Given:

• Since we consider bias as additional weight thus the weight vector is 1x3 we have to fix the dimensionality of the input vector x1, x2, x3 and x4from 2x1 to be 3x1 to perform the multiplication.


11100100

X).( bXWfY

x1x2

x3x4

x1 x2 y0 0 00 1 11 0 11 1 1

111101011001

X

5.03.05.0)0( W


• Update weight vector for iteration 1


OK

Wrong

1,1101

].5.03.15.0[3.)1(

yXW

1,3.2111

].5.03.15.0[4.)1(

yXW

OK

OK

Wrong1,5.0100

].5.03.15.0[1.)1(

yXW

0,5.0100

.5.03.05.01.)0(

yXW

0,2.0110

.5.03.05.02.)0(

yXW

5.03.15.0

)..( 2)0()1( XyWW ydis

TT





5.03.15.0

1)..()1()2( XyyWWdis

TT

1,8.0110

].5.03.15.0[2.)2(

yXW

0,0101

].5.03.15.0[3.)2(

yXW

OK

Wrong

5.03.15.1


TT

1,5.0100

].5.03.15.1[1.)3(

yXW

1,3.3111

].5.03.15.1[4.)3(

yXW OK

Wrong



• The weights learning has converged at 4 iterations7Neural Networks Dr. Randa Elanwar

0,5.0100

].5.03.15.2[1.)4(

yXW

1,8.0110

].5.03.15.1[2.)4(

yXW

5.03.15.1


TT

1,1101

].5.03.15.1[3.)4(

yXW

1,3.2111

].5.03.15.1[4.)4(

yXW

OK

OK

OK

OK

Solving Linearly Separable Functions (Batch mode)


• Add w for all misclassified inputs together in 1 step


0,0101

].5.03.05.0[3.)0(

yXW

0,3.0111

].5.03.05.0[4.)0(

yXW

OK

Wrong

Wrong

Wrong

4)..(3)..(2)..()0()1( XyXyXy yyyWWdisdisdis

TT

5.23.25.2

111

101

110

5.03.05.0

)1(WT

0,5.0100

.5.03.05.01.)0(

yXW

0,2.0110

.5.03.05.02.)0(

yXW





1,5.2100

].5.23.25.2[1.)1(

yXW

1,8.4110

].5.23.25.2[2.)1(

yXW

1,5101

].5.23.25.2[3.)1(

yXW

1,3.7111

].5.23.25.2[4.)1(

yXW

Wrong

OK

OK

OK

5.13.25.2


TT





1,5.1100

].5.13.25.2[1.)2(

yXW

1,8.3110

].5.13.25.2[2.)2(

yXW

1,4101

].5.13.25.2[3.)2(

yXW

1,3.6111

].5.13.25.2[4.)2(

yXW

OK

OK

OK

Wrong

5.03.25.2


TT





OK

OK

OK

Wrong

5.03.25.2


TT

1,5.0100

].5.03.25.2[1.)3(

yXW

1,8.2110

].5.03.25.2[2.)3(

yXW

1,3101

].5.03.25.2[3.)3(

yXW

1,3.5111

].5.03.25.2[4.)3(

yXW


• The weights learning has also converged at 4 iterations but with different final values


OK

OK

OK

OK0,5.0100

].5.03.25.2[1.)4(

yXW

1,8.1110

].5.03.25.2[2.)4(

yXW

1,2101

].5.03.25.2[3.)4(

yXW

1,3.4111

].5.03.25.2[4.)4(

yXW


• Example: Consider linearly separable patterns where class C1 consists of the two patterns [1 0]T and [1 1]T and C2 consists of the two patterns [0 0]T and [0 1]T. Use the perceptron algorithm with = 1 and w(0)= [0 1 -1/2]T to design the line separating the two classes.


X2

X1

X4

X3

X2

X1

initial

finalIt is very important to graph the problem to define the initial line and assign the direction of positive and negative

Initial weight X2-1/2 = 0, intersects vertical axis at (0,1/2) and is parallel to horizontal axis

When X2>1/2 we get +ve value thus the positive direction is above the line

+ve

-ve


• Initially x2 and x3 are in the correct side while x1 and x4 are in the wrong side.

• Thus we assume x1 and x2 have to be on the +ve side and x3and x4 have to be on the –ve side

• Note that sometimes you are not given the activation function f. In such case: you can compute outputs and update weights by polarity (sign) instead of the activation function value:

If the W.X >0 and it is wrong w = .(-1).X

If the W.X <0 and it is wrong w = .(+1).X



• Again since the weight vector is 1x3 we have to fix the dimensionality of the input vector x1, x2, x3 and x4 from 2x1 to be 3x1 to perform the multiplication.



5.0101

].5.010[1.)0(

XW -ve and it have to be +ve

5.011

1)0()0()1( Xw WWWTTT

5.2111

].5.011[2.)1(

XW

OK

5.0100

].5.011[3.)1(

XW +ve and it have to be -ve






5.011


5.0110

].5.011[4.)2(

XW +ve and it have to be -ve

5.101


5.0101

].5.101[1.)3(

XW -ve and it have to be +ve

5.002


5.1111

].5.002[2.)4(

XW

OK


• The new straight line equation is 2X1-1/2 = 0 a vertical line intersecting the horizontal axis at 1/4 and is parallel to the vertical axis

• The solution has converged in 4 iterations.


5.0100

5.0023.)4(

XW OK

5.0110

5.0024.)4(

XW OK

5.1101

5.0021.)4(

XW OK


• Again since the weight vector is 3x1 we have to fix the dimensionality of the input vector x1, x2, x3 and x4 to be 1x3 to perform the multiplication.

• Update weight vector for iteration 1. Add w for all misclassified inputs together in 1 step


+ve and it have to be -ve

-ve and it have to be +ve

OK

OK

4.1.)0()1( XXWWTT

5.001

110

101

5.010

)1(WT

5.0101

].5.010[1.)0(

XW

1111

].5.010[2.)0(

XW

5.0100

].5.010[3.)0(

XW

5.0110

].5.010[4.)0(

XW


• The new straight line equation is X1-1/2 = 0 a vertical line intersecting the horizontal axis at 1/2 and is parallel to the vertical axis

• The solution has converged in 1 iteration.


OK

OK

OK

OK5.0101

].5.001[1.)1(

XW

5.0111

].5.001[2.)1(

XW

5.0100

].5.001[3.)1(

XW

5.0110

].5.001[4.)1(

XW

Non linear problems

• XOR problem

• No way to draw a line to separate the positive from negative examples


Input1 Input2 Output

0 0 0

0 1 1

1 0 1

1 1 0

Non linear problems

• XOR problem

• The only way to separate the positive from negative examples is to draw 2 lines (i.e., we need 2 straight line equations) or nonlinear region to capture one type only


+ve

+ve-ve

-ve+ve-ve

cba yx 22

Non linear problems

• To implement the nonlinearity we need to insert one or more extra layer of nodes between the input layer and the output layer (Hidden layer)


introduction to neural networks (under graduate course) lecture 5 of 9

Education