ai-lecture 12 - simple perceptron

8/12/2019 AI-Lecture 12 - Simple Perceptron

1/24

Neural Networks


2/24

Pattern Classification Characteristics in Pattern Recognition

Perceptron

Simple Perceptron for Pattern Classification

Perceptron: Algorithm

Perceptron: Example Perceptron: Matlab code

Assignment 1


3/24


4/24

Pattern Recognition

Classes and Classification

The goalof the pattern recognition process is

to identify, or classify, input patterns.

Character recognition may involve the task of

identifying an image scanned from a written page of

text as one of 26 possible characters of the alphabet. In speech recognition, pattern recognition may

involve the discrimination of vectors representing

spoken words.

The pattern being recognized is classifiedas

belonging to one of a small number of classes.

For example, the 26 characters may be the classes in

character recognition.

The scanned input image is classified as one of the

alphabet characters.


5/24

Characteristics in Pattern Recognition

1. Data Space Partitioning

Pattern recognition results in a partitioning and

labeling of regions in the inpu t space.

If the input to a pattern recognition system is an N

dimensional vector, the input space is a spacespanned by all input vectors (each input vector can

be viewed as a point, or pattern, in the N dimensional

input space).

Let X be the set of all input patterns, and C be the set

of classes.

Formally, a pattern recognition system P(X) performs

a mapping from the input space to the set of possible

classes.

P(X) : X C


6/24


1. Data Space Partitioning(Input Data Space Partitioning for two dimensional data)

A

C

B A

C

Estimated

DecisionBoundaryfor Class C

PointsCorrectlyClassified

as Class C

PointsMisclassified

as Class B


7/24


2. Adaptivity Adaptation occurs during training when examples of

various classes from a training set are presented tothe pattern recognition system along with theircorrect classification.

The system learns from the training process byassociating an input example in the training set withthe correct class.

After being trained with a class label, the system canoften later correctly classify the same pattern and

improve on similar mistakes.

Thus the performance of a pattern recognitionsystem improves as more training data isrepresented and it becomes more and more capable

of correctly classifying a wider range of inputs.


8/24


3. Generalization If an adaptive system is trained using a set of

patterns, T, and can correctly classify patterns not inT, the process behind this behavior is calledgeneralization.

Generalization based on adaptation is recognized asbeing the essence of intelligent systems.

Usually, the training set T, available to a patternrecognition system is very small compared to thetotal number of examples in the input space.

By using a well selected set that captures most ofthe variations expected from the inputs, a welldesigned pattern recognition system should be ableto learn important properties in distinguishing

patterns of different classes.


9/24


4. Input Space Complexity Minimum error boundaries between classes

in pattern recognition can be extremely

complex and difficult to characterize.

Since the unknown boundaries can beextremely irregular, adaptive systems are

unlikely to find the solution through

learning merely based in a small set of

examples.


10/24

Perceptrons had perhaps the most far-reaching impact of any of the early neuralnets.

A number of different types of Perceptronshave been used and described by various

workers. The original perceptrons had three layers of

neurons sensory units, associator units and aresponse unit forming an approximate modelof a retina.

Under suitable assumptions, its iterativelearning procedure can be proved to convergeto the correct weights i.e., the weights thatallow the net to produce the correct outputvalue for each of the training input patterns.


11/24

The architecture of asimple perceptron for

performing singleclassification is shownin the figure.

The goal of the net isto classify each inputpattern as belonging,or not belonging, to aparticular class.

Belonging is signifiedby the output unitgiving a response of

+1; not belonging isindicated by aresponse of -1.

A zero output meansthat it is not decisive.

X1

X2

l

l

l

Xn

1

Y2

b

w1

w2

wn

xif

xif

xif

xf

1

0

1

)(


12/24

1. Initialize weights, bias and the threshold . Also

set the learning rate such that ( 0 <


13/24


14/24

11 1 1 0 0 1 1 1 1

Input Bias

yin y tx1 x2 w1 w2 b

1 0 1 2 1 -1 0 1 00 1 1 1 1 -1 0 0 -1

0 0 1 -1 -1 -1 0 0 -1

Net

Input

Output Target Weights

0 0 0

2 1 1 1 -1 -1 1 1 1 01 0 1 1 1 -1 0 1 -1

0 1 1 0 0 -1 0 0 -2

0 0 1 -2 -1 -1 0 0 -2


15/24

10 1 1 1 1 1 1 2 3 -4

Input Bias

yin y tx1 x2 w1 w2 b

1 0 1 -2 -1 -1 2 3 -4

0 1 1 -1 -1 -1 2 3 -4

0 0 1 -4 -1 -1 2 3 -4

Net

InputOutput Target Weights

Thus the positive response is given by all the pointssuch that

And the negative response is given by all the points

such that2.0432 21 xx

2.0432 21 xx


16/24

We note that the output of the Perceptron is1(positive), if the net input yin is greater than .

Also that the output is -1(negative), if the net inputyin is less than .

We also know thatyin= b + x1*w1 + x2*w2

in this case Thus we have

2*x1 + 3*x24 > 0.2 (+ve output)

2*x1 + 3*x24 < -0.2 (-ve output) Removing the inequalities and simplifying them givesus the following two equations:

2*x1 + 3*x2 = 4.2

2*x1 + 3*x2 = 3.8


17/24


18/24


19/24

The following slides contain the

Matlab source code for the

Simple Perceptron Studied inthe class.

Also find the first assignment in

these slides


20/24

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Purpose:% This Matlab code is based on Perceptron learning rule% Part 1: Training data and various control parameters are defined% Part 2: Perceptron learning rule is used for training of the net% Part 3: The resulting network is tested for the input data

% Part 4: Some exercises%%%%%%%%%%%%%%%%%%%%%%%%% Part 1 %%%%%%%%%%%%%%%%%%clear all% Initialize various parametersw(1) = 0 % Initialize weights, w(1) is bias weight.w(2) = 0; w(3) = 0alpha = 1 % Learning Ratetheta = 0.2 % Length of non-decisive regionstopFlag = -1; % Stoping flag, -1 = do not stop and +1 = stopwtChange = 1; % Weight change Flag, -1 = no change, +1 = changeepsilon = 0.001; % Termination Criteriaepoch = 0; % Epoch counter

% define training patternsnoOfTrainingPatterns = 4;

s(1,:) = [1 1 1]; t(1,1) = 1; s(2,:) = [1 1 0]; t(2,1) = -1;s(3,:) = [1 0 1]; t(3,1) = -1; s(4,:) = [1 0 0]; t(4,1) = -1;


21/24

%%%%%%%%%%%%%%%%%% Part 2 %%%%%%%%%%%%%%%%%%

% Step 1: While stopping condition is false, do steps 1 to 6while stopFlag < 0

% Step 2: For each training pair s:t, do steps 3 to 5wtChange = -1;for i=1:noOfTrainingPatterns

%Step 3: Set activations of each input unitx = s(i,:); outDesired = t(i,:);

% Step 4: Compute activation of the output unit% Calculate the net inputyin = x*w';if yin > theta

y = 1.;

elseif yin < -thetay = -1.;else

y = 0.;end


22/24

% Step 5: Update biases and weights

if abs(outDesired-y) > epsilon

w = w + (alpha*outDesired).*x;wtChange = 1.;

end

epoch, x, yin, y, outDesired, wend

if wtChange < 0

stopFlag = 1.;end

epoch = epoch + 1;

end % of while loop


23/24

%%%%%%%%%% Part 3 %%%%%%%%%%%%

% Lets pick one pattern at random from the training data.p=fix(1+rand*(noOfTrainingPatterns-1))

x=s(p,:)outDesired=t(p,:)

% For testing, we simply give this as input to the trained net% and find out the output.yin = x*w';

if yin > thetay = 1.;

elseif yin < -thetay = -1.;

elsey = 0.;

end

% print y and the desired outputy, outDesired

Assignment


24/24

AssignmentNote: Please submit your assignment (both in hard- and soft copy form) by

4:00pm Thursday, January 01st, 2013).Please carry out the followingexercises by doing appropriate modification in the above code.

Exercise 1: Consider the training data for the Perceptron used aboveInputs target

x1 x2 x3 bias t1 1 1 1 11 1 0 1 -11 0 1 1 -10 1 1 1 -1

(i.e. there is one more input neuron now). Verify that iterations convergeafter 26 epochs for learning rate equal to 1.0 and theta equal to 0.1, andthat the converged set of weights and bias is w1=2, w2 =3, w3 =4 and b=-8.

Exercise 2: Plot the changes in the separating lines as they occur in Exercise1.Exercise 3: Using the above code, find the weights required to perform the

following classification:Vectors (1,1,1,1) and (-1,1,-1,-1) are members of the class (and

therefore have target value 1);Vectors (1,1,1,-1) and (1,-1,-1,1) are not members of the class (and

have target value -1).Use a learning rate of 1 and starting weights of 0. Using each of the trainingvectors as input, test the response of the net. Display the number of epochstaken to each the convergence. Also plot the maximum error in each epoch.

ai-lecture 12 - simple perceptron

Documents