CSCI 4410Lecture 11: Introduction to Neural Networksadapted from Kathy Swigger

Biological NeuronBiological Neuron

The Neuron - A Biological Information Processor

dentrites - the receivers soma - neuron cell body (sums input

signals) axon - the transmitter synapse - point of transmission neuron activates after a certain

threshold is metLearning occurs via electro-chemical

changes in effectiveness of synaptic junction.

Biological NeuronBiological Neuron

Advantage of the BrainAdvantage of the Brain

Inherent Advantages of the Brain: “distributed processing and

representation”

Parallel processing speeds Fault tolerance Graceful degradation Ability to generalize

PrehistoryW.S. McCulloch & W. Pitts (1943). “A logical

calculus of the ideas immanent in nervous activity”, Bulletin of Mathematical Biophysics, 5, 115-137.

1 1 1 01

0 1 0 0

000

x & yyx

outputinputs

Truth Table for Logical AND

x+y-21 * -2

y * +1

x * +1

if sum<0 : 0

else : 1

inp

uts

weig

hts

sum output

• This seminal paper pointed out that simple artificial “neurons” could be made to perform basic logical operations such as AND, OR and NOT.

Nervous Systems as Logical Circuits

Groups of these “neuronal” logic gates could carry out any computation, even though each neuron was very limited.

x+y-11 * -1

y * +1

x * +1

if sum<0 : 0

else : 1

inp

uts

weig

hts

sum output1 1 1 01

0 1 0 0

110

x | yyx

outputinputs

Truth Table for Logical OR

• Could computers built from these simple units reproduce the computational power of biological brains?

• Were biological neurons performing logical operations?

The Perceptron

It obeyed the following rule:

If the sum of the weighted inputs exceeds a threshold, output 1, else output -1.

output

inputs

weig

ht

s sum

Σxi wi

*

Frank Rosenblatt (1962). Principles of Neurodynamics, Spartan, New York, NY.

Subsequent progress was inspired by the invention of learning rules inspired by ideas from neuroscience…

Rosenblatt’s Perceptron could automatically learn to categorise or classify input vectors into types.

1 if Σ inputi * weighti > threshold

-1 if Σ inputi * weighti < threshold

Networks Network parameters are adapted so

that it discriminates between classes For m classes, the classifier partitions

the feature space into m decision regions

The separation of the classes is the decision boundary.

In more than 2 dimensions this is a surface

Networks For 2 classes can view net output as a

discriminant function y(x, w) where:y(x, w) = 1 if x in C1

y(x, w) = - 1 if x in C2

Need some training data with known classes to generate an error function for the network

Need a (supervised) learning algorithm to adjust the weights

Linear discriminant functions

-4 -2 0 2 4 6 8 10 12 14-4

-2

0

2

4

6

8

Feature 1

Featu

re 2

TWO-CLASS DATA IN A TWO-DIMENSIONAL FEATURE SPACE

DecisionBoundary

DecisionRegion 1

Decision Region 2

A linear discriminant function is a mapping which partitions feature space using a linear function.

Simple form of classifier: “separate the two classes using a straight line in feature space”

The Perceptron as a Classifier

For d-dimensional data perceptron consists of d-weights, a bias and a thresholding activation function. For 2D data we have:

w1

w2

w0

a = w0 + w1 x1 + w2 x2 Activate {-1, +1}

1. Weighted Sum of the inputs

2. Pass thru Activation function:T(a)= -1 if a < 0 T(a)= 1 if a >= 0

x1

1Output= classdecision

If we group the weights as a vector we therefore have the net output given by:

Output = w . x + w0 (bias or threshold)

x2

View the bias as another weight from an input which is constantly on

Common Activation Function Choices

Nodes representing boolean functions

Using the step activation function from the previous slide.

Network LearningStandard procedure for training the weights is by gradient descent

For this process we have a set of training data from known classes to be used in conjunction with an error function (eg sum of squares error) to specify an error for each instantiation of the network

Then do: w new = w old - error function

where: the error function is: T – O where T = desired output and O is actual output.

This moves us downhill

Illustration of Gradient Descent

w1

w0

E(w)

Illustration of Gradient Descent

w1

w0

E(w)

Illustration of Gradient Descent

w1

w0

E(w)

Direction of steepestdescent = direction ofnegative gradient

Illustration of Gradient Descent

w1

w0

E(w)

Original point inweight space

New point inweight space

Example

1Bias

2

W0 = -1

W1 = .5

W2 = .3

Ouput= sum (weights * input) + threshold (bias)Output = (2*.5) + (1* .3) + -1 = .3Activation = 1 if >0 and 0 if < 0

Updating the functionsWi(t+1) = Wi(t) + Wi(t) (t+1) = (t) + (t)

W(t)i = (T-O) I i

(t) = (T-O)

Error = T-O (where T=Desired and O = actual output

W1(t+1) = .5 +(0-1)(2) = -1.5W2(t+1) = .3 + (0-1)(1) = -.7 (t+1) = -1 + (0-1) = -2

1

The Fall of the PerceptronMarvin Minsky & Seymour Papert (1969).

Perceptrons, MIT Press, Cambridge, MA.

• Before long researchers had begun to discover the Perceptron’s limitations.

• Unless input categories were “linearly separable”, a perceptron could not learn to discriminate between them.

• Unfortunately, it appeared that many important categories were not linearly separable.

• E.g., those inputs to an XOR gate that give an output of 1 (namely 10 & 01) are not linearly separable from those that do not (00 & 11).

The Fall of the Perceptron

Successful

Unsuccessful

Many Hours in the Gym per Week

Few Hours in the Gym

per Week

Footballers

Academics

In this example, a perceptron would not be able to discriminate between the footballers and the academics…

…despite the simplicity of their relationship:

Academics =Successful XOR Gym

Multi-Layered Networks

Feed-forward : Links can only go in one direction.

Recurrent : Arbitrary topologies can be formed

from links .

Feedforward Network

I2

t = -0.5W24= -1

H4

W46 = 1 t = 1.5

H6

W67 = 1

t = 0.5

I1

t = -0.5W13 = -1

H3

W35 = 1t = 1.5

H5

O7

W57 = 1

W25 = 1

W16 = 1

Feedforward Networks Arranged in layers.

Each unit is linked only to the unit in next layer.

No units are linked between the same layer, back to previous layers or skip a layer.

Computations can proceed uniformly from input to output units.

No internal state exists.

Multi-layer networks

Have one or more layers of hidden units

With hidden layer, it is possible to implement any function.

Recurrent Networks The brain is not a feed-forward network.

Allows activation to be fed back to previous layers.

Can become unstable or oscillate.

May take long time to compute a stable output.

Learning process is much more difficult.

Can implement more complex designs.

Can model systems with state.

Back propagation Training

Multi-layer Networks - the XOR function

• XOR can be written:-

• x XOR y = (x AND NOT y) OR (y AND NOT x)

Multi-Layer Networks

• The Single layer perceptron could not solve XOR, because it couldn’t ‘draw a line’ to separate the two classes

• This multi-layer perceptron ‘draws’ an extra line

x

y x AND NOT y

y AND NOT x

OR the above

Decision Boundaries

Can draw arbitrarily complex decision boundaries with multi-layered networks

But how do we train them / change the weights ?

Backpropagation

How do we assign an error / blame to a neuron hidden in a layer far away from the output nodes ?

The trick is to feed the information in…

Work out the errors at the output nodes…

Then Propagate the errors backwards through the layers

New Threshold Function

• The Backprop algorithm requires sensitive measurement of error and a smoothly varying function (has a derivative everywhere)

• we replace the sign function with a smooth function

• A popular choice is the sigmoid

Sigmoid Function

Defined by, Oj = 1 / ( 1 + e-Nj )

where Nj= sum of the (weights*inputs) + bias

Backpropagation

We now use the Generalized Delta Rule for altering weights in the networks :-

wij (t+1) = wij(t) + wij(t)

wij (t) = (learning rate) (err)j Oi

j (t+1) = j + j

j (t) = (learning rate) (err) jTwo rules for updating weights(with sigmoid function) :-

Make the changes…

1) (err) j = O j (1 – Oj )(Tj – Oj)

for nodes in the output layer

2) (err)j = O j (1 – Oj )(k (err)k wjk)

for nodes on hidden layers

Essentials of BackProp

• The equations for changing the weights are derived by trying to assign an amount of blame to weights deep in the network

•The sensitivity of ‘Error’ for output layer is calculated with respect to nodes and weights in hidden layers

•In practice, simply show training sets to the input layer, compare the results at the output layer ( T - O ) and used the two rules for weight adjustments. (1) for weights leading to output layer, (2) otherwise

Training Algorithm 1

Step 0: Initialize the weights to small random values

Step 1: Feed the training sample through the network and determine the final output

Nj= Sum of the (weights * Input) + threshold

(bias) Step 2: Compute the error for each

output unit, for unit j it is:1) (err) j = O j (1 – Oj )(Tj – Oj)

Training Algorithm (cont.)

Step 3: Calculate the weight correction term for each output unit, for unit j it is:

Δwij = (learning rate) (err)j O i

A small constant

Hidden layer signal

Training Algorithm 3

Step 4: Propagate the delta terms (errors) back through the weights of the hidden units where the delta input for the jth hidden unit is:

(err)j = O j (1 – Oj )( (err)k wjk)

k

Training Algorithm 4 Step 5: Calculate the weight correction

term for the hidden units:

Step 6: Update the weights:

Step 7: Test for stopping (maximum cylces, small changes, etc)

Δwij = (lrate)(err)j Oi

wij(t+1) = wij(t) + Δwij

Options

There are a number of options in the design of a backprop system Initial weights – best to set the

initial weights (and all other free parameters) to random numbers inside a small range of values (say –0.5 to 0.5)

Number of cycles – tend to be quite large for backprop systems

Number of neurons in the hidden layer – as few as possible

The numbers

I1 1I2 1W13 .1W14 -.2W23 .3W24 .4W35 .5W45 -.43 .24 -.35 .4

Output!Where N= input of nodeO= Activation function = threshold value of node

Backpropgating!

XOR Architecture

x

y

fv21 v11

v31

fv22 v12

v32

fw21 w11

w31

1

1

1

Initial Weights Randomly assign small weight

values:

x

y

f.21 -.3

.15

f-.4 .25

.1

f-.2 -.4

.3

1

1

1

Feedfoward – 1st Pass

x

y

f.21 -.3

.15

f-.4 .25

.1

f-.2 -.4

.3

1

1

1

Training Case: (0 0)

0

0

1

1

s1 = -.3(1) + .21(0) + .25(0) = -.3

Oj = -sj 1

1 + e

Activation function f:

f = .43

s2 = .25(1) -.4(0) + .1(0)

f = .56

1

s3 = -.4(1) - .2(.43) +.3(.56) = -.318

f = .42(not 0)

Backpropagate

0

0

f.21 -.3

.15

f-.4 .25

.1

f-.2 -.4

.3

1

1

1

err3 = .42 (1 – .42) (0-.42)

= -.102

_in1 = 3w13 = -.102(-.2) = .02

formula2= .43(1-.43).02 = .005

_in2 = 3w12 = -.102(.3) = -.03

formula2 = -.56(1-.56)(.03) = -.007

1) (errdrv) j = O j (1 – Oj )(Tj – Oj)

2) (errdrv)j = O j (1 – Oj )( (errdrv)k wjk)

f=.42

f=.56

f=.43

Update the Weights – First Pass

0

0

f.21 -.3

.15

f-.4 .25

.1

f-.2 -.4

.3

1

1

1

Wij(t+1)= wij(t)+ wij(t) wij(t)= (lrate)(errdrv)j Oi

Wt = .3 +.28=.58 wij(t)= .5*(.102)*.56=.28

Applications

Sentinel

GOAL: Initial design and development of a neural network-based misuse detection system which explores the use of this technology in the identification of instances of external attacks on a computer network.

DNSSMTP POP

HTTP FTP

Internet

SENTINEL Intrusion Response

t = 180

h t t p : / / w w w . c n n . c

a d m I n I s t r a t o r

h t t p : / / e s e t . s c I s

t e l n e t c I s , n o v a .

l o g o u t

s u p e r v I s o r

h t t p : / / w w w . g t I s c

a n o n y m o u s

h t t p : / / w w w . m i c r o

f t p : / / a s t r o . g a t c

………………

….

f t p : / / a s t r o . g a t c

h t t p : / / w w w . g t I s c

a n o n y m o u s

s u p e r v I s o r

h t t p : / / e s e t . s c I s

Raw Data and

Destination Port of

Network Packets

Self-Organizing Map

NetworkStream

Neural Network Output

Results of SOM Event Classification

Results Tested with data sets containing 6, 12, 18, and 0

“attacks” in each 180 event data set Successfully detected >= 12 “attacks” in test cases Failed to “alert” in lower number of “attacks” (per

design)

Hybrid NN Test Results

0.00

0.20

0.40

0.60

0.80

1.00

6 12 18 0

FTP Attempts in Data Set

An Automatic Screening System for Diabetic Retinopathy

Visual Features: Nerve: Circular yellow and bright area

from which vessels emerge Macula: Dark elliptic red area Microaneurysms: Small scattered red

and dark spots (in the order of 10x10 pixels in our database of images)

Haemorrhages: Larger red blots Cotton spots (exudates): Yellow blots

Example

Cotton Spots:

Approach

A moving window searches for the matching pattern

Match

Miss

Miss …NN trained torecognize nerves

Miss or Match

Test window

How long should you train the net?

The goal is to achieve a balance between correct responses for the training patterns and correct responses for new patterns. (That is, a balance between memorization and generalization.)

If you train the net for too long, then you run the risk of overfitting to the training data.

In general, the network is trained until it reaches an acceptable error rate (e.g., 95%).

One approach to avoid overfitting is to break up the data into a training set and a training test set. The weight adjustments are based on the training set. However, at regular intervals the test set is evaluated to see if the error is still decreasing. When the error begins to increase on the test set, training is terminated.