
2/48

2/48

Neuron Abstraction Neurons transduce signalselectrical to

chemical, and from chemical back again

to electrical. Each synapseis associated with what we

call the synaptic efficacythe

efficiency with which a signal istransmitted from the presynaptic topostsynaptic neuron

S


3/48

3/48

Neuron Abstraction

S


4/48

4/48

Neuron Abstraction:

Activations and Weights thejth artificial neuron

that receives input

signals si, from possiblyn different sources

an internal activationxjwhich is a linearweighted aggregation ofthe impinging signals,modified by an internal

threshold, j

S


5/48

5/48

Neuron Abstraction:

Activations and Weights the j th artificial neuron

that connection

weightswijmodel the

synaptic efficacies of

various interneuron

synapses.

S


6/48

6/48

Notation: wijdenotes the

weight from neuron

i to neuronj .S


7/48

7/48

Neuron Abstraction: Signal Function The activation of the neuron

is subsequently transformed

through asignal functionS()

Generates the output signal

sj = S(xj ) of the neuron.

S


8/48

8/48

Neuron Abstraction: Signal Function a signal function may

typically be

binary threshold

linear threshold

sigmoidal

Gaussian

probabilistic.

S


9/48

9/48

Activations Measure Similarities

The activation xjis simply the binner

product of the impinging signal vector

S = (s0, . . . , sn)T, with the neuronal weight

vectorWj = (w0j , . . . ,wnj )T

Adaptiv

eFilter


10/48

10/48

Neuron Signal Functions:

Binary Threshold Signal Function Net positive

activations translate

to a +1 signal value

Net negative

activations translate

to a 0 signal value.


11/48

11/48


Binary Threshold Signal Function The threshold logic

neuron is a two state

machine

sj = S(xj ) {0, 1}


12/48

12/48


Binary Threshold Signal Function


13/48

13/48

Threshold Logic Neuron (TLN)

in Discrete Time

The updated signal value S(xjk+1) at time instant

k + 1 is generated from the neuron activation

xik+1

, sampled at time instant k + 1. The response of the threshold logic neuron as a

two-state machine can be extended to thebipolarcase where the signals are

sj{1, 1}


14/48

14/48

Threshold Logic Neuron (TLN)

in Discrete Time

The resulting signal function is then none

other than thesignum function, sign(x)commonly encountered in communication

theory.


15/48

15/48

Interpretation of Threshold

From the point of view of the net activation xj

the signal is +1 ifxj = qj+ j 0, or qj j;

and is 0 ifqj< j.


16/48

16/48

Interpretation of Threshold

The neuron thus compares the net external

input qj ifqjis greater than the negative threshold, it fires

+1, otherwise it fires 0.


17/48

17/48

Linear Threshold Signal Function

j = 1/xmis the slope

parameterof thefunction

Figure plotted for xm =2 and j = 0.5. .


18/48

18/48

Linear Threshold Signal Function

Sj (xj ) = max(0,min(jxj , 1))

Note that in thiscourse we assume thatneurons within anetwork arehomogeneous.


19/48

19/48

Sigmoidal Signal Function

jis a gain scale factor In the limit, as j

the smooth logistic

function approachesthe non-smooth binary

threshold function.


20/48

20/48

Sigmoidal Signal Function

The sigmoidalsignal function hassome very usefulmathematical

properties. It is monotonic

continuous

bounded


21/48

21/48

Gaussian Signal Function

jis the Gaussian spreadfactor and cj is thecenter.

Varying the spreadmakes the functionsharper or more diffuse.


22/48

22/48

Gaussian Signal Function

Changing the centershifts the function to theright or left along theactivation axis

This function is anexample of a non-monotonicsignal

function


23/48

23/48

Stochastic Neurons

The signal is assumed to be two state

sj {0, 1} or {1, 1}

Neuron switches into these states depending

upon aprobabilistic function of its

activation,P(xj).


24/48

24/48

Summary of Signal Functions


25/48

25/48

Neural Networks Defined

Artificial neural networks are massively

parallel adaptive networks of simple

nonlinear computing elements calledneurons which are intended to abstract and

model some of the functionality of the

human nervous system in an attempt topartially capture some of its computational

strengths.


26/48

26/48

Eight Components of Neural

Networks

Neurons. These can be of three types:

Input: receive external stimuli

Hidden: compute intermediate functions Output: generate outputs from the network

Activation state vector. This is a vector of

the activation levelxi of individual neuronsin the neural network,

X= (x1, . . . , xn)T Rn.


27/48

27/48


Networks

Signal function. A function that generates theoutput signal of the neuron based on its

activation. Pattern of connectivity. This essentially determines the

inter-neuron connection architecture or the graph of thenetwork. Connections which model the inter-neuron

synaptic efficacies, can be excitatory (+)

inhibitory ()

absent (0).


28/48

28/48


Networks

Activity aggregation rule. A way ofaggregating activity at a neuron, and isusually computed as an inner product of theinput vector and the neuron fan-in weightvector.


29/48

29/48


Networks

Activation rule. A function that determinesthe new activation level of a neuron on the

basis of its current activation and itsexternal inputs.

Learning rule. Provides a means ofmodifying connection strengths based both

on external stimuli and networkperformance with an aim to improve thelatter.


30/48

30/48


Networks

Environment. The environments withinwhich neural networks can operate could be

deterministic (noiseless) or stochastic (noisy).


31/48

31/48

Architectures:

Feedforward and Feedback Local groups of neurons can be connected in

either,

afeedforwardarchitecture, in which the networkhas no loops, or

afeedback(recurrent) architecture, in which loops

occur in the network because of feedbackconnections.


32/48

32/48

Architectures:

Feedforward and Feedback


33/48

33/48

Neural Networks Generate Mappings

Multilayered networks that associate vectors from

one space to vectors of another space are calledheteroassociators.

Map or associate two different patterns with one

anotherone as input and the other as output.

Mathematically we write,f: Rn Rp.

f: Rn Rp


34/48

34/48

Neural Networks Generate Mappings

When neurons in a

single field connect

back onto themselvesthe resulting network is

called an autoassociator

since it associates asingle pattern in Rn with

itself.

f: Rn Rn


35/48

35/48

Activation and Signal State Spaces For ap-dimensional field of neurons, the

activation state space is Rp.

The signal state space is the Cartesian crossspace,

Ip = [0, 1] [0, 1] p times = [0, 1]p Rpifthe neurons have continuous signal functions in

the interval [0, 1]

[1, 1]pif the neurons have continuous signalfunctions in the interval [1, 1].


36/48

36/48

Activation and Signal State Spaces For the case when the neuron signal

functions are binary threshold, the signal

state space is Bp = {0, 1} {0, 1}p times = {0, 1}p Ip Rp

{1, 1}pwhen the neuron signal functions are

bipolar threshold.


37/48

37/48

Feedforward vs Feedback:

Multilayer Perceptrons

Organized into different layers

Unidirectional connections

memory-less: output depends only on the present

input

X Rn

S = f (X)


38/48

38/48


Multilayer Perceptrons

Possess no dynamics

Demonstrate powerful properties

Universal function approximation

Find widespread applications in pattern

classification.

X Rn

S = f (X)


39/48

39/48


Recurrent Neural Networks

Non-linear dynamical

systems

New state of thenetwork is a function

of the current input

and the present stateof the network


40/48

40/48


Recurrent Neural Networks

Possess a rich repertoire

of dynamics

Capable of performingpowerful tasks such as

pattern completion

topological featuremapping

pattern recognition


41/48

41/48

More on Feedback Networks

Network activations and signals are in a flux ofchange until they settle down to a steady value

Issue of Stability: Given a feedback networkarchitecture we must ensure that the networkdynamics leads to behavior that can be interpretedin a sensible way.


42/48

42/48

More on Feedback Networks

Dynamical systems have variants of

behavior like

fixed point equilibriawhere the system

eventually converges to a fixed point

Chaotic dynamicswhere the system

wanders aimless in state space


43/48

43/48

Summary of Major Neural Networks

Models


44/48

44/48

Summary of Major Neural Networks

Models


45/48

45/48

Salient Properties of Neural

Networks

Robustness Ability to operate, albeit withsome performance loss, in the event of

damage to internal structure. Associative Recall Ability to invoke related

memories from one concept.

For e.g. a friends name elicits vivid mental

pictures and related emotions


46/48

46/48

Salient Properties of Neural

Networks

Function Approximation and GeneralizationAbility to approximate functions using

learning algorithms by creating internalrepresentations and hence not requiring themathematical model of how outputs dependon inputs. So neural networks are often

referred to as adaptive function estimators.


47/48

47/48

Application Domains of Neural

Networks

Associative RecallFault Tolerance


48/48

Application Domains of Neural

NetworksFunction Approximation

Control

Prediction

3.neuron network architecture

Documents