Ad

alin

e a

nd M

adal

ine

Sign

al p

roce

ssin

g de

velo

ped

as a

n en

gine

erin

g di

scip

line

wit

h th

e ad

vent

of

elec

tron

ic c

omm

unic

atio

n. I

niti

ally

, ana

log

filt

ers

usin

gito

r (R

LC

) ci

rcui

ts w

ere

desi

gned

to

rem

ove

nois

e fr

om t

he c

omm

unic

atio

nsi

gnal

s. T

oday

, si

gnal

pro

cess

ing

has

evol

ved

into

a m

any-

face

ted

tech

nolo

gy,

wit

h th

e em

phas

is h

avin

g sh

ifte

d fr

om t

uned

cir

cuit

impl

emen

tatio

n to

dig

ital

sign

al p

roce

ssor

s (D

SPs)

tha

t ca

n pe

rfor

m t

he s

ame

type

s of

fil

teri

ng a

pplic

a-tio

ns b

y ex

ecut

ing

conv

olut

ion

filte

rs i

mpl

emen

ted

in s

oftw

are.

Th

e ba

sis

for

the

indu

stry

rem

ains

the

des

ign

and

impl

emen

tatio

n of

filt

ers

to p

erfo

rm n

oise

rem

oval

fro

m i

nfor

mat

ion-

bear

ing

sign

als.

In t

his

chap

ter,

we

will

foc

us o

n a

spec

ific

typ

e of

filt

er,

calle

d th

e A

da-

line

(and

the

mul

tiple

-Ada

line,

or

Mad

alin

e) d

evel

oped

by

Ber

nard

Wid

row

of

Stan

ford

Uni

vers

ity.

As

we

wil

l se

e, t

he A

dalin

e m

odel

is

sim

ilar

to t

hat

of a

sing

le P

E i

n an

AN

S.

2.1

RE

VIE

W O

F S

IGN

AL

PR

OC

ES

SIN

G

We

begi

n ou

r di

scus

sion

of

the

Ada

line

and

Mad

alin

e ne

twor

ks w

ith

a re

view

of b

asic

sig

nal-

proc

essi

ng t

heor

y.

An

unde

rsta

ndin

g of

thi

s m

ater

ial

is e

ssen

-tia

l if

we

are

to a

ppre

ciat

e th

e op

erat

ion

and

appl

icat

ions

of

thes

e ne

twor

ks.

How

ever

, th

is m

ater

ial

is a

lso

typi

call

y co

vere

d as

par

t of

an

unde

rgra

duat

ecu

rric

ulum

in

info

rmat

ion

codi

ng a

nd d

ata

com

mun

icat

ion.

T

here

fore

, re

ader

sal

read

y co

mfo

rtab

le w

ith

sign

al-p

roce

ssin

g co

ncep

ts m

ay s

kip

this

fir

st s

ectio

nw

itho

ut f

ear

of m

issi

ng m

ater

ial

rele

vant

to

the

Ada

line

and

Mad

alin

e to

pics

.Fo

r th

ose

read

ers

who

ar

e no

t fa

mil

iar

wit

h th

e te

chni

ques

com

mon

ly u

sed

to i

mpl

emen

t el

ectr

onic

com

mun

icat

ions

and

sig

nal

proc

essi

ng,

we

shal

l be

-gi

n by

des

crib

ing

brie

fly

the

data

-enc

odin

g an

d m

odul

atio

n sc

hem

es u

sed

in a

nam

plit

ude-

mod

ulat

ion

(AM

) ra

dio

tran

smis

sion

. A

s pa

rt o

f th

is d

iscu

ssio

n, w

esh

all

illu

stra

te t

he n

eed

for

filte

rs i

n th

e co

mm

unic

atio

ns i

ndus

try.

We

will

the

n

45

46

Ad

alin

e a

nd

revi

ew t

he c

once

pts

of t

he f

requ

ency

dom

ain,

the

fou

r ba

sic

filt

er t

ypes

, an

dFo

urie

r an

alys

is.

Thi

s pr

elim

inar

y se

ctio

n co

nclu

des

with

a b

rief

ove

rvie

w o

fdi

gita

l si

gnal

pro

cess

ing,

bec

ause

man

y of

the

conc

epts

rea

lize

d in

dig

ital

fil

ters

are

dire

ctly

app

lica

ble

to t

he A

dali

ne a

nd M

adal

ine

(and

man

y ot

her)

neu

ral

netw

orks

.

Si

gnal

Pro

cess

ing

and

Filte

rs

Sig

nal

proc

essi

ng i

s an

eng

inee

ring

dis

cipl

ine

that

dea

ls p

rim

aril

y w

ith

the

im-

plem

enta

tion

of f

ilte

rs t

o re

mov

e or

red

uce

unw

ante

d fr

eque

ncy

com

pone

nts

from

an

info

rmat

ion-

bear

ing

sign

al.

Let

's c

onsi

der,

for

exa

mpl

e, a

n A

M r

a-di

o br

oadc

ast.

Ele

ctro

nic

com

mun

icat

ion

tech

niqu

es, w

heth

er f

or a

udio

sig

nals

or o

ther

dat

a, c

onsi

st o

f si

gnal

enc

odin

g an

d m

odul

atio

n.

Info

rmat

ion

to b

e t

his

case

, au

dibl

e so

unds

, su

ch a

s vo

ice

or b

e en

-co

ded

elec

tron

ical

ly b

y an

ana

log

sign

al t

hat

exac

tly r

epro

duce

s th

e fr

eque

ncie

san

d am

plit

udes

of t

he o

rigi

nal s

ound

s. S

ince

the

soun

ds b

eing

enc

oded

rep

rese

nta

cont

inuu

m f

rom

sile

nce

thro

ugh

voic

e to

mus

ic,

the

inst

anta

neou

s fr

eque

ncy

of t

he e

ncod

ed s

igna

l w

ill

vary

wit

h ti

me,

ra

ngin

g fr

om 0

to

appr

oxim

atel

y10

,000

her

tz (

Hz)

.R

athe

r th

an a

ttem

pt t

o tr

ansm

it th

is e

ncod

ed s

igna

l di

rect

ly,

we

tran

sfor

mth

e si

gnal

int

o a

form

mor

e su

itabl

e fo

r ra

dio

tran

smis

sion

. W

e ac

com

plis

h th

istr

ansf

orm

atio

n by

mod

ulat

ing

the

ampl

itud

e of

a h

igh-

freq

uenc

y ca

rrie

r si

gnal

wit

h th

e an

alog

inf

orm

atio

n si

gnal

. T

his

proc

ess

is i

llus

trat

ed i

n Fi

gure

2.1

.H

ere,

th

e ca

rrie

r is

not

hing

mor

e th

an a

sin

e w

ave

wit

h a

freq

uenc

y m

uch

grea

ter

than

the

inf

orm

atio

n si

gnal

. Fo

r A

M r

adio

, th

e ca

rrie

r fr

eque

ncy

wil

l be

in t

he r

ange

of

550

to (

KH

z).

Sinc

e th

e fr

eque

ncy

of t

he c

arri

eris

sig

nifi

cant

ly g

reat

er t

han

is t

he m

axim

um f

requ

ency

of t

he i

nfor

mat

ion

sign

al,

litt

le i

nfor

mat

ion

is l

ost

by t

his

mod

ulat

ion.

T

he m

odul

ated

sig

nal

can

then

be

tran

smit

ted

to a

rec

eivi

ng s

tati

on (

or b

road

cast

to a

nyon

e w

ith

a ra

dio

rece

iver

),w

here

the

sig

nal

is d

emod

ulat

ed a

nd i

s re

prod

uced

as

soun

d.T

he m

ost o

bvio

us r

easo

n fo

r a

filt

er in

AM

rad

io is

tha

t dif

fere

nt p

eopl

e ha

vedi

ffer

ent p

refe

renc

es i

n m

usic

and

ent

erta

inm

ent.

The

refo

re,

the

gove

rnm

ent

and

the

com

mun

icat

ion

indu

stry

hav

e al

low

ed m

any

diff

eren

t ra

dio

stat

ions

to

op-

erat

e in

the

sam

e ge

ogra

phic

al a

rea,

so

that

eve

ryon

e's

tast

es i

n en

tert

ainm

ent

can

be a

ccom

mod

ated

. W

ith s

o m

any

diff

eren

t ra

dio

stat

ions

all

broa

dcas

ting

in c

lose

pro

xim

ity,

how

is

it th

at w

e ca

n li

sten

to

only

one

sta

tion

at a

tim

e?Th

e an

swer

is

to a

llow

eac

h re

ceiv

er t

o be

tun

ed b

y th

e us

er t

o a

sele

ctab

le f

re-

quen

cy.

In t

unin

g th

e ra

dio,

we

are

esse

ntia

lly c

hang

ing

the

freq

uenc

y-re

spon

sech

arac

teri

stic

s of

a b

andp

ass

filt

er i

nsid

e th

e ra

dio.

T

his

filt

er a

llow

s on

ly t

hesi

gnal

s fr

om t

he s

tatio

n in

whi

ch w

e ar

e in

tere

sted

to

pass

, w

hile

eli

min

atin

gal

l th

e ot

her

sign

als

bein

g br

oadc

ast

wit

hin

the

spec

trum

of

the

AM

rad

io.

To i

llus

trat

e ho

w t

he b

andp

ass

filt

er o

pera

tes,

we

wil

l ch

ange

our

ref

eren

cefr

om t

he t

ime

dom

ain

to t

he f

requ

ency

dom

ain.

W

e be

gin

by c

onst

ruct

ing

atw

o-ax

is g

raph

, w

here

the

x a

xis

repr

esen

ts i

ncre

asin

g fr

eque

ncie

s an

d th

e y

axis

rep

rese

nts

decr

easi

ng a

tten

uati

on i

n a

unit

cal

led

the

deci

bel

(dB

). S

uch

a

2.1

Rev

iew

of

Sig

nal

Pro

cess

ing

47

1.0

0.5

(a)

-0.5

-1.0

0.05

0.

10.15

0.2

Car

rier

wav

e

(b)

(c)

0.05

0.1

0.15

Wav

e co

ntai

ning

info

rmat

ion

0.05

0.1

0.15

0.2

1.1

0.9

0.8

0.7

1.5

1.0

0.5

-0.5

-1.0

-1.5

Am

plitu

de-m

odul

ated

wav

e

Figu

re 2

.1

Typ

ical

in

form

atio

n-en

codi

ng

and

ampl

itude

-mod

ulat

ion

tech

niq

ues

for

ele

ctro

nic

com

munic

atio

n a

re

(a)

The

carr

ier

wave

has

a f

requency

muc

h h

ighe

r th

an t

hat

of

(b)

the

info

rmatio

n-b

earing

(c

) T

he c

arr

ier

wave

is

mod

ulat

edby

th

e i

nfo

rma

tion

-be

ari

ng s

ignal.

0.2

48

Ad

alin

e a

nd

Mad

alin

e

grap

h is

ill

ustr

ated

in

Figu

re 2

.2(a

). F

or t

he A

M r

adio

exa

mpl

e, l

et u

s im

agin

eth

at t

here

are

sev

en A

M r

adio

sta

tions

, la

bele

d A

thr

ough

G,

oper

atin

g in

the

area

whe

re w

e ar

e li

sten

ing.

T

he f

requ

enci

es a

t w

hich

the

se s

tatio

ns t

rans

mit

are

grap

hed

as v

erti

cal

line

s lo

cate

d on

the

fre

quen

cy a

xis

at t

he p

oint

cor

re-

spon

ding

to

thei

r tr

ansm

itti

ng,

or c

arri

er,

freq

uenc

y. T

he a

mpl

itude

of

the

lines

,as

ill

ustr

ated

in

Figu

re 2

.2(a

), i

s al

mos

t 0

dB,

indi

catin

g th

at e

ach

stat

ion

istr

ansm

itti

ng a

t fu

ll p

ower

, an

d ea

ch c

an b

e re

ceiv

ed e

qual

ly w

ell.

Now

we

wil

l a

ban

dpas

s fi

lter

to s

elec

t on

e of

the

sev

en s

tatio

ns.

The

freq

uenc

y re

spon

se o

f a

typi

cal

band

pass

filt

er i

s ill

ustr

ated

in

Figu

re 2

.2(b

).N

otic

e th

at t

he f

requ

ency

-res

pons

e cu

rve

is s

uch

that

all

freq

uenc

ies

that

fal

lou

tsid

e th

e in

vert

ed n

otch

are

atte

nuat

ed t

o ve

ry s

mal

l m

agni

tude

s, w

here

asfr

eque

ncie

s w

ithin

the

pass

band

are

allo

wed

to p

ass

with

ver

y lit

tlehe

nce

the

nam

e "b

andp

ass

filte

r."

To t

une

our

radi

o re

ceiv

er t

o an

y on

e of

the

seve

n br

oadc

astin

g st

atio

ns,

we

sim

ply

adju

st t

he f

requ

ency

res

pons

e of

the

filte

r su

ch t

hat

the

carr

ier

freq

uenc

y of

the

des

ired

sta

tion

is w

ithin

the

pass

band

. CD C o i

0-

f

[ 3

C

[

EE

F- C B

500

700

900

1300

Frequency

(KHz)

1500

17

00

(b)

CO

Figu

re 2

.2

-20

500

700

900

1300

1500

1700

Fre

quen

cy (

KH

z)

Th

ese

are

fre

qu

en

cy d

omai

n g

rap

hs

of (

a) A

M r

ad

io r

ecep

tion

of s

even

d

iffe

ren

t st

atio

ns,

(b)

the

fre

qu

en

cy r

espo

nse

of t

hetu

nin

g fil

ter

and

the

mag

nitu

de o

f th

e re

ceiv

ed

sig

na

ls a

fte

rfilte

rin

g.

2.1

Rev

iew

of

Sig

nal

Pro

cess

ing

49

As

anot

her

exam

ple

of t

he u

se o

f fi

lters

in

the

com

mun

icat

ion

indu

stry

,co

nsid

er t

he p

robl

em o

f ec

ho s

uppr

essi

on i

n lo

ng-d

ista

nce

tele

phon

e co

mm

u-ni

catio

n.

As

indi

cate

d in

Fig

ure

2.3,

the

pro

blem

is

caus

ed b

y th

e in

tera

ctio

nbe

twee

n th

e am

plif

iers

and

ser

ies

coup

ling

use

d on

bot

h en

ds o

f th

e li

ne,

and

the

dela

y tim

e re

quir

ed t

o tr

ansm

it t

he v

oice

inf

orm

atio

n be

twee

n th

e sw

itch

-in

g of

fice

and

the

com

mun

icat

ions

sat

ellit

e in

geo

stat

iona

ry o

rbit

, 23

,000

mil

esab

ove

the

eart

h.

Spec

ific

ally

, yo

u he

ar a

n ec

ho o

f yo

ur o

wn

voic

e in

the

tel

e-ph

one

whe

n yo

u sp

eak.

The

sig

nal

carr

ying

you

r vo

ice

arri

ves

at t

he r

ecei

ving

tele

phon

e ap

prox

imat

ely

270

mill

isec

onds

aft

er y

ou s

peak

. Th

is d

elay

is

the

amou

nt o

f tim

e re

quir

ed b

y th

e m

icro

wav

e si

gnal

to

trav

el t

he 4

6,00

0 m

iles

betw

een

the

tran

smitt

ing

stat

ion,

the

sat

ellit

e, a

nd t

he r

ecei

ving

sta

tion

on t

hegr

ound

. O

nce

rece

ived

and

rou

ted

to t

he d

estin

atio

n te

leph

one,

the

sig

nal

isag

ain

ampl

ifie

d an

d re

prod

uced

as

soun

d on

the

rec

eivi

ng h

ands

et.

Unf

ortu

-na

tely

, it

is a

lso

ofte

n pi

cked

up

by t

he t

rans

mitt

er a

t th

e re

ceiv

ing

end,

due

to i

mpe

rfec

tions

in

the

devi

ces

used

to

deco

uple

the

inc

omin

g si

gnal

s.

It c

anth

en b

e a

nd f

ed b

ack

to y

ou a

ppro

xim

atel

y 1/

2 se

cond

aft

er y

ousp

oke.

The

res

ult

is e

cho.

O

bvio

usly

, a

sim

ple

band

pass

filt

er c

anno

t be

use

dto

rem

ove

the

echo

, be

caus

e th

ere

is n

o w

ay t

o di

stin

guis

h th

e ec

hoed

sig

nal

from

val

id s

igna

ls.

To s

olve

pro

blem

s su

ch a

s th

ese,

the

com

mun

icat

ions

ind

ustr

y ha

s de

vel-

oped

man

y di

ffer

ent

type

s of

filt

ers.

Th

ese

filte

rs n

ot o

nly

are

used

in

23,3

00 m

iles

Inco

min

g si

gnal

retra

nsm

itted

due

to c

oupl

ing

leak

age

Ret

urni

ng s

igna

l is

orig

inal

dela

yed

by 5

00 m

illis

econ

ds,

resu

lting

in a

n "e

cho"

2.3

E

cho

ca

n o

ccu

r in

lo

ng

-dis

tan

ce t

ele

com

mu

nic

atio

ns.

50A

dal

ine a

nd

Mad

alin

e

tron

ic c

omm

unic

atio

ns,

but

also

hav

e an

app

licat

ion

base

tha

t in

clud

es r

adar

and

sona

r im

agin

g, e

lect

roni

c w

arfa

re,

and

med

ical

tec

hnol

ogy.

H

owev

er,

all

the

appl

icat

ion-

spec

ific

fi

lter

im

plem

enta

tions

can

be

grou

ped

into

fou

r ge

n-er

al f

ilte

r ty

pes:

lo

wpa

ss,

ban

dpas

s, a

nd b

ands

top.

The

cha

ract

eris

ticfr

eque

ncy

resp

onse

of

the

se

filt

ers

is

depi

cted

in

Fi

gure

2.

4.

The

ad

aptiv

efi

lter

, w

hich

is

the

su

bjec

t of

the

rem

aind

er o

f th

e ch

apte

r,

has

char

acte

ris-

tics

uni

que

to t

he a

ppli

cati

on i

t se

rves

. It

can

rep

rodu

ce t

he c

hara

cter

istic

s of

any

of t

he f

our

basi

c fi

lter

typ

es,

alon

e or

in

com

bina

tion.

A

s w

e sh

all

show

late

r, th

e ad

aptiv

e fi

lter

is i

deal

ly s

uite

d to

the

tel

epho

ne-e

cho

prob

lem

jus

tdi

scus

sed.

Low

pass

filt

er

Hig

hpas

s fil

ter

Ban

dpas

s fil

ter

Ban

dsto

p fil

ter

c t t c CD

Freq

uenc

y

Freq

uenc

y

Freq

uenc

y

Fre

quen

cy

Figu

re 2

.4

Fre

qu

en

cy-r

esp

on

se c

ha

ract

eri

stic

s of

the

fou

r ba

sic

filte

r ty

pes

are

sh

ow

n.

2.1

Rev

iew

of

Sig

nal

Pro

cess

ing

51

2.1.

2 F

ou

rier

An

alys

is a

nd

th

e F

req

uen

cy D

omai

n

To

anal

yze

a si

gnal

-pro

cess

ing

prob

lem

tha

t re

quir

es a

filt

er,

we

mus

t le

ave

the

tim

e do

mai

n an

d fi

nd a

too

l fo

r tr

ansl

atin

g ou

r fi

lter

mod

els

into

the

fre

quen

cydo

mai

n,

beca

use

mos

t of

the

si

gnal

s w

e w

ill

anal

yze

cann

ot b

e co

mpl

etel

yun

ders

tood

in

the

tim

e do

mai

n.

For

exam

ple,

mos

t si

gnal

s co

nsis

t no

t on

ly o

fa

fund

amen

tal

freq

uenc

y, b

ut a

lso

harm

onic

s th

at m

ust

be c

onsi

dere

d, o

r th

eyco

nsis

t of

man

y di

scre

te f

requ

ency

com

pone

nts

that

mus

t be

acc

ount

ed f

or b

yth

e fi

lters

we

desi

gn.

The

re a

re m

any

tool

s th

at w

e ca

n us

e to

hel

p un

ders

tand

the

freq

uenc

y-do

mai

n na

ture

of

sign

als.

One

of t

he m

ost

com

mon

ly u

sed

is t

heFo

urie

r se

ries.

It

has

bee

n sh

own

that

any

per

iodi

c si

gnal

can

be

mod

eled

as

an i

nfin

ite

serie

s of

sin

es a

nd c

osin

es.

The

Fou

rier

ser

ies,

whi

ch d

escr

ibes

the

freq

uenc

y-do

mai

n na

ture

of

perio

dic

sign

als,

is

give

n by

the

equ

atio

n

x(t)

= +

whe

re i

s th

e fu

ndam

enta

l fr

eque

ncy

of th

e si

gnal

in

the

time

dom

ain,

and

the

coef

fici

ents

, a

nd a

re n

eede

d to

mod

ulat

e th

e am

plitu

de o

f th

e in

divi

dual

term

s of

the

serie

s.T

his

serie

s is

use

ful

for

desc

ribi

ng t

he d

iscr

ete

freq

uenc

y co

mpo

nent

s th

atco

mpr

ise

a no

ntri

vial

per

iodi

c si

gnal

. A

s an

illu

stra

tion,

a s

quar

e w

ave

can

bede

com

pose

d in

to a

sum

mat

ion

of f

requ

ency

ele

men

ts c

onta

inin

g no

thin

g m

ore

than

sin

e w

aves

of

diff

eren

t am

plitu

de a

nd f

requ

ency

, as

is

illus

trat

ed i

n Fi

g-ur

e 2.

5. S

ince

a s

quar

e w

ave

is u

sefu

l for

repr

esen

ting

bina

ry in

form

atio

n in

dat

atr

ansm

issi

on,

it is

im

port

ant

that

we

unde

rsta

nd th

e fr

eque

ncy-

dom

ain

natu

re o

fsu

ch a

sig

nal.

From

ins

pect

ion

in t

he t

ime

dom

ain,

we

can

obse

rve

that

the

squa

re w

ave

is i

deal

ly s

uite

d to

bin

ary

data

rep

rese

ntat

ion

beca

use

ther

e ar

e tw

odi

stin

ct s

tate

s (a

1 a

nd a

0),

and

the

tran

sitio

n ti

me

betw

een

stat

es is

neg

ligib

le.

It i

s di

ffic

ult,

how

ever

, to

obt

ain

a pe

rfec

t sq

uare

wav

e in

any

pra

ctic

alel

ectr

onic

cir

cuit,

due

in

part

to

the

effe

cts

of t

he t

rans

mitt

ing

med

ia o

n th

esi

gnal

. T

o ill

ustr

ate

why

thi

s is

so,

con

side

r th

e Fo

urie

r se

ries

expa

nsio

n

x(t)

= +

- +

- +

whi

ch d

escr

ibes

a t

ypic

al s

quar

e w

ave.

As

illus

trat

ed i

n Fi

gure

2.5

, if

we

alge

brai

cally

add

tog

ethe

r th

e fi

rst

thre

esi

nuso

idal

com

pone

nts

of t

his

Four

ier

serie

s, w

e pr

oduc

e a

sign

al t

hat

alre

ady

stro

ngly

res

embl

es t

he s

quar

e w

ave.

How

ever

, w

e sh

ould

not

ice

that

the

res

ul-

tant

sig

nal

also

exh

ibit

s ri

pple

s in

bot

h ac

tive

regi

ons.

The

se r

ippl

es w

ill r

emai

nto

som

e ex

tent

, un

less

we

com

plet

e th

e in

fini

te s

erie

s.

Sinc

e th

at i

s ob

viou

sly

not

prac

tical

, w

e m

ust

even

tual

ly t

runc

ate

the

seri

es a

nd s

ettle

for

som

e am

ount

of r

ippl

e in

the

res

ulti

ng s

igna

l.It

tur

ns

out

that

thi

s tr

unca

tion

exa

ctly

cor

resp

onds

to

the

beha

vior

we

obse

rved

whe

n tr

ansm

itti

ng a

squ

are

wav

e ac

ross

an

elec

trom

agne

tic m

edia

. A

s i

s im

poss

ible

to

have

a m

ediu

m o

f in

fini

te b

andw

idth

, it

follo

ws

that

it

is t

o tr

ansm

it al

l th

e fr

eque

ncy

com

pone

nts

of a

squ

are

wav

e.

Thu

s,

52

Ad

alin

e a

nd

1.0J I

:

1

;5

0.1!-1

.5-1

.0-0

.5

0.25

0.5

1.0

1.5

0.5

1.0

1.5

-0.5

-0.7

5

0.2

0.1

0.5

1.0

1.5

Figu

re 2

.5

The

fir

st t

hree

fre

quen

cy-d

omai

n co

mpo

nent

s of

a s

quar

e w

ave

are

show

n.

Not

ice

that

the

sin

e w

aves

eac

h ha

ve d

iffer

ent

mag

nitu

des,

as

indi

cate

d by

the

coor

dina

tes

on th

e e

ven

thou

gh t

hey

are g

raph

ed t

o the

sam

e h

eigh

t.

whe

n w

e tr

ansm

it a

peri

odic

squ

are

wav

e, w

e ca

n ob

serv

e th

e fr

eque

ncy-

dom

ain

effe

cts

in t

he t

ime-

dom

ain

sign

al a

s ov

ersh

oot,

unde

rsho

ot,

and

ripp

le.

Thi

s ex

ampl

e sh

ows

that

the

Four

ier s

erie

s ca

n be

a p

ower

ful t

ool i

n he

lpin

gus

to

unde

rsta

nd t

he f

requ

ency

-dom

ain

natu

re o

f an

y pe

riodi

c si

gnal

, an

d to

pred

ict

ahea

d of

tim

e w

hat

tran

smis

sion

eff

ects

we

mus

t co

nsid

er a

s w

e de

sign

filte

rs f

or o

ur s

igna

l-pr

oces

sing

app

licat

ions

.W

e ca

n al

so a

pply

Fou

rier

ana

lysi

s to

ape

riod

ic s

igna

ls,

by e

valu

atin

g th

eFo

urie

r in

tegr

al,

whi

ch i

s gi

ven

by

2.1

Rev

iew

of

Sig

nal

Pro

cess

ing

53

We

wil

l no

t, ho

wev

er,

bela

bor

this

poi

nt.

Our

pur

pose

her

e is

mer

ely

to u

nder

-st

and

the

freq

uenc

y-do

mai

n na

ture

of s

igna

ls.

Rea

ders

inte

rest

ed in

inv

esti

gati

ngFo

urie

r an

alys

is f

urth

er a

re r

efer

red

to K

apla

n

F

ilter

Im

ple

men

tatio

n a

nd

Dig

ital

Sig

nal

Pro

cess

ing

Ear

ly i

mpl

emen

tatio

ns o

f th

e fo

ur b

asic

filt

ers

wer

e pr

edom

inan

tly t

uned

RL

Cci

rcui

ts.

Thi

s ap

proa

ch h

ad a

bas

ic l

imita

tion,

how

ever

, in

tha

t th

e fi

lters

had

only

a v

ery

smal

l ra

nge

of a

djus

tabi

lity.

A

side

fro

m o

ur b

eing

abl

e to

cha

nge

the

reso

nant

fre

quen

cy o

f th

e fi

lter

by a

djus

ting

a va

riab

le c

apac

itor

or i

nduc

tor,

the

filte

rs w

ere

pret

ty m

uch

fixe

d on

ce i

mpl

emen

ted,

lea

ving

litt

le r

oom

for

chan

ge a

s ap

plic

atio

ns b

ecam

e m

ore

soph

istic

ated

.Th

e ne

xt s

tep

in t

he e

volu

tion

of f

ilter

des

ign

cam

e ab

out w

ith t

he a

dven

t of

digi

tal

com

pute

r sy

stem

s, a

nd, j

ust

rece

ntly

, w

ith t

he a

vaila

bilit

y of

mic

roco

m-

pute

r ch

ips

with

arc

hite

ctur

es c

usto

m-t

ailo

red

for

sign

al-p

roce

ssin

g ap

plic

atio

ns.

The

basi

c co

ncep

t und

erly

ing

digi

tal

filte

r im

plem

enta

tion

is t

he i

dea

that

a c

on-

tinuo

us a

nalo

g si

gnal

can

be

sam

pled

per

iodi

cally

, qu

antiz

ed,

and

proc

esse

dby

a f

airl

y st

anda

rd c

ompu

ter

syst

em.

This

app

roac

h, i

llust

rate

d in

Fig

ure

2.6,

over

cam

e th

e lim

itatio

n of

fix

ed im

plem

enta

tion,

bec

ause

cha

ngin

g th

e fi

lter

was

sim

ply

a m

atte

r of

rew

ritin

g th

e so

ftw

are

for

the

com

pute

r. W

e w

ill t

here

fore

conc

entra

te o

n w

hat

goes

on

with

in t

he s

oftw

are

sim

ulat

ion

of th

e an

alog

fil

ter.

We

assu

me

that

the

com

pute

r im

plem

enta

tion

of t

he f

ilter

is

a di

scre

te-

time,

lin

ear,

time-

inva

rian

t sy

stem

. Sy

stem

s th

at s

atis

fy t

hese

con

stra

ints

can

perf

orm

a t

rans

form

atio

n on

an

inpu

t si

gnal

, ba

sed

on s

ome

pred

efin

ed c

rite

ria,

Orig

inal

sig

nal

Tim

e

t

Dis

cret

e sa

mpl

es

Tim

e

Fig

ure

2.6

D

iscr

ete-

time

sam

plin

g of

a c

on

tinu

ou

s si

gn

al

is s

ho

wn

.

54A

dal

ine a

nd

to p

rodu

ce a

n ou

tput

tha

t co

rres

pond

s to

the

inp

ut a

s th

ough

it

had

pass

edth

roug

h an

ana

log

filt

er.

Thu

s, a

com

pute

r, e

xecu

ting

a p

rogr

am t

hat

appl

ies

a gi

ven

tran

sfor

mat

ion

oper

atio

n, t

o di

scre

te,

digi

tize

d ap

prox

imat

ions

of

aco

ntin

uous

inp

ut s

igna

l, c

an p

rodu

ce a

n ou

tput

val

ue y

(n)

for

each

inp

utsa

mpl

e, w

here

is

the

disc

rete

tim

este

p va

riab

le.

In i

ts r

ole

in p

erfo

rmin

g th

istr

ansf

orm

atio

n, t

he c

ompu

ter

can

be t

houg

ht o

f as

a d

igita

l fi

lter

. M

oreo

ver,

any

filt

er c

an b

e co

mpl

etel

y ch

arac

teri

zed

by i

ts r

espo

nse,

h(n

), t

o th

e un

it i

mpu

lse

func

tion,

rep

rese

nted

as

M

ore

prec

isel

y,

=

The

bene

fit

of th

is f

orm

ulat

ion

is t

hat,

once

the

sys

tem

res

pons

e to

the

uni

tim

puls

e is

kno

wn,

the

sys

tem

out

put

for

any

inpu

t is

giv

en b

y

y(n)

=

—

whe

re i

s th

e sy

stem

inp

ut.

Thi

s eq

uatio

n is

mea

ning

ful

to u

s in

tha

t it

desc

ribe

s a

conv

olut

ion

sum

betw

een

the

inpu

t si

gnal

and

the

unit

impu

lse

resp

onse

of

the

syst

em.

The

pro

- c

an b

e pi

ctur

ed a

s a

win

dow

slid

ing

past

a s

cene

of

inte

rest

. A

s ill

ustr

ated

in F

igur

e 2.

7, f

or e

ach

time

step

, th

e sy

stem

out

put

is p

rodu

ced

by t

rans

posi

ngan

d sh

iftin

g o

ne p

ositi

on t

o th

e ri

ght.

The

sum

mat

ion

is t

hen

perf

orm

edov

er a

ll no

nzer

o va

lues

of

for

the

fin

ite l

engt

h of

the

filte

r. In

thi

s m

anne

r,w

e ca

n re

aliz

e th

e fi

lter

by

repe

titiv

ely

perf

orm

ing

floa

ting-

poin

t m

ultip

licat

ions

and

addi

tions

, co

uple

d w

ith s

ampl

e tim

e de

lays

and

shi

ft o

pera

tions

. R

epet

itive

,m

athe

mat

ical

ope

ratio

ns a

re w

hat

com

pute

rs d

o be

st;

ther

efor

e, t

he c

onvo

lutio

nsu

m p

rovi

des

us w

ith

a m

echa

nism

for

bui

ldin

g th

e di

gita

l eq

uiva

lent

of

anal

ogfi

lters

. R

eade

rs i

nter

este

d in

lea

rnin

g m

ore

abou

t di

gita

l si

gnal

pro

cess

ing

are

refe

rred

to

Opp

enhe

im a

nd S

chaf

er [

5] o

r H

amm

ing

It is

suf

fici

ent

for

our

purp

oses

to n

ote

that

the

con

volu

tion

sum

is

apr

oduc

ts o

pera

tion

sim

ilar

to t

he t

ype

of o

pera

tion

an A

NS

PE p

erfo

rms

whe

nco

mpu

ting

its i

nput

act

ivat

ion

sign

al.

Spec

ific

ally

, th

e A

dalin

e us

es e

xact

ly t

his

cal

cula

tion,

with

out t

he s

ampl

e tim

e de

lays

and

shi

ft o

pera

tions

,to

det

erm

ine

how

muc

h in

put

stim

ulat

ion

it re

ceiv

es f

rom

an

inst

anta

neou

s in

put

sign

al.

As

we

shal

l se

e in

the

nex

t se

ctio

n, t

he A

dalin

e ex

tend

s th

e ba

sic

filt

erop

erat

ion

one

step

fur

ther

, in

tha

t it

has

impl

emen

ted

wit

hin

itsel

f a

mea

nsof

ada

ptin

g th

e w

eigh

ting

coef

fici

ents

to

allo

w i

t to

inc

reas

e or

dec

reas

e th

est

imul

atio

n it

rece

ives

the

nex

t tim

e it

is p

rese

nted

wit

h th

e sa

me

sign

al.

The

abi

lity

of t

he A

dalin

e to

ada

pt i

ts w

eigh

ting

coef

fici

ents

is

extr

emel

yus

eful

. W

hen

wri

ting

a di

gita

l fi

lter

pro

gram

on

a co

mpu

ter,

the

pro

gram

mer

mus

t kn

ow e

xact

ly h

ow t

o sp

ecif

y th

e fi

lteri

ng a

lgor

ithm

and

wha

t th

e de

tails

of

the

sign

al c

hara

cter

istic

s ar

e. I

f mod

ific

atio

ns a

re d

esir

ed, o

r if

the

sign

al c

hara

c-te

rist

ics

chan

ge, r

epro

gram

min

g is

requ

ired

. W

hen

the

prog

ram

mer

use

s an

Ada

-lin

e, t

he p

robl

em s

hift

s to

one

of

bein

g ab

le t

o sp

ecif

y th

e de

sire

d ou

tput

sig

nal,

2.2

Ad

alin

e a

nd

th

e A

dap

tive

Lin

ear

Com

bin

er55

(a)

(b)

0.2 0.1

I

I I

•-

•

" , .

I T

, ,

01

23

45

67

n

x(n)

1.00

0.50

- • • •

• • • •

nx(

n)1

1 1

1 0.

5

5

0.5

6

0.5

7

05

(c)

|.05

|.10

I .05

.10

|.2b

.30

.25

y(0)

=

|.1

5 =

=

0.30

= =

Fig

ure

2.7

C

on

volu

tion s

um

ca

lcu

latio

n i

s (

a) T

he p

roce

ss b

egin

sby

de

term

inin

g t

he d

esi

red

resp

on

se o

f th

e f

ilte

r to

th

e u

nit

imp

uls

e f

un

ctio

n

at e

igh

t d

iscr

ete

tim

est

ep

s.

(b)

The

in

put

sign

al i

s sa

mpl

ed a

nd q

uant

ized

eig

ht

(c)

The

out

put

of

the f

ilte

r is

pro

duce

d f

or

ea

ch b

y m

ulti

plic

atio

n o

fea

ch te

rm in

(a)

with

the

corr

espo

ndin

g va

lue

of (b

) fo

r a

ll va

lidtim

este

ps.

give

n a

part

icul

ar i

nput

sig

nal.

The

Ada

line

take

s th

e in

put

and

the

desi

red

out-

put,

and

adju

sts

itse

lf s

o th

at i

t ca

n pe

rfor

m t

he d

esir

ed t

rans

form

atio

n. F

urth

er-

mor

e, t

he s

igna

l ch

arac

teri

stic

s ch

ange

, th

e A

dalin

e ca

n ad

apt

auto

mat

ical

ly.

We

shal

l no

w e

xpan

d th

ese

idea

s, a

nd b

egin

our

inve

stig

atio

n of

the

Ada

line.

2.2

AD

AL

INE

A

ND

T

HE

A

DA

PT

IVE

LIN

EA

R C

OM

BIN

ER

Ada

line

is

a de

vice

con

sist

ing

of a

sin

gle

proc

essi

ng e

lem

ent;

as

such

, it

is t

echn

ical

ly a

neu

ral

netw

ork.

N

ever

thel

ess,

it

is a

ver

y im

port

ant

stru

ctur

eth

at d

eser

ves

clos

e st

udy.

M

oreo

ver,

we

wil

l sh

ow h

ow i

t ca

n fo

rm t

he b

asis

°f a

net

wor

k in

a l

ater

sec

tion.

56A

dal

ine a

nd

Mad

alin

e

The

ter

m A

dalin

e is

an

acro

nym

; ho

wev

er,

its

mea

ning

has

cha

nged

som

e-w

hat

over

the

yea

rs.

Initi

ally

cal

led

the

AD

Apt

ive

Lin

ear

NE

uron

, it

beca

me

the

AD

Apt

ive

LIN

ear

Ele

men

t, w

hen

neur

al n

etw

orks

fel

l ou

t of

fav

or i

n th

ela

te 1

960s

. It

is

alm

ost

iden

tica

l in

str

uctu

re t

o th

e ge

nera

l PE

des

crib

ed i

nC

hapt

er 1

. F

igur

e 2.

8 sh

ows

the

Ada

line

str

uctu

re.

The

re a

re t

wo

basi

c m

od-

ific

atio

ns r

equi

red

to m

ake

the

gene

ral

PE s

truc

ture

int

o an

Ada

line.

T

he f

irst

mod

ific

atio

n is

the

add

itio

n of

a c

onne

ctio

n w

ith

wei

ght,

whi

ch w

e re

fer

to a

s th

e bi

as t

erm

. T

his

term

is

a w

eigh

t on

a c

onne

ctio

n th

at h

as i

ts i

nput

valu

e al

way

s eq

ual

to

1.

The

inc

lusi

on o

f su

ch a

ter

m i

s la

rgel

y a

mat

ter

ofex

peri

ence

. W

e sh

ow i

t he

re f

or c

ompl

eten

ess,

but

it

wil

l no

t ap

pear

in

the

disc

ussi

on o

f th

e ne

xt s

ectio

ns.

We

shal

l re

surr

ect

the

idea

of

a bi

as t

erm

in

Cha

pter

3,

on t

he b

ackp

ropa

gatio

n ne

twor

k.T

he s

econ

d m

odif

icat

ion

is t

he a

ddit

ion

of a

bip

olar

con

diti

on o

n th

e ou

tput

.T

he d

ashe

d bo

x in

Fig

ure

2.8

encl

oses

a p

art

of th

e A

dalin

e ca

lled

the

adap

tive

linea

r co

mbi

ner

(AL

C).

If th

e ou

tput

of t

he A

LC

is p

ositi

ve, t

he A

dalin

e ou

tput

is +

1. I

f th

e A

LC

out

put

is n

egat

ive,

the

Ada

line

out

put

is —

1.

Bec

ause

muc

hof

the

int

eres

ting

pro

cess

ing

take

s pl

ace

in t

he A

LC

por

tion

of

the

Ada

line

,w

e sh

all

conc

entr

ate

on t

he A

LC.

Lat

er,

we

shal

l ad

d ba

ck t

he b

inar

y ou

tput

cond

itio

n.Th

e pr

oces

sing

don

e by

the

ALC

is

that

of

the

typi

cal

proc

essi

ng e

lem

ent

desc

ribe

d in

the

pre

viou

s ch

apte

r. T

he A

LC

per

form

s a

y

+1

outp

ut-

Ada

ptiv

e lin

ear

com

bine

rI

Figu

re 2

.8

The

com

plet

e A

da

line

cons

ists

of t

he a

dapt

ive

linea

r co

mbi

ner,

in

the

dash

ed

box,

and

a

bip

ola

r o

utp

ut

fun

ctio

n.

Th

eadaptiv

e l

inear

com

bine

r re

sem

bles

the

gen

eral

PE

desc

ribed

in C

ha

pte

r

2.2

Ad

alin

e a

nd

th

e A

dap

tive

Lin

ear

Co

mb

iner

5

7

lati

on u

sing

the

inp

ut a

nd w

eigh

t ve

ctor

s, a

nd a

pplie

s an

out

put

func

tion

to

get

a si

ngle

out

put

valu

e. U

sing

the

not

atio

n in

Fig

ure

2.8,

y =

whe

re i

s th

e bi

as w

eigh

t. If

we

mak

e th

e id

enti

fica

tion

, =

1,

we

can

rew

rite

the

pre

cedi

ng e

quat

ion

as

or,

in v

ecto

r no

tati

on,

y =

(2.1)

The

out

put

func

tion

in

this

cas

e is

the

ide

ntity

fun

ctio

n,

as i

s th

e ac

tiva

tion

func

tion

. Th

e us

e of

the

iden

tity

func

tion

as

both

out

put

and

acti

vati

on f

unct

ions

mea

ns t

hat t

he o

utpu

t is

the

sam

e as

the

act

ivat

ion,

whi

ch i

s th

e sa

me

as t

he n

etin

put

to t

he u

nit.

The

Ada

line

(or

the

AL

C)

is A

DA

ptiv

e in

the

sen

se t

hat

ther

e ex

ists

aw

ell-

defi

ned

proc

edur

e fo

r m

odif

ying

the

wei

ghts

in

orde

r to

all

ow t

he d

evic

eto

giv

e th

e co

rrec

t ou

tput

val

ue f

or t

he g

iven

inp

ut.

Wha

t ou

tput

val

ue i

sco

rrec

t de

pend

s on

the

par

ticu

lar

proc

essi

ng f

unct

ion

bein

g pe

rfor

med

by

the

devi

ce.

The

Ada

line

(or

the

AL

C)

is L

inea

r be

caus

e th

e ou

tput

is a

sim

ple

line

arfu

ncti

on o

f th

e in

put

valu

es.

It i

s a

NE

uron

onl

y in

the

ver

y li

mit

ed s

ense

of

the

PEs

desc

ribe

d in

the

pre

viou

s ch

apte

r. T

he A

dali

ne c

ould

als

o be

sai

d to

be

a E

lem

ent,

avoi

ding

the

NE

uron

iss

ue a

ltoge

ther

. In

the

nex

t se

ctio

n, w

elo

ok a

t a

met

hod

to t

rain

the

Ada

line

to p

erfo

rm a

giv

en p

roce

ssin

g fu

ncti

on.

2.2.

1 T

he

LMS

Lea

rnin

g R

ule

Giv

en a

n in

put

vect

or,

x, i

t is

str

aigh

tfor

war

d to

det

erm

ine

a se

t of

wei

ghts

, w

hich

wil

l re

sult

in a

par

ticu

lar

outp

ut v

alue

, y.

Su

ppos

e w

e ha

ve a

set

of in

put

vect

ors,

XL

}, e

ach

havi

ng i

ts o

wn,

per

haps

uni

que,

cor

rect

or d

esir

ed o

utpu

t va

lue,

k =

The

pro

blem

of

find

ing

a si

ngle

wei

ght

vect

or t

hat

can

succ

essf

ully

ass

ocia

te e

ach

inpu

t ve

ctor

wit

h it

s de

sire

d ou

tput

valu

e is

no

long

er s

impl

e. I

n th

is s

ectio

n, w

e de

velo

p a

met

hod

calle

d th

e le

ast-

mea

n-sq

uare

(L

MS)

lea

rnin

g ru

le,

whi

ch i

s on

e m

etho

d of

fin

ding

the

des

ired

wei

ght

vect

or.

We

refe

r to

thi

s pr

oces

s of

fin

ding

the

wei

ght

vect

or a

s tr

aini

ngth

e A

LC.

The

lear

ning

rul

e ca

n be

em

bedd

ed i

n th

e de

vice

its

elf,

whi

ch c

an t

hen

self-

adap

t as

inp

uts

and

desi

red

outp

uts

are

pres

ente

d to

it.

Smal

l ad

just

men

tsar

e m

ade

to t

he w

eigh

t va

lues

as

each

com

bina

tion

is

proc

esse

dun

til

the

AL

C g

ives

cor

rect

out

puts

. In

a s

ense

, th

is p

roce

dure

is

a tr

ue t

rain

ing

proc

edur

e, b

ecau

se w

e do

not

nee

d to

cal

cula

te t

he v

alue

of

the

wei

ght

vect

orex

plic

itly

. B

efor

e de

scri

bing

the

tra

inin

g pr

oces

s in

det

ail,

let

's p

erfo

rm t

heca

lcul

atio

n m

anua

lly.

58

Ad

alin

e a

nd

Cal

cula

tion

of w

*.

To b

egin

, le

t's s

tate

the

pro

blem

a l

ittle

dif

fere

ntly

: G

iven

exam

ples

, o

f so

me

proc

essi

ng f

unct

ion

that

ass

o-ci

ates

inp

ut v

ecto

rs,

wit

h (o

r m

aps

to)

the

desi

red

outp

ut v

alue

s, w

hat

is t

he b

est

wei

ght

vect

or,

for

an

AL

C t

hat

perf

orm

s th

is m

appi

ng?

To a

nsw

er t

his

ques

tion,

we

mus

t fi

rst

defi

ne w

hat

it is

tha

t co

nstit

utes

the

best

wei

ght

vect

or.

Cle

arly

, on

ce t

he b

est

wei

ght

vect

or i

s fo

und,

we

wou

ldlik

e th

e ap

plic

atio

n of

eac

h in

put

vect

or t

o re

sult

in t

he p

reci

se,

corr

espo

ndin

gou

tput

val

ue.

Thu

s, w

e w

ant t

o el

imin

ate,

or

at le

ast t

o m

inim

ize,

the

diff

eren

cebe

twee

n th

e de

sire

d ou

tput

and

the

act

ual

outp

ut f

or e

ach

inpu

t ve

ctor

. Th

eap

proa

ch w

e se

lect

her

e is

to

min

imiz

e th

e m

ean

squa

red

erro

r fo

r th

e se

t of

inpu

t ve

ctor

s.If

the

act

ual

outp

ut v

alue

is

fo

r th

e i

nput

vec

tor,

the

n th

e co

rre-

spon

ding

err

or t

erm

is

—

The

mea

n sq

uare

d er

ror,

or

expe

ctat

ion

valu

e of

the

err

or,

is d

efin

ed b

y

(2.2)

k=\

whe

re L

is

the

num

ber

of i

nput

vec

tors

in

the

trai

ning

Usi

ng E

q. w

e ca

n ex

pand

the

mea

n sq

uare

d er

ror

as f

ollo

ws:

= =•

(2.3)

(2.4)

In g

oing

fro

m E

q. (

2.3)

to

Eq.

(2.4

), w

e ha

ve m

ade

the

assu

mpt

ion

that

the

trai

ning

set

is

stat

istic

ally

sta

tiona

ry,

mea

ning

tha

t an

y ex

pect

atio

n va

lues

var

ysl

owly

wit

h re

spec

t to

tim

e. T

his

assu

mpt

ion

allo

ws

us t

o fa

ctor

out

the

wei

ght

vect

ors

from

the

exp

ecta

tion

valu

e te

rms

in E

q. (

2.4)

.

Exe

rcis

e 2.

1: G

ive

the

deta

ils o

f th

e de

riva

tion

that

lea

ds f

rom

Eq.

(2.

3),

toE

q. (

2.4)

alo

ng w

ith

the

just

ific

atio

n fo

r ea

ch s

tep.

Why

are

the

fac

tors

dk

and

lef

t to

geth

er i

n th

e la

st t

erm

in

Eq.

(2.

4),

rath

er t

han

show

n as

the

pro

duct

of t

he t

wo

sepa

rate

exp

ecta

tion

val

ues?

Def

ine

a m

atri

x R

= c

alle

d th

e in

put

corr

elat

ion

mat

rix,

and

ave

ctor

p

Furt

her,

mak

e th

e id

entif

icat

ion

£ =

U

sing

the

sede

fini

tion

s, w

e ca

n re

wri

te E

q. (

2.4)

as

(2.5

)

Thi

s eq

uatio

n sh

ows

as

an e

xpli

cit

func

tion

of

the

wei

ght

vect

or,

w.

In o

ther

wor

ds,

=

and

Ste

arns

use

the

not

atio

n, ]

, fo

r th

e ex

pect

atio

n va

lue;

als

o, t

he t

erm

exe

mpl

ars

wil

l so

met

imes

be

seen

as

a sy

nony

m f

or t

rain

ing

set.

2.2

Ad

alin

e a

nd

th

e A

dap

tive

Lin

ear

Co

mb

iner

59

To f

ind

the

wei

ght

vect

or c

orre

spon

ding

to

the

min

imum

m

ean

squa

red

erro

r, w

e di

ffer

enti

ate

Eq.

(2

.5),

eval

uate

the

res

ult

at a

nd s

et t

he r

esul

teq

ual

to z

ero:

= -

2p

- 2

p =

0

=

p

=

(2.6)

(2.7) (2.8)

Not

ice

that

, al

thou

gh i

s a

scal

ar,

is

a v

ecto

r. Eq

uatio

n (2

.6)

is a

nex

pres

sion

of

the

grad

ient

of

whi

ch i

s th

e ve

ctor

(2.9)

_

All

tha

t w

e ha

ve d

one

by t

he p

roce

dure

is

to s

how

tha

t w

e ca

n fi

nd a

poi

ntw

here

the

slo

pe o

f th

e fu

ncti

on,

is

zero

. In

gen

eral

, th

at p

oint

may

be

am

inim

um o

r a

max

imum

poi

nt.

In t

he e

xam

ple

that

fol

low

s, w

e sh

ow a

sim

ple

case

whe

re t

he A

LC

has

onl

y tw

o w

eigh

ts.

In t

hat

situ

atio

n, t

he g

raph

of

is a

par

abol

oid.

Fur

ther

mor

e, i

t mus

t be

conc

ave

upw

ard,

sin

ce a

ll co

mbi

nati

ons

of w

eigh

ts m

ust

resu

lt in

a n

onne

gativ

e va

lue

for

the

mea

n sq

uare

d er

ror,

Thi

s re

sult

is g

ener

al a

nd i

s ob

tain

ed r

egar

dles

s of

the

dim

ensi

on o

f th

e w

eigh

tve

ctor

. In

the

cas

e of

dim

ensi

ons

high

er t

han

two,

the

par

abol

oid

is k

now

n as

aSu

ppos

e w

e ha

ve

an

AL

C

wit

h tw

o in

puts

and

var

ious

oth

er q

uant

itie

sde

fine

d as

fol

low

s:

R =

3 1

1 4]

= 1

0

Rat

her

than

inv

erti

ng R

, w

e us

e E

q. (

2.7)

to

find

the

opt

imum

wei

ght

vect

or:

3 1

1 4

Thi

s eq

uati

on r

esul

ts i

n tw

o eq

uati

ons

for

w*

and

w*

+

= 5

The

solu

tion

is w

* (

1,

The

gra

ph o

f a

s a

func

tion

of

the

two

wei

ghts

is s

how

n in

Fig

ure

2.9.

2.2

: Sh

ow t

hat

the

min

imum

val

ue o

f th

e m

ean

squa

red

erro

r ca

n be

as

L

60 a

nd

Fig

ure

2.9

F

or

an

AL

C

with

on

ly

two

wei

ghts

, th

e er

ror

surf

ace

is

a

para

bolo

id.

The

wei

ghts

tha

t m

inim

ize

the

err

or

occ

ur

at t

hebo

ttom

of

the p

arab

oloi

dal

surf

ace.

Exe

rcis

e 2.

3:

Det

erm

ine

an e

xpli

cit

equa

tion

for

as

a fu

ncti

on o

f a

ndus

ing

the

exam

ple

in t

he t

ext.

Use

it

to f

ind

the

opt

imum

wei

ght

vect

or,

the

min

imum

mea

n sq

uare

d er

ror,

and

pro

ve t

hat

the

para

bolo

id i

sco

ncav

e up

war

d.

In t

he n

ext

sect

ion,

we

shal

l ex

amin

e a

met

hod

for

find

ing

the

optim

umw

eigh

t ve

ctor

by

an i

tera

tive

pro

cedu

re.

Thi

s pr

oced

ure

allo

ws

us t

o av

oid

the

ofte

n-di

ffic

ult

calc

ulat

ions

nec

essa

ry t

o de

term

ine

the

wei

ghts

man

uall

y.

Fin

ding

w*

by t

he M

etho

d of

Ste

epes

t D

esce

nt.

As

you

mig

ht i

mag

ine,

the

anal

ytic

al c

alcu

lati

on t

o de

term

ine

the

opti

mum

wei

ghts

for

a p

robl

em i

s ra

ther

diff

icul

t in

gen

eral

. N

ot o

nly

does

the

mat

rix

man

ipul

atio

n ge

t cu

mbe

rsom

e fo

rla

rge

dim

ensi

ons,

but

als

o ea

ch c

ompo

nent

of

R a

nd p

is

itsel

f an

exp

ecta

tion

valu

e. T

hus,

exp

lici

t ca

lcul

atio

ns o

f R

and

p r

equi

re k

now

ledg

e of

the

stat

isti

csof

the

inpu

t si

gnal

s. A

bet

ter

appr

oach

wou

ld b

e to

let

the

AL

C f

ind

the

opti

mum

wei

ghts

its

elf

by h

avin

g it

sear

ch o

ver

the

wei

ght

surf

ace

to f

ind

the

min

imum

.A

pur

ely

rand

om s

earc

h m

ight

not

be

prod

ucti

ve o

r ef

fici

ent,

so w

e sh

all

add

som

e in

tell

igen

ce t

o th

e pr

oced

ure.

2.2

an

d t

he A

dap

tive L

inea

r C

om

bin

er61

Beg

in b

y as

sign

ing

arbi

trar

y va

lues

to

the

wei

ghts

. Fr

om t

hat

poin

t on

the

wei

ght

surf

ace,

det

erm

ine

the

dire

ctio

n of

the

ste

epes

t sl

ope

in t

he d

ownw

ard

dire

ctio

n. C

hang

e th

e w

eigh

ts s

ligh

tly

so t

hat

the

new

wei

ght

vect

or l

ies

fart

her

dow

n th

e su

rfac

e. R

epea

t th

e pr

oces

s un

til

the

min

imum

has

bee

n re

ache

d. T

his

proc

edur

e is

ill

ustr

ated

in

Figu

re I

mpl

icit

in t

his

met

hod

is t

he a

ssum

ptio

nth

at w

e kn

ow w

hat

the

wei

ght

surf

ace

look

s li

ke i

n ad

vanc

e. W

e do

not

kno

w,

but

we

wil

l se

e sh

ortl

y ho

w t

o ge

t ar

ound

thi

s pr

oble

m.

Typ

ical

ly,

the

wei

ght

vect

or d

oes

not

init

iall

y m

ove

dire

ctly

tow

ard

the

min

imum

poi

nt.

The

cros

s-se

ctio

n of

the

par

abol

oida

l w

eigh

t su

rfac

e is

usu

ally

elli

ptic

al,

so t

he n

egat

ive

grad

ient

may

not

poi

nt d

irec

tly

at t

he m

inim

um p

oint

,at

lea

st i

niti

ally

. Th

e si

tuat

ion

is i

llust

rate

d m

ore

clea

rly

in t

he c

onto

ur p

lot

ofth

e w

eigh

t su

rfac

e in

Fig

ure

Figu

re 2

.10

We c

an

use t

his

di

agra

m

to vi

sua

lize t

he st

eepe

st-d

esce

ntm

etho

d.

An

initi

al

sele

ctio

n fo

r th

e w

eig

ht

vect

or

resu

ltsin

a

n

err

or,

T

he st

ee

pe

st-d

esc

en

t m

etho

d co

nsi

sts

of

slid

ing th

is p

oin

t dow

n the s

urf

ace

tow

ard

the b

otto

m,

alw

ays

mo

vin

g i

n t

he d

ire

ctio

n o

f th

e s

teepest

do

wn

wa

rd s

lope.

62 a

nd

2.3.

Fig

ure

In

the c

onto

ur

plot

of

the w

eig

ht

surf

ace

of

Fig

ure

the

dir

ect

ion

of s

teep

est

de

sce

nt

is p

erp

en

dic

ula

r to

the

con

tour

line

s at

ea

ch p

oint

, an

d th

is d

ire

ctio

n do

es n

ot a

lwa

ys p

oin

tto

the m

inim

um

poin

t.

Bec

ause

the

wei

ght

vect

or i

s va

riab

le i

n th

is p

roce

dure

, w

e w

rite

it

as a

nex

plic

it fu

ncti

on o

f th

e ti

mes

tep,

t.

The

init

ial

wei

ght

vect

or i

s de

note

dan

d th

e w

eigh

t ve

ctor

at

times

tep

t is

At e

ach

step

, th

e ne

xt w

eigh

t ve

ctor

is c

alcu

late

d ac

cord

ing

to

1) =

+(2

.10)

whe

re i

s th

e ch

ange

in

at

the

tim

este

p.W

e ar

e lo

okin

g fo

r th

e di

rect

ion

of t

he s

teep

est

desc

ent

at e

ach

poin

t on

the

surf

ace,

so

we

need

to

calc

ulat

e th

e gr

adie

nt o

f th

e su

rfac

e (w

hich

giv

es t

hedi

rect

ion

of t

he s

teep

est

slo

pe).

T

he n

egat

ive

of t

he g

radi

ent

is i

n th

edi

rect

ion

of s

teep

est

desc

ent.

To g

et t

he m

agni

tude

of

the

chan

ge,

mul

tipl

y th

egr

adie

nt b

y a

suita

ble

cons

tant

, T

he a

ppro

pria

te v

alue

for

wil

l be

dis

cuss

edla

ter.

Thi

s pr

oced

ure

resu

lts

in t

he f

ollo

win

g ex

pres

sion

:

(2.1

1)

2.2

A

dal

ine

and

th

e A

dap

tive

Lin

ear

Co

mb

iner

All

tha

t is

nec

essa

ry t

o co

mpl

ete

the

disc

ussi

on i

s to

det

erm

ine

the

valu

e of

at

each

suc

cess

ive

iter

atio

n st

ep.

The

val

ue o

f w

as d

eter

min

ed a

naly

tica

lly

in t

he p

revi

ous

sect

ion.

Equ

atio

n (2

.6)

or E

q. (

2.9)

cou

ld b

e us

ed h

ere

to d

eter

min

e b

ut w

ew

ould

hav

e th

e sa

me

prob

lem

tha

t w

e ha

d w

ith

the

anal

ytic

al d

eter

min

atio

nof

W

e w

ould

nee

d to

kno

w b

oth

R a

nd p

in

adva

nce.

T

his

know

ledg

eis

equ

ival

ent

to k

now

ing

wha

t th

e w

eigh

t su

rfac

e lo

oks

like

in

adva

nce.

To

circ

umve

nt t

his

diff

icul

ty,

we

use

an a

ppro

xim

atio

n fo

r th

e gr

adie

nt t

hat

can

bede

term

ined

fro

m i

nfor

mat

ion

that

is

know

n ex

plic

itly

at

each

ite

ratio

n.Fo

r ea

ch s

tep

in t

he i

tera

tion

pro

cess

, w

e pe

rfor

m t

he f

ollo

win

g:

1. A

pply

an

inpu

t ve

ctor

, t

o th

e A

dalin

e in

puts

.

2.

Det

erm

ine

the

valu

e of

the

err

or s

quar

ed,

usi

ng t

he c

urre

nt v

alue

of

the

wei

ght

vect

or

(2.1

2)

3.

Cal

cula

te a

n ap

prox

imat

ion

to b

y us

ing

as

an a

ppro

xim

atio

nfo

r

(2

.13)

=

(2.1

4)

whe

re w

e ha

ve u

sed

Eq.

to

calc

ulat

e th

e gr

adie

nt e

xpli

citl

y.

4. U

pdat

e th

e w

eigh

t ve

ctor

acc

ordi

ng t

o E

q. u

sing

Eq.

as

the

appr

oxim

atio

n fo

r th

e gr

adie

nt:

+ 1

) =

+

(2.1

5)

5. R

epea

t st

eps

1 th

roug

h 4

wit

h th

e ne

xt i

nput

vec

tor,

unti

l th

e er

ror

has

been

redu

ced

to a

n ac

cept

able

val

ue.

Equ

atio

n (2

.15)

is

an e

xpre

ssio

n of

the

LM

S a

lgor

ithm

. T

he p

aram

eter

dete

rmin

es t

he s

tabi

lity

and

spee

d of

con

verg

ence

of

the

wei

ght

vect

or t

owar

dth

e m

inim

um-e

rror

val

ue.

Bec

ause

an

appr

oxim

atio

n of

the

gra

dien

t ha

s be

en u

sed

in E

q. t

hepa

th t

hat

the

wei

ght

vect

or t

akes

as

it m

oves

dow

n th

e w

eigh

t su

rfac

e to

war

dth

e m

inim

um w

ill

not b

e as

sm

ooth

as

that

ind

icat

ed i

n Fi

gure

Fig

ure

show

s an

exa

mpl

e of

how

a s

earc

h pa

th m

ight

loo

k w

ith

the

LM

S al

gori

thm

of

Eq.

(2.

15).

Cha

nges

in

the

wei

ght

vect

or m

ust

be k

ept

rela

tive

ly s

mal

l on

eac

hite

ratio

n.

If c

hang

es a

re t

oo l

arge

, th

e w

eigh

t ve

ctor

cou

ld w

ande

r ab

out

the

surf

ace,

nev

er f

indi

ng t

he m

inim

um,

or f

indi

ng i

t on

ly b

y ac

cide

nt r

athe

r th

anas

a r

esul

t of

a s

tead

y co

nver

genc

e to

war

d it

. T

he f

unct

ion

of t

he p

aram

eter

is t

o pr

even

t th

is a

imle

ss s

earc

hing

. In

the

nex

t se

ctio

n, w

e sh

all

disc

uss

the

para

met

er,

and

oth

er p

ract

ical

con

side

rati

ons.

64

Ad

alin

e a

nd

Fig

ure

T

he h

ypo

the

tica

l pat

h t

ake

n b

y a

we

igh

t ve

cto

r a

s it

sea

rch

es

for

the

min

imu

m

err

or

usi

ng t

he

alg

ori

thm

is

no

t a

smo

oth

cu

rve b

eca

use

the

gra

die

nt

is b

ein

g a

pp

roxi

ma

ted

at e

ach p

oint

. N

ote a

lso t

hat

step

siz

es

get

smal

ler

as t

hem

inim

um

-err

or

solu

tion is

ap

pro

ach

ed

.

2.2.

2 P

ract

ical

Con

side

ratio

ns

The

re a

re s

ever

al q

uest

ions

to

cons

ider

whe

n w

e ar

e at

tem

ptin

g to

use

the

AL

Cto

sol

ve a

par

ticu

lar

prob

lem

:

• H

ow m

any

trai

ning

vec

tors

are

req

uire

d to

sol

ve a

par

ticu

lar

prob

lem

?

• H

ow i

s th

e ex

pect

ed o

utpu

t ge

nera

ted

for

each

tra

inin

g ve

ctor

?

• W

hat

is t

he a

ppro

pria

te d

imen

sion

of

the

wei

ght

vect

or?

• W

hat

shou

ld b

e th

e in

itia

l va

lues

for

the

wei

ghts

?

• Is

a b

ias

wei

ght

requ

ired

?

• W

hat

happ

ens

if t

he s

igna

l st

atis

tics

var

y w

ith

tim

e?

2.2

an

d t

he

Ad

apti

ve L

inea

r C

om

bin

er65

• W

hat

is t

he a

ppro

pria

te v

alue

for

• H

ow d

o w

e de

term

ine

whe

n to

sto

p tr

aini

ng?

The

ans

wer

s to

the

se q

uest

ions

dep

end

on t

he s

peci

fic

prob

lem

bei

ng a

ddre

ssed

,so

it

is d

iffi

cult

to

give

wel

l-de

fine

d re

spon

ses

that

app

ly i

n al

l ca

ses.

Mor

eove

r,fo

r a

spec

ific

cas

e, t

he a

nsw

ers

are

not

nece

ssar

ily

inde

pend

ent.

Con

side

r th

e di

men

sion

of

the

wei

ght

vect

or.

If t

here

are

a w

ell-

defi

ned

num

ber

of f

rom

mul

tipl

e t

here

wou

ld b

e on

e w

eigh

tfo

r eac

h in

put.

The

ques

tion

wou

ld b

e w

heth

er to

add

a b

ias

wei

ght.

Figu

rede

pict

s th

is c

ase,

wit

h th

e bi

as t

erm

add

ed,

in a

som

ewha

t st

anda

rd f

orm

tha

tsh

ows

the

vari

abil

ity

of t

he w

eigh

ts,

the

erro

r te

rm,

and

the

feed

back

fro

mth

e ou

tput

to

the

wei

ghts

. A

s fo

r th

e bi

as t

erm

its

elf,

inc

ludi

ng i

t so

met

imes

help

s co

nver

genc

e of

the

wei

ghts

to

an a

ccep

tabl

e so

luti

on.

It i

s pe

rhap

s be

stth

ough

t of

as

an e

xtra

deg

ree

of fr

eedo

m,

and

its

use

is l

arge

ly a

mat

ter

ofex

peri

men

tatio

n w

ith t

he s

peci

fic

appl

icat

ion.

A s

itua

tion

dif

fere

nt f

rom

the

pre

viou

s pa

ragr

aph

aris

es i

f th

ere

is o

nly

asi

ngle

inp

ut s

igna

l, sa

y fr

om a

sin

gle

elec

troc

ardi

ogra

ph (

EK

G)

sens

or.

For

= 1

(bi

as in

put)

des

ired

outp

ut

Fig

ure

2.1

3

This

figure

show

s a s

tandard

dia

gra

m o

f the A

LC

with

multi

ple

inpu

ts a

nd

a bi

as t

erm

. W

eigh

ts a

re i

nd

ica

ted

as v

ari

ab

lere

sist

ors

to

em

phas

ize

the

adap

tive

natu

re

of

the

devi

ce.

Calc

ula

tion o

f the e

rror,

is s

how

n e

xp

licitly

as the

additi

on

of

a n

egativ

e o

f th

e o

utp

ut

sig

na

l to

th

e d

esir

ed o

utp

ut

valu

e.

66A

dal

ine

and

exam

ple,

an

AL

C c

an b

e us

ed t

o re

mov

e no

ise

from

the

inp

ut s

igna

l in

ord

er t

ogi

ve a

cle

aner

sig

nal a

t the

out

put.

In a

cas

e su

ch a

s th

is o

ne, t

he A

LC

is

arra

nged

in a

con

figu

rati

on k

now

n as

a t

rans

vers

e fi

lter

. In

thi

s co

nfig

urat

ion,

the

inp

utsi

gnal

is

sam

pled

at

seve

ral

poin

ts i

n ti

me,

rat

her

than

fro

m s

ever

al s

enso

rs a

ta

sing

le t

ime.

Fig

ure

sho

ws

the

AL

C a

rran

ged

as a

tra

nsve

rse

filt

er.

For

the

tran

sver

se f

ilte

r, e

ach

addi

tion

al s

ampl

e in

tim

e re

pres

ents

ano

ther

degr

ee o

f fr

eedo

m t

hat

can

be u

sed

to f

it t

he i

nput

sig

nal

to t

he d

esir

ed o

utpu

tsi

gnal

. T

hus,

if

you

cann

ot g

et a

goo

d fi

t w

ith

a sm

all

num

ber

of s

ampl

es,

try

a fe

w m

ore.

O

n th

e ot

her

hand

, if

you

get

good

con

verg

ence

wit

h yo

ur f

irst

Fig

ure

2.1

4 In

an

A

LC

ar

rang

ed

as

a tr

an

sve

rse

filte

r,

the

sam

ple

s a

re p

rovi

de

d b

y n—

1,

pre

sum

ab

ly e

qual

, tim

e d

ela

ys,

T

he

AL

C s

ees

the

sig

na

l at

the

cu

rre

nt

time,

as

we

ll as

its v

alu

e at

th

e p

revi

ou

s n

- 1

sam

ple

tim

es.

W

hen

data

is

initi

ally

applie

d,

rem

em

ber

to w

ait

at

least

for

data

to b

epr

esen

t at

all

of t

he A

LC

's i

nput

s.

2.2

A

dal

ine a

nd

th

e A

dap

tive L

inea

r C

om

bin

er

67

choi

ce,

try

one

wit

h fe

wer

sam

ples

to

see

whe

ther

you

get

a s

igni

fica

nt s

peed

upin

con

verg

ence

and

stil

l ha

ve s

atis

fact

ory

resu

lts (

you

may

be

surp

rise

d to

fin

dth

at t

he r

esul

ts a

re b

ette

r in

som

e ca

ses)

. M

oreo

ver,

the

bia

s w

eigh

t is

pro

babl

ysu

perf

luou

s in

thi

s ca

se.

Ear

lier,

we

allu

ded

to a

rela

tions

hip

betw

een

trai

ning

tim

e an

d th

e di

men

sion

of t

he w

eigh

t ve

ctor

, es

peci

ally

for

the

sof

twar

e si

mul

atio

ns t

hat

we

cons

ider

in t

his

text

: M

ore

wei

ghts

gen

eral

ly m

ean

long

er t

rain

ing

tim

es.

Thi

s eq

uati

onm

ust

be c

onst

antly

bal

ance

d ag

ains

t ot

her

fact

ors,

suc

h as

the

acc

epta

bilit

y of

the

solu

tion.

As

stat

ed i

n th

e pr

evio

us p

arag

raph

, us

ing

mor

e w

eigh

ts d

oes

not

alw

ays

resu

lt in

a b

ette

r so

lutio

n. F

urth

erm

ore,

the

re a

re o

ther

fac

tors

tha

t af

fect

both

the

tra

inin

g ti

me

and

the

acce

ptab

ility

of

the

solu

tion.

The

para

met

er i

s on

e fa

ctor

tha

t ha

s a

sign

ific

ant

effe

ct o

n tr

aini

ng.

If i

s to

o la

rge,

con

verg

ence

wil

l ne

ver

take

pla

ce,

no m

atte

r ho

w l

ong

is t

hetr

aini

ng p

erio

d.

If t

he s

tatis

tics

of t

he i

nput

sig

nal

are

know

n, i

t is

pos

sibl

e to

show

tha

t th

e va

lue

of i

s re

stri

cted

to

the

rang

e

0

whe

re i

s th

e la

rges

t eig

enva

lue

of th

e m

atri

x R

, the

inp

ut c

orre

latio

n m

atri

xdi

scus

sed

in S

ectio

n A

lthou

gh i

t is

not

alw

ays

reas

onab

le t

o ex

pect

thes

e st

atis

tics

to b

e kn

own,

the

re a

re c

ases

whe

re t

hey

can

be e

stim

ated

. T

hete

xt b

y W

idro

w a

nd S

tear

ns c

onta

ins

man

y ex

ampl

es.

In t

his

text

, w

e pr

opos

e a

mor

e he

uris

tic a

ppro

ach:

Pic

k a

valu

e fo

r s

uch

that

a w

eigh

t do

es n

ot c

hang

eby

mor

e th

an a

sm

all

frac

tion

of it

s cu

rren

t val

ue.

Thi

s ru

le i

s ad

mitt

edly

vag

ue,

but

expe

rien

ce a

ppea

rs t

o be

the

bes

t te

ache

r fo

r se

lect

ing

an a

ppro

pria

te v

alue

for

As

trai

ning

pro

ceed

s, t

he e

rror

val

ue w

ill

dim

inis

h (h

opef

ully

), r

esul

ting

in s

mal

ler

and

smal

ler

wei

ght

chan

ges,

and

, he

nce,

in

a sl

ower

con

verg

ence

tow

ard

the

min

imum

of t

he w

eigh

t sur

face

. It

is s

omet

imes

use

ful t

o in

crea

se t

heva

lue

of d

urin

g th

ese

peri

ods

to s

peed

con

verg

ence

. B

ear

in m

ind,

how

ever

,th

at a

lar

ger

may

mea

n th

at t

he w

eigh

ts m

ight

bou

nce

arou

nd t

he b

otto

m o

fth

e w

eigh

t su

rfac

e, g

ivin

g an

ove

rall

erro

r th

at i

s un

acce

ptab

le.

Her

e ag

ain,

expe

rien

ce i

s ne

cess

ary

to e

nabl

e us

to

judg

e ef

fect

ivel

y.O

ne m

etho

d of

com

pens

atin

g fo

r di

ffer

ence

s in

pro

blem

s is

to

use

norm

al-

ized

inp

ut v

ecto

rs.

Inst

ead

of u

se •

A

noth

er t

acti

c is

to

scal

e th

ede

sire

d ou

tput

val

ue.

Thes

e m

etho

ds h

elp

part

icul

arly

whe

n w

e ar

e se

lect

ing

init

ial

wei

ght

valu

es o

r a

valu

e fo

r I

n m

ost

case

s, w

eigh

ts c

an b

e in

itia

lize

d to

rand

om v

alue

s of

sm

all

real

bet

wee

n -1

.0 a

nd T

he v

alue

of

is

usua

lly

best

kep

t si

gnif

ican

tly

less

tha

n 1;

a v

alue

of

or

even

0.0

5 m

aybe

rea

sona

ble

for

som

e b

ut v

alue

s co

nsid

erab

ly le

ss m

ay b

e re

quir

ed.

The

que

stio

n of

whe

n to

sto

p tr

aini

ng i

s la

rgel

y a

mat

ter

of th

e re

quir

emen

tson

the

out

put

of t

he s

yste

m.

You

det

erm

ine

the

amou

nt o

f er

ror

that

you

can

on

the

outp

ut s

igna

l, an

d tr

ain

unti

l th

e ob

serv

ed e

rror

is

cons

iste

ntly

less

tha

n th

e re

quir

ed v

alue

. Si

nce

the

mea

n sq

uare

d er

ror

is t

he v

alue

use

d to

deri

ve t

he t

rain

ing

algo

rith

m,

that

is

the

quan

tity

tha

t us

uall

y de

term

ines

whe

n

68A

dal

ine

and

Mad

alin

e

a sy

stem

has

con

verg

ed t

o it

s m

inim

um e

rror

sol

utio

n. A

lter

nati

vely

, ob

serv

ing

indi

vidu

al e

rror

s is

oft

en n

eces

sary

, si

nce

the

syst

em p

erfo

rman

ce m

ay h

ave

a re

quir

emen

t th

at n

o er

ror

exce

ed a

cer

tain

am

ount

. N

ever

thel

ess,

a m

ean

squa

red

erro

r th

at f

alls

as

the

iter

atio

n nu

mbe

r in

crea

ses

is p

roba

bly

your

bes

tin

dica

tion

tha

t th

e sy

stem

is

conv

ergi

ng t

owar

d a

solu

tion

.W

e us

uall

y as

sum

e th

at t

he i

nput

sig

nals

are

sta

tist

ical

ly s

tati

onar

y, a

nd,

ther

efor

e,

is e

ssen

tially

a

cons

tant

afte

r th

e op

timum

val

ues

have

been

det

erm

ined

. D

urin

g tr

aini

ng,

wil

l ho

pefu

lly

decr

ease

tow

ard

a st

able

solu

tion.

Su

ppos

e, h

owev

er,

that

the

inp

ut s

igna

l st

atis

tics

chan

ge s

omew

hat

over

tim

e, o

r un

derg

o so

me

disc

onti

nuit

y: A

dditi

onal

tra

inin

g w

ould

be

requ

ired

to c

ompe

nsat

e.O

ne

way

to

deal

w

ith

this

sit

uati

on

is t

o ce

ase

or r

esum

e tr

aini

ng c

on-

diti

onal

ly,

base

d on

the

cur

rent

val

ue o

f

If t

he s

igna

l st

atis

tics

cha

nge,

trai

ning

can

be

rein

itiat

ed u

ntil

is

aga

in r

educ

ed t

o an

acc

epta

ble

valu

e.T

his

met

hod

pres

umes

tha

t a

met

hod

of e

rror

mea

sure

men

t is

ava

ilabl

e.Pr

ovid

ed t

hat t

he i

nput

sig

nals

are

sta

tist

ical

ly s

tati

onar

y, c

hoos

ing

the

num

-be

r of

inp

ut v

ecto

rs t

o us

e du

ring

tra

inin

g m

ay b

e re

lativ

ely

sim

ple.

Y

ou c

anus

e re

al,

inp

uts

as t

rain

ing

vect

ors,

pro

vide

d th

at y

ou k

now

the

desi

red

outp

ut f

or e

ach

inpu

t ve

ctor

. If

it

is p

ossi

ble

to i

dent

ify

a sa

mpl

e of

inpu

t ve

ctor

s th

at a

dequ

atel

y re

prod

uces

the

sta

tistic

al d

istr

ibut

ion

of t

he a

ctua

lin

puts

, it

may

be

poss

ible

to

trai

n on

thi

s se

t in

a s

hort

er t

ime.

Th

e ac

cura

cyof

the

tra

inin

g de

pend

s on

how

wel

l th

e se

lect

ed s

et o

f tr

aini

ng v

ecto

rs m

odel

sth

e di

stri

buti

on o

f th

e en

tire

inpu

t si

gnal

spa

ce.

The

oth

er,

rela

ted

ques

tion

is h

ow t

o go

abo

ut d

eter

min

ing

the

desi

red

outp

ut

for

a gi

ven

inpu

t ve

ctor

. A

s w

ith

man

y qu

esti

ons

disc

usse

d in

thi

sse

ctio

n, t

his

depe

nds

on t

he s

peci

fic

deta

ils o

f th

e pr

oble

m.

Fort

unat

ely,

for

som

e pr

oble

ms,

kn

owin

g th

e de

sire

d re

sult

is e

asy

com

pare

d to

fin

ding

an

algo

rith

m f

or t

rans

form

ing

the

inpu

ts i

nto

the

desi

red

resu

lt.

The

AL

C w

ill

ofte

n so

lve

the

diff

icul

t pa

rt.

The

"eas

y" p

art

is l

eft

to t

he e

ngin

eer.

Exe

rcis

e 2.

4:

A l

owpa

ss f

ilte

r ca

n be

con

stru

cted

wit

h an

Ada

line

havi

ng t

wo

wei

ghts

. C

onsi

der

a si

mpl

e ca

se o

f th

e re

mov

al o

f a

rand

om n

oise

fro

m a

cons

tant

sig

nal.

The

con

stan

t si

gnal

lev

el i

s C

=

3,

an

d th

e ra

ndom

noi

sesi

gnal

has

a c

onst

ant

pow

er,

— —

0.0

25.

Ass

ume

that

the

ran

dom

noi

seis

com

plet

ely

wit

h th

e co

nsta

nt in

put s

igna

l. C

alcu

late

the

optim

umw

eigh

t ve

ctor

and

the

mea

n sq

uare

d er

ror

in t

he o

utpu

t af

ter

the

opti

mum

wei

ght

vect

or h

as b

een

foun

d.

By

find

ing

the

eige

nval

ues

of t

he m

atri

x, R

, de

term

ine

the

max

imum

val

ue o

f th

e co

nsta

nt f

or u

se i

n th

e L

MS

algo

rith

m.

2.3

AP

PL

ICA

TIO

NS

O

F

AD

AP

TIV

ES

IGN

AL

PR

OC

ES

SIN

G

Up

to n

ow,

we

have

bee

n co

ncer

ned

wit

h th

e A

dali

ne m

inus

the

thr

esho

ldco

ndit

ion

on t

he o

utpu

t. In

Sec

tion

2.4,

on

the

Mad

alin

e, w

e w

ill

repl

ace

the

thre

shol

d co

ndit

ion

and

exam

ine

netw

orks

of

Ada

lines

. In

thi

s se

ctio

n, w

e w

ill

L

2.3

Ap

plic

atio

ns

of

Ad

apti

ve S

igna

l P

roce

ssin

g

look

at

a fe

w e

xam

ples

of

adap

tive

sig

nal

proc

essi

ng u

sing

onl

y th

e A

LC

por

tion

of t

he A

dali

ne.

2.3.

1 E

cho

Can

cella

tion

in

Tel

epho

ne C

ircu

its

You

may

hav

e ex

peri

ence

d th

e ph

enom

enon

of

echo

in

tele

phon

e co

nver

sati

ons:

you

hear

the

wor

ds y

ou s

peak

int

o th

e m

outh

piec

e a

frac

tion

of

a se

cond

lat

erin

the

ear

phon

e of

the

tele

phon

e. T

he e

cho

tend

s to

be

mos

t no

ticea

ble

on l

ong-

dist

ance

cal

ls,

espe

cial

ly t

hose

ove

r sa

tell

ite

link

s w

here

tra

nsm

issi

on d

elay

s ca

nbe

a s

igni

fica

nt f

ract

ion

of a

sec

ond.

Tele

phon

e ci

rcui

ts

cont

ain

devi

ces

calle

d hy

brid

s th

at

are

inte

nded

to

isol

ate

inco

min

g si

gnal

s fr

om o

utgo

ing

sign

als,

thu

s av

oidi

ng t

he e

cho

effe

ct.

Unf

ortu

nate

ly,

thes

e ci

rcui

ts d

o no

t alw

ays

perf

orm

per

fect

ly,

due

to c

ause

s su

chas

im

peda

nce

mis

mat

ches

, re

sult

ing

in s

ome

echo

bac

k to

the

spe

aker

. E

ven

whe

n th

e ec

ho s

igna

l ha

s be

en a

ttenu

ated

by

a su

bsta

ntia

l am

ount

, it

still

may

be a

udib

le,

and

henc

e an

ann

oyan

ce t

o th

e sp

eake

r.C

erta

in e

cho-

supp

ress

ion

devi

ces

rely

on

rela

ys t

hat

open

and

clo

se c

ircu

its

in t

he o

utgo

ing

lines

so

that

inc

omin

g vo

ice

sign

als

are

not

sent

bac

k to

the

spea

ker.

Whe

n tr

ansm

issi

on d

elay

s ar

e lo

ng,

as w

ith

sate

llit

e co

mm

unic

atio

ns,

thes

e ec

ho s

uppr

esso

rs c

an r

esul

t in

a l

oss

of p

arts

of

wor

ds.

Thi

s ch

oppy

-sp

eech

eff

ect

is p

erha

ps m

ore

fam

ilia

r th

an t

he e

cho

effe

ct.

An

adap

tive

fil

ter

can

be u

sed

to r

emov

e th

e ec

ho e

ffec

t w

itho

ut t

he c

hopp

ines

s of

the

rela

ys u

sed

in o

ther

ech

o su

ppre

ssio

n ci

rcui

ts [

9, 7

J.Fi

gure

is

a b

lock

dia

gram

of

a te

leph

one

circ

uit

wit

h an

ad

aptiv

efi

lter

use

d as

an

echo

-sup

pres

sion

dev

ice.

T

he e

cho

is c

ause

d by

a l

eaka

ge o

fth

e in

com

ing

voic

e si

gnal

to

the

outp

ut l

ine

thro

ugh

the

hybr

id c

ircu

it.

Thi

sle

akag

e ad

ds t

o th

e ou

tput

sig

nal

com

ing

from

the

mic

roph

one.

Th

e ou

tput

of

the

adap

tive

fil

ter,

is

subt

ract

ed f

rom

the

out

goin

g si

gnal

, s

+ w

here

s i

sth

e ou

tgoi

ng p

ure

voic

e si

gnal

and

is

the

nois

e, o

r ec

ho c

ause

d by

lea

kage

of

the

inco

min

g vo

ice

sign

al t

hrou

gh t

he h

ybri

d ci

rcui

t. T

he s

ucce

ss o

f th

e ec

hoca

ncel

latio

n de

pend

s on

how

wel

l th

e ad

aptiv

e fi

lter

can

mim

ic t

he l

eaka

geth

roug

h th

e hy

brid

cir

cuit

.N

otic

e th

at t

he i

nput

to

the

filt

er i

s a

copy

of

the

inco

min

g si

gnal

, n,

and

that

the

err

or i

s a

copy

of

the

outg

oing

sig

nal,

E s

+ -

y

(2.1

6)

We

assu

me

that

y i

s co

rrel

ated

wit

h th

e no

ise,

but

not

wit

h th

e pu

re v

oice

sign

al,

s. I

f th

e qu

anti

ty,

y,

is n

onze

ro,

som

e ec

ho s

till

rem

ains

in

the

out-

goin

g si

gnal

. Sq

uari

ng a

nd t

akin

g ex

pect

atio

n va

lues

of

both

sid

es o

f E

q.gi

ves

=

+ +

-

(2.1

7)

= +

-

(2.1

8)

Equ

atio

n (2

.18)

fol

low

s, s

ince

s i

s no

t co

rrel

ated

wit

h ei

ther

y o

r r

esul

ting

in t

he l

ast

term

in

Eq.

bei

ng e

qual

to

zero

.

70

Ad

alin

e an

d

Voi

ce s nois

e,

Hyb

ridci

rcui

tA

dapt

ive

filte

r

To earp

hone

Figu

re 2

.15

1

Ada

ptiv

efil

ter

r

Hyb

ridci

rcui

t

eT

oea

rpho

ne

Voi

cesi

gnal

Th

is f

igu

re

is a

sc

he

ma

tic o

f a

tele

ph

on

e ci

rcu

it us

ing

anad

aptiv

e fil

ter

to c

ance

l ec

ho.

The

ada

ptiv

e fil

ter

is d

epic

ted

as a

box

; th

e sl

ante

d ar

row

rep

rese

nts

the

adju

stab

le w

eigh

ts.

The

sign

al p

ower

, {.

s2 }, i

s de

term

ined

by

the

sour

ce o

f th

e vo

ice

say,

som

e am

plif

ier

at t

he t

elep

hone

sw

itch

ing

stat

ion

loca

l to

the

sen

der.

Thu

s,

is n

ot d

irec

tly a

ffec

ted

by c

hang

es i

n

The

adap

tive

filte

r at

tem

pts

to m

inim

ize

and

, in

doi

ng s

o,

min

imiz

es

((n'

- t

he p

ower

of

the

unca

ncel

ed n

oise

on

the

outg

oing

lin

e.Si

nce

ther

e is

onl

y on

e in

put

to t

he a

dapt

ive

filt

er,

the

devi

ce w

ould

be

conf

igur

ed a

s a

tran

sver

se f

ilte

r. W

idro

w a

nd S

tear

ns [

9] s

ugge

st s

ampl

ing

the

inco

min

g si

gnal

at

a ra

te o

f 8

KH

z an

d us

ing

128

wei

ght

valu

es.

2.3.

2 O

ther

Ap

plic

atio

ns

Rat

her

than

go

into

the

det

ails

of t

he m

any

appl

icat

ions

tha

t can

be

addr

esse

d by

thes

e ad

aptiv

e fi

lter

s, w

e re

fer

you

once

aga

in t

o th

e ex

cell

ent

text

by

Wid

row

and

Stea

rns.

In

thi

s se

ctio

n, w

e sh

all

sim

ply

sugg

est

a fe

w b

road

are

as w

here

adap

tive

filte

rs c

an b

e us

ed i

n ad

ditio

n to

the

ech

o-ca

ncel

latio

n ap

plic

atio

n w

eha

ve d

iscu

ssed

.Fi

gure

sho

ws

an a

dapt

ive

filt

er th

at i

s us

ed p

redi

ct th

e fu

ture

val

ue o

fa

sign

al b

ased

on

its

pres

ent

valu

e. A

sec

ond

exam

ple

is s

how

n in

Fig

ure

In t

his

exam

ple,

the

ada

ptiv

e fi

lter

lea

rns

to r

epro

duce

the

out

put

from

som

epl

ant

base

d on

inp

uts

to t

he s

yste

m.

Thi

s co

nfig

urat

ion

has

man

y us

es a

s an

adap

tive

con

trol

sys

tem

. T

he p

lant

cou

ld r

epre

sent

man

y th

ings

, in

clud

ing

ahu

man

ope

rato

r. I

n th

at c

ase,

the

ada

ptiv

e fi

lter

cou

ld l

earn

how

to

resp

ond

toch

angi

ng c

ondi

tion

s by

wat

chin

g th

e hu

man

ope

rato

r. E

vent

uall

y, s

uch

a de

vice

mig

ht r

esul

t in

an

auto

mat

ed c

ontr

ol s

yste

m,

leav

ing

the

hum

an f

ree

for

mor

eim

port

ant

Ano

ther

use

ful

appl

icat

ion

of t

hese

dev

ices

is

in

adap

tive

bea

m-f

orm

ing

ante

nna

arra

ys.

Alt

houg

h th

e te

rm a

nten

na i

s us

uall

y as

soci

ated

wit

h el

ectr

o-

as

trai

ning

ano

ther

ada

ptiv

e fi

lter

wit

h th

e St

anda

rd &

Poo

rs 5

00.

2.3

Ap

plic

atio

ns o

f A

dap

tive S

igna

l P

roce

ssin

g71

Cur

rent

sig

nal

Pre

dict

ion o

f

curr

ent

sign

al

Pas

t sig

nal

Fig

ure

2.1

6

Th

is s

chem

atic

show

s an

adaptiv

e f

ilter

used

to p

redic

t si

gnal

valu

es.

The

input s

ignal u

sed to

tra

in the n

etw

ork

is a

dela

yed

valu

e of

the

actu

al s

igna

l; th

at i

s, i

t is

the

sig

nal

at s

ome

past

time.

The

exp

ecte

d ou

tput

is

the

curr

ent

valu

e of

the

sig

nal.

The

ada

ptiv

e fil

ter

atte

mpt

s to

min

imiz

e th

e e

rro

r be

twee

n its

outp

ut a

nd t

he c

urre

nt s

igna

l, ba

sed

on a

n in

put

of th

e si

gnal

valu

e fr

om s

ome

time

in t

he p

ast.

Onc

e th

e fil

ter

is c

orr

ect

lypr

edic

ting

the

curr

ent

sign

al

base

d on

the

pa

st s

igna

l, th

ecu

rrent

sign

al c

an b

e u

sed d

ire

ctly

as

an i

nput

with

out

the

dela

y.

The f

ilter

will

then m

ake

a p

redic

tion o

f th

e f

utu

resi

gnal

valu

e.

Inpu

t si

gnal

s

Pre

dict

ion

of

plan

t ou

tput

Figu

reT

his

e

xam

ple

sh

ow

s an

a

da

ptiv

e fil

ter

used

to

m

odel

th

eo

utp

ut

fro

m a

sys

tem

, ca

lled t

he

pla

nt.

Inputs

to t

he f

ilter

are

the

sam

e as

tho

se t

o th

e pl

ant.

The

filt

er

ad

just

s its

wei

ghts

base

d o

n t

he d

iffere

nce

bet

wee

n i

ts o

utp

ut

and t

he

outp

ut

ofth

e p

lant.

72A

dal

ine

and

mag

neti

c ra

diat

ion,

we

broa

den

the

defi

niti

on h

ere

to i

nclu

de a

ny s

pati

al a

rray

of s

enso

rs.

The

bas

ic t

ask

here

is

to l

earn

to

stee

r th

e ar

ray.

At

any

give

n ti

me,

a si

gnal

may

be

arri

ving

fro

m a

ny g

iven

dir

ectio

n, b

ut a

nten

nae

usua

lly

are

dire

ctio

nal

in t

heir

rec

epti

on c

hara

cter

isti

cs:

The

y re

spon

d to

sig

nals

in

som

edi

rect

ions

, bu

t no

t in

oth

ers.

T

he a

nten

na a

rray

wit

h ad

aptiv

e fi

lter

s le

arns

to

adju

st i

ts d

irec

tion

al c

hara

cter

isti

cs i

n or

der

to r

espo

nd t

o th

e in

com

ing

sign

alno

mat

ter

wha

t th

e di

rect

ion

is,

whi

le r

educ

ing

its

resp

onse

to

unw

ante

d no

ise

sign

als

com

ing

in f

rom

oth

er d

irec

tion

s.O

f co

urse

, w

e ha

ve o

nly

touc

hed

on t

he n

umbe

r of

app

licat

ions

for

the

sede

vice

s.

Unl

ike

man

y ot

her

neur

al-n

etw

ork

arch

itec

ture

s, t

his

is a

rel

ativ

ely

mat

ure

devi

ce w

ith

a lo

ng h

isto

ry o

f su

cces

s.

In t

he n

ext

sect

ion,

we

repl

ace

the

bina

ry o

utpu

t co

ndit

ion

on t

he A

LC

cir

cuit

so

that

the

lat

ter

beco

mes

, on

ceag

ain,

the

com

plet

e A

dali

ne.

2.4

TH

E M

AD

AL

INE

As

you

can

see

from

the

dis

cuss

ion

in C

hapt

er 1

, th

e A

dalin

e re

sem

bles

the

perc

eptr

on c

lose

ly;

it a

lso

has

som

e of

the

sam

e li

mit

atio

ns a

s th

e pe

rcep

tron

.Fo

r ex

ampl

e, a

tw

o-in

put

Ada

line

can

not

com

pute

the

XO

R f

unct

ion.

C

om-

bini

ng A

dalin

es i

n a

laye

red

stru

ctur

e ca

n ov

erco

me

this

dif

ficu

lty,

as

we

did

inC

hapt

er 1

wit

h th

e pe

rcep

tron

. Su

ch a

str

uctu

re i

s il

lust

rate

d in

Fig

ure

2.18

.

Exe

rcis

e 2.

5:

Wha

t lo

gic

func

tion

is

bein

g co

mpu

ted

by t

he s

ingl

e A

dali

ne i

nth

e ou

tput

lay

er o

f Fi

gure

Con

stru

ct a

thr

ee-i

nput

Ada

line

that

com

pute

sth

e m

ajor

ity

func

tion

.

2.4.

1 M

adal

ine

Arc

hit

ectu

re

Mad

alin

e is

the

acr

onym

for

Man

y A

dali

nes.

Arr

ange

d in

a m

ulti

laye

red

arch

i-te

ctur

e as

ill

ustr

ated

in

Figu

re 2

.19,

the

Mad

alin

e re

sem

bles

the

gen

eral

neu

ral-

netw

ork

stru

ctur

e sh

own

in C

hapt

er I

n th

is c

onfi

gura

tion

, th

e M

adal

ine

coul

dbe

pre

sent

ed w

ith

a la

rge-

dim

ensi

onal

inp

ut t

he p

ixel

val

ues

from

a ra

ster

sca

n.

Wit

h su

itab

le t

rain

ing,

the

net

wor

k co

uld

be t

augh

t to

res

pond

wit

h a

bina

ry o

n on

e of

sev

eral

out

put

node

s, e

ach

of w

hich

cor

resp

onds

to

a di

ffer

ent

cate

gory

of

inpu

t im

age.

E

xam

ples

of

such

cat

egor

izat

ion

are

dog,

arm

adil

lo,

jave

lina

} an

d {F

logg

er,

Tom

Cat

, E

agle

,

In s

uch

ane

twor

k, e

ach

of f

our

node

s in

the

out

put

laye

r co

rres

pond

s to

a s

ingl

e cl

ass.

For

a gi

ven

inpu

t pa

ttern

, a

node

wou

ld h

ave

a o

utpu

t if

the

inp

ut p

atte

rnco

rres

pond

ed t

o th

e cl

ass

repr

esen

ted

by t

hat

part

icul

ar n

ode.

T

he o

ther

thr

eeno

des

wou

ld h

ave

a o

utpu

t. If

the

inp

ut p

atte

rn w

ere

not

a m

embe

r of

any

know

n cl

ass,

the

res

ults

fro

m t

he n

etw

ork

coul

d be

am

bigu

ous.

To t

rain

su

ch

a ne

twor

k,

we

mig

ht b

e te

mpt

ed t

o be

gin

wit

h th

e L

MS

algo

rith

m a

t th

e ou

tput

lay

er.

Sinc

e th

e ne

twor

k is

pre

sum

ably

tra

ined

wit

hpr

evio

usly

ide

ntif

ied

inpu

t pa

tter

ns,

the

desi

red

outp

ut v

ecto

r is

kno

wn.

W

hat

2.4

T

he M

adal

ine

73

Fig

ure

2.1

8 M

an

y A

da

line

s (t

he

Ma

da

line

) ca

n

com

pu

te

the

X

OR

fun

ctio

n o

f tw

o i

np

uts

. N

ote t

he a

dd

itio

n o

f th

e b

ias

term

s to

ea

ch A

da

line

. A

pos

itive

an

alo

g ou

tput

fro

m a

n A

LC

re

sults

in a

+1 o

utpu

t fro

m t

he a

sso

cia

ted A

da

line

; a n

eg

ativ

e a

na

log

outp

ut r

esu

lts i

n a

Lik

ew

ise

, a

ny

inpu

ts t

o th

e d

evi

ce t

ha

ta

re b

ina

ry i

n na

ture

mus

t us

e ±1

ra

the

r th

an

1 an

d 0.

we

do n

ot k

now

is

the

desi

red

outp

ut f

or a

giv

en n

ode

on o

ne o

f th

e hi

dden

laye

rs.

Furt

herm

ore,

the

LM

S al

gori

thm

wou

ld o

pera

te o

n th

e an

alog

out

puts

of t

he A

LC

, no

t on

the

bip

olar

out

put

valu

es o

f th

e A

dali

ne.

For

thes

e re

ason

s,a

diff

eren

t tr

aini

ng s

trat

egy

has

been

dev

elop

ed f

or t

he M

adal

ine.

2.4

.2

Th

e T

rain

ing

Alg

ori

thm

It i

s po

ssib

le t

o de

vise

a m

etho

d of

tra

inin

g a

str

uctu

re b

ased

on

the

LM

S al

gori

thm

; ho

wev

er,

the

met

hod

reli

es o

n re

plac

ing

the

line

ar t

hres

hold

outp

ut f

unct

ion

wit

h a

cont

inuo

usly

dif

fere

ntia

ble

func

tion

(th

e th

resh

old

func

-tio

n is

dis

cont

inuo

us a

t 0;

hen

ce,

it is

not

dif

fere

ntia

ble

ther

e).

We

tak

e up

the

stud

y of

thi

s m

etho

d in

the

nex

t ch

apte

r.

For

now

, w

e co

nsid

er a

met

hod

know

n as

Mad

alin

e ru

le I

I (M

RII

). T

he o

rigi

nal

Mad

alin

e ru

le w

as a

n ea

rlie

r

74

Adal

ine a

nd

Mad

alin

e

Out

put l

ayer

of

Hid

den la

yer

of M

adal

ines

Figu

re

Man

y A

da

line

s ca

n b

e jo

ine

d in

a l

ayer

ed n

eura

l ne

twor

ksu

ch a

s th

is o

ne.

met

hod

that

we

shal

l no

t di

scus

s he

re.

Det

ails

can

be

foun

d in

ref

eren

ces

give

nat

the

end

of

this

cha

pter

. r

esem

bles

a p

roce

dure

wit

h ad

ded

inte

llige

nce

in t

hefo

rm o

f a

min

imum

dis

turb

ance

pri

ncip

le.

Sinc

e th

e ou

tput

of

the

netw

ork

is a

ser

ies

of b

ipol

ar u

nits

, tr

aini

ng a

mou

nts

to r

educ

ing

the

num

ber

of i

ncor

-re

ct o

utpu

t no

des

for

each

tra

inin

g in

put

patte

rn.

The

min

imum

dis

turb

ance

prin

cipl

e en

forc

es t

he n

otio

n th

at t

hose

nod

es t

hat

can

affe

ct t

he o

utpu

t er

ror

whi

le i

ncur

ring

the

lea

st c

hang

e in

the

ir w

eigh

ts s

houl

d ha

ve p

rece

denc

e in

the

lear

ning

pro

cedu

re.

Thi

s pr

inci

ple

is e

mbo

died

in

the

foll

owin

g al

gori

thm

:

1.

App

ly a

tra

inin

g ve

ctor

to

the

inpu

ts o

f th

e M

adal

ine

and

prop

agat

e it

thro

ugh

to t

he o

utpu

t un

its.

2. C

ount

the

num

ber

of i

ncor

rect

val

ues

in t

he o

utpu

t la

yer;

cal

l th

is n

umbe

rth

e er

ror.

3.

For

all

units

on

the

outp

ut l

ayer

,

a.

Sele

ct t

he f

irst

pre

viou

sly

unse

lect

ed n

ode

who

se a

nalo

g ou

tput

is

clos

-es

t to

zer

o.

(Thi

s no

de i

s th

e no

de t

hat

can

reve

rse

its

bipo

lar

outp

ut

2.4

T

he M

adal

ine

75

wit

h th

e le

ast

chan

ge i

n it

s t

he t

erm

min

imum

dis

tur-

banc

e.)

b. C

hang

e th

e w

eigh

ts o

n th

e se

lect

ed u

nit

such

tha

t th

e bi

pola

r ou

tput

of

the

unit

cha

nges

.c.

Pr

opag

ate

the

inpu

t ve

ctor

for

war

d fr

om t

he i

nput

s to

the

out

puts

onc

eag

ain.

d. I

f th

e w

eigh

t ch

ange

res

ults

in

a re

duct

ion

in t

he n

umbe

r of

err

ors,

acce

pt t

he w

eigh

t ch

ange

; ot

herw

ise,

res

tore

the

ori

gina

l

4. R

epea

t st

ep 3

for

all

laye

rs e

xcep

t th

e in

put

laye

r.

5.

For

all

unit

s on

the

out

put

laye

r,a.

Sel

ect

the

prev

ious

ly u

nsel

ecte

d pa

ir o

f un

its

who

se a

nalo

g ou

tput

s ar

ecl

oses

t to

zer

o.b.

App

ly a

wei

ght

corr

ectio

n to

bot

h un

its,

in

orde

r to

cha

nge

the

bipo

lar

outp

ut o

f ea

ch.

c.

Prop

agat

e th

e in

put

vect

or f

orw

ard

from

the

inp

uts

to t

he o

utpu

ts.

d. I

f th

e w

eigh

t ch

ange

res

ults

in

a re

duct

ion

in t

he n

umbe

r of

err

ors,

acce

pt t

he w

eigh

t ch

ange

; ot

herw

ise,

res

tore

the

ori

gina

l w

eigh

ts.

6. R

epea

t st

ep 5

for

all

laye

rs e

xcep

t th

e in

put

laye

r.

If n

eces

sary

, th

e se

quen

ce i

n st

eps

5 an

d 6

can

be r

epea

ted

wit

h tr

iple

tsof

uni

ts,

or q

uadr

uple

ts o

f un

its,

or

even

lar

ger

com

bina

tions

, un

til

satis

fact

ory

resu

lts a

re o

btai

ned.

Pr

elim

inar

y in

dica

tions

are

tha

t pa

irs

are

adeq

uate

for

mod

est-

size

d ne

twor

ks w

ith

up t

o 25

uni

ts p

er l

ayer

At

the

time

of t

his

wri

ting,

the

was

stil

l un

derg

oing

exp

erim

enta

tion

to d

eter

min

e its

con

verg

ence

cha

ract

eris

tics

and

oth

er p

rope

rtie

s.

Mor

eove

r, a

new

lea

rnin

g al

gori

thm

, h

as b

een

deve

lope

d. i

s si

mil

ar t

o M

RII

,bu

t the

ind

ivid

ual u

nits

hav

e a

cont

inuo

us o

utpu

t fun

ctio

n, r

athe

r th

an th

e bi

pola

rth

resh

old

func

tion

In

the

next

sec

tion,

we

shal

l us

e a

Mad

alin

e ar

chite

ctur

eto

exa

min

e a

spec

ific

pro

blem

in

patte

rn r

ecog

niti

on.

2.4.

3 A

Mad

alin

e fo

r T

ran

slat

ion

-In

vari

ant

Pat

tern

Rec

og

niti

on

Var

ious

Mad

alin

e st

ruct

ures

hav

e be

en u

sed

rece

ntly

to

dem

onst

rate

the

app

li-

cabi

lity

of

this

arc

hite

ctur

e to

ada

ptiv

e pa

ttern

rec

ogni

tion

havi

ng t

he p

rope

rtie

sof

tra

nsla

tion

inv

aria

nce,

rot

atio

n in

vari

ance

, an

d sc

ale

inva

rian

ce.

The

se t

hree

prop

ertie

s ar

e es

sent

ial

to a

ny r

obus

t sy

stem

tha

t w

ould

be

calle

d on

to

rec-

ogni

ze o

bjec

ts i

n th

e fi

eld

of v

iew

of

opti

cal

or i

nfra

red

sens

ors,

for

exa

mpl

e.R

emem

ber,

how

ever

, th

at e

ven

hum

ans

do n

ot a

lway

s in

stan

tly

reco

gniz

e ob

-je

cts

that

hav

e be

en r

otat

ed t

o un

fam

ilia

r or

ient

atio

ns,

or t

hat

have

bee

n sc

aled

sign

ific

antl

y sm

alle

r or

lar

ger

than

the

ir e

very

day

size

. T

he p

oint

is

that

the

rem

ay b

e al

tern

ativ

es t

o tr

aini

ng i

n in

stan

tane

ous

reco

gniti

on a

t al

l an

gles

and

scal

e fa

ctor

s.

Be

that

as

it m

ay,

it i

s po

ssib

le t

o bu

ild

neur

al-n

etw

ork

devi

ces

that

exh

ibit

the

se c

hara

cter

isti

cs t

o so

me

degr

ee.

76

Ad

alin

e a

nd

Figu

re 2

.20

show

s a

port

ion

of a

net

wor

k th

at i

s us

ed t

o im

plem

ent

tran

sla-

tion

-inv

aria

nt r

ecog

niti

on o

f a

patte

rn T

he r

etin

a is

a 5

-by-

5-pi

xel

arra

y on

whi

ch b

it-m

appe

d re

pres

enta

tion

of p

atte

rns,

suc

h as

the

let

ters

of

the

alph

abet

,ca

n be

pla

ced.

T

he p

orti

on o

f th

e ne

twor

k sh

own

is c

alle

d a

slab

. U

nlik

e a

laye

r, a

sla

b do

es n

ot c

omm

unic

ate

wit

h ot

her

slab

s in

the

net

wor

k, a

s w

ill

bese

en s

hort

ly.

Eac

h A

dali

ne i

n th

e sl

ab r

ecei

ves

the

iden

tical

25

inpu

ts f

rom

the

retin

a, a

nd c

ompu

tes

a bi

pola

r ou

tput

in

the

usua

l fa

shio

n; h

owev

er,

the

wei

ghts

on t

he 2

5 A

dali

nes

shar

e a

uniq

ue r

elat

ions

hip.

Con

side

r th

e w

eigh

ts o

n th

e to

p-le

ft A

dalin

e as

bei

ng a

rran

ged

in a

squ

are

mat

rix

dupl

icat

ing

the

pixe

l ar

ray

on t

he r

etin

a.

The

Ada

line

to

the

imm

edia

te

Mad

alin

e sl

ab

Ret

ina

Figu

re 2

.20

T

his

sin

gle

sla

b of

Ad

alin

es

will

giv

e th

e s

ame

outp

ut

(eith

er+

1

or -1

) fo

r a p

art

icula

r patte

rn o

n t

he r

etin

a,

regard

less

of

the

ho

rizo

nta

l or

vert

ica

l a

lign

me

nt

of th

at

pattern

on

the

retin

a.

All

25

ind

ivid

ua

l A

da

line

s a

re c

on

ne

cte

d t

o a

single

Adalin

e t

hat

com

pute

s th

e m

ajo

rity

fu

nct

ion

: If

mos

tof

the i

nputs

are

+1,

the m

ajo

rity

ele

ment

resp

onds

with

a+

1

outp

ut.

The n

etw

ork

de

rive

s its

tra

nsl

atio

n-i

nva

ria

nce

pro

pe

rtie

s fr

om

th

e

pa

rtic

ula

r co

nfig

ura

tion

of

the

weig

hts

.S

ee the t

ext

for

deta

ils.

2.4

T

he M

adal

ine

77

righ

t of

the

top

-lef

t pi

xel

has

the

iden

tica

l se

t of

wei

ght

valu

es,

but

tran

slat

edon

e pi

xel

to t

he r

ight

: T

he r

ight

mos

t co

lum

n of

wei

ghts

on

the

firs

t un

it w

raps

arou

nd t

o th

e le

ft t

o be

com

e th

e le

ftm

ost

colu

mn

on t

he s

econ

d un

it.

Sim

ilar

ly,

the

unit

bel

ow t

he t

op-l

eft

unit

als

o ha

s th

e id

enti

cal

wei

ghts

, bu

t tr

ansl

ated

one

pixe

l do

wn.

T

he b

otto

m r

ow o

f w

eigh

ts o

n th

e fi

rst

unit

bec

omes

the

top

row

of

the

unit

und

er i

t. T

his

tran

slat

ion

cont

inue

s ac

ross

eac

h ro

w a

nd d

own

each

col

umn

in a

sim

ilar

man

ner.

Fig

ure

2.21

ill

ustr

ates

som

e of

thes

e w

eigh

tm

atri

ces.

B

ecau

se o

f th

is r

elat

ions

hip

amon

g th

e w

eigh

t m

atri

ces,

a

sing

lepa

tter

n on

the

ret

ina

wil

l el

icit

iden

tical

res

pons

es f

rom

the

sla

b, i

ndep

ende

nt

Key

wei

ght m

atrix

: top

row

, le

ft co

lum

n W

eigh

t m

atrix

: top

row

, 2n

d co

lum

nw

w

w

w

12

13

14

15

Wei

ght m

atrix

: 2n

d ro

w,

left

colu

mn

W

W

W

W51

52

53

45

W

W

W

W22

23

24

25

w12

W13

35 4544

W32

33

34

35

W

W42

43

44

45

Wei

ght m

atrix

: 5th

row

, 5t

h co

lum

n

Fig

ure

2.2

1

The w

eig

ht

matr

ix i

n t

he u

pper

left i

s th

e k

ey w

eig

ht

matr

ix.

All

othe

r w

eigh

t m

atric

es o

n th

e sl

ab a

re d

eriv

ed f

rom

thi

sm

atr

ix.

The

matr

ix t

o th

e

right

of

the

key

weig

ht

matr

ixre

pre

sents

the m

atr

ix o

n the

direct

ly to

the r

ight o

f the

one w

ith t

he k

ey w

eigh

t m

atr

ix.

Not

ice that

the f

ifth c

olum

nof

the k

ey w

eigh

t m

atrix

has

wra

pped

aro

und t

o b

ecom

e th

efir

st c

olu

mn,

with

the o

ther

colu

mns

shift

ing o

ne s

pace

to

the r

ight.

The m

atr

ix b

elo

w t

he k

ey

weig

ht

matr

ix is

the

one o

n t

he A

da

line d

irect

ly b

elo

w t

he A

da

line w

ith t

he k

ey

we

igh

t m

atr

ix.

The

ma

trix

dia

gonal

to t

he k

ey

weig

ht

matr

ixre

pre

sents

th

e m

atr

ix o

n t

he A

da

line a

t th

e l

ower

rig

ht o

f th

esl

ab.

78

Adal

ine a

nd

of th

e pa

ttern

's t

rans

lati

onal

pos

itio

n on

the

ret

ina.

We

enco

urag

e yo

u to

ref

lect

on t

his

resu

lt fo

r a

mom

ent

(per

haps

sev

eral

mom

ents

), t

o co

nvin

ce y

ours

elf

ofits

val

idit

y.T

he m

ajor

ity

node

is

a si

ngle

Ada

line

tha

t co

mpu

tes

a bi

nary

out

put

base

don

the

out

puts

of

the

maj

orit

y of

the

Ada

lines

con

nect

ing

to i

t. B

ecau

se o

f th

etr

ansl

atio

nal

rela

tion

ship

am

ong

the

wei

ght

vect

ors,

the

pla

cem

ent

of a

par

ticu

lar

patte

rn a

t an

y lo

catio

n on

the

ret

ina

wil

l re

sult

in t

he i

dent

ical

out

put

from

the

maj

ority

ele

men

t (w

e im

pose

the

res

tric

tion

that

pat

tern

s th

at e

xten

d be

yond

the

reti

na b

ound

arie

s w

ill

wra

p ar

ound

to

the

oppo

site

sid

e, j

ust

as t

he v

ario

usw

eigh

t m

atri

ces

are

deri

ved

from

the

key

wei

ght

Of

cour

se,

a pa

ttern

diff

eren

t fr

om t

he f

irst

may

elic

it a

diff

eren

t re

spon

sefr

om t

he m

ajor

ity e

lem

ent.

Bec

ause

onl

y tw

o re

spon

ses

are

poss

ible

, the

sla

b ca

n di

ffer

entia

te tw

o cl

asse

s on

inpu

t pa

tter

ns.

In t

erm

s of

hyp

ersp

ace,

a s

lab

is c

apab

le o

f di

vidi

ngin

to t

wo

regi

ons.

To o

verc

ome

the

limita

tion

of o

nly

two

poss

ible

cla

sses

, th

e re

tina

can

beco

nnec

ted

to m

ultip

le s

labs

, eac

h ha

ving

dif

fere

nt k

ey w

eigh

t mat

rice

s (W

idro

wan

d W

inte

r's t

erm

for

the

wei

ght

mat

rix

on t

he t

op-l

eft

elem

ent

of e

ach

slab

).G

iven

the

bin

ary

natu

re o

f th

e ou

tput

of

each

sla

b, a

sys

tem

of

n sl

abs

coul

ddi

ffer

enti

ate

2"

diff

eren

t pa

ttern

cla

sses

. Fi

gure

2.2

2 sh

ows

four

suc

h sl

abs

prod

ucin

g a

four

-dim

ensi

onal

out

put c

apab

le o

f dis

tingu

ishi

ng d

iffe

rent

inp

ut-

patte

rn c

lass

es w

ith t

rans

latio

nal

inva

rian

ce.

Let

's r

evie

w t

he b

asic

ope

ratio

n of

the

tra

nsla

tion

inv

aria

nce

netw

ork

inte

rms

of a

spe

cifi

c ex

ampl

e. C

onsi

der t

he le

tters

A P

, as

the

inp

ut p

atte

rns

we

wou

ld l

ike

to i

dent

ify

rega

rdle

ss o

f th

eir

or

left

-rig

ht t

rans

lati

onon

the

5-b

y-5-

pixe

l re

tina.

The

se t

rans

late

d re

tina

patte

rns

are

the

inpu

ts t

o th

esl

abs

of t

he n

etw

ork.

E

ach

retin

a pa

ttern

res

ults

in

an o

utpu

t pa

ttern

fro

m t

hein

vari

ance

net

wor

k th

at m

aps

to o

ne o

f th

e 16

inp

ut c

lass

es (

in t

his

case

, ea

chcl

ass

repr

esen

ts a

let

ter)

. B

y us

ing

a lo

okup

tab

le,

or o

ther

met

hod,

we

can

asso

ciat

e th

e 16

pos

sibl

e ou

tput

s fr

om t

he i

nvar

ianc

e ne

twor

k w

ith o

ne o

f th

e16

pos

sibl

e le

tters

tha

t ca

n be

ide

ntif

ied

by t

he n

etw

ork.

So f

ar,

noth

ing

has

been

sai

d co

ncer

ning

the

val

ues

of t

he w

eigh

ts o

n th

eA

dali

nes

of t

he v

ario

us s

labs

in

the

syst

em.

Tha

t is

bec

ause

it

is n

ot a

ctua

lly

nece

ssar

y to

tra

in t

hose

nod

es i

n th

e us

ual

sens

e.

In f

act,

each

key

wei

ght

mat

rix

can

be c

hose

n at

ran

dom

, pro

vide

d th

at e

ach

inpu

t-pa

ttern

clas

s re

sult

ina

uniq

ue o

utpu

t ve

ctor

fro

m t

he i

nvar

ianc

e ne

twor

k.

Usi

ng t

he e

xam

ple

of t

hepr

evio

us p

arag

raph

, an

y tr

ansl

atio

n of

one

of t

he l

ette

rs s

houl

d re

sult

in t

he s

ame

outp

ut f

rom

the

inv

aria

nce

netw

ork.

Fu

rthe

rmor

e, a

ny p

atte

rn f

rom

a d

iffe

rent

clas

s (i

.e.,

a di

ffer

ent

lette

r) m

ust

resu

lt in

a d

iffe

rent

out

put

vect

or f

rom

the

netw

ork.

Thi

s re

quir

emen

t m

eans

tha

t, if

you

pic

k a

rand

om k

ey w

eigh

t m

atri

xfo

r a

part

icul

ar s

lab

and

find

tha

t tw

o le

tters

giv

e th

e sa

me

outp

ut p

atte

rn,

you

can

sim

ply

pick

a d

iffe

rent

wei

ght

mat

rix.

As

an a

lter

nati

ve t

o ra

ndom

sel

ectio

n of

key

wei

ght

mat

rice

s, i

t m

ay b

epo

ssib

le t

o op

tim

ize

sele

ctio

n by

em

ploy

ing

a tr

aini

ng p

roce

dure

bas

ed o

n th

e I

nves

tiga

tion

s in

thi

s ar

ea a

re o

ngoi

ng a

t th

e ti

me

of t

his

wri

ting

2.5

S

imula

ting

th

e A

dal

ine

79

Ret

ina

Figu

re 2

.22

Eac

h of t

he four

slab

s in

the

sys

tem

depic

ted

her

e w

ill p

roduce

a +

1 o

r a —

1 o

utpu

t va

lue f

or

eve

ry p

atte

rn t

hat

appe

ars

onth

e r

etin

a.

The

outp

ut

vect

or

is a

fo

ur-

dig

it b

ina

ry n

umbe

r,so

the

sys

tem

ca

n p

ote

ntia

lly d

iffe

ren

tiate

up

to 1

6 di

ffere

ntcl

ass

es

of i

nput

pat

tern

s.

L

2.5

SIM

UL

AT

ING

TH

E A

DA

LIN

E

As

we

shal

l fo

r th

e im

plem

enta

tion

of a

ll ot

her

netw

ork

sim

ulat

ors

we

wil

lpr

esen

t, w

e sh

all

begi

n th

is s

ectio

n by

des

crib

ing

how

the

gen

eral

dat

a st

ruc-

ture

s ar

e us

ed t

o m

odel

the

Ada

line

uni

t an

d M

adal

ine

netw

ork.

Onc

e th

e ba

sic

arch

itec

ture

has

bee

n pr

esen

ted,

we

wil

l des

crib

e th

e al

gori

thm

ic p

roce

ss n

eede

dto

pro

paga

te s

igna

ls t

hrou

gh t

he A

dali

ne.

The

sec

tion

conc

lude

s w

ith

a di

scus

-si

on o

f th

e al

gori

thm

s ne

eded

to

caus

e th

e A

dalin

e to

sel

f-ad

apt

acco

rdin

g to

the

lear

ning

law

s de

scri

bed

prev

ious

ly.

2.5.

1 A

dal

ine

Dat

a S

tru

ctu

res

It i

s ap

prop

riat

e th

at t

he A

dali

ne i

s th

e fi

rst

test

of

the

sim

ulat

or d

ata

stru

ctur

esw

e pr

esen

ted

in C

hapt

er 1

for

tw

o re

ason

s:

1.

Sinc

e th

e fo

rwar

d pr

opag

atio

n of

sig

nals

thr

ough

the

sin

gle

Ada

line

is

vir-

tual

ly i

dent

ical

to

the

forw

ard

prop

agat

ion

proc

ess

in m

ost

of t

he o

ther

netw

orks

we

wil

l st

udy,

it

is b

enef

icia

l fo

r us

to

obse

rve

the

Ada

line

to

80

Ad

alin

e a

nd

gain

a b

ette

r un

ders

tand

ing

of w

hat

is h

appe

ning

in

each

uni

t of

a l

arge

rne

twor

k.

2. B

ecau

se t

he A

dali

ne i

s no

t a

netw

ork,

its

im

plem

enta

tion

exe

rcis

es t

heve

rsat

ility

of

the

netw

ork

stru

ctur

es w

e ha

ve d

efin

ed.

As

we

have

alr

eady

se

en,

the

Ada

line

is

only

a

sing

le p

roce

ssin

g un

it.

The

refo

re,

som

e of

the

gen

eral

ity w

e bu

ilt

into

our

net

wor

k st

ruct

ures

wil

l no

tbe

req

uire

d. S

peci

fica

lly,

the

re w

ill

be n

o re

al n

eed

to h

andl

e m

ulti

ple

unit

s an

dla

yers

of

unit

s fo

r th

e A

dalin

e.

Nev

erth

eles

s, w

e w

ill

incl

ude

the

use

of t

hose

stru

ctur

es,

beca

use

we

wou

ld l

ike

to b

e ab

le t

o ex

tend

the

Ada

line

easi

ly i

nto

the

Mad

alin

e.W

e be

gin

by d

efin

ing

our

netw

ork

reco

rd a

s a

stru

ctur

e th

at w

ill c

onta

inal

l th

e pa

ram

eter

s th

at w

ill b

e us

ed g

loba

lly,

as w

ell

as p

oint

ers

to l

ocat

e th

edy

nam

ic a

rray

s th

at w

ill

cont

ain

the

netw

ork

data

. In

the

cas

e of

the

Ada

line,

a go

od c

andi

date

str

uctu

re f

or t

his

reco

rd w

ill t

ake

the

form

record Adaline =

: float;

"layer;

output :

end record

for stability

to input

to output

Not

e th

at,

even

tho

ugh

ther

e is

onl

y on

e un

it in

the

Ada

line,

we

wil

l us

etw

o la

yers

to

mod

el t

he n

etw

ork.

Thu

s, t

he i

np

ut

and

ou

tpu

t po

inte

rs w

illpo

int

to d

iffe

rent

lay

er r

ecor

ds.

We

do t

his

beca

use

we

will

use

the

in

pu

tla

yer

as s

tora

ge f

or h

oldi

ng th

e in

put

sign

al v

ecto

r to

the

Ada

line.

The

re w

ill b

eno

con

nect

ions

ass

ocia

ted

wit

h th

is l

ayer

, as

the

inp

ut w

ill

be p

rovi

ded

by s

ome

othe

r pro

cess

in

the

syst

em (

e.g.

, a

time-

mul

tiple

xed

con

vert

er,

or a

n ar

ray

of s

enso

rs).

Con

vers

ely,

the

ou

tpu

t la

yer

will

con

tain

one

wei

ght

arra

y to

mod

el t

heco

nnec

tions

bet

wee

n th

e in

pu

t an

d th

e o

utp

ut

(rec

all t

hat o

ur d

ata

stru

ctur

espr

esum

e th

at P

Es p

roce

ss i

nput

con

nect

ions

pri

mar

ily).

K

eepi

ng i

n m

ind

that

we

wou

ld l

ike

to e

xten

d th

is s

truc

ture

eas

ily t

o ha

ndle

the

Mad

alin

e ne

twor

k,w

e w

ill

reta

in t

he i

ndir

ectio

n to

the

con

nect

ion

wei

ght

arra

y pr

ovid

ed b

y th

e a

rray

des

crib

ed i

n C

hapt

er

Not

ice

that

, in

the

cas

e of

the

Ada

line,

how

ever

, th

e a

rray

will

con

tain

onl

y on

e va

lue,

the

poin

ter

to t

he i

nput

con

nect

ion

arra

y.T

here

is

one

othe

r th

ing

to c

onsi

der

that

may

var

y be

twee

n A

dalin

e un

its.

As

we

have

see

n pr

evio

usly

, th

ere

are

two

part

s to

the

Ada

line

str

uctu

re:

the

linea

r A

LC

and

the

bip

olar

Ada

line

unit

s.

To

dist

ingu

ish

betw

een

them

, w

ede

fine

an

enum

erat

ed t

ype

to c

lass

ify

each

Ada

line

neu

ron:

type NODE_TYPE :

We

now

hav

e ev

eryt

hing

we

need

to

defi

ne t

he l

ay

er

reco

rd s

truc

ture

for

the

Ada

line

. A

pro

toty

pe s

truc

ture

for

thi

s re

cord

is

as f

ollo

ws.

2.5

Sim

ula

ting

th

e

record layer =

activation : NODE_TYPE

of Adaline

to unit output

weights :

access to weight

end record

Fin

ally

, th

ree

dyna

mic

ally

allo

cate

d ar

rays

are

nee

ded

to c

onta

in t

he o

utpu

tof

the

Ada

line

uni

t, th

e a

nd t

he c

onne

ctio

n w

eig

hts

val

ues.

We

wil

l no

t sp

ecif

y th

e st

ruct

ure

of t

hese

arr

ays,

oth

er t

han

to i

ndic

ate

that

the

ou

ts a

nd w

eig

hts

arr

ays

wil

l bo

th c

onta

in f

loat

ing-

poin

t va

lues

, w

here

as t

he a

rray

will

sto

re m

emor

y ad

dres

ses

and

mus

t th

eref

ore

cont

ain

mem

ory

poin

ter

type

s.

The

entir

e da

ta s

truc

ture

for

the

Ada

line

sim

ulat

or i

sde

pict

ed i

n Fi

gure

2.2

3.

2.5.

2 S

igna

l P

ropa

gatio

n T

hrou

gh t

he A

dalin

e

If s

igna

ls a

re t

o be

pro

paga

ted

thro

ugh

the

Ada

line

suc

cess

full

y, t

wo

acti

viti

esm

ust

occu

r:

We

mus

t ob

tain

the

inp

ut s

igna

l ve

ctor

to

stim

ulat

e th

e A

dali

ne,

and

the

Ada

line

mus

t pe

rfor

m

its

inpu

t-su

mm

atio

n an

d ou

tput

-tra

nsfo

rmat

ion

func

tion

s.

Sinc

e th

e or

igin

of

the

inpu

t si

gnal

vec

tor

is s

omew

hat

appl

icat

ion

spec

ific

, w

e w

ill

pres

ume

that

the

use

r w

ill

prov

ide

the

code

nec

essa

ry t

o ke

epth

e da

ta l

ocat

ed i

n th

e a

rray

in

the

in

pu

ts l

ayer

cur

rent

.W

e sh

all

now

con

cent

rate

on

the

mat

ter

of c

ompu

ting

the

inp

ut s

timul

atio

nva

lue

and

tran

sfor

min

g it

to t

he a

ppro

pria

te o

utpu

t. W

e ca

n ac

com

plis

h th

ista

sk t

hrou

gh t

he a

ppli

cati

on o

f tw

o al

gori

thm

ic f

unct

ions

, w

hich

we

wil

l na

me

and

The

alg

orit

hms

for

thes

e fu

ncti

ons

are

as f

ollo

ws:

weig

hts

Fig

ure

2.2

3 T

he A

da

line s

imu

lato

r d

ata

str

uct

ure

is

sho

wn

.

82

Adal

ine a

nd

Madalin

e

function

(INPUTS

WEIGHTS

return float

var sum

float;

temp : float;

ins :

wts :

i integer;

begin

sum = 0;

ins = INPUTS;

wts = WEIGHTS'

input

connection

for i = 1 to

do

all weights in

temp = ins[i] *

sum = sum + temp;

modulated

end do

end function;

the modulated

function compute_output (INPUT : float;

ACT : NODE TYPE) return float

begin

if (ACT = linear)

then return (INPUT)

else if

(INPUT >= 0.0)

then return

else return (-1.0)

end function;

the Adaline is a linear

just return the

the input is

return a binary

return a binary

2.5.

3 A

dapt

ing

the

Ada

line

Now

tha

t our

sim

ulat

or c

an f

orw

ard

prop

agat

e si

gnal

inf

orm

atio

n, w

e tu

rn o

ur a

t-te

ntio

n to

the

impl

emen

tatio

n of

the

lear

ning

alg

orith

ms.

Her

e ag

ain

we

assu

me

that

the

inp

ut s

igna

l pa

ttern

is

plac

ed i

n th

e ap

prop

riat

e ar

ray

by a

n ap

plic

atio

n-sp

ecif

ic p

roce

ss.

Dur

ing

trai

ning

, ho

wev

er,

we

wil

l ne

ed t

o kn

ow w

hat

the

targ

et o

utpu

t i

s fo

r ev

ery

inpu

t ve

ctor

, so

tha

t w

e ca

n co

mpu

te t

he e

rror

term

for

the

Ada

line

.R

ecal

l th

at,

duri

ng t

rain

ing,

the

alg

orit

hm r

equi

res

that

the

Ada

line

upda

te

its

wei

ghts

af

ter

ever

y fo

rwar

d pr

opag

atio

n fo

r a

new

in

put

patte

rn.

We

mus

t al

so c

onsi

der

that

the

Ada

line

appl

icat

ion

may

nee

d to

ada

pt t

he

2.5

Sim

ulat

ing

the

S3

Ada

line

whi

le i

t is

run

ning

. B

ased

on

thes

e ob

serv

atio

ns,

ther

e is

no

need

to s

tore

or

accu

mul

ate

erro

rs a

cros

s al

l pa

tter

ns w

ithi

n th

e tr

aini

ng a

lgor

ithm

.T

hus,

we

can

desi

gn t

he t

rain

ing

algo

rith

m m

erel

y to

ada

pt t

he w

eigh

ts f

or a

sing

le p

atte

rn.

How

ever

, th

is d

esig

n de

cisi

on p

lace

s on

the

app

lica

tion

pro

-gr

am

the

resp

onsi

bilit

y fo

r de

term

inin

g w

hen

the

Ada

line

ha

s tr

aine

d su

ffi-

cien

tly. Thi

s ap

proa

ch i

s us

uall

y ac

cept

able

bec

ause

of

the

adva

ntag

es i

t of

fers

ove

rth

e im

plem

enta

tion

of

a se

lf-c

onta

ined

tra

inin

g lo

op.

Spec

ific

ally

, it

mea

ns t

hat

we

can

use

the

sam

e tr

aini

ng f

unct

ion

to a

dapt

the

Ada

line

init

iall

y or

whi

leit

is o

n-li

ne.

The

gen

eral

ity

of t

he a

lgor

ithm

is

a pa

rtic

ular

ly u

sefu

l fe

atur

e,in

tha

t th

e ap

plic

atio

n pr

ogra

m m

erel

y ne

eds

to d

etec

t a

cond

itio

n re

quir

ing

adap

tatio

n.

It c

an t

hen

sam

ple

the

inpu

t th

at c

ause

d th

e er

ror

and

gene

rate

the

corr

ect

resp

onse

"on

the

fly

," p

rovi

ded

we

have

som

e w

ay o

f kn

owin

g th

atth

e er

ror

is i

ncre

asin

g an

d ca

n ge

nera

te t

he c

orre

ct d

esir

ed v

alue

s to

acc

om-

mod

ate

retr

aini

ng.

Thes

e va

lues

, in

tur

n, c

an t

hen

be i

nput

to

the

Ada

line

trai

ning

alg

orit

hm,

thus

ada

ptat

ion

at r

un t

ime.

F

inal

ly,

it al

so r

e-du

ces

the

hous

ekee

ping

cho

res

that

mus

t be

per

form

ed b

y th

e si

mul

ator

, si

nce

we

wil

l no

t ne

ed t

o m

aint

ain

a lis

t of

exp

ecte

d ou

tput

s fo

r al

l tr

aini

ng p

at-

tern

s. We

mus

t no

w d

efin

e al

gori

thm

s to

com

pute

the

squ

ared

err

or t

erm

the

appr

oxim

atio

n of

the

gra

dien

t of

the

err

or s

urfa

ce,

and

to u

pdat

e th

e co

n-ne

ctio

n w

eigh

ts t

o th

e A

dalin

e.

We

can

agai

n si

mpl

ify

mat

ters

by

com

bin-

ing

the

com

puta

tion

of t

he e

rror

and

the

upd

ate

of t

he c

onne

ctio

n w

eigh

tsin

to

one

func

tion

, as

th

ere

is

no

need

to

co

mpu

te

the

form

er

wit

hout

perf

orm

ing

the

latt

er.

We

now

pr

esen

t th

e al

gori

thm

s to

ac

com

plis

h th

ese

func

tions

:

function

(A : Adaline; TARGET : float)

return float

var tempi : float;

temp2 : float;

err : float;

term for

begin

tempi =

temp2 =

(tempi,

err

absolute (TARGET -

return

end function;

function

(A : Adaline; ERR : float)

return void

var grad : float;

gradient of the

ins :

to inputs

wts :

to weights

i : integer;

84

Ad

alin

e a

nd

begin

ins =

start of input

= A.

start of

for i = 1 to

do

all connections,

grad = -2 * err *

=

- grad *

end

end

2.5.

4 C

om

ple

ting

the

Ad

alin

e S

imul

ator

The

algo

rith

ms

we

have

jus

t de

fine

d ar

e su

ffic

ient

to

impl

emen

t an

Ada

line

sim

ulat

or i

n bo

th l

earn

ing

and

oper

atio

nal

mod

es.

To o

ffer

a c

lean

int

erfa

ceto

any

ext

erna

l pr

ogra

m t

hat

mus

t ca

ll ou

r si

mul

ator

to

perf

orm

an

Ada

line

func

tion

, w

e ca

n co

mbi

ne t

he m

odul

es w

e ha

ve d

escr

ibed

int

o tw

o hi

gher

-lev

elfu

ncti

ons.

The

se f

unct

ions

will

per

form

the

tw

o ty

pes

of a

ctiv

ities

the

Ada

line

mus

t per

form

: a

nd

function

var tempi : float;

(A : Adaline) return void

begin

tempi =

A.

=

end function;

function adapt_Adaline

return float

var err : float;

(A : Adaline; TARGET : float)

until

begin

input

err =

(A,

(A,

end

2.5.

5

Mad

alin

e S

imu

lato

r Im

ple

men

tatio

n

As

we

have

dis

cuss

ed e

arlie

r, t

he M

adal

ine

netw

ork

is s

impl

y a

colle

ctio

n of

bina

ry A

dali

ne u

nits

, co

nnec

ted

toge

ther

in

a la

yere

d st

ruct

ure.

How

ever

, ev

enth

ough

the

y sh

are

the

sam

e ty

pe o

f pr

oces

sing

uni

t, t

he l

earn

ing

stra

tegi

es

2.5

S

imula

ting

th

e A

dalin

e85

men

ted

for

the

Mad

alin

e ar

e si

gnif

ican

tly

diff

eren

t, a

s de

scri

bed

in S

ectio

n 2.

5.2.

Prov

idin

g th

at a

s a

guid

e, a

long

wit

h th

e di

scus

sion

of t

he d

ata

stru

ctur

es n

eede

d,w

e le

ave

the

algo

rith

m d

evel

opm

ent

for

the

Mad

alin

e ne

twor

k to

you

as

an e

x-er

cise

. In t

his

rega

rd,

you

shou

ld n

ote

that

the

lay

ered

str

uctu

re o

f th

e M

adal

ine

lend

s its

elf d

irec

tly

to o

ur s

imul

ator

dat

a st

ruct

ures

. A

s il

lust

rate

d in

Fig

ure

2.24

,w

e ca

n im

plem

ent

a la

yer

of A

dali

ne u

nits

as

easi

ly a

s w

e cr

eate

d a

sing

leA

dali

ne.

The

maj

or d

iffe

renc

es h

ere

wil

l be

the

len

gth

of t

he a

rray

s in

the

lay

er

reco

rds

(sin

ce t

here

will

be

mor

e th

an o

ne A

dalin

e ou

tput

per

laye

r),

and

the

leng

th a

nd n

umbe

r of

con

nect

ion

arra

ys (

ther

e w

ill b

e on

e w

eig

hts

arra

y fo

r ea

ch A

dalin

e in

the

lay

er,

and

the

arr

ay w

ill b

eex

tend

ed b

y on

e sl

ot f

or e

ach

new

weig

hts

arr

ay).

Sim

ilarl

y, t

here

will

be

mor

e la

yer

reco

rds

as t

he d

epth

of

the

Mad

alin

ein

crea

ses,

an

d, f

or e

ach

laye

r, th

ere

wil

l be

a c

orre

spon

ding

inc

reas

e in

the

num

ber

of w

eig

hts

, an

d a

rray

s.

Bas

ed o

n th

ese

ob-

serv

atio

ns,

one

fact

tha

t be

com

es i

mm

edia

tely

per

cept

ible

is

the

com

bina

tori

algr

owth

of

both

mem

ory

cons

umed

and

com

pute

r ti

me

requ

ired

to

supp

ort

a li

n-ea

r gr

owth

in

netw

ork

size

. T

his

rela

tion

ship

bet

wee

n co

mpu

ter

reso

urce

s an

dm

odel

siz

ing

is t

rue

not

only

for

the

Mad

alin

e, b

ut f

or a

ll A

NS

mod

els

we

wil

lst

udy.

It i

s fo

r the

se r

easo

ns th

at w

e ha

ve s

tress

ed o

ptim

izat

ion

in d

ata

stru

ctur

es.

ou

tpu

ts

Ma

da

lin

e

activation

outs

weights

We

°3

ight p

ou

tpu

ts

we

igh

ts

Fig

ure

2.2

4

Ma

da

line

data

str

uct

ure

s a

re s

ho

wn

.

86

Ada

line

and

Pro

gram

min

g E

xerc

ises

2.1.

Ext

end

the

Ada

line

sim

ulat

or to

inc

lude

the

bia

s un

it,

0, a

s de

scri

bed

in t

hete

xt.

2.2.

Ext

end

the

sim

ulat

or t

o im

plem

ent

a th

ree-

laye

r M

adal

ine

usin

g th

e al

go-

rith

ms

disc

usse

d in

Sec

tion

2.3

.2.

Be

sure

to

use

the

bina

ry A

dali

ne t

ype.

Test

the

oper

atio

n of

you

r si

mul

ator

by

trai

ning

it to

sol

ve th

e X

OR

pro

blem

desc

ribe

d in

the

tex

t.

2.3.

We

have

ind

icat

ed t

hat

the

netw

ork

stab

ility

ter

m,

can

gre

atly

aff

ect

the

abili

ty o

f the

Ada

line

to c

onve

rge

on a

sol

utio

n. U

sing

fou

r di

ffer

ent

valu

esfo

r o

f yo

ur o

wn

choo

sing

, tr

ain

an A

dalin

e to

elim

inat

e no

ise

from

an

inpu

t si

nuso

id r

angi

ng f

rom

0 t

o (

one

way

to

do t

his

is t

o us

e a

scal

edra

ndom

-num

ber

gene

rato

r to

prov

ide

the

nois

e).

Gra

ph th

e cu

rve

of tr

aini

ngite

ratio

ns v

ersu

s

Sugg

este

d R

eadi

ngs

The

auth

orita

tive

text

by

Wid

row

and

Ste

arns

is

the

stan

dard

ref

eren

ce t

o th

em

ater

ial

cont

aine

d in

thi

s ch

apte

r

The

ori

gina

l de

lta-

rule

der

ivat

ion

isco

ntai

ned

in a

196

0 pa

per

by W

idro

w a

nd H

off

[6],

whi

ch i

s al

so r

epri

nted

in

the

colle

ctio

n ed

ited

by A

nder

son

and

Ros

enfe

ld

Bib

liogr

aphy

Jam

es A

. And

erso

n an

d E

dwar

d R

osen

feld

, edi

tors

. F

oun-

datio

ns o

f Res

earc

h. M

IT P

ress

, C

ambr

idge

, M

A,

1988

.

[2]

Dav

id A

ndes

, B

erna

rd W

idro

w,

Mic

hael

and

Eri

c W

an.

Aro

bust

alg

orith

m f

or t

rain

ing

anal

og n

eura

l ne

twor

ks.

In P

roce

edin

gs o

fth

e In

tern

atio

nal

Join

t C

onfe

renc

e on

Neu

ral

Net

wor

ks,

page

s I-

533-

I-53

6, J

anua

ry

1990

.

[3]

Ric

hard

W.

Ham

min

g.

Dig

ital

Filt

ers.

Pr

entic

e-H

all,

Engl

ewoo

d C

liffs

,N

J, 1

983.

[4]

Wilf

red

Kap

lan.

Adv

ance

d C

alcu

lus,

3rd

edi

tion.

Add

ison

-Wes

ley,

Rea

ding

,M

A,

1984

.

[5]

Ala

n V

. O

ppen

heim

ari

d R

onal

d W

. Sc

hafe

r. S

igna

l P

roce

ssin

g.Pr

entic

e-H

all,

Eng

lew

ood

Cli

ffs,

NJ,

19

75.

[6]

Ber

nard

Wid

row

and

Mar

cian

E.

Hof

f. A

dapt

ive

swit

chin

g ci

rcui

ts.

In 7

960

WE

SCO

N C

onve

ntio

n R

ecor

d, N

ew Y

ork,

pag

es 1

960.

IR

E.

[7]

Ber

nard

Wid

row

and

Rod

ney

Win

ter.

Neu

ral

nets

for

ada

ptiv

e fi

lter

ing

and

adap

tive

patte

rn r

ecog

nitio

n. C

ompu

ter,

Mar

ch 1

988.

Bib

liog

rap

hy

[8]

Win

ter

and

Ber

nard

Wid

row

. M

AD

AL

INE

RU

LE

II:

A t

rain

ing

algo

rith

m f

or n

eura

l ne

twor

ks.

In P

roce

edin

gs o

f th

e IE

EE

Sec

ond

In-

tern

atio

nal

Con

fere

nce

on N

etw

orks

, S

an D

iego

, C

A,

July

19

88.

[9]

Ber

nard

Wid

row

and

Sam

uel

D.

Stea

rns.

Ada

ptiv

e Si

gnal

Pro

cess

ing.

Sig

nal

Proc

essi

ng S

erie

s. P

rent

ice-

Hal

l, E

ngle

woo

d C

liff

s, N

J, 1

985.