R

1

UN

IVE

RS

ITY

OF

OS

LO

DE

PA

RT

ME

NT

OF

INFO

RM

AT

ICS

D. G

esbe

rt: I

N35

7 S

tati

stic

al S

ign

al P

roce

ssin

g1 o

f 21

IN35

7:A

DA

PT

IVE

FIL

TE

RS

Co

urs

eb

oo

k:C

hap

.9St

atis

tica

lDig

ital

Sign

alP

roce

ssin

gan

dm

od

elin

g,M

.Hay

es19

96(a

lso

bu

ilds

on

Ch

ap7.

2).

Dav

idG

esb

ert

Sign

alan

dIm

age

Pro

cess

ing

Gro

up

(DSB

)h

ttp

://w

ww

.ifi.u

io.n

o/~

gesb

ert

Mar

ch20

03

UN

IVE

RS

ITY

OF

OS

LO

DE

PA

RT

ME

NT

OF

INFO

RM

AT

ICS

D. G

esbe

rt: I

N35

7 S

tati

stic

al S

ign

al P

roce

ssin

g2 o

f 21

Ou

tlin

e•

Mo

tiva

tio

ns

for

adap

tive

filt

erin

g

•T

he

adap

tive

FIR

filt

er

•St

eep

est

des

cen

tan

do

pti

miz

atio

nth

eory

•St

eep

est

des

cen

tin

adap

tive

filt

erin

g

•T

he

LMS

algo

rith

m

•Pe

rfo

rman

ceo

fLM

S

•T

he

RLS

algo

rith

m

•Pe

rfo

rman

ceo

fRLS

•E

xam

ple

:Ad

apti

veb

eam

form

ing

inm

ob

ilen

etw

ork

s

UN

IVE

RS

ITY

OF

OS

LO

DE

PA

RT

ME

NT

OF

INFO

RM

AT

ICS

D. G

esbe

rt: I

N35

7 S

tati

stic

al S

ign

al P

roce

ssin

g3 o

f 21

Mo

tiva

tio

ns

for

adap

tive

filt

erin

gG

oal

:“E

xten

din

go

pti

mu

m(e

x:W

ien

er)

filt

ers

toth

eca

sew

her

eth

ed

ata

isn

ots

tati

on

ary

or

the

un

der

lyin

gsy

stem

isti

me

vary

ing”

{d(n

)}d

esir

edra

nd

om

pro

cess

(un

ob

serv

ed)

may

be

no

nst

atio

nar

y{x

0(n

)}o

bse

rved

ran

do

mp

roce

ss,m

ayb

en

on

stat

ion

ary

{x2(n

)}o

bse

rved

ran

do

mp

roce

ss,m

ayb

en

on

stat

ion

ary

. . .{x

p−1(n

)}o

bse

rved

ran

do

mp

roce

ss,m

ayb

en

on

stat

ion

ary

d(n)

d(n)

^

e(n)

x (

n)x

(n)

x (

n)

2 p−10

filte

r

erro

r si

gnal

desi

red

sign

al

estim

ated

sig

na l

p ob

serv

atio

ns

Wn

filte

r W

mus

t be

adju

sted

ove

r tim

e n

UN

IVE

RS

ITY

OF

OS

LO

DE

PA

RT

ME

NT

OF

INFO

RM

AT

ICS

D. G

esbe

rt: I

N35

7 S

tati

stic

al S

ign

al P

roce

ssin

g4 o

f 21

Cas

eso

fno

nst

atio

nar

ity

Th

efi

lterW

mu

stb

ead

just

edov

erti

me

and

isd

eno

tedW

(n)

ino

rder

totr

ack

no

nst

atio

nar

ity:

•E

xam

ple

1:To

fin

dth

ew

ien

erso

luti

on

toth

elin

ear

pre

dic

tio

no

fsp

eech

sign

al.T

he

spee

chsi

gnal

isn

on

stat

ion

ary

bey

on

dap

pro

x20

ms

ofo

bse

rvat

ion

s.d

(n),{x

i(n

)}ar

en

on

stat

ion

ary.

•E

xam

ple

2:To

fin

dth

ead

apti

veb

eam

form

erth

attr

acks

the

loca

tio

no

fam

ob

ileu

ser,

ina

wir

eles

sn

etw

ork

.d(n

)is

stat

ion

ary

(seq

uen

ceo

fm

od

ula

tio

nsy

mb

ols

),b

ut{x

i(n

)}ar

en

ot

bec

ause

the

chan

nel

isch

angi

ng.

UN

IVE

RS

ITY

OF

OS

LO

DE

PA

RT

ME

NT

OF

INFO

RM

AT

ICS

D. G

esbe

rt: I

N35

7 S

tati

stic

al S

ign

al P

roce

ssin

g5 o

f 21

Ap

roac

hes

toth

ep

rob

lem

Two

solu

tio

ns

totr

ack

filt

erW

(n):

•(A

dap

tive

filt

erin

g)O

ne

has

alo

ng

trai

nin

gsi

gnal

ford

(n)

and

on

ead

just

sW

(n)

tom

inim

ize

the

pow

ero

fe(n

)co

nti

nu

ou

sly.

•(B

lock

filt

erin

g)O

ne

split

sti

me

into

sho

rtti

me

inte

rval

sw

her

eth

ed

ata

isap

pro

xim

atel

yst

atio

nar

y,an

dre

-co

mp

ute

the

Wie

ner

solu

tio

nfo

rev

ery

blo

ck.

UN

IVE

RS

ITY

OF

OS

LO

DE

PA

RT

ME

NT

OF

INFO

RM

AT

ICS

D. G

esbe

rt: I

N35

7 S

tati

stic

al S

ign

al P

roce

ssin

g6 o

f 21

Vect

or

Form

ula

tio

n(t

ime-

vary

ing

filt

er)

W(n

)=

[w0(n

),w

1(n

),..,w

p−1(n

)]T

X(n

)=

[x0(n

),x

1(n

),..,x

p−1(n

)]T

d̂(n

)=W

(n)TX

(n)

wh

ereT

isth

etr

ansp

ose

op

erat

or.

UN

IVE

RS

ITY

OF

OS

LO

DE

PA

RT

ME

NT

OF

INFO

RM

AT

ICS

D. G

esbe

rt: I

N35

7 S

tati

stic

al S

ign

al P

roce

ssin

g7 o

f 21

Tim

eva

ryin

go

pti

mu

mli

nea

rfi

lter

ing

e(n

)=d

(n)−d̂

(n)

J(n

)=E|e

(n)|2

vari

esw

ith

nd

ue

ton

on

-sta

tio

nar

ity

wh

ereE

()is

the

exp

ecta

tio

n.

Fin

dW

(n)

such

thatJ

(n)

ism

inim

um

atti

men

.W(n

)is

the

op

tim

um

lin

ear

filt

erin

the

Wie

ner

sen

seat

tim

en

.

UN

IVE

RS

ITY

OF

OS

LO

DE

PA

RT

ME

NT

OF

INFO

RM

AT

ICS

D. G

esbe

rt: I

N35

7 S

tati

stic

al S

ign

al P

roce

ssin

g8 o

f 21

Fin

din

gth

eso

luti

on

Th

eso

luti

onW

(n)

isgi

ven

by

the

tim

eva

ryin

gW

ien

er-H

op

feq

uat

ion

s.

Rx(n

)W(n

)=r dx(n

)w

her

e(1

)

Rx(n

)=E

(X(n

)∗X

(n)T

)(2

)r dx(n

)=E

(d(n

)X(n

)∗)

(3)

UN

IVE

RS

ITY

OF

OS

LO

DE

PA

RT

ME

NT

OF

INFO

RM

AT

ICS

D. G

esbe

rt: I

N35

7 S

tati

stic

al S

ign

al P

roce

ssin

g9 o

f 21

Ad

apti

veA

lgo

rith

ms

Th

eti

me-

vary

ing

stat

isti

csu

sed

in(1

)ar

eu

nkn

own

bu

tcan

be

esti

mat

ed.A

dap

tive

algo

rith

ms

aim

ates

tim

atin

gan

dtr

acki

ng

the

solu

tio

nW

(n)

give

nth

eo

bse

rvat

ion

s{x

i(n

)},i

=0..p−

1an

da

trai

nin

gse

qu

ence

ford

(n).

Two

key

app

roac

hes

:

•St

eep

est

des

cen

t(a

lso

calle

dgr

adie

nt

sear

ch)

algo

rith

ms.

•R

ecu

rsiv

eLe

astS

qu

ares

(RLS

)alg

ori

thm

.

Trac

kin

gis

form

ula

ted

by: (n

+1)

=W

(n)

+∆W

(n)

(4)

wh

ere

∆W

(n)

isth

eco

rrec

tio

nap

plie

dto

the

filt

erat

tim

en

.

UN

IVE

RS

ITY

OF

OS

LO

DE

PA

RT

ME

NT

OF

INFO

RM

AT

ICS

D. G

esbe

rt: I

N35

7 S

tati

stic

al S

ign

al P

roce

ssin

g10

of 2

1

Stee

pes

td

esce

nti

no

pti

miz

atio

nth

eory

Ass

um

pti

on

s:St

atio

nar

yca

se.

Idea

:“Lo

cale

xtre

ma

ofc

ost

fun

ctio

nJ(

W)c

anb

efo

un

db

yfo

llow

ing

the

pat

hw

ith

the

larg

estg

rad

ien

t(d

eriv

ativ

e)o

nth

esu

rfac

eo

fJ(W

).”

•W

(0)

isan

arb

itra

ryin

itia

lpo

int

•W

(n+

1)=W

(n)−µ

δJ δW∗| W

=W

(n)

wh

ereµ

isa

smal

lste

p-s

ize

(µ<<

1).

Bec

ause

J()

isq

uad

rati

ch

ere,

ther

eis

on

lyo

ne

loca

lmin

imu

mto

war

dw

hic

hW

(n)

will

con

verg

e.

UN

IVE

RS

ITY

OF

OS

LO

DE

PA

RT

ME

NT

OF

INFO

RM

AT

ICS

D. G

esbe

rt: I

N35

7 S

tati

stic

al S

ign

al P

roce

ssin

g11

of 2

1

Th

est

eep

est

des

cen

tWie

ner

algo

rith

mD

eriv

atio

no

fth

egr

adie

nt

exp

ress

ion

:

J(W

)=E

(e(n

)e(n

)∗)

wh

ere

e(n

)=d

(n)−d̂

(n)

=d

(n)−W

TX

(n)

δJ δW∗

=E

(δe(n

)

δW∗e(n

)∗+e(n

)δe(n

)∗

δW∗

)

δJ δW∗

=E

(0+e(n

)δe(n

)∗

δW∗

)

δJ δW∗

=−E

(e(n

)X(n

)∗)

UN

IVE

RS

ITY

OF

OS

LO

DE

PA

RT

ME

NT

OF

INFO

RM

AT

ICS

D. G

esbe

rt: I

N35

7 S

tati

stic

al S

ign

al P

roce

ssin

g12

of 2

1

Th

est

eep

est

des

cen

tWie

ner

algo

rith

mA

lgo

rith

m:

•W

(0)

isan

arb

itra

ryin

itia

lpo

int

•W

(n+

1)=W

(n)

+µE

(e(n

)X(n

)∗)

W(n

)w

illco

nve

rge

toW

o=

R−

1xr dx

(wie

ner

solu

tio

n)

if0<µ<

2/λmax

(max

eige

nva

lue

ofR

x.(

see

p.50

1fo

rp

roo

f).

Pro

ble

m:E

(e(n

)X(n

)∗)

isu

nkn

own

!

UN

IVE

RS

ITY

OF

OS

LO

DE

PA

RT

ME

NT

OF

INFO

RM

AT

ICS

D. G

esbe

rt: I

N35

7 S

tati

stic

al S

ign

al P

roce

ssin

g13

of 2

1

Th

eL

east

Mea

nSq

uar

e(L

MS)

Alg

ori

thm

Idea

:E(e

(n)X

(n)∗

)is

rep

lace

db

yit

sin

stan

tan

eou

sva

lue.

•W

(0)

isan

arb

itra

ryin

itia

lpo

int

•W

(n+

1)=W

(n)

+µe(n

)X(n

)∗

•R

epea

twit

hn

+2.

.

UN

IVE

RS

ITY

OF

OS

LO

DE

PA

RT

ME

NT

OF

INFO

RM

AT

ICS

D. G

esbe

rt: I

N35

7 S

tati

stic

al S

ign

al P

roce

ssin

g14

of 2

1

Th

eL

east

Mea

nSq

uar

e(L

MS)

Alg

ori

thm

Lem

ma:W

(n)

will

con

verg

ein

the

mea

nto

war

dW

o=

R−

1xr dx,i

f0<µ<

2/λmax,(

see

p.50

7)ie

.:

(W(n

))→

R−

1xr dx

wh

enn→∞

(5)

Imp

ort

ant

Rem

arks

:

•T

he

vari

ance

ofW

(n)

aro

un

dit

sm

ean

isfu

nct

ion

ofµ

.

•µ

allo

ws

atr

ade-

off

bet

wee

nsp

eed

ofco

nve

rgen

cean

dac

cura

cyo

fth

ees

tim

ate.

•A

smal

lµre

sult

sin

larg

erac

cura

cyb

ut

slow

erco

nve

rgen

ce.

•T

he

algo

rith

mis

der

ived

un

der

the

assu

mp

tio

no

fsta

tio

nar

ity,

bu

tca

nb

eu

sed

inn

on

-sta

tio

nar

yen

viro

nm

ent

asa

trac

kin

gm

eth

od

.

UN

IVE

RS

ITY

OF

OS

LO

DE

PA

RT

ME

NT

OF

INFO

RM

AT

ICS

D. G

esbe

rt: I

N35

7 S

tati

stic

al S

ign

al P

roce

ssin

g15

of 2

1

Afa

ster

-co

nve

rgin

gal

gori

thm

Idea

:bu

ilda

run

nin

ges

tim

ate

oft

he

stat

isti

csRx(n

),r dx(n

),an

dso

lve

the

Wie

ner

Ho

pfe

qu

atio

nat

each

tim

e:

x(n

)W(n

)=r dx(n

)(6

)

Wh

ere

Rx(n

)=

k=n ∑ k

=0

λn−kX

(k)∗X

(k)T

(7)

r dx(n

)=

k=n ∑ k

=0

λn−kd

(k)X

(k)∗

(8)

wh

ereλ

isth

efo

rget

tin

gfa

ctor

(λ<

1cl

ose

to1)

UN

IVE

RS

ITY

OF

OS

LO

DE

PA

RT

ME

NT

OF

INFO

RM

AT

ICS

D. G

esbe

rt: I

N35

7 S

tati

stic

al S

ign

al P

roce

ssin

g16

of 2

1

Rec

urs

ive

leas

t-sq

uar

es(R

LS)

Toav

oid

inve

rtin

ga

mat

rix

aea

chst

ep,o

nes

fin

ds

are

curs

ive

solu

tio

nfo

rW

(n).

Rx(n

)=λRx(n−

1)+X

(n)∗X

(n)T

(9)

r dx(n

)=λr dx(n−

1)+d

(n)X

(n)∗

(10)

W(n

)=W

(n−

1)+

∆W

(n−

1)(1

1)

Qu

esti

on

:How

tod

eter

min

eth

eri

ght

corr

ecti

on

∆W

(n−

1)??

An

swer

:Usi

ng

the

mat

rix

inve

rsio

nle

mm

a(W

oo

db

ury

’sid

enti

ty)

UN

IVE

RS

ITY

OF

OS

LO

DE

PA

RT

ME

NT

OF

INFO

RM

AT

ICS

D. G

esbe

rt: I

N35

7 S

tati

stic

al S

ign

al P

roce

ssin

g17

of 2

1

Mat

rix

inve

rsio

nle

mm

aW

ed

efin

eP

(n)

=Rx(n

)−1.T

he

M.I

.L.i

su

sed

tou

pd

ate

P(n−

1)to

P(n

)d

irec

tly:

A+uvH

)−1

=A−

1−

A−

1uvH

A−

1

1+vH

A−

1u

(12)

We

app

lyto

Rx(n

)−1

=(λ

Rx(n−

1)+X

(n)∗X

(n)T

)−1

UN

IVE

RS

ITY

OF

OS

LO

DE

PA

RT

ME

NT

OF

INFO

RM

AT

ICS

D. G

esbe

rt: I

N35

7 S

tati

stic

al S

ign

al P

roce

ssin

g18

of 2

1

Mat

rix

inve

rsio

nle

mm

a

Rx(n

)−1

=λ−

1Rx(n−

1)−

1−λ−

1Rx(n−

1)−

1X

(n)∗X

(n)T

Rx(n−

1)−

1

1+λ−

1X

(n)T

Rx(n−

1)−

1X

(n)∗

(13)

(14)

UN

IVE

RS

ITY

OF

OS

LO

DE

PA

RT

ME

NT

OF

INFO

RM

AT

ICS

D. G

esbe

rt: I

N35

7 S

tati

stic

al S

ign

al P

roce

ssin

g19

of 2

1

Th

eR

LS

algo

rith

m

W(0

)=

0(1

5)P

(0)

=δ−

1I

(16)

Z(n

)=

P(n−

1)X

(n)∗

(17)

G(n

)=

Z(n

)

λ+X

(n)TZ

(n)

(18)

α(n

)=d

(n)−W

(n−

1)TX

(n)

(19)

W(n

)=W

(n−

1)+α

(n)G

(n)

(20)

P(n

)=

1 λ(P

(n−

1)−G

(n)Z

(n)H

)(2

1)

wh

ereδ<<

1is

asm

alla

rbit

rary

init

ializ

atio

np

aram

eter

UN

IVE

RS

ITY

OF

OS

LO

DE

PA

RT

ME

NT

OF

INFO

RM

AT

ICS

D. G

esbe

rt: I

N35

7 S

tati

stic

al S

ign

al P

roce

ssin

g20

of 2

1

RL

Svs

.LM

SC

om

ple

xity

:RLS

mo

reco

mp

lex

bec

ause

ofm

atri

xm

ult

ipli

cati

on

s.L

MS

sim

ple

rto

imp

lem

ent.

Co

nve

rgen

cesp

eed

:LM

Ssl

ower

bec

ause

dep

end

so

nam

plit

ud

eo

fgr

adie

nt

and

eige

nva

lue

spre

ado

fco

rrel

atio

nm

atri

x.R

LSis

fast

erb

ecau

seit

po

ints

alw

ays

atth

eri

ght

solu

tio

n(i

tso

lves

the

pro

ble

mex

actl

yat

each

step

).

Acc

ura

cy:I

nLM

Sth

eac

cura

cyis

con

tro

lled

via

the

step

sizeµ

.In

RL

Svi

ath

efo

rget

tin

gfa

cto

rλ

.In

bo

thca

ses

very

hig

hac

cura

cyin

the

stat

ion

ary

regi

me

can

be

ob

tain

edat

the

loss

ofc

on

verg

ence

spee

d.

UN

IVE

RS

ITY

OF

OS

LO

DE

PA

RT

ME

NT

OF

INFO

RM

AT

ICS

D. G

esbe

rt: I

N35

7 S

tati

stic

al S

ign

al P

roce

ssin

g21

of 2

1

Ap

pli

cati

on

Th

eLM

Sap

plie

dto

the

pro

ble

mo

fad

apti

veb

eam

form

ing.

..

Tob

ed

evel

op

ped

incl

ass.