lecture 06 - oprimum filters

7/20/2008

1

1

Lecture 06: Optimum filters

Instructor:Instructor:Dr. Gleb V. Tcheslavski

Contact:[email protected]

Office Hours:

Room 2030

Class web site:

Summer II 2008ELEN 5301 Adv. DSP and Modeling

http://www.ee.lamar.edu/gleb/adsp/Index.htm

Dear, it seems like I solved the problem of water purification!

2

Problem statement: Wiener filter

The problem of estimation (or extraction) of one signal from another arises quite often. In many applications, the desired signal (speech, radar signal, EEG, image, etc ) is not available or observed directly Instead the desired signal may be noisyetc.) is not available or observed directly. Instead, the desired signal may be noisy or destroyed. In some simple situations, it may be possible to design a classical filter (LPF, HPF, BPF) to resolve the desired signal from the data. However, these filters are rarely optimum in the sense of producing the best estimate of the signal. Therefore, optimum digital filters – including Wiener and Kalman filters –are of interest.

The discrete Wiener filter is designed to recover the desired signal dn from noisy observations


noisy observations

n n nx d v= + (6.2.1)

Assuming that both dn and xn are wss random processes, Wiener considered the problem of designing the filter W(z) that produces the minimum mean-square (MMS) error estimate of dn.

7/20/2008

2

3

Problem statement: Wiener filter

Therefore: { }2nE eξ = (6.3.1)

ˆwhere ˆn n ne d d= − (6.3.2)

Thus, the problem is to find a filter that minimizes ξ. We begin by considering the general problem of Wiener filtering, where an LTI filter W(z) minimizing (6.3.2) needs to be designed. Depending upon the relationship between xn and dn, a number of different problems may be solved with Wiener filters. Some of them are:

1. Filtering: given xn = dn + vn, need to estimate dn with a causal filter; i.e., from the current and past values of x ;


the current and past values of xn;2. Smoothing: the same as filtering except that the filter can be non-causal;3. Prediction: if dn = xn+1 and W(z) is a causal filter, the Wiener filter becomes a

linear predictor. Thus, the filter produces a prediction (estimate) of the future value of the signal in terms of a linear combination of its previous values;

4. Deconvolution: when xn = dn∗gn + vn with gn being the unit-pulse response of an LTI filter, the Wiener filter becomes a deconvolution filter.

4

FIR Wiener filter

We need to design an FIR Wiener filter producing the MMS error estimate of a desired process dn by filtering a set of observations of a statistically related process xprocess xn.

Assuming that xn and dn are jointly wss with known autocorrelations rx,k and rd,kand known cross-correlation rdx,k; denoting the unit-pulse response of the Wiener filter by wn; and assuming a p-1 order filter, the filter transfer function will be

1

0( )

pn

nn

W z w z−

−

=

= ∑Therefore, for the input xn, the filter output will be

(6.4.1)


p n p1

0

ˆp

n l n ll

d w x−

−=

= ∑ (6.4.2)

To minimize the mean-square error (that does not depend on n)

{ } { }22 ˆn n nE e E d dξ = = − (6.4.3)

7/20/2008

3

5

FIR Wiener filter

its derivative must be zero according to the optimization theory:

{ }*

* 0 0 1 1neE E kξ ⎧ ⎫∂∂ ∂ ⎨ ⎬ (6.5.1){ }* * * 0, 0,1,..., 1n

n n nk k k

E e e E e k pw w wξ

= = = = −⎨ ⎬∂ ∂ ∂⎩ ⎭complex conjugate

With 1

0

p

n n l n ll

e d w x−

−=

= − ∑it follows that *

**n

n ke xw −

∂= −

∂

(6.5.2)

(6.5.3)


kw∂and, therefore:

{ }* 0, 0,1,..., 1n n kE e x k p− = = −

Which is known as the orthogonality principle or the projection theorem.

(6.5.4)

6

FIR Wiener filter

Therefore, combining (6.5.2) and (6.5.4):

{ } { }1

* * 0 0 1 1p

E d E k−

∑ (6 6 1){ } { }0

0, 0,1,..., 1n n k l n l n kl

E d x w E x x k p− − −=

− = = −∑ (6.6.1)

Since xn and dn are jointly wss:

{ }{ }

*

*

( )

( )

n l n k x

n n k dx

E x x r k l

E d x r k

− −

−

= −

=(6 6 1) becomes

(6.6.2)

(6.6.3)


(6.6.1) becomes

1

0( ) ( ), 0,1,..., 1

p

l x dxl

w r k l r k k p−

=

− = = −∑Which is a set of p equations in the p unknowns wk, k = 0,1,…p-1 called the Wiener-Hopf equations.

(6.6.4)

7/20/2008

4

7

FIR Wiener filter

Since the autocorrelation sequence is conjugate symmetric:*( ) ( )x xr k r k= − (6.7.1)( ) ( )x x

the Wiener-Hopf equations in matrix form are* *

0*

1

1

(0)(0) (1) ( 1)(1)(1) (0) ( 2)

( 1)( 1) ( 2) (0)

dxx x x

dxx x x

p dxx x x

w rr r r pw rr r r p

w r pr p r p r −

⎡ ⎤ ⎡ ⎤− ⎡ ⎤⎢ ⎥ ⎢ ⎥ ⎢ ⎥−⎢ ⎥ ⎢ ⎥ ⎢ ⎥=⎢ ⎥ ⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎢ ⎥ ⎢ ⎥−− − ⎢ ⎥⎢ ⎥ ⎣ ⎦⎣ ⎦⎣ ⎦

(6.7.2)


or, in a more compact form:

x dx=R w r (6.7.3)

Where Rx is a p x p Hermitian Toeplitz matrix of autocorrelations, w is the vector of filter coefficients, and rdx is the vector of cross-correlations between dn and xn.

8

FIR Wiener filter

The MMS error in the estimate of dn can be found with

{ } { }*1 1

2 * * *p p

E E d E d Eξ− −⎧ ⎫⎡ ⎤ ⎧ ⎫⎪ ⎪

⎨ ⎬ ⎨ ⎬⎢ ⎥∑ ∑{ } { }0 0

n n n l n l n n l n n ll l

E e E e d w x E e d w E e xξ − −= =

⎪ ⎪= = − = −⎨ ⎬ ⎨ ⎬⎢ ⎥⎣ ⎦ ⎩ ⎭⎪ ⎪⎩ ⎭

∑ ∑ (6.8.1)

and since wk is the solution to the Wiener-Hopf equation, it follows that

{ }* 0n n kE e x − =

Therefore: { }1

* *min

p

n n n l n l nE e d E d w x dξ−

−

⎧ ⎫⎛ ⎞= = −⎨ ⎬⎜ ⎟

⎝ ⎠⎩ ⎭∑

(6.8.2)

(6.8.3)


0l=⎝ ⎠⎩ ⎭and evaluating the expected values

1*

min0

(0) ( )p

d l dxl

r w r lξ−

=

= − ∑ (6.8.4)

7/20/2008

5

9

FIR Wiener filter

Using vector notation:

(0) Hξmin (0) Hd dxrξ = − r w (6.9.1)

Finally, since

1x dx−=w R r

The MMS error can be written as

(6.9.2)


The MMS error can be written as

1min (0) H

d dx x dxrξ −= − r R r (6.9.3)

10

FIR Wiener filter: Filtering

Filtering problem implies that a signal dn needs to be estimated from an observation corrupted by noise

dn n nx d v= +Assume that noise is a zero mean process uncorrelated with dn. Then

{ }* 0n n kE d v − =

and { } { } { }* * *( ) ( )dx n n k n n k n n k dr k E d x E d d E d v r k− − −= = + =

(6.10.1)

(6.10.2)

(6.10.3)


Since { } [ ][ ]{ }**( )x n k n n k n k n nr k E x x E d v d v+ + += = + +

for vn and dn uncorrelated, it follows that

( ) ( ) ( )x d vr k r k r k= +

(6.10.4)

(6.10.5)

7/20/2008

6

11

FIR Wiener filter: Filtering (Ex)

Therefore, with Rd the autocorrelation matrix for dn, Rv the autocorrelation matrix for vn, and rdx = rd = [rd(0),…rd(p – 1)]T, the Wiener-Hopf equations become

[ ]d v d+ =R R w rFurther simplifications are possible when more information about the statistic of the signal is available.

(6.11.1)

Example: let dn be an (AR) process with an autocorrelation sequence

( ) kdr k α=

h 0 1 d th ti i i l t d hit ith i

(6.11.2)


where 0 < α < 1 and the corrupting noise vn is uncorrelated white with a variance σv

2 and

n n nx d v= +We need to design a 1st order FIR Wiener filter to reduce the noise.

10 1( )W z w w z−= +

(6.11.3)

(6.11.4)

12


The Wiener-Hopf equations are

0(0) (1) (0)x x dxr r rw⎡ ⎤ ⎡ ⎤⎡ ⎤⎢ ⎥ ⎢ ⎥⎢ ⎥ (6 12 1)

1(1) (0) (1)x x dxr r rw=⎢ ⎥ ⎢ ⎥⎢ ⎥

⎣ ⎦⎣ ⎦ ⎣ ⎦(6.12.1)

Since noise is uncorrelated with the signal

( ) ( ) kdx dr k r k α= =

and 2( ) ( ) ( ) kx d v v kr k r k r k α σ δ= + = +

Therefore, the Wiener-Hopf equations become

(6.12.2)

(6.12.3)


p q2

02

1

111

v

v

ww

σ ααα σ

⎡ ⎤+ ⎡ ⎤ ⎡ ⎤=⎢ ⎥ ⎢ ⎥ ⎢ ⎥+ ⎣ ⎦⎣ ⎦⎣ ⎦

The Wiener filter is

( )2 2

02 22 2

1

11

1v

vv

ww

σ αασσ α

⎡ ⎤+ −⎡ ⎤= ⎢ ⎥⎢ ⎥

⎣ ⎦ + − ⎣ ⎦

(6.12.4)

(6.12.5)

7/20/2008

7

13


Then the Wiener filter is( )2 2 2 11

( ) v v zW z

σ α ασ −+ − += (6 13 1)

( )22 2( )

1 v

W zσ α+ −

(6.13.1)

For a particular case of α = 0.8 and σv2 = 1, the Wiener filter becomes

( ) 0.4048 0.2381j jW e eω ω−= +

which is a LPF with a magnitude response

(6.13.2)


magnitude response

14


We also note that the power spectrum of dn is

( )21jP e ω α−

(6 14 1)( ) ( )21 2 cosj

dP eα α ω

=+ −

For the specific case of α = 0.8 and σv2 = 1

( ) 0.361.64 1.6cos

jdP e ω

ω=

−

(6.14.1)

(6.14.2)

The power spectrum of the desired signal


of the desired signal is decreasing with frequency. Spectrum of noise is constant, therefore, LPF should increase SNR.

7/20/2008

8

15


The mean square error will be

{ }2 2

2 * * 2i 0 1 2

1(0) (0) (1) vd d dE e r w r w r σ αξ σ + −

= = − − = (6.15.1){ } ( )min 0 1 22 2(0) (0) (1)

1n d dx dx v

v

E e r w r w rξ σσ α+ −

For the specific case of α = 0.8 and σv2 = 1, ξmin = 0.4048.

Prior to filtering, since rd(0) = σd2 = 1 and σv

2 = 1, the power in dn equals to power in vn, then SNR = 1 = 0 dB. After filtering, the power in the signal d’n = wn∗dn is

{ } [ ]2 00 1

1

1' 0.3748

1T

n d

wE d w w

wα

α⎡ ⎤⎡ ⎤

= = =⎢ ⎥⎢ ⎥⎣ ⎦ ⎣ ⎦

w R w

(6.15.1)

(6.15.2)


And the noise power is

{ } [ ]2 00 1

1

' 0.2206Tn v

wE v w w

w⎡ ⎤

= = =⎢ ⎥⎣ ⎦

w R w (6.15.3)

(6.15.4)

The SNR at the filter output is0.374810lg 2.3020.2206

SNR dB= =

16

FIR Wiener filter: Linear prediction

With noise-free observations, linear prediction is the problem of finding the MMS estimate (prediction) of xn+1 using a linear combination of the current and p-1 previous values of xprevious values of xn.

Therefore, an FIR linear predictor of order p-1 has the form:

1

10

ˆp

n k n kk

x w x−

+ −=

= ∑ (6.16.1)


The linear predictor may be implemented by the Wiener filter by setting dn = xn+1.

Since

{ } { }* *1( ) ( 1)dx n n k n n k xr k E d x E x x r k− + −= = = + (6.16.2)

7/20/2008

9

17


The Wiener-Hopf equations for the optimum linear predictor are

* * (1)(0) (1) ( 1) w r⎡ ⎤ ⎡ ⎤ ⎡ ⎤

(6.17.1)

0*

1

1

(1)(0) (1) ( 1)(2)(1) (0) ( 2)

( )( 1) ( 2) (0)

xx x x

xx x x

p xx x x


w r pr p r p r −

⎡ ⎤ ⎡ ⎤− ⎡ ⎤⎢ ⎥ ⎢ ⎥ ⎢ ⎥−⎢ ⎥ ⎢ ⎥ ⎢ ⎥=⎢ ⎥ ⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎢ ⎥ ⎢ ⎥

− − ⎢ ⎥⎢ ⎥ ⎣ ⎦⎣ ⎦⎣ ⎦

The mean-square error is


(6.17.2)

1*

min0

(0) ( 1)p

d k xk

r w r kξ−

=

= − +∑

18

FIR Wiener filter: Linear prediction: Example

For the same (AR) process with an autocorrelation sequence

( ) kdr k α= (6.18.1)

the 1st order predictor is

1 0 1 1ˆn n nx w x w x+ −= +and the Wiener-Hopf equations are

02

1

11

ww

α αα α

⎡ ⎤⎡ ⎤ ⎡ ⎤=⎢ ⎥⎢ ⎥ ⎢ ⎥

⎣ ⎦ ⎣ ⎦⎣ ⎦

(6.18.2)

(6.18.3)


The predictor coefficients are

022

1

111 01

ww

α α αα αα

−⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤= =⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥−− ⎣ ⎦ ⎣ ⎦ ⎣ ⎦⎣ ⎦

(6.18.4)

7/20/2008

10

19

FIR Wiener filter: Linear prediction: Example

Therefore, the predictor is

1ˆn nx xα+ = (6.19.1)

The mean-square linear prediction error is2

min 0 1(0) (1) (2) 1x x xr w r w rξ α= − − = −

We observe that as α increases, the correlation between samples of xn increases and the prediction error decreases. For uncorrelated samples, α = 0 and ξmin = 1, which is equal to the variance of xn. The optimum predictor in this case will be

(6.19.2)


q n p p

1ˆ 0nx + =

That is the mean value of the process.

(6.19.3)

20


A more realistic situation is when the signal is contaminated by noise The linear predictor needsnoise. The linear predictor needs to estimate (predict) the signal in the presence of noise.

With the inputn n ny x v= +

The linear prediction will be

(6.20.1)

( )1 1p p− −

∑ ∑


( )10 0

ˆn k n k k n k n kk k

x w y w x v+ − − −= =

= = +∑ ∑The Wiener-Hopf equations are

y dy=R w r

(6.20.2)

(6.20.3)

7/20/2008

11

21


If the noise vn is uncorrelated with the signal xn, then Ry, the autocorrelation matrix for yn, is

{ }*( ) ( ) ( )k E k k+ (6 21 1){ }( ) ( ) ( )y n n k x vr k E y y r k r k−= = + (6.21.1)

and rdy(k), the cross-correlation vector between dn and yn, is

{ } { }* *1( ) ( 1)dy n n k n n k xr k E d y E x y r k− + −= = = +

Therefore, the only difference between linear prediction with and without noise is i th t l ti t i f th i t i l Wh i i l t d ith

(6.21.2)


in the autocorrelation matrix for the input signal. When noise is uncorrelated with the signal, Rx is replaced with Ry = Rx + Rv.

22


The problem of one-step linear prediction (when xn+1 is predicted) can be generalized to the problem of multistep prediction, when xn+α is predicted as a linear combination of the p values x x xlinear combination of the p values xn, xn-1,… xn-p+1.

1

0

ˆp

n k n kk

x w xα

−

+ −=

= ∑ (6.22.1)


Where α is a positive integer. Only the cross-correlation term will change compared to the one-step prediction:

{ }*( ) ( )dx n n k xr k E x x r kα α+ −= = + (6.22.2)

7/20/2008

12

23


The Wiener-Hopf equations become

* * ( )(0) (1) ( 1) w rr r r p α⎡ ⎤ ⎡ ⎤ ⎡ ⎤0*

1

1

( )(0) (1) ( 1)( 1)(1) (0) ( 2)

( 1)( 1) ( 2) (0)

xx x x

xx x x

p xx x x


w r pr p r p r

αα

α−

⎡ ⎤ ⎡ ⎤− ⎡ ⎤⎢ ⎥ ⎢ ⎥ ⎢ ⎥+−⎢ ⎥ ⎢ ⎥ ⎢ ⎥=⎢ ⎥ ⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎢ ⎥ ⎢ ⎥+ −− − ⎢ ⎥⎢ ⎥ ⎣ ⎦⎣ ⎦⎣ ⎦

or in the matrix form:

x x α=R w r (6.23.2)

(6.23.1)


,x x α

where rx,α is the autocorrelation vector beginning with rx(α). The MMS error is1

*min ,

0

(0) ( ) (0)p

Hx k x x x

k

r w r k r αξ α−

=

= − + = −∑ r w (6.23.3)

24


The multistep predictor can also be implemented as a one-step predictor using a linearpredictor using a linear combination of the values of xnover the interval from n-α-p-2 to n-α+1.

1

1 10

ˆp

n k n kk

x w x α

−

+ − − +=

= ∑ (6.24.1)

Assuming that the delay α is a free parameter the prediction problem can be


Assuming that the delay α is a free parameter, the prediction problem can be viewed as finding the filter coefficients AND the delay α that minimize MS error.

7/20/2008

13

25

FIR Wiener filter: Noise cancellation

The problem of noise cancellation is similar to the filtering problem since the goal is to recover a signal degraded by noise (btw, the signal is assumed to be recorded by a primary sensor) However unlike the filtering where the noiserecorded by a primary sensor). However, unlike the filtering where the noiseautocorrelation is known, noise parameters need to be estimated by a secondary sensor that is placed within the noise field. Although the noise is measured by a secondary sensor, it will be correlated


,with the noise coming from the primary sensor but the two processes are not equal.

Since v1,n ≠ v2,n, it is not possible to estimate dn by simply subtracting v2,n from xn. Instead, the noise canceller contains a Wiener filter estimating the noise from the sequence received from the secondary sensor. This estimate is then subtracted from the primary signal to form an estimate

1,ˆ nv

1,ˆ ˆn n nd x v= − (6.25.1)

26

FIR Wiener filter: Noise cancellation

The Wiener-Hopf equations for the noise cancellation system are

=R w r (6 26 1)2 1 2v v v=R w r (6.26.1)

Where Rv2 is the autocorrelation matrix of v2,n and rv1v2 is the cross-correlation between the needed noise signal v1,n and the Wiener filter input v2,n. The cross-correlations are

{ } ( ){ } { } { }1 2

* * * *1, 2, 2, 2, 2,( )v v n n k n n n k n n k n n kr k E v v E x d v E x v E d v− − − −= = − = −

Assuming that v2 n and dn are uncorrelated:

(6.26.2)


g 2,n n

{ }1 2 2

*2,( ) ( )v v n n k xvr k E x v r k−= =

The Wiener-Hopf equations are

2 2v xv=R w r

(6.26.3)

(6.26.4)

7/20/2008

14

27

FIR Wiener filter: Noise cancellation (Example)

Assume that the desired signal is a sinusoid:

( )( )0sinnd nω φ= +with ω0 = 0.05π and that the noise sequences are

1, 1, 1

2, 2, 1

0.80.6

n n n

n n n

v v gv v g

−

−

= +

= − +where gn is zero-mean, unit variance white noise uncorrelated with dn.

xn dn

(6.27.1)

(6.27.2)


1,n n nx d v= +v2,nThe observed signal:

(6.27.3)

28


The sample (estimated) autocorrelation is11ˆ ( )

N

k−

∑2 2, 2,0

( )v n n kn

r k v vN −

=

= ∑

2

1

2,0

1ˆ ( )N

xv n n kn

r k x vN

−

−=

= ∑The sample (estimated) cross-correlation is

(6.28.1)

(6.28.2)


Note: in typical applications, signal or (and) noise are often nonstationary. Therefore, the use of LTI Wiener filter is not optimum. However, an adaptive Wiener filter may provide effective noise cancellation in nonstationary environments.

7/20/2008

15

29


Output of a 6th

order Wiener filter


Output of a 12th order Wiener filter

30

IIR Wiener filter

The goal is to design an IIR Wiener filter that for the sequence xn would produce an output yn = xn∗hn that is as seque ce n ou d p oduce a output yn n n t at s asclose as possible – in the mean-square sense – to the desired process dn. We notice that for the IIR Wiener filter, there are an infinite number of unknown coefficients to be found: hn for all n.


7/20/2008

16

31

Noncausal IIR Wiener filter

For a noncausal (unconstrained) IIR Wiener filter, the problem is to find the unit sample response of the filter

∞

( ) nn

nH z h z−

=−∞

= ∑that minimizes the mean-square error

{ }2nE eξ =

Where the mean-square error is∞

(6.31.1)

(6.31.2)


ˆn n n n l n l

l

e d d d h x∞

−=−∞

= − = − ∑ (6.31.3)

32


This problem can be solved similarly to the FOR Wiener filter problem: by equating the derivative of the mean-square error with respect to hk

* to zero for each k:

ξ∂ { }** 0,n n kk

E e x khξ

−

∂= − = − ∞ < < ∞

∂Which is equivalent to

{ }* 0,n n kE e x k− = − ∞ < < ∞

The last equation (6.32.2) is called the orthogonality principle and it is identical to

(6.32.1)

(6.32.2)


the orthogonality principle for an FIR filter except that here the equality must hold for all k. Therefore:

{ } { }* * ,l n l n k n n kl

h E x x E d x k∞

− − −=−∞

= − ∞ < < ∞∑ (6.32.3)

We note that the expectation in the lhs is the autocorrelation, and in the rhs –cross-correlation.

7/20/2008

17

33


( ) ( ),l x dxl

h r k l r k k∞

=−∞

− = − ∞ < < ∞∑ (6.33.1)

l ∞

Which are the Wiener-Hopf equations of the noncausal IIR Wiener filter. We observe that the only difference compared to the FIR case is the summation limit and the range of values for k. We can also notice that

( ) ( )k x dxh r k r k∗ =In the frequency domain:

( ) ( ) ( )j j jω ω ω

(6.33.2)


( ) ( ) ( )j j jx dxH e P e P eω ω ω=

Therefore, the frequency response of the IIR Wiener filter is

( ) ( )( )

jdxj

jx

P eH e

P e

ωω

ω=

(6.33.3)

(6.33.4)

34


The transfer function is( )( )( )

dxP zH zP z

= (6.34.1)( )xP z

We observe that the denominator is an auto-spectrum, while the numerator is a cross-spectrum.The mean-square error is

*min (0) ( )d l dx

l

r h r lξ∞

=−∞

= − ∑U i P l’ th

(6.34.2)


Using Parseval’s theorem

( ) ( )*min

1(0)2

j jd dxr H e P e d

πω ω

π

ξ ωπ −

= − ∫ (6.34.3)

7/20/2008

18

35


Since ( )1(0)

2j

d dr P e dπ

ω ωπ

= ∫ (6.35.1)

π−

we may rewrite the MS error as

( ) ( ) ( )*min

12

j j jd dxP e H e P e d

πω ω ω

π

ξ ωπ −

⎡ ⎤= −⎣ ⎦∫

The error can also be expressed in terms of the complex variable z as

1 1

(6.35.2)


( ) ( ) ( ) ( )* * 1 * * 1min

1 1(0) 1 ( ) 12 2d dx d dx

C C

r H z P z z dz P z H z P z z dzj j

ξπ π

− −⎡ ⎤= − = −⎣ ⎦∫ ∫(6.35.3)

where the contour, C, may be taken to be the unit circle.

36

Noncausal IIR Wiener filter: Smoothing

We need a MMS estimate of a process dn using the noisy observation

n n nx d v= + (6.36.1)n n nx d vWe need to find auto- and cross-spectra. Assuming that dn and vn are uncorrelated zero mean random processes, the autocorrelation is

( ) ( ) ( )x d vr k r k r k= +and the power spectrum (auto-spectrum) is

( ) ( ) ( )j j jx d vP e P e P eω ω ω= +

( )

(6.36.2)

(6.36.3)


( ) ( ) ( )x d v

The cross-correlation

{ } { } { }* * *( ) ( )dx n n k n n k n n k dr k E d x E d d E d v r k− − −= = + =

and the cross-spectrum

( ) ( )j jdx dP e P eω ω=

(6.36.4)

(6.36.5)

7/20/2008

19

37


Therefore, the IIR Wiener filter is

( ) ( )jdj

P eH

ωω (6 37 1)( ) ( )

( ) ( )j

j jd v

H eP e P e

ωω ω

=+

We note that over the frequency bands where the filter magnitude response is approximately one. Therefore, over these frequency bands, a little attenuation takes place. Since

( ) ( )j jd vP e P eω ω

( ) ( )j jdx dP e P eω ω=

(6.37.1)

(6.37.2)


and is real, then

( ) ( ) ( ) ( ) ( )*min

1 1 12 2

j j j j jd dx dP e H e P e d P e H e d

π πω ω ω ω ω

π π

ξ ω ωπ π− −

⎡ ⎤ ⎡ ⎤= − = −⎣ ⎦ ⎣ ⎦∫ ∫(6.37.3)

38


Finally, combining with the filter frequency response equation, the MS error of a noncausal IIR Wiener filter smoother is

( ) ( )( ) ( ) ( ) ( )min

1 12 2

jvj j j

d vj jd v

P eP e d P e H e d

P e P e

ωπ πω ω ω

ω ωπ π

ξ ω ωπ π− −

= =+∫ ∫

or, in the z-domain:

1min

1 ( ) ( )vP z H z z dzξ −= ∫

(6.38.1)

(6.38.2)


min ( ) ( )2 v

Cjξ

π ∫

7/20/2008

20

39

Causal IIR Wiener filter

To be able to realize an IIR Wiener filter, we needs to constrain it to be causal. For the causal filter, its unit pulse response hn is zero for n < 0. Therefore:

∞

0

ˆn n n k n k

k

d x h h x∞

−=

= ∗ = ∑Using the same procedure as before, we can find the Wiener-Hopf equations for the causal IIR Wiener filter:

0( ) ( ); 0l x dx

l

h r k l r k k∞

− = ≤ ≤ ∞∑

(6.39.1)

(6.39.2)


0l=

The important difference between this result and the one for the noncausal IIR filter is the summation limit. The restriction on k as being non-negative implies that the cross-correlation rdx(k) cannot be expressed as the convolution of hk and rx(k).

40


We start the filter design at the special case when the input to the filter is unit variance white noise εn. Denoting the Wiener filter coefficients by gn, the Wiener-Hopf equations areHopf equations are

0( ) ( ); 0l d

lg r k l r k kε ε

∞

=

− = ≤ ≤ ∞∑Since rε(k) = δ(k), the lhs reduces to gk. Therefore, the causal Wiener filter for white noise is:

( )n d ng r n uε=where un is the unit step function. The z-domain solution is

(6.40.1)

(6.40.2)


where un is the unit step function. The z domain solution is

[ ]( ) ( )dG z P zε +=

Here “+” indicates the “positive-time part” of the sequence whose z-transform is contained within the brackets.

(6.40.3)

7/20/2008

21

41


However, in most of the practical applications, it is unlikely that the input to a Wiener filter will be white noise. Assuming that the input xn is a random process with a rational power spectrum with no poles and zeros on the unit circle we maywith a rational power spectrum with no poles and zeros on the unit circle, we may perform a spectral factorization and write Px as follows:

( )2 * *0( ) ( ) 1xP z Q z Q zσ= (6.41.1)

where Q(z) is minimum phase and of the form

1 21 2

( )( ) 1 ...( )

N zQ z q z q zD z

− −= + + + = (6.41.2)


where N(z) and D(z) are minimum phase monic polynomials. If xn is filtered with a filter having a transfer function of the form

0

1( )( )

F zQ zσ

= (6.41.3)

42


the power spectrum of the output process εn will be

( ) ( ) ( ) ( )* *1 1P z P z F z F z= = (6 42 1)( ) ( ) ( ) ( )1 1xP z P z F z F zε

Therefore, the output process εn is white noise and F(z) is called a whitening filter. We notice that since Q(z) is minimum phase, then F(z) is stable and causal and has a stable and causal inverse F-1(z). As a result, xn may be recovered from εn by filtering with the inverse filter F-1(z). In other words, there is no loss of information in the linear transformation producing white noise from xn.

Let H(z) be the causal Wiener filter with an input xn having a rational spectrum and producing the MMS estimate of d Suppose that the input is filtered with a cascade

(6.42.1)


producing the MMS estimate of dn. Suppose that the input is filtered with a cascade of three filters F(z), F-1(z), and H(z) where F(z) is the causal whitening filter for xnand F-1(z) is its causalinverse.

7/20/2008

22

43


The cascade G(z) = F-1(z)H(z) is the causal IIR Wiener filter producing the MMS estimate of dn from the white noise εn. The causality of G(z) follows from the fact that both F-1(z) and G(z) are causalthat both F (z) and G(z) are causal.The cross-correlation between dn and εn is

{ }*

* *( ) ( )d n n k n l n k l l dxl l

r k E d E d f x f r k lε ε∞ ∞

− − −=−∞ =−∞

⎧ ⎫⎡ ⎤⎪ ⎪= = = +⎨ ⎬⎢ ⎥⎣ ⎦⎪ ⎪⎩ ⎭∑ ∑ (6.43.1)

Therefore, the cross-power spectral density is

( ) ( ) ( ) ( )* *1 dxP zP P F (6 43 2)


( ) ( ) ( ) ( )( )* *

0

11d dxP z P z F z

Q zε σ= = (6.43.2)

and the causal Wiener filter for estimating dn from εn is

( ) ( )( )* *

0

11

dxP zG z

Q zσ+

⎡ ⎤⎢ ⎥=⎢ ⎥⎣ ⎦

(6.43.3)

44


Since the causal Wiener filter is a cascade of two filters

( ) ( ) ( )H z F z G z= (6.44.1)( ) ( ) ( )then

( ) ( )( )

( )2 * *0

11

dxP zH z

Q z Q zσ+

⎡ ⎤⎢ ⎥=⎢ ⎥⎣ ⎦

(6.44.2)

In the case of real processes, hn is real and causal IIR Wiener filter in the form:

( ) ( )1 dxP zH

⎡ ⎤⎢ ⎥


( ) ( )( )

( )20

11

dxH zQ z Q zσ

+

= ⎢ ⎥⎣ ⎦

Finally, the MS error for the causal IIR Wiener filter is

*min

0

(0) ( )d l dxl

r h r lξ∞

=

= −∑

(6.44.3)

(6.44.4)

7/20/2008

23

45


In the frequency domain the error is

( ) ( ) ( )*1 j j jP H P dπ

ω ω ωξ ⎡ ⎤∫

( ) ( )* * 1min

1 ( ) 12 d dx

C

P z H z P z z dzj

ξπ

−⎡ ⎤= −⎣ ⎦∫

In the z-domain the error is

( ) ( ) ( )min 2j j j

d dxP e H e P e dω ω ω

π

ξ ωπ −

⎡ ⎤= −⎣ ⎦∫ (6.45.1)

(6.45.2)


We observe that the expressions for the causal IIR Wiener filter error in the frequency and z-domains are exactly the same as the corresponding expressions for the non-causal IIR Wiener filter. In the time-domain description the difference arises in the summation limits.

46


We notice that a noncausal Wiener filter may be expressed using spectral factorization as

( ) ( )dP z( ) ( )( ) ( )2 * *

0 1dx

nc

P zH z

Q z Q zσ=

Or, viewed as a cascade of two filters, the noncausal Wiener filter is

( ) ( )( )

( )2 * *0

11

dxnc

P zH z

Q z Q zσ

⎡ ⎤⎢ ⎥=⎢ ⎥⎣ ⎦

Th l Wi filt b i l t d b th t t h b l W

(6.46.1)

(6.46.2)


The non-causal Wiener filter can be implemented by the structure shown below. We note that the first filter is the causal whitening filter generating white noise εn while the secondis noncausal filter to produce the MMS dn.

7/20/2008

24

47


A causal IIR Wiener filter is formed by taking the causal part of [Pdx(z)/Q*(1/z*)] as shown below.


48

Causal IIR Wiener filter: filtering

Considering a problem of estimation of a process dn from the noisy observation

x d v= + (6.48.1)n n nx d v+and assuming that the noise vn is uncorrelated with dn:

( ) ( )dx dP z P z=the causal Wiener filter becomes

( ) ( )( )

( )2 * *0

11

dP zH z

Q z Q zσ

⎡ ⎤⎢ ⎥=⎢ ⎥⎣ ⎦

( )

(6.48.2)

(6.48.3)


( ) ( )0 1Q z Q zσ+

⎢ ⎥⎣ ⎦

where ( ) ( ) ( ) ( ) ( )2 * *0 1x d vP z P z P z Q z Q zσ= + =

However, to evaluate (built) the actual Wiener filter, the expressions for power spectral densities Pd(z) and Pv(z) are required.

(6.48.4)

7/20/2008

25

49

Causal IIR Wiener filter: filtering (Example)

We need to estimate a signal dn generated by

10 8d d w= + (6.49.1)10.8n n nd d w− +from its noisy observation

n n nx d v= +Where vn is unit variance white noise uncorrelated with dn and wn is white noise with variance σ2

w = 0.36. Therefore, rd(k) = 0.8|k|. To find the optimum causal IIR Wiener filter, we begin from observation that

( ) ( )P P

(6.49.1)

(6.49.2)


( ) ( )( ) ( ) ( ) ( ) 1

dx d

x d v d

P z P z

P z P z P z P z

=

= + = +Therefore, with

( ) ( )( )1

0.361 0.8 1 0.8dP z

z z−=

− −

(6.49.3)

(6.49.4)

(6.49.5)

50


The power spectrum of xn is

( ) ( )( )11 0.5 1 0.50.361 1 6z z

P−− −

( ) ( )( )( )( )( )( )1 1

1 1.61 0.8 1 0.8 1 0.8 1 0.8xP z

z z z z− −= + =

− − − −

Since xn is real, then:( ) ( ) ( )2 1

0xP z Q z Q zσ −=

with ( )1

20 1

1 0.51.6,1 0 8

zQ zz

σ−

−

−= =

(6.50.1)

(6.50.2)

(6.50.3)


1 0.8z−Since the causal IIR Wiener filter is

( ) ( )( )

( )2 10

1 dxP zH z

Q z Q zσ −

+

⎡ ⎤⎢ ⎥=⎢ ⎥⎣ ⎦

(6.50.4)

7/20/2008

26

51


we can express ( )( ) ( )( )

( )( )1 1

1

1 0.80.361 0.51 0.8 1 0.8

dxP z zzQ z z z− −

−=

−− −

( )( )1

1 11 1

0.36 0.6 0.31 0.8 0.51 0.8 0.5

zz zz z

−

− −− −= = +

− −− −

Therefore: ( )( ) 11

0.61 0.8

dxP zzQ z −−

+

⎡ ⎤⎢ ⎥ =

−⎢ ⎥⎣ ⎦Positive-time part

and the Wiener filter is

(6.51.1)

(6.51.2)


and the Wiener filter is

( ) ( )( ) ( )

1

11 1

1 0.81 0.6 0.3751.6 1 0.51 0.5 1 0.8

zH z

zz z

−

−− −

−= =

−− −

or ( )0.375 1 2 nn nh u=

(6.51.3)

(6.51.4)

52


Since ( ) ( ) ( )D̂ z H z X z=

th ti t f d b t d i l f ll

(6.52.1)

the estimate of dn may be computed recursively as follows

1ˆ ˆ0.5 0.375n n nd d x−= +

Finally, the mean-square error can be estimated as

( ){ } ( )2

min0 0

3 1ˆ (0) ( ) 1 0.8 0.37508 2

ll

n n d l dxl l

E d d r h r lξ∞ ∞ ⎛ ⎞= − = − = − =⎜ ⎟

⎝ ⎠∑ ∑

(6.52.2)

(6.52.3)


{ }0 08 2l l= = ⎝ ⎠

For the comparison, for the 2nd-order FIR Wiener filter, the MS error was 0.4048. We conclude that using all previous observations of xn only slightly improves the performance of the Wiener filter.

7/20/2008

27

53


For another comparison, we compute a noncausal Wiener filter as

( ) ( ) ( ) 0.36 1.6dx dP z P zH ( ) ( )

( )( )( ) ( )( )11 0.5 1 0.5

dx d

x x

H zP z P z z z−

= = =− −

The unit pulse response is

3 110 2

n

nh ⎛ ⎞= ⎜ ⎟⎝ ⎠

The MS error can be computed as

(6.53.1)

(6.53.2)


( )min0

3 1 3(0) ( ) 1 2 0.8 0.310 2 10

kk

d l dxl k

r h r lξ∞ ∞

=−∞ =

⎛ ⎞= − = − + =⎜ ⎟⎝ ⎠

∑ ∑Which is lower than for the causal filter as it should be expected.

(6.53.3)

54


An interesting observation from the result for that particular non-causal IIR filter is that the recursive estimator can be rewritten as

1 1 1ˆ ˆ ˆ ˆ0.5 0.375 0.8 0.375 0.8n n n n n nd d x d x d− − −

⎡ ⎤= + = + −⎣ ⎦Therefore, the MMS estimate of dn is based on all observations of xn up to time n.Similarly, the MMS estimate of dn-1 is based on all observations of xn up to time n-1. If we have an estimate of dn-1, we may “predict” the estimate for of dn! In this situation, we may “predict” the next measurement of xn.When the next actual measurement of xn arrives, we may compare it to the predicted value. The prediction error will be

(6.54.1)


ˆn n nx xα = −This error is called an innovation process and represents the “new information”. In other words, it represents the part that cannot be predicted. Therefore, the estimate of dn can be corrected by the new information. This approach is related to Kalman filtering.

(6.54.2)

7/20/2008

28

55

Causal IIR Wiener filter: linear prediction

We need to derive an optimum linear predictor in the form

ˆ h∞

∑10

ˆn k n kk

x h x+ −=

= ∑that would produce the best estimate for xn+1 based on xk for all k ≤ n. Since the infinite number of past signal values is used, we expect a better prediction than an FIR predictor produces.

For the linear prediction problem 1n nd x +=

(6.55.1)

(6.55.2)


And the cross-correlation is

{ } { }* *1( ) ( 1)dx n n k n n k xr k E d x E x x r k− + −= = = +

Therefore:

( ) ( )dx xP z zP z=

(6.55.3)

(6.55.4)

56


The Wiener predictor is then

( ) ( )1 xzP zH z⎡ ⎤⎢ ⎥= (6 56 1)( ) ( ) ( )2 * *

0 1H z

Q z Q zσ+

⎢ ⎥=⎢ ⎥⎣ ⎦

However, since ( ) ( ) ( )2 * *0 1xP z Q z Q zσ=

The linear predictor can be simplified as

( ) ( ) ( )1H z zQ zQ z +

= ⎡ ⎤⎣ ⎦

(6.56.1)

(6.56.2)

(6.56.3)


( )QRecalling that Q(z) is a monic polynomial (0th-order coefficient = 1),

( ) 1 21 21 ...Q z q z q z− −= + + + (6.56.4)

7/20/2008

29

57


we observe that the positive-time part of zQ(z) is

( ) 1 2 1 2zQ z z q q z q z q q z q z− − − −⎡ ⎤= + + + + = + + +⎡ ⎤⎣ ⎦ ⎣ ⎦( )( )

1 2 2 1 2 2... ...

1

zQ z z q q z q z q q z q z

z Q z+ +

⎡ ⎤= + + + + = + + +⎡ ⎤⎣ ⎦ ⎣ ⎦

= −⎡ ⎤⎣ ⎦The causal IIR linear predictor becomes

( ) ( ) ( ) ( )1 11 1H z z Q z z

Q z Q z⎡ ⎤

= − = −⎡ ⎤ ⎢ ⎥⎣ ⎦⎣ ⎦

(6.57.1)

(6.57.2)


The MMS error is

( ) ( ) ( )* * 1min

1 12 d dx

C

P z H z P z z dzj

ξπ

−⎡ ⎤= −⎣ ⎦∫ (6.57.3)

58


Since ( ) ( ) ( ) ( )d x dx xP z P z and P z zP z= =

th MMS ill b

(6.58.1)

the MMS error will be

( ) ( ) ( )1 * * 1min

1 12 x x

C

P z z H z P z z dzj

ξπ

− −⎡ ⎤= −⎣ ⎦∫Since the power spectrum is symmetric, i.e.

( ) ( )* *1x xP z P z=

(6.58.2)

(6.58.3)


the error becomes

( ) ( )1 1min

1 12 x

C

P z z H z z dzj

ξπ

− −⎡ ⎤= −⎣ ⎦∫ (6.58.4)

7/20/2008

30

59


( )⎡ ⎤⎛ ⎞

Substituting the transfer function for the causal IIR Wiener predictor leads to

( ) ( )( )( )

( )

1 1min

2 * * 1 20 0 0

1 1 11 12 2

1 12

xx

C C

C

P zP z z dz z dz

j Q z j Q z

Q z z dz qj

ξπ π

σ σπ

− −

−

⎡ ⎤⎛ ⎞= − − =⎢ ⎥⎜ ⎟⎜ ⎟⎢ ⎥⎝ ⎠⎣ ⎦

= =

∫ ∫

∫

Finally since Q(z) is monic q0 = 1 and therefore the MMS error is

(6.59.1)


Finally, since Q(z) is monic, q0 = 1 and, therefore, the MMS error is

2min 0ξ σ= (6.59.2)

60


The spectral factorization suggests that for a wss random process, whose power spectrum is a real-valued, positive, and periodic function of frequency, the f ll i f t i ti h ldfollowing factorization holds:

( ) ( ) ( )2 * *0 1xP z Q z Q zσ=

where ( )1 ln22

0

jxP e d

e

πω

π

ωπσ −

∫=

Th f

(6.60.1)

(6.60.2)


Therefore:( )1 ln

2min

jxP e d

e

πω

π

ωπξ −

∫=

which is known as the Kolmogorov-Szegö formula.

(6.60.3)

7/20/2008

31

61

Causal IIR Wiener filter: linear prediction of an AR process

Assuming that xn is an autoregressive* AR(p) process with a power spectrum

( )20P σ( )

( ) ( )0* *1xP z

A z A z=

where( )

11

pk

kk

A z a z−

=

= + ∑is a minimum phase polynomial having all its roots inside the unit circle.The optimum linear predictor is

(6.61.1)

(6.61.2)


* - AR processes will be explained in details later.

( ) ( ) 1 11 21 ... p

pH z z A z a a z a z− − += − = − − − −⎡ ⎤⎣ ⎦which happens to be an FIR filter! Therefore, only the last p values out of an infinite number of past signal samples are used to predict xn+1.

(6.61.3)

62

Causal IIR Wiener filter: linear prediction of an AR process

A random autoregressive AR(p) process satisfies a difference equation of the form:

1 1 2 2 ...n n n p n p nx a x a x a x w− − −= − − − − +

where wn is white noise. Since wn+1 cannot be predicted from xn or its previous values (noise is assumed as uncorrelated with the signal), predicting xn+1, we can use the model for xn and ignore the noise at best.

1 1 2 1 1ˆ ...n n n p n px a x a x a x+ − − += − − − −

(6.62.1)

(6.62.2)


7/20/2008

32

63

Causal IIR Wiener filter: linear prediction of an AR process (Ex)

Consider a real-valued AR(2) process

0 9 0 2x x x w= + (6 63 1)1 20.9 0.2n n n nx x x w− −= − +where wn is unit-variance zero-mean white noise. Since

(6.63.1)

( )( ) ( )1

1xP z

A z A z−=

where ( ) 1 21 0.9 0.2A z z z− −= − +

Th ti li di t i

(6.63.2)

(6.63.3)


The optimum linear predictor is

( ) ( ) 1 2 11 0.9 0.2 0.9 0.2H z z A z z z z z− − −⎡ ⎤= − = − = −⎡ ⎤⎣ ⎦ ⎣ ⎦

and the prediction of xn+1 is formed as

1 1ˆ 0.9 0.2n n nx x x+ −= −

(6.63.4)

(6.63.5)

64


A specific realization of xn (solid line) and its optimal prediction (dotted line) are shown below


For this particular case, the average squared error is

[ ]1

21 1

0

1 ˆ 0.0324N

n nn

x xN

ξ−

+ +=

= − =∑ (6.64.1)

7/20/2008

33

65


However, in practice, the statistics of xn is never known. Therefore, more realistically would be to estimate the AR parameters* first for the given data:

1

0

1ˆ ( )N

x n n kn

r k x xN

−

−=

= ∑where N = 200. Then we find

ˆ ˆ ˆ(0) 2.1904 (1) 1.5462 (2) 0.8670x x xr r r= = =The normal equations* are

1ˆ2.1904 1.5462 1.5462a⎡ ⎤⎡ ⎤ ⎡ ⎤⎢ ⎥⎢ ⎥ ⎢ ⎥

(6.65.1)

(6.65.2)

(6 65 3)


* - AR parameters estimation will be explained in details later.

2ˆ1.5462 2.1904 0.8670a= −⎢ ⎥⎢ ⎥ ⎢ ⎥

⎣ ⎦ ⎣ ⎦⎣ ⎦which are solved to find estimates for a1 and a2 next

1

2

ˆ 0.8500ˆ 0.2042aa

−⎡ ⎤ ⎡ ⎤=⎢ ⎥ ⎢ ⎥

⎣ ⎦⎣ ⎦

(6.65.3)

(6.65.4)

66


We observe that the estimated AR parameters are not equal to the true ones and, therefore, the predictor becomes

1 1ˆ 0.85 0.2042n n nx x x+ −= −Next, instead of using the predictor on the data that was used to estimate the AR parameters, we apply the predictor to the next 200 data values. The result is shown below:

(6.66.1)


7/20/2008

34

67

Causal IIR Wiener filter: deconvolution

The deconvolution problem is concerned with the recovery of the signal dn that has been convolved with a filter gn that may not be precisely known:

n n nx d g= ∗Such distortions are often introduced in the process of measuring or recording data: slightly out of focus cameras, band-limited communication channels. For instance, a moving during the exposure camera would introduce distortion. If the “blurring” function gn is perfectly known and has an inverse gn

-1, such that1

n n ng g δ−∗ =

(6.67.1)

(6.67.2)


then, in theory, it is not hard to recover dn from xn:

( ) ( )( )

1j

jn n n j

X ed x g D e

G e

ωω

ω−= ∗ ⇔ =

However, in practice, precise knowledge of gn is generally not available.

(6.67.3)

68


Another problem is that the frequency response G(ejω) is zero (or very small) at some frequencies. Therefore, G(ejω) may be either noninvertible or ill-conditioned.

In addition, noise may be introduced in the measurement process, and, therefore, more accurate model for the observed process would be

n n n nx d g w= ∗ +where wn is additive noise that is often assumed as uncorrelated with dn. In this situation, even if the inverse filter exists and is well-behaved, after the inverse filter is applied to xn, the restored signal is

( )jω

(6.68.1)


( ) ( ) ( )( ) ( ) ( )ˆ

jj j j j

j

W eD e D e D e V e

G e

ωω ω ω ω

ω= + = +

noise

or, in the time domain: ˆn n nd d v= +

(6.68.2)

(6.68.3)

7/20/2008

35

69


Therefore, the signal estimate gets additional “filtered noise” term…

An alternative approach to the deconvolution problem is to design a Wiener filterAn alternative approach to the deconvolution problem is to design a Wiener filter producing the MMS estimate of dn from xn. Let hn be a noncausal IIR LTI filter producing the estimate of dn.

ˆn n n l n l

ld x h h x

∞

−=−∞

= ∗ = ∑The solution to the Wiener-Hopf equations in the frequency domain is

( ) ( )jdxj

P eH e

ωω =

(6.69.1)

(6 69 2)


( ) ( )jx

H eP e ω

=

Since wn is uncorrelated with dn:

( ) ( ) ( ) ( )2j j j jx d wP e P e G e P eω ω ω ω= +

(6.69.2)

(6.69.3)

70

Also, the cross-psd is


( ) ( ) ( )*j j jdx dP e P e G eω ω ω= (6.70.1)( ) ( ) ( )

Therefore, the optimum Wiener filter for deconvolution is given by

( ) ( ) ( )( ) ( ) ( )

*

2

j jdj

j j jd w

P e G eH e

P e G e P e

ω ωω

ω ω ω=

+

Moreover, assuming that G(ejω) is non-zero for all ω and that its inverse exists:

( )1jP e ω⎡ ⎤

⎢ ⎥

(6.70.2)


( ) ( )( )

( ) ( ) ( ) 21 dj

j j j jd w

P eH e

G e P e P e G eω

ω ω ω ω

⎢ ⎥= ⎢ ⎥+⎢ ⎥⎣ ⎦Since the power spectrum of the filtered noise is

( ) ( ) ( ) 2j j jv wP e P e G eω ω ω=

(6.70.3)

(6.70.4)

7/20/2008

36

71


The term in the square brackets of (6.70.3) may be represented as

( ) ( )jdP e ω

( ) ( )( ) ( )

djj j

d v

eF e

P e P eω

ω ω=

+

which is the noncausal IIR Wiener smoothing filter for estimating dn from

n n ny d v= +Therefore, the deconvolving Wiener filter may be viewed as a cascade of the “inverse degradation” filter followed by a noncausal Wiener smoothing filter that

d th filt d i

(6.71.1)

(6.71.2)


reduces the filtered noise.

72

Discrete Kalman filter

The primary limitation to Wiener filters is that both the signal and noise processes must be jointly wss. Most of practical signals are nonstationary, which limits

li ti f Wi filtapplications of Wiener filters.

Recall that for recovering an AR(1) process of the form

1 1n n nx a x w−= +from its noisy observation

n n ny x v= +where vn and wn are uncorrelated white noise processes, the optimum linear

i f i ll f h f k ld b d i h

(6.72.1)

(6.72.2)


estimate of xn using all of the measurements of yk, k ≤ n could be computed with a recursion:

[ ]1 1 1 1ˆ ˆ ˆn n n nx a x K y a x− −= + −where K is a constant Kalman gain minimizing the MS error

{ }2ˆn nE x x−

(6.72.3)

(6.72.4)

7/20/2008

37

73


However, the mentioned above again uses a wss assumption. For a time-varying situation, the optimum estimate may be found as

1,1 1 1,1 1ˆ ˆ ˆn n n n n n nx a x K y a x− − − −⎡ ⎤= + −⎣ ⎦where Kn is a time-varying gain.The above result can be extended to estimation of AR(p) processes of the form

1

p

n k n k nk

x a x w−=

= +∑

(6.73.1)

(6.73.2)


1k=

That is measured in the presence of additive noise

n n ny x v= + (6.73.3)

74


Let denote a p-dimensional state vector

1

nxx

⎡ ⎤⎢ ⎥⎢ ⎥1

1

nn

n p

x

x

−

− +

⎢ ⎥=⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦

x

Then equations (6.73.2) and (6.73.3) may be expressed in the form:

1 2 1 11 0 0 0 0

p pa a a a−⎡ ⎤ ⎡ ⎤⎢ ⎥ ⎢ ⎥

(6.74.1)


1

1 0 0 0 00 1 0 0 0

0 0 1 0 0

n n nw−

⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎢ ⎥= +⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦

x x (6.74.2)

7/20/2008

38

75


and [ ]1 0 0n n ny v= +x (6.75.1)

Using matrix notation, we have

1n n n

Tn n ny v

−= +

= +

x Ax w

c xwhere A is a p x p state transition matrix,

[ ]0 0 Tn nw=w

(6.75.2)

(6.75.3)

(6.75.4)


is a vector noise process, and c is a unit vector of length p.We notice that for the case of AR(1) process, the optimum estimate of the state vector xn using all the previous measurements, may be found as

1 1ˆ ˆ ˆTn n n ny− −⎡ ⎤= + −⎣ ⎦x Ax K c Ax

where K is a Kalman gain vector.

(6.75.5)

76


The equation (6.75.2) is applicable to stationary AR(p) processes only but can be easily generalized to nonstationary processes as follows

1 1n n n n− −= +x A x wwhere An-1 is a time-varying p x p state transition matrix and wn is a vector of zero-mean white noise process with

{ } ( )wHn n

n k nE

k n =⎧

= ⎨ ≠⎩

Qw w

0

(6.76.1)

(6.76.2)


⎩In addition, let yn be a vector of observations of length q formed as

n n n n= +y C x vwhere Cn is a time-varying q x p matrix, and vn is a vector of zero-mean white noise statistically independent of wn with

(6.76.3)

7/20/2008

39

77


{ } ( )vHn n

n k nE

k n =⎧

= ⎨ ≠⎩

Qv v

0(6.77.1)

k n ≠⎩0The optimum linear estimate for the time-varying case would be

1 1 1 1ˆ ˆ ˆTn n n n n n ny− − − −⎡ ⎤= + −⎣ ⎦x A x K c A x

With the appropriate Kalman gain matrix Kn, this recursion corresponds to the discrete Kalman filter.Assuming that An, Cn, Qv(n), and Qw(n) are known and denoting the best |ˆ n nx

(6.77.2)


g n, n, Qv( ), Qw( ) glinear estimate of xn at time n given the observations yi for I = 1,2,…,n and denoting the best linear estimate of xn at time n given the observations yi for I = 1,2,…,n-1, the corresponding state estimation errors are

|n n

| 1ˆ n n−x

| |

| 1 | 1

ˆ

ˆn n n n n

n n n n n− −

= −

= −

e x x

e x x(6.77.3)

78


Denoting the error covariance matrices as

{ }HE=P e e{ }{ }

| | |

| 1 | 1 | 1

n n n n n n

Hn n n n n n

E

E− − −

=

=

P e e

P e e(6.78.1)

we wish to solve the following problem:Suppose that we are given an estimate of the state x0 and that the error covariance matrix for this estimate P0|0 is known. When the measurement y1 is available, we need to update and find the estimate of the state at time n = 1 minimizing the MS error

0|0x̂

0|0x̂ 1|1x̂


1 minimizing the MS error

{ } { } { }12 2

1 1|1 1|1 ,1|10

p

ii

E tr E eξ−

=

= = = ∑e P

Once such an estimate is found and the error covariance P1|1 is evaluated, the estimation is repeated for the next observation y2.

(6.78.2)

7/20/2008

40

79


The solution to this problem is derived in two steps. First, given we will find , which is the best estimate of xn without the observation yn. Next, given

d ill ti t

1| 1ˆ n n− −x| 1ˆ n n−x

x̂yn, and we will estimate xn.| 1n n−xIn the first step, since no new measurements are used to estimate xn, all we know is that

1 1n n n n− −= +x A x wSince wn is a zero-mean white noise and its values are unknown, we may predict xn as follows:

| 1 1 1| 1ˆ ˆn n n n n− − − −=x A x

(6.79.1)

(6.79.2)


| 1 1 1| 1n n n n nwhich has an estimation error

| 1 | 1 1 1 1 1| 1 1 1| 1ˆ ˆn n n n n n n n n n n n n n n− − − − − − − − − −= − = + − = +e x x A x w A x A e w (6.79.3)

80


Since wn has zero mean, if is an unbiased estimate of xn-1, i.e.1| 1ˆ n n− −x

{ } 0E{ }1| 1 0n nE − − =ethen is an unbiased estimate of xn:| 1ˆ n n−x

{ }| 1 0n nE − =eSince the estimation error en-1|n-1 is uncorrelated with white noise wn, then

( )HP A P A Q

(6.80.1)

(6.80.2)


| 1 1 1| 1 1 ( )Hn n n n n n w n− − − − −= +P A P A Q

where Qw(n) is the covariance matrix for the noise process wn. This completes the first step of the Kalman filter.

(6.80.3)

7/20/2008

41

81


In the second step, we incorporate the new measurement yn into the estimateA new linear estimate is formed next as

1| 1ˆ n n− −x

'| | 1ˆ ˆn n n n n n n−= +x K x K y

where Kn and Kn’ are matrices that need to be specified. The error can be found as

[ ]' '| | 1 | 1

' '| 1

ˆn n n n n n n n n n n n n n n n n

n n n n n n n n n

− −

−

⎡ ⎤= − − = − − − +⎣ ⎦⎡ ⎤= − − + −⎣ ⎦

e x K x K y x K x e K C x v

I K K C x K e K v

Since { } { }0 0E and E e= =v

(6.81.1)

(6.81.2)

(6 81 3)


{ } { }| 10 0n n nE and E e −= =v

then will be unbiased for any xn only if|ˆ n nx'n n n= −K I K C

(6.81.3)

(6.81.4)

82


(6.82.1)

Which leads to the estimate in form:

[ ]| | 1ˆ ˆ= − +x I K C x K y (6.82.1)[ ]| | 1n n n n n n n n− +x I K C x K yor

| | 1 | 1ˆ ˆ ˆn n n n n n n n n− −⎡ ⎤= + −⎣ ⎦x x K y C x (6.82.2)

And the error is[ ]'

| | 1 | 1n n n n n n n n n n n n n− −= − = − −e K e K v I K C e K vSince vn is uncorrelated with wn, then vn is uncorrelated with xn. Also, vn is

(6.82.3)


uncorrelated with en|n-1:

{ }| 1 0n n nE − =e vTherefore, the error covariance matrix for en|n-1 is

{ } [ ] [ ]| | | | 1 ( )HH Hn n n n n n n n n n n n n v nE n−= = − − +P e e I K C P I K C K Q K

(6.82.4)

(6.82.5)

7/20/2008

42

83


Next, we need to find the value for the Kalman gain Kn that minimizes the MS error

{ }{ }|n n ntrξ = Ptrace

where the trace function is

1( )

n

iii

tr A a=

= ∑

(6.83.1)

(6.83.2)

Therefore, we need to differentiate ξn with respect to Kn, set the derivative to zero, and solve for Kn. Using the matrix differentiation formulas


( ) Hd trd

=KA AK

and ( ) 2Hd trd

=KAK KAK

(6.83.3)

(6.83.4)

84


we obtain

( ) [ ]| | 12 2 ( ) 0Hn n n n n n n n v

d tr nd −= − − + =P I K C P C K QK

(6.84.1)( ) [ ]| | 1n n n n n n n n vdKSolving for Kn gives the expression for the Kalman gain

1

| 1 | 1 ( )H Hn n n n n n n n v n

−

− −⎡ ⎤= +⎣ ⎦K P C C P C Q

Next, we can simplify the expression for the error covariance:

[ ]=P I K C P

(6.84.2)

(6 84 3)


[ ]| | 1n n n n n n−= −P I K C PFinally, the initial conditions for n = 0 can be chosen as

{ }{ }

0|0 0

0|0 0 0

ˆH

E

E

=

=

x x

P x x

(6.84.3)

(6.84.4)

(6.84.5)

7/20/2008

43

85


In summary:


86

Discrete Kalman filter (Ex)

Let xn be an AR(1) process

0 8x x w= + (6.86.1)10.8n n nx x w−= +where wn is white noise with a variance σw

2 = 0.36 and let

n n ny x v= +be noisy measurements of xn where vn is unit variance white noise that is uncorrelated with wn. Thus, with An = 0.8 and Cn = 1, the Kalman filter state equation is

(6.86.1)

(6.86.2)

[ ]ˆ ˆ ˆ0 8 0 8x x K y x= + (6 86 3)


Here the state vector is a scalar. Therefore, the Kalman gain can be computed with scalar equations

[ ]1 10.8 0.8n n n n nx x K y x− −= + −

2| 1 | 1

1

| 1 | 1

0.8 0.36

1

n n n n

n n n n n

P P

K P P

− −

−

− −

= +

⎡ ⎤= +⎣ ⎦

(6.86.3)

(6.86.4)

(6.86.5)

7/20/2008

44

87

Discrete Kalman filter (Ex)

(6.87.1)[ ]| | 11n n n n nP K P −= −

With { }With { } { }20 0 0|0 0ˆ 0 1x E x and P E x= = = =

the Kalman gain and the error covariances are shown for the first few values of n.

We observe that after a few iterations the

(6.87.2)


We observe that after a few iterations the Kalman filter settles into its steady state solution:

[ ]1 1ˆ ˆ ˆ0.8 0.375 0.8n n n nx x y x− −= + − (6.87.3)

88


The goal of the discrete Kalman filter is to use the measurements yn to estimate the state xn of a dynamicmeasurements yn to estimate the state xn of a dynamic system.

The Kalman filter is a very powerful recursive estimation algorithm that has found applications in various areas including radar tracking, estimation and prediction of target trajectories, adaptive equalization of telephone channels and f di di i h l d ti t


fading dispersive channels, adaptive antenna arrays…

lecture 06 - oprimum filters

Documents