adaptive and statistical signal processing

8/9/2019 Adaptive and Statistical Signal Processing

1/21

Page 2

Adaptive and Statistical Signal ProcessingContents

Course No:

1. Random Variables and Vectors

2. Random Variables and Vectors, Intro to Estimation

3. MVU, Cramer Rao Lower Bound

4. Linear Models, Best Linear Unbiased Estimator

5. Least Squares and Maximum Likelihood Estimation

6. Introduction to Bayes Estimation7. Linear Bayes Estimation

8. Linear Bayes Estimation, Stochastic Processes

9. Stochastic Processes10. Wiener Filter

11. Wiener Filter

12. Least Squares Filter


2/21

Page 3

Minimum Variance Unbiased Estimator (MVU)

Remember: Quality Criterion for estimator:

mean squared error: should be minimal

let then

with e and e 2 as the mean and variance of the estimation error e,respectively, e is also called the bias of the estimator.

For a minimum variance unbiased estimatior (MVU) our aim is to find

an estimator that has zero bias ( e =0), i.e. with minimumvariance e 2 of the error

Please note a MVU does not necessarily is the estimator with theminimum mean squared error, in fact there might be an estimatorthat is biased ( e 0) but has a lower mean squared error.However, from a practical point of view, MUVs are often easier tofind that estimators minimizing the mean squared error.

Often it is even impossible to find an MVU. In this case we even haveto use additional constraints, e.g. to find best linear unbiasedestimator (BLUE)

))(( 2 E

=e222 ))(( ee E =

=)( E


3/21


4/21


5/21Page 6

Cramer-Rao-Lower Bound

Provide an easy way to determine a lower bound on theestimator performance, i.e. a better estimator cannot exist

The MVU estimator does not necessarily attain the CRLB

Sometimes the best estimator is provided by the CRLB directly


6/21Page 7


The Cramer-Rao Lower Bound (CRLB) is given by 1)

This means that the variance of every unbiased estimator mustbe higher or equal than the CRLB

If the CRLB exists and one can write

then the MVU estimator is given byand has the variance

where I( x ) is the so called Fischer information:

i.e. then the MVU satisfies the CRLB with equality1) if the pdf p( x ; ) satisfies the regularity condition:

allfor0

);(ln =

x p E

=

2

2 );(ln(x)

x p E I

)( xg= )(1x I


7/21Page 8


cf. Information Theory: Information is log b p(x)

Intuitive explanation for

The Fischer information can be seen as the amount of information about that is available in the data

The more information is available for estimation, the lower theestimation variance!

Important properties of ln p( x ; )

non-negativity

additive for independent RV:

CLRB lowers when using additional (independent) random variables

)(1

)var(

I

=

=1

0

)];[(ln);(ln N

n

n x p p x

=

=

1

02

2

2

2 )];[(ln);(ln N

n

n x p E

p E

x


8/21Page 9

Efficient Estimators

An unbiased estimator is said to be efficient if it attains the CRLB:

it uses all the available data efficiently

An efficient estimator is always the MVU but the MVU is not necessarilyan efficient estimator

Taken from Kay: Fundamentals of Statistical Signal Processing, Vol 1: Estimation Theory, Prentice Hall, Upper Saddle River


9/21Page 10


Example: DC Level in white gaussian noise:

x[n] = A + w[n]

w[n] is WGN with variance 2

Taking the first derivative

with as the sample mean.

=

=

1

0

22

22

)][(2

1exp

)2(

1);(

N

n N An x A p

x

)(

)][(

1

)][(2

1

])2ln[(

);(ln

2

1

02

1

0

2

222

A x N

An x An x A A

A p N

n

N

n

N

=

=

=

=

=

x

=

==1

0

][1)( N

n

n x N

xg x


10/21

Page 11


Differentiating again gives:

This leads to the CLRB:

By using the result from the first derivative one obtains:

This means for DC level in WGN the sample mean is the MVUestimator!

22

2 );(ln

N A p =

x

N

A2

)var(

))()(()(2 Ag A I A x N = x


11/21

Page 12

General CRLB for Signals in White Noise

Assume a deterministic signal with an unknown parameter isobserved in WGN as:

The pdf of x depending on the parameter is given as:

Differentiating once produces:

and a second differentiation results in:

1,...,1,0][];[][ =+= N nnwnsn x

=

=

1

0

22

22

]);[][(

2

1exp

)2(

1);(

N

n

N nsn x p

x

=

=

1

022

2 ];[]);[][(

1);(ln N

n

nsnsn x

p

x

=

=

1

0

2

2

2

22

2 ];[];[]);[][(

1);(ln N

n

nsnsnsn x

p

x


12/21

Page 13

General CRLB for Signals in White Noise

Taking the expectation value results in:

This leads to the CRLB for signals in White Noise:

The form of the bound demonstrates the importance of thesignal dependence on .

Signals that change rapidly as the unknown parameter changesresult in accurate estimators

E.g. as we have seen with the DC level in WGN: s[n;]=

produces a CRLB of 2 /N.

21

0

2

];[)var(

=

N

n

ns

21

022

2 ];[1);(ln

=

=

N

n

ns p E

x


13/21

Page 14

Transformation of Parameters

Assume the we wish to estimate a parameter that is a functiong( ) of some more fundamental parameter and already knowthe CRLB for

Then the CRLB for g() can be obtained by (without proof):Let and be an estimator of

Then the CRLB for is

2

2

2

);(ln)var(

x p E

g

)( g=


14/21

Page 15

Transformation of Parameters

But: be carefull!

Non-linear transformations destroy the efficiency

e.g. DC Level in WGN, Estimator for A 2

Square of the sample mean: might be a resonable estimator

But is not even unbiased anymore

On the other hand: affine (linear) transformations

preserve efficiency!

2 x

22

222

)var()()( A N A x x E x E +=+=

bagg +== )()(


15/21

Page 16

CRLB for Vector Paramters

Commonly: vector of parameters =[ 1 , 2 ,., p ] T

Without proof the CRLB is found as the [i,i] element of theinverse of the Fischer Information Matrix I ( )

With the Fischer Information Matrix defined by:

A Fischer Information Matrix is always symmetric

In practice I ( ) is assumed to be positive definite and henceinvertible

[ ]iii )()var( 1 I

[ ] p j pi p E ji

ij ,...,1 and,...,1for);(ln

)(2

==

=

xI


16/21

Page 17

CRLB for Vector Paramters

More formally: The covariance matrix of any unbiasedestimator satisfies: 1)

Furthermore an unbiased estimator may be found thatattains the bound if and only if

Again this MVU estimator is then efficientHere p( x ) is a p dimensional function!

The CRLB represents a powerful tool to find efficient estimatorsfor vector parameters.

1) if the pdf p( x ; ) satisfies the regularity condition:

xallfor0

);(ln =

p E

0IC

)(1)

C )

))()(();(ln xI

x =

g p

)( x g=


17/21

Page 18

Example: Line Fitting

Consider the problem x[n] = A + Bn + w[n]

w[n] is again WGN

The parameter vector

from which the first derivatives follow as:

=

=

1

0

22

22

)][(2

1exp

)2(

1);(

N

n N Bn An x p

x

],[ B A=

n Bn An x B

p

Bn An x A

p

N

n

N

n

=

=

=

=

1

02

1

02

)][(1);(ln

)][(1);(ln

x

x


18/21


19/21


20/21

Page 21


The CRLB for B is lower than for A (for N>2)

The lower bound for estimation of B decrease with oder 1/N 3oposed to 1/N for A

B is easier to estimate than A Intuitive Explanation: changes of B are magnified by n

Finding Estimators for A and B:

The derivatives

can be rewritten as (after some manipulations) as

=

=

=

=

n Bn An x

Bn An x

B

p A

p p

N

n

N

n1

02

1

02

)][(1

)][(1

);(ln

);(ln);(ln

x

x

x

+

+

+

=

=

B B A A

N N N N

N N N N

N

B p A

p

p

)1(12

)1(6 )1(

6

)1(

)12(2

);(ln

);(ln

);(ln2

2 x

x

x


21/21

Page 22


with

This means for these estimators the CRLB is satisfied withequality, hence they are efficient MVU estimators.

=

=

=

=

+

+=

+

+

=

1

0

1

02

1

0

1

0

][)1(

12][

)1(6

][)1(

6][

)1()12(2

N

n

N

n

N

n

N

n

nnx N N

n x N N

B

nnx N N

n x N N N

A

adaptive and statistical signal processing

Documents