Estimation theory

Johan E. Carlson

Dept. of Computer Science, Electrical and Space EngineeringLuleå University of Technology

Lecture 1

1 / 26

Outline1. General course information

2. Introduction (Chapter 1)2.1. Estimation theory in Signal Processing

2.2. Problem formulation

2.3. Assessing estimator performance

3. Minimum variance unbiased estimation (Chapter 2)3.1. Unbiased estimators

3.2. Existence of the MVUB estimator

3.3. Finding the MVUB estimator

3.4. Extension to a vector parameter

2 / 26

Notes

Notes

General course informationCourse web page:http://staff.www.ltu.se/~johanc/estimation_theory/

Text book:Steven M. Kay, Fundamentals of Statistical Signal Processing:Estimation Theory, Vol 1. Prentice Hall, 1993.ISBN10: 0133457117Examination:

Completion of theoretical homework assignments (writtensolutions to be handed in to me).Completion of computer assignments (short lab reports to me).

3 / 26

Estimation in Signal Processing

Modern estimation theory is central to many electrical system, e.g.Radar, Sonar, AcousticsSpeechImage and videoBiomedicine (Biomedical engineering)Communications (Channel estimation, synchronization, etc.)Automatic controlSeismology

4 / 26

Notes

Notes

Estimation in Signal Processing

Example: Ultrasonic characterization of thin multi-layeredmaterials

Estimate thickness of layersLocate flaws/cracks in internal layers

5 / 26

Estimation in Signal Processing

Simply put, given an observed N -point data set

{x[0], x[1], . . . , x[N − 1]}

which depends on an unknown parameter θ, wee define theestimator

θ = g(x[0], x[1], . . . , x[N − 1]),

where g is some function.

6 / 26

Notes

Notes

Mathematical problem formulation

Since data are inherently random, we will describe them in termsof probability density functions (PDF), i.e.

p(x[0], x[1], . . . , x[N − 1]; θ),

where semicolon (;) denotes that the PDF is parameterized by theunknown parameter θ.

The estimation problem is thus to find (infer ) the value of θ from theobservations. The PDF should be chosen so that

It takes into account any prior knowledge or constraintsIt is mathematically tractable

7 / 26

Mathematical problem formulation

Example: The Dow-Jones index

0 20 40 60 80 1002800

2900

3000

3100

3200

Day number

Dow

−Jo

nes

aver

age

It appears it is "on average increasing".

8 / 26

Notes

Notes

Mathematical problem formulation

Example: The Dow-Jones indexA reasonable model could then be

x[n] = A+Bn+ w[n], n = 0, 1, . . . , N − 1,

where w[n] is white Gaussian noise (WGN), i.e. each sample ofw[n] has the PDF N (0, σ2), and is uncorrelated with all the othersamples. The unknown parameters can arranged in the vectorθ = [A B]T . The PDF of x[n] then is

p(x;θ) =1

(2πσ2)N2

exp

[− 1

2σ2

N−1∑n=0

(x[n]−A−Bn)2],

where x = [x[0], x[1], . . . , x[N − 1]]T

9 / 26

Mathematical problem formulation

Example: The Dow-Jones indexThe straight line assumption is consistent with data. A modelsthe offset and B models the linear increase over time.The choice of the Gaussian PDF makes the modelmathematically tractable.Here the parameters are assumed to be unknown, butdeterministic.One could also assume that θ is random, but constrained, sayA is in [2800, 3200], and that it is uniformly distributed in thisinterval.This would lead to a Bayesian approach, andthe joint PDF will be

p(x;θ) = p(x|θ)p(θ)

10 / 26

Notes

Notes

Assessing estimator performance

Consider the following signal (a realization of a DC voltagecorrupted by noise).

0 20 40 60 80 100−1

0

1

2

3

n

x[n]

A realistic signal model would then be

x[n] = A+ w[n],

where w[n] is N (0, σ2).11 / 26

Assessing estimator performance

How to estimate A?

A reasonable estimator would be the sample mean

A =1

N

N−1∑n=0

x[n]

How close will A to A?Are there any better estimators than the sample mean?

12 / 26

Notes

Notes

Assessing estimator performance

Another estimator could be

A = x[0]

Intuitively, this should not perform as well, since we’re notusing all available data.But, for any given realization of x[n] it might actually be closerto the true A than the sample mean.So, how do we assess the performance?We need to consider the estimators from a statisticalperspective!Let’s consider E(A) and var(A)

13 / 26

Assessing estimator performance

For the first estimator

E(A) = E

(1

N

N−1∑n=0

x[n]

)

=1

N

N−1∑n=0

E(x[n])

= A

For the second estimator

E(A) = E(x[0]

= A

14 / 26

Notes

Notes

Assessing estimator performance

For the first estimator

var(A) = var

(1

N

N−1∑n=0

x[n]

)

=1

N2

N−1∑n=0

var(x[n])

=1

N2Nσ2

=σ2

N

For the second estimator

var(A) = var(x[0])

= σ2

15 / 26

Assessing estimator performance

So, the expected value of both estimators, E(A) = A, i.e. theyare both unbiased.The variance of the second estimator is σ2, which is largerthan the variance of the first estimator.It appears that indeed the sample mean is a better estimatethan x[0]!

16 / 26

Notes

Notes

Minimum variance unbiased (MVUB) estimation

Let’s start by considering estimation of unknown butdeterministic parameters.We will restrict the search for estimators to those who onaverage yield the true parameter value, i.e. to unbiasedestimators.For all possible unbiased estimators, we will then look for theone with the minimum variance, i.e. the minimum varianceunbiased estimator (MVUB).

17 / 26

Unbiased estimatorsAn estimator is said to be unbiased if

E(θ) = θ,

for all possible values of θ.

If θ = g(x), this means that

E(θ) =

∫g(x)p(x;θ)dx = θ,

for all θ.

18 / 26

Notes

Notes

The minimum variance criterionLet’s look at a natural optimality criterion, known as the meansquare error (MSE)

mse(θ) = E[(θ − θ)2

]

19 / 26

The minimum variance criterionUnfortunately, this criterion often leads to unrealizable estimators,since

mse(θ) = E

{[(θ − E(θ)

)+(E(θ)− θ

)]2}= var(θ) +

[E(θ)− θ

]2= var(θ) + b2(θ),

which shows that the MSE depends both on the variance of theestimator and on the bias. If the bias depends on the parameteritself, we’re in trouble!

Let’s restrict ourselves to search only forunbiased estimators!

20 / 26

Notes

Notes

Existence of the MVUB estimatorDoes an MVUB estimator always exist?No!

21 / 26

Existence of the MVUB estimatorA counterexample of to the existence

Assume that we have to independent observations x[0] and x[1]with PDF

x[0] ∼ N (θ, 1)

x[1] ∼{N (θ, 1), θ ≥ 0N (θ, 2), θ < 0

22 / 26

Notes

Notes

Existence of the MVUB estimatorThe two estimators

θ1 =1

2(x[0] + x[1])

θ2 =2

3x[0] +

1

3

can easily be shown to be unbiased. The variances are

var(θ1) =1

4(var(x[0]) + var(x[1]))

var(θ2) =4

9var(x[0]) +

1

9var(x[1])

23 / 26

Existence of the MVUB estimatorAs a result (looking back at the PDFs), we have that

var(θ1) =

{1836 , θ ≥ 02436 , θ < 0

var(θ2) =

{2036 , θ ≥ 02436 , θ < 0

So, for θ ≥ 0, the minimum variance is 18/36 (estimator 1) and forθ < 0 it is 24/36 (estimator 2). Hence, no single estimator will havethe uniformly minimum variance for all θ.

24 / 26

Notes

Notes

Finding the MVUB estimator

There is no (known) universal method that will always producethe MVUB estimator.There are some possible approaches though

Determine the Cramer-Rao lower bound (CRLB) and check ifsome estimator satisfies it (Chapter 3 and 4).Apply the Rao-Blackwell-Lehmann-Scheffe (RBLS) theorem(Chapter 5).Further restrict the estimators to be also linear. Then find theMVUB estimator within this class. (Chapter 6).

25 / 26

Extension to a vector parameter

If θ = [θ1, θ2, . . . , θp]T is a vector of unknown parameters, we say

that an estimator is unbiased if

E(θi) = θi,

for i = 1, 2, . . . , p. By defining

E(θ) =

E(θ1)

E(θ2)...

E(θp)

,we can define un unbiased estimator as having the property

E(θ) = θ.

26 / 26

Notes

Notes