ece 6540, lecture 07ece6540/slides/2016_ece6540_class007.pdf · sufficient statistics in the end,...
TRANSCRIPT
-
Variations
ECE 6540, Lecture 07Cramer-Rao Lower Bounds
-
Last Time Rao-Blackwell-Lehmann-Scheffe Theorem
Complete Sufficient Statistics
2
-
Sufficient Statistics Rao-Blackwell-Lehmann-Scheffe Theorem
If �𝜽𝜽 is an unbiased estimator of 𝜽𝜽 and 𝑇𝑇 𝒙𝒙 is a sufficient statistic for 𝜽𝜽, then�𝜽𝜽 = E �𝜽𝜽|𝑇𝑇 𝒙𝒙 is
1) A valid estimator for 𝜽𝜽 (not dependent on 𝜽𝜽)2) Unbiased3) Of lesser or equal variance than that of �𝜽𝜽, for all 𝜽𝜽 (each element of �𝜽𝜽 has
lesser or equal variance)4) If the sufficient statistic is complete, then �𝜽𝜽 is the MVU estimator
3
𝑇𝑇 𝒙𝒙 represents more information
-
Sufficient Statistics A Complete Statistics A statistic is complete if there is only one function𝑔𝑔 ⋅ of the statistic t = 𝑇𝑇 𝑥𝑥
that is unbiased.
That is, A statistic is complete if only one 𝑔𝑔 ⋅ exists such that
𝐸𝐸 𝑔𝑔 𝑇𝑇 𝒙𝒙 = 𝜽𝜽
4
-
Sufficient Statistics In the end, Applying Neyman-Fisher factorization can be difficult…
Proving that a statistic is complete can be difficult… (unless you have an exponential family PDF)
Rao-Blackwell-Lehmann-Scheffe Theorem can be difficult…
What else can we do????
5
-
Cramer-Rao Lower Bounds
6
-
Cramer-Rao Lower Bounds Question: How could we know if an estimator is a MVUB estimator or is close to an MVUB estimator?
7
-
Cramer-Rao Lower Bounds The Cramer-Rao Lower Bound (CRLB) Sets a lower bound on the variance of any unbiased estimator
As a result: An estimator that achieves the CRLB is the MVUB estimator
The CRLB tells sets a benchmark for estimator variance. — Estimators with a variance “close” to the CRLB are almost the MVUB
estimator. — No estimator can have a variance lower than the CRLB.
𝜃𝜃
Estimator variance
Cramer Rao Lower Bound
Common type of analysis
8
-
Cramer-Rao Lower Bounds The Cramer-Rao Lower Bound (CRLB) Sets a lower bound on the variance of any unbiased estimator
Warning about bounds! Bounds usually tell us what we cannot achieve, not what we can achieve.
— i.e., we do not obtain the maximum lower bound
In some scenarios, bounds can be very “loose” — Therefore the Cramer-Rao Lower Bound may not be achievable
(or possibly even that informative)
𝜃𝜃
Estimator variance
Cramer Rao Lower Bound
9
-
Cramer-Rao Lower Bounds The Cramer-Rao Lower Bound (for a single parameter) is defined by
Assume 𝐸𝐸 �̂�𝜃 − 𝐸𝐸 𝜃𝜃 = 0 (unbiased)
Assume 𝐸𝐸 𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜃𝜃𝜕𝜕𝜃𝜃
= 0 , for all𝜃𝜃 (regularity condition)
Then
var �̂�𝜃 ≥1
𝐼𝐼 𝜃𝜃 , 𝐼𝐼 𝜃𝜃 = −𝐸𝐸𝜕𝜕2 ln 𝑝𝑝 𝒙𝒙;𝜃𝜃
𝜕𝜕𝜃𝜃2
Fisher Information
Log-Likelihood functionThe score
10
-
Cramer-Rao Lower Bounds The Cramer-Rao Lower Bound (for a vector parameter) is defined by
Assume 𝐸𝐸 �𝜽𝜽− 𝐸𝐸 𝜽𝜽 = 0 (unbiased)
Assume 𝐸𝐸 𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽
= 0 , for all𝜃𝜃 (regularity condition)
Then
𝐂𝐂�𝜽𝜽 − 𝑰𝑰−1 𝜽𝜽 ≽ 0 , 𝐼𝐼𝑖𝑖𝑖𝑖 𝜽𝜽 = −𝐸𝐸𝜕𝜕2 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽
𝜕𝜕𝜃𝜃𝑖𝑖𝜕𝜕𝜃𝜃𝑖𝑖
FisherInformation
Matrix
Log-Likelihood functionThe score
Positive Semidefinitive
11
-
Cramer-Rao Lower Bounds Positive Semidefinite Matrices A matrix 𝑴𝑴 is positive semidefinite if
𝒙𝒙𝐻𝐻𝑴𝑴𝒙𝒙 ≥ 0 , for all 𝑥𝑥 ∈ ℂ𝑁𝑁
Also, A matrix 𝑴𝑴 is positive semidefinite if every eigenvalue of 𝑴𝑴≥ 0
Therefore,
𝐂𝐂�𝜽𝜽 − 𝑰𝑰−1 𝜽𝜽 ≽ 0 is equivalent to x𝐻𝐻𝐂𝐂�𝜽𝜽𝑥𝑥 ≥ x𝐻𝐻𝑰𝑰−1 𝜽𝜽 𝑥𝑥 for all 𝑥𝑥 ∈ ℂ𝑁𝑁
Or the eigenvalues of 𝐂𝐂�𝜽𝜽 − 𝑰𝑰−1 𝜽𝜽 are all ≥ 0
12
-
Cramer-Rao Lower Bounds A Looser Cramer-Rao Lower Bound (for a vector parameter) is defined by
Assume 𝐸𝐸 �𝜽𝜽− 𝐸𝐸 𝜽𝜽 = 0 (unbiased)
Assume 𝐸𝐸 𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽
= 0 , for all𝜃𝜃 (regularity condition)
Then
var �𝜽𝜽𝑖𝑖 ≥ 𝑰𝑰−1 𝜽𝜽 𝑖𝑖𝑖𝑖, 𝐼𝐼𝑖𝑖𝑖𝑖 𝜽𝜽 = −𝐸𝐸𝜕𝜕2 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽
𝜕𝜕𝜃𝜃𝑖𝑖𝜕𝜕𝜃𝜃𝑖𝑖
FisherInformation
Matrix
Log-Likelihood functionThe score
13
-
Cramer-Rao Lower Bounds A efficient estimator An estimator is said to be efficient if it meets the Cramer-Rao lower bound.
— Covariance matrix = the inverse Fisher information matrix— 𝐂𝐂�𝜽𝜽 = 𝑰𝑰−1 𝜽𝜽
14
-
Cramer-Rao Lower Bounds Cramer-Rao Lower Bound Proof (for one parameter)
Assume �̂�𝜃 is unbiased and has a PDF 𝑝𝑝 𝑥𝑥;𝜃𝜃
𝐸𝐸 �̂�𝜃 𝑥𝑥 = �−∞
∞
�̂�𝜃 𝑥𝑥 𝑝𝑝 𝑥𝑥;𝜃𝜃 𝑑𝑑𝑥𝑥 = 𝜃𝜃
�−∞
∞
�̂�𝜃 𝑥𝑥𝜕𝜕𝑝𝑝 𝑥𝑥;𝜃𝜃𝜕𝜕𝜃𝜃 𝑑𝑑𝑥𝑥 =
𝜕𝜕𝜃𝜃𝜕𝜕𝜃𝜃
�−∞
∞
�̂�𝜃 𝑥𝑥𝜕𝜕 ln 𝑝𝑝 𝑥𝑥;𝜃𝜃
𝜕𝜕𝜃𝜃 𝑝𝑝 𝑥𝑥;𝜃𝜃 𝑑𝑑𝑥𝑥 = 1
�−∞
∞
�̂�𝜃 𝑥𝑥 − 𝜃𝜃𝜕𝜕 ln 𝑝𝑝 𝑥𝑥;𝜃𝜃
𝜕𝜕𝜃𝜃 𝑝𝑝 𝑥𝑥;𝜃𝜃 𝑑𝑑𝑥𝑥 = 1
�−∞
∞
�̂�𝜃 𝑥𝑥 − 𝜃𝜃 2𝑝𝑝 𝑥𝑥;𝜃𝜃 𝑑𝑑𝑥𝑥 �−∞
∞𝜕𝜕 ln 𝑝𝑝 𝑥𝑥;𝜃𝜃
𝜕𝜕𝜃𝜃
2
𝑝𝑝 𝑥𝑥;𝜃𝜃 𝑑𝑑𝑥𝑥 ≥ 1
𝜕𝜕 ln 𝑝𝑝 𝑥𝑥;𝜃𝜃𝜕𝜕𝜃𝜃
=𝜕𝜕𝑝𝑝 𝑥𝑥; 𝜃𝜃𝜕𝜕𝜃𝜃
1𝑝𝑝 𝑥𝑥; 𝜃𝜃
Express as integral
Take derivative of both sides
Cauchy-SchwartzInequality
Apply regularity condition [new term is equal to 0]
15
-
Cramer-Rao Lower Bounds Cramer-Rao Lower Bound Proof (for one parameter)
Assume �̂�𝜃 is unbiased and has a PDF 𝑝𝑝 𝑥𝑥;𝜃𝜃
�−∞
∞
�̂�𝜃 − 𝜃𝜃 2𝑝𝑝 𝑥𝑥;𝜃𝜃 𝑑𝑑𝑥𝑥 �−∞
∞𝜕𝜕 ln 𝑝𝑝 𝑥𝑥;𝜃𝜃
𝜕𝜕𝜃𝜃
2
𝑝𝑝 𝑥𝑥;𝜃𝜃 𝑑𝑑𝑥𝑥 ≥ 1
var �̂�𝜃 𝐸𝐸𝜕𝜕 ln 𝑝𝑝 𝑥𝑥;𝜃𝜃
𝜕𝜕𝜃𝜃
2
≥ 1
var �̂�𝜃 ≥1
𝐸𝐸𝜕𝜕 ln 𝑝𝑝 𝑥𝑥;𝜃𝜃
𝜕𝜕𝜃𝜃
2
var �̂�𝜃 ≥1
𝐸𝐸𝜕𝜕2 ln 𝑝𝑝 𝑥𝑥;𝜃𝜃
𝜕𝜕𝜃𝜃2
Express as expectations
Simplify
See next slide for this proof
16
-
Cramer-Rao Lower Bounds
𝐸𝐸𝜕𝜕 ln 𝑝𝑝 𝑥𝑥;𝜃𝜃
𝜕𝜕𝜃𝜃= 0
�−∞
∞𝜕𝜕 ln 𝑝𝑝 𝑥𝑥; 𝜃𝜃
𝜕𝜕𝜃𝜃𝑝𝑝 𝑥𝑥;𝜃𝜃 = 0
𝜕𝜕𝜕𝜕𝜃𝜃
�−∞
∞𝜕𝜕 ln 𝑝𝑝 𝑥𝑥;𝜃𝜃
𝜕𝜕𝜃𝜃𝑝𝑝 𝑥𝑥;𝜃𝜃 = 0
�−∞
∞𝜕𝜕2 ln 𝑝𝑝 𝑥𝑥;𝜃𝜃
𝜕𝜕𝜃𝜃2𝑝𝑝 𝑥𝑥; 𝜃𝜃 +
𝜕𝜕 ln 𝑝𝑝 𝑥𝑥; 𝜃𝜃𝜕𝜕𝜃𝜃
𝜕𝜕𝑝𝑝 𝑥𝑥; 𝜃𝜃𝜕𝜕𝜃𝜃
= 0
�−∞
∞𝜕𝜕2 ln 𝑝𝑝 𝑥𝑥; 𝜃𝜃
𝜕𝜕𝜃𝜃2𝑝𝑝 𝑥𝑥;𝜃𝜃 +
𝜕𝜕 ln 𝑝𝑝 𝑥𝑥;𝜃𝜃𝜕𝜕𝜃𝜃
𝜕𝜕 ln 𝑝𝑝 𝑥𝑥; 𝜃𝜃𝜕𝜕𝜃𝜃
𝑝𝑝 𝑥𝑥; 𝜃𝜃 = 0
𝐸𝐸𝜕𝜕2 ln 𝑝𝑝 𝑥𝑥; 𝜃𝜃
𝜕𝜕𝜃𝜃2+ 𝐸𝐸
𝜕𝜕 ln 𝑝𝑝 𝑥𝑥; 𝜃𝜃𝜕𝜕𝜃𝜃
2
= 0
𝐸𝐸𝜕𝜕2 ln 𝑝𝑝 𝑥𝑥; 𝜃𝜃
𝜕𝜕𝜃𝜃2= −𝐸𝐸
𝜕𝜕 ln 𝑝𝑝 𝑥𝑥; 𝜃𝜃𝜕𝜕𝜃𝜃
2
Regularity condition
Express as integral
Take derivative of both sides
Evaluate derivative
𝜕𝜕 ln 𝑝𝑝 𝑥𝑥;𝜃𝜃𝜕𝜕𝜃𝜃 =
𝜕𝜕𝑝𝑝 𝑥𝑥;𝜃𝜃𝜕𝜕𝜃𝜃
1𝑝𝑝 𝑥𝑥; 𝜃𝜃
Express as expectation
17
-
Cramer-Rao Lower Bounds Example: DC signal in Gaussian noise Consider a length 𝑁𝑁 signal defined by
𝒚𝒚 = 𝐴𝐴𝟏𝟏+𝒘𝒘𝒘𝒘~𝒩𝒩 0,𝜎𝜎2𝑰𝑰
Determine the Cramer-Rao Lower Bound.
18
-
Cramer-Rao Lower Bounds Example: DC signal in Gaussian noise Consider a length 𝑁𝑁 signal defined by
𝒚𝒚 = 𝐴𝐴𝟏𝟏+𝒘𝒘𝒘𝒘~𝒩𝒩 0,𝜎𝜎2𝑰𝑰
Determine the Cramer-Rao Lower Bound.
ln 𝑝𝑝 𝒙𝒙;𝐴𝐴 = − ln 2𝜋𝜋 𝑁𝑁𝜎𝜎2 − 12𝜎𝜎2
𝒙𝒙 − 𝐴𝐴𝟏𝟏 𝑇𝑇𝑰𝑰 𝒙𝒙− 𝐴𝐴𝟏𝟏
ln 𝑝𝑝 𝒙𝒙;𝐴𝐴 = − ln 2𝜋𝜋 𝑁𝑁𝜎𝜎2 − 12𝜎𝜎2
𝒙𝒙𝑇𝑇𝒙𝒙− 2𝐴𝐴𝒙𝒙𝑇𝑇𝟏𝟏+ 𝐴𝐴2𝟏𝟏𝑇𝑇𝟏𝟏
𝜕𝜕2 ln 𝑝𝑝 𝒙𝒙;𝐴𝐴
𝜕𝜕𝐴𝐴2= − 1
2𝜎𝜎22𝟏𝟏𝑇𝑇𝟏𝟏 = −𝑁𝑁/𝜎𝜎2
var �̂�𝐴 ≥ 1−𝐸𝐸 𝜕𝜕
2 ln 𝑝𝑝 𝒙𝒙;𝜃𝜃𝜕𝜕𝐴𝐴2
= 𝜎𝜎2
𝑁𝑁
19
-
Cramer-Rao Lower Bounds Example: DC signal in Gaussian noise Consider a length 𝑁𝑁 signal defined by
𝒚𝒚 = 𝐴𝐴+𝒘𝒘𝒘𝒘~𝒩𝒩 0,𝜎𝜎2𝑰𝑰
Consider the estimation
�̂�𝐴 =1𝑁𝑁�𝑛𝑛=1
𝑁𝑁
𝑦𝑦𝑛𝑛
Is this an efficient estimator of 𝐴𝐴?
20
-
Cramer-Rao Lower Bounds Example: DC signal in Gaussian noise Consider a length 𝑁𝑁 signal defined by
𝒚𝒚 = 𝐴𝐴+𝒘𝒘𝒘𝒘~𝒩𝒩 0,𝜎𝜎2𝑰𝑰
Consider the estimation
�̂�𝐴 =1𝑁𝑁�𝑛𝑛=1
𝑁𝑁
𝑦𝑦𝑛𝑛
Is this an efficient estimator of 𝐴𝐴?
var �̂�𝐴 = var1𝑁𝑁�𝑛𝑛=1
𝑁𝑁
𝑦𝑦𝑛𝑛 =1𝑁𝑁2
�𝑛𝑛=1
𝑁𝑁
var 𝑦𝑦𝑛𝑛 =1𝑁𝑁2
𝑁𝑁𝜎𝜎2 =𝜎𝜎2
𝑁𝑁
The variance matches the Cramer-Rao Bound, so the estimator is efficient
21
-
Cramer-Rao Lower Bounds Example: Two values with Gaussian noise Consider a length 2 signal defined by
𝒚𝒚 = 𝑨𝑨+𝒘𝒘 , 𝑨𝑨 = 𝐴𝐴1 𝐴𝐴2 𝑻𝑻
𝒘𝒘~𝒩𝒩 0,𝑪𝑪 , 𝑪𝑪 = 2 11 3𝜎𝜎2
5 Determine the Cramer-Rao Lower Bound.
22
-
Cramer-Rao Lower Bounds Example: Two values with Gaussian noise Consider a length 2 signal defined by
𝒚𝒚 = 𝑨𝑨+𝒘𝒘 , 𝑨𝑨 = 𝐴𝐴1 𝐴𝐴2 𝑻𝑻
𝒘𝒘~𝒩𝒩 0,𝑪𝑪 , 𝑪𝑪 = 2 11 31
5𝜎𝜎2, 𝑪𝑪−𝟏𝟏 =
1𝜎𝜎2
3 −1−1 2
Determine the Cramer-Rao Lower Bound.
ln 𝑝𝑝 𝒙𝒙;𝑨𝑨 = − ln 2𝜋𝜋 𝑪𝑪 − 12𝜎𝜎2
𝒙𝒙 − 𝑨𝑨 𝑇𝑇𝑪𝑪−𝟏𝟏 𝒙𝒙− 𝑨𝑨
ln 𝑝𝑝 𝒙𝒙;𝑨𝑨 = − ln 2𝜋𝜋 𝑪𝑪 − 12𝜎𝜎2
𝒙𝒙𝑇𝑇𝑪𝑪−1𝒙𝒙− 2𝐴𝐴𝒙𝒙𝑇𝑇𝑪𝑪−1𝑨𝑨+𝑨𝑨𝑇𝑇𝑪𝑪−1𝑨𝑨
𝜕𝜕2 ln 𝑝𝑝 𝒙𝒙;𝑨𝑨
𝜕𝜕𝐴𝐴12= −3/𝜎𝜎2
𝜕𝜕2 ln 𝑝𝑝 𝒙𝒙;𝑨𝑨
𝜕𝜕𝐴𝐴1𝜕𝜕𝐴𝐴2= 1
𝜎𝜎2
𝜕𝜕2 ln 𝑝𝑝 𝒙𝒙;𝑨𝑨
𝜕𝜕𝐴𝐴22= −2/𝜎𝜎2
3𝐴𝐴12 − 2𝐴𝐴1𝐴𝐴2 + 2𝐴𝐴22
𝑰𝑰 𝑨𝑨 = 3/𝜎𝜎2 −1/𝜎𝜎2
−1/𝜎𝜎2 2/𝜎𝜎2
𝑰𝑰−1 𝑨𝑨 = 2 11 3𝜎𝜎2
523
-
Cramer-Rao Lower Bounds Example: Two values with Gaussian noise Consider a signal defined by
𝒚𝒚 = 𝑨𝑨+𝒘𝒘
𝒘𝒘~𝒩𝒩 0,𝑪𝑪 , 𝑪𝑪 = 2 11 3𝜎𝜎2
5 Consider the estimator
�𝑨𝑨 = 𝒚𝒚
Is this an efficient estimator of 𝑨𝑨?
24
-
Cramer-Rao Lower Bounds Example: Two values with Gaussian noise Consider a signal defined by
𝒚𝒚 = 𝑨𝑨+𝒘𝒘
𝒘𝒘~𝒩𝒩 0,𝑪𝑪 , 𝑪𝑪 = 2 11 3𝜎𝜎2
5 Consider the estimator
�𝑨𝑨 = 𝒚𝒚
Is this an efficient estimator of 𝑨𝑨?
𝐂𝐂�̂�𝐴 = 𝐂𝐂𝒚𝒚 = 𝑪𝑪 =2 11 3
𝜎𝜎2
5The co-variance matrix matches the Cramer-Rao Bound, so the estimator is efficient
25
-
Cramer-Rao Lower Bounds and Linear Models
-
Cramer-Rao Lower Bounds and Linear Models Relationship between the score function and the MVUB estimator
Score function: 𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽
Regularity condition: E 𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽
= 0
Fisher Information: 𝑰𝑰 𝜽𝜽 = E 𝜕𝜕2 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽𝜕𝜕𝜽𝜽𝑻𝑻
Score function + efficient estimator: 𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽
= 𝑰𝑰 𝜽𝜽 𝒈𝒈 𝒙𝒙 − 𝜽𝜽
IF the score function can be represented in this way, then 𝒈𝒈 𝒙𝒙 is the MVUB estimator
27
-
Cramer-Rao Lower Bounds and Linear Models Consider the standard linear model signal of length N 𝒙𝒙 = 𝑯𝑯𝜽𝜽+𝒘𝒘 , 𝒘𝒘~𝒩𝒩 0,𝜎𝜎2𝑰𝑰
Determine the Cramer-Rao Lower Bound for 𝜽𝜽.
28
-
Cramer-Rao Lower Bounds and Linear Models Consider the standard linear model signal of length N 𝒙𝒙 = 𝑯𝑯𝜽𝜽+𝒘𝒘 , 𝒘𝒘~𝒩𝒩 0,𝜎𝜎2𝑰𝑰
Determine the Cramer-Rao Lower Bound for 𝜽𝜽.
𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽 =
𝜕𝜕𝜕𝜕𝜽𝜽 − ln 2𝜋𝜋𝜎𝜎
2 𝑁𝑁.2 −1
2𝜎𝜎2 𝒙𝒙 −𝑯𝑯𝜽𝜽𝑇𝑇 𝒙𝒙−𝑯𝑯𝜽𝜽
𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜃𝜃𝜕𝜕𝜽𝜽 =
12𝜎𝜎2
𝜕𝜕𝜕𝜕𝜽𝜽 𝒙𝒙
𝑇𝑇𝒙𝒙− 2𝒙𝒙𝑇𝑇𝑯𝑯𝜽𝜽+ 𝜽𝜽𝑇𝑇𝑯𝑯𝑇𝑇𝑯𝑯𝜽𝜽
𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽 =
12𝜎𝜎2
−2𝑯𝑯𝑇𝑇𝒙𝒙+ 2𝑯𝑯𝑇𝑇𝑯𝑯𝜽𝜽
𝜕𝜕2 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽𝜕𝜕𝜽𝜽𝑇𝑇 = 𝑰𝑰 𝜽𝜽 =
1𝜎𝜎2𝑯𝑯
𝑇𝑇𝑯𝑯
Score function: Gradient of the likelihood function
Hessian of the likelihood function
29
-
Cramer-Rao Lower Bounds and Linear Models Relationship between the score function and the MVUB estimator
Score function: 𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽
= 1𝜎𝜎2
−𝑯𝑯𝑇𝑇𝒙𝒙+𝑯𝑯𝑇𝑇𝑯𝑯𝜽𝜽
Regularity condition: E 𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽
= 0
Fisher Information: 𝑰𝑰 𝜽𝜽 = E 𝜕𝜕2 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽𝜕𝜕𝜽𝜽𝑻𝑻
= 1𝜎𝜎2𝑯𝑯𝑇𝑇𝑯𝑯
Score function + efficient estimator:
𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽 = 𝑰𝑰 𝜽𝜽 𝒈𝒈 𝒙𝒙 − 𝜽𝜽
𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽
=1𝜎𝜎2𝑯𝑯
𝑇𝑇𝑯𝑯 𝒈𝒈 𝒙𝒙 − 𝜽𝜽𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽
𝜕𝜕𝜽𝜽 =1𝜎𝜎2𝑯𝑯
𝑇𝑇𝑯𝑯 𝑯𝑯𝑇𝑇𝑯𝑯 −1𝑯𝑯𝑇𝑇𝒙𝒙− 𝜽𝜽
30
-
Cramer-Rao Lower Bounds and Linear Models The MVUB Estimator is therefore
�𝜽𝜽 𝒙𝒙 = 𝑯𝑯𝑇𝑇𝑯𝑯 −1𝑯𝑯𝑇𝑇𝒙𝒙 = 𝑯𝑯†𝒙𝒙
Question: What is this above? Have we seen it before?
31
-
Cramer-Rao Lower Bounds and Linear Models Consider the standard linear model signal of length N 𝒙𝒙 = 𝑯𝑯𝜽𝜽+𝒘𝒘 , 𝒘𝒘~𝒩𝒩 0,𝑪𝑪
Determine the Cramer-Rao Lower Bound for 𝜽𝜽.
32
-
Cramer-Rao Lower Bounds and Linear Models Consider the standard linear model signal of length N 𝒙𝒙 = 𝑯𝑯𝜽𝜽+𝒘𝒘 , 𝒘𝒘~𝒩𝒩 0,𝑪𝑪
Determine the Cramer-Rao Lower Bound for 𝜽𝜽.
𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽 =
𝜕𝜕𝜕𝜕𝜽𝜽 − ln 2𝜋𝜋𝜎𝜎
2 𝑁𝑁.2 −1
2𝜎𝜎2 𝒙𝒙 −𝑯𝑯𝜽𝜽𝑇𝑇𝑪𝑪−1 𝒙𝒙−𝑯𝑯𝜽𝜽
𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜃𝜃𝜕𝜕𝜽𝜽 =
12
𝜕𝜕𝜕𝜕𝜽𝜽 𝒙𝒙
𝑇𝑇𝒙𝒙− 2𝒙𝒙𝑇𝑇𝑪𝑪−1𝑯𝑯𝜽𝜽+ 𝜽𝜽𝑇𝑇𝑯𝑯𝑇𝑇𝑪𝑪−1𝑯𝑯𝜽𝜽
𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽 =
12−2𝑯𝑯𝑇𝑇𝑪𝑪−1𝒙𝒙+ 2𝑯𝑯𝑇𝑇𝑪𝑪−1𝑯𝑯𝜽𝜽
𝜕𝜕2 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽𝜕𝜕𝜽𝜽𝑇𝑇 = 𝑰𝑰 𝜽𝜽 = 𝑯𝑯
𝑇𝑇𝑪𝑪−1𝑯𝑯
Score function: Gradient of the likelihood function
Hessian of the likelihood function
33
-
Cramer-Rao Lower Bounds and Linear Models Relationship between the score function and the MVUB estimator
Score function: 𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽
= 12−2𝑯𝑯𝑇𝑇𝑪𝑪−1𝒙𝒙+ 2𝑯𝑯𝑇𝑇𝑪𝑪−1𝑯𝑯𝜽𝜽
Regularity condition: E 𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽
= 0
Fisher Information: 𝑰𝑰 𝜽𝜽 = E 𝜕𝜕2 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽𝜕𝜕𝜽𝜽𝑻𝑻
= 𝑯𝑯𝑇𝑇𝑪𝑪−1𝑯𝑯
Score function + efficient estimator:
𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽 = 𝑰𝑰 𝜽𝜽 𝒈𝒈 𝒙𝒙 − 𝜽𝜽
𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽 =
1𝜎𝜎2𝑯𝑯
𝑇𝑇𝑪𝑪−1𝑯𝑯 𝒈𝒈 𝒙𝒙 − 𝜽𝜽𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽
𝜕𝜕𝜽𝜽=
1𝜎𝜎2𝑯𝑯𝑇𝑇𝑪𝑪−1𝑯𝑯 𝑯𝑯𝑇𝑇𝑪𝑪−1𝑯𝑯 −1𝑯𝑯𝑇𝑇𝑪𝑪−1𝒙𝒙− 𝜽𝜽
34
-
Cramer-Rao Lower Bounds and Linear Models The MVUB Estimator is therefore
�𝜽𝜽 𝒙𝒙 = 𝑯𝑯𝑇𝑇𝑪𝑪−1𝑯𝑯 −1𝑯𝑯𝑇𝑇𝑪𝑪−1𝒙𝒙
Question: This is an interesting result. How are some ways we can interpret this?
35
-
Cramer-Rao Lower Bounds and Linear Models The MVUB Estimator is therefore
�𝜽𝜽 𝒙𝒙 = 𝑯𝑯𝑇𝑇𝑪𝑪−1𝑯𝑯 −1𝑯𝑯𝑇𝑇𝑪𝑪−1𝒙𝒙
Question: What is our problem when 𝑯𝑯 = 𝟏𝟏 = 1 1 … 1 𝑇𝑇?
36
-
Cramer-Rao Lower Bounds and Linear Models The MVUB Estimator is therefore
�𝜽𝜽 𝒙𝒙 = 𝑯𝑯𝑇𝑇𝑪𝑪−1𝑯𝑯 −1𝑯𝑯𝑇𝑇𝑪𝑪−1𝒙𝒙
Question: What is our problem when 𝑯𝑯 = 𝟏𝟏 = 1 1 … 1 𝑇𝑇? 𝒙𝒙 = 𝟏𝟏𝜃𝜃 = 𝒘𝒘
�̂�𝜃 𝒙𝒙 =𝟏𝟏𝑇𝑇𝑪𝑪−1𝒙𝒙𝟏𝟏𝑇𝑇𝑪𝑪−1𝟏𝟏
=𝟏𝟏𝑇𝑇𝑫𝑫𝑇𝑇𝑫𝑫𝒙𝒙𝟏𝟏𝑇𝑇𝑫𝑫𝑇𝑇𝑫𝑫𝟏𝟏
=𝑫𝑫𝟏𝟏 𝑇𝑇𝑫𝑫𝒙𝒙𝑫𝑫𝟏𝟏 𝑇𝑇𝑫𝑫𝟏𝟏
=𝑫𝑫𝟏𝟏 𝑇𝑇𝒙𝒙𝒙𝑫𝑫𝟏𝟏 𝑇𝑇𝑫𝑫𝟏𝟏
=𝒅𝒅𝑇𝑇𝒙𝒙𝒙𝒅𝒅 𝟐𝟐
Now a scalar value! Pre-whitenedSince C is symmetric, it has a square root
An averaging operation
37
VariationsLast TimeSufficient Statistics Sufficient Statistics Sufficient Statistics Slide Number 6Cramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsSlide Number 26Cramer-Rao Lower Bounds and Linear ModelsCramer-Rao Lower Bounds and Linear ModelsCramer-Rao Lower Bounds and Linear ModelsCramer-Rao Lower Bounds and Linear ModelsCramer-Rao Lower Bounds and Linear ModelsCramer-Rao Lower Bounds and Linear ModelsCramer-Rao Lower Bounds and Linear ModelsCramer-Rao Lower Bounds and Linear ModelsCramer-Rao Lower Bounds and Linear ModelsCramer-Rao Lower Bounds and Linear ModelsCramer-Rao Lower Bounds and Linear Models