point estimation notes of stat 6205 by dr. fan. overview section 6.1 point estimation maximum...

Point Point EstimationEstimation

Notes of STAT 6205 by Dr. Fan

OverviewOverview• Section 6.1• Point estimation• Maximum likelihood estimation• Methods of moments• Sufficient statistics

o Definitiono Exponential familyo Mean square error (how to choose an estimator)

6205-Ch6 2

Big Picture• Goal: To study the unknown distribution of a

population• Method: Get a representative/random sample and

use the information obtained in the sample to make statistical inference on the unknown features of the distribution

• Statistical Inference has two parts: o Estimation (of parameters)o Hypothesis testing

• Estimation:o Point estimation: use a single value to estimate a parametero Interval estimation: find an interval covering the unknown parameter

6205-Ch6 3

Point EstimatorPoint Estimator• Biased/unbiased: an estimator is called unbiased

if its mean is equal to the parameter of estimate; otherwise, it is biased

• Example: X_bar is unbiased for estimating mu

6205-Ch6 4

Maximum Likelihood Estimation

• Given a random sample X1, X2, …, Xn from a distribution f(x; ) where is a (unknown) value in the parameter space

• Likelihood function vs. joint pdf

• Maximum Likelihood Estimator (m.l.e.) of , denoted as is the value which maximizes the likelihood function, given the sample X1, X2, …, Xn.

n

iixfxL

1

);();(

6205-Ch6 5

Examples/Exercises• Problem 1: To estimate p, the true probability of heads

up for a given coin.• Problem 2: Let X1, X2, …, Xn be a random sample

from a Exp(mu) distribution. Find the m.l.e. of mu.• Problem 3: Let X1, X2, …, Xn be a random sample

from a Weibull(a=3,b) distribution. Find the m.l.e. of b.

• Problem 4: Let X1, X2, …, Xn be a random sample from a N(,^2) distribution. Find the m.l.e. of and .

• Problem 5: Let X1, X2, …, Xn be a random sample from a Weibull(a,b) distribution. Find the m.l.e. of a and b.

6205-Ch6 6

Method of MomentsMethod of Moments• Idea: Set population moments = sample

moments and solve for parameters

• Formula: When the parameter is r-dimensional, solve the following equations for :

6205-Ch6 7

n

i

ki

k ,...r,knXXE1

21for /)(

Examples/ExercisesExamples/ExercisesGiven a random sample from a population

•Problem 1: Find the m.m.e. of for a Exp() population.

•Exercise 1: Find the m.m.e. of and for a N(^2) population.

6205-Ch6 8

Sufficient StatisticsSufficient Statistics• Idea: The “sufficient” statistic contains all

information about the unknown parameter; no other statistic can provide additional information as to the unknown parameter.

• If for any event A, P[A|Y=y] does not depend on the unknown parameter, then the statistic Y is called “sufficient” (for the unknown parameter).

• Any one-to-one mapping of a sufficient statistic Y is also sufficient.

• Sufficient statistics do not need to be estimators of the parameter.

6205-Ch6 9

Sufficient StatisticsSufficient Statistics

6205-Ch6 10

Examples/ExercisesExamples/ExercisesLet X1, X2, …, Xn be a random sample from f(x)

Problem: Let f be Poisson(a). Prove that1.X-bar is sufficient for the parameter a2.The m.l.e. of a is a function of the sufficient statistic

Exercise: Let f be Bin(n, p). Prove that X-bar is sufficient for p (n is known). Hint: find a sufficient statistic Y for p and then show that X-bar is a function of Y

6205-Ch6 11

Exponential FamilyExponential Family

6205-Ch6 12

Examples/ExercisesExamples/Exercises

Example 1: Find a sufficient statistic for p for Bin(n, p)

Example 2: Find a sufficient statistic for a for Poisson(a)

Exercise: Find a sufficient statistic for for Exp()

6205-Ch6 13

Joint Sufficient StatisticsJoint Sufficient Statistics

Example: Prove that X-bar and S^2 are joint sufficient statistics for and of N(, ^2)

6205-Ch6 14

Application of SufficienceApplication of Sufficience

6205-Ch6 15

ExampleExampleConsider a Weibull distribution with parameter(a=2, b)

1)Find a sufficient statistic for b

2)Find an unbiased estimator which is a function of the sufficient statistic found in 1)

6205-Ch6 16

Good Estimator?Good Estimator?• Criterion: mean square error

6205-Ch6 17

ExampleExample• Which of the following two estimator of variance

is better?

6205-Ch6 18