Econometrics

1

Lecture 3: Properties of the OLS Estimator For the OLS estimator yXXXb ')'( 1−= define:

• Vector of OLS residuals

⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢

⎣

⎡

−

−

=−=

∑

∑

=

=

K

kknkn

K

kkk

bxy

bxy

Xbye

1

111

Econometrics

2

• Vector of OLS fitted or predicted values

⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢

⎣

⎡

==

∑

∑

=

=

K

kknk

K

kkk

bx

bx

Xby

1

11

ˆ

Econometrics

3

Residuals and fitted values have properties

a. 0' =eX

b. If the regression has an intercept

bxy '= with

⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢

⎣

⎡=

=

∑

∑

=

=

n

iiK

n

ii

xn

xn

x

1

11

1

)1(1

c. If the regression has an intercept

yy ˆ=

Econometrics

4

d. If the regression has an intercept

∑∑ ∑== =

+−=−n

ii

n

i

n

iii eyyyy

1

22

1 1

2 )ˆ()(

Econometrics

5

Proofs and remarks: Ad a. 0' =eX Proof: 0'')('' =−=−= XbXyXXbyXeX This implies that each column of X is orthogonal to vector e , i.e. for

Kk ,,1…=

∑=

=n

iiik ex

10

If regression has intercept (and hence X has column equal to ι , a vector of 1-s), then

∑=

===n

ii eee

10'ι

Econometrics

6

The fundamental assumption in CLR model 0)|( =XE ε implies that 0)'( =εXE Hence OLS makes sample analog of )'( εXE equal to 0.

Econometrics

7

Note MyyXXXXIXbye =−=−= − )')'(( 1 with ')'( 1 XXXXIM −−= The matrix M has a number of properties 0=MX

MM =' M is symmetric MM =2 M is idempotent

Econometrics

8

Also PyyXXXXXby === − ')'(ˆ 1 with ')'( 1 XXXXP −= Note

PP =' P is symmetric PP =2 P is idempotent Because Xby =ˆ the matrix P projects yon the space spanned by the columns of X . For this reason P is called the least squares projection matrix. Projection matrices are idempotent, because if we project again nothing changes.

Econometrics

9

Because 0' =eX and M is also a projection matrix, e is the projection of y on space orthogonal to the columns of X . Note eyy += ˆ Also 0'''ˆ == eXbey i.e. y and e are orthogonal and y is the sum of two orthogonal vectors.

Econometrics

10

In graph (one observation with 2xxyb = )

y

bxy =ˆ

bxye −=

Econometrics

11

Ad b: bxy '= Proof: If we write [ ]KxxX 1ι= with kx the −k th column of X , then the normal equations can be written as

y

x

xXb

x

x

KK⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

′

′=

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

′

′ 22

'' ιι

Econometrics

12

The first equation is yXb '' ιι = or

yn

Xbn

'1'1 ιι =

or ybx ='

Econometrics

13

Ad c: yy ˆ= Proof: Because eyy += ˆ we have

en

yn

yn

'1ˆ'1'1 ιιι +=

or

yy =ˆ In words: sample average of y is predicted exactly.

Econometrics

14

Ad d: ∑∑ ∑== =

+−=−n

ii

n

i

n

iii eyyyy

1

22

1 1

2 )ˆ()(

Proof: Because iii eyyyy +−=− ˆ , we have

∑ ∑∑∑= ===

−++−=−n

i

n

iiii

n

ii

n

ii yyeeyyyy

1 1

2

1

2

1

2 )ˆ(2)ˆ()(

The last term is 0.

Econometrics

15

Terminology

∑=

−n

ii yy

1

2)( Total Sum of Squares (TSS)

∑=

−n

ii yy

1

2)ˆ( Explained Sum of Squares (ESS)

∑=

n

iie

1

2 Residual Sum of Squares (RSS)

Hence RSSESSTSS +=

Econometrics

16

If we divide by TSS

TSSRSS

TSSESS

−= 1

Coefficient of determination

∑

∑

=

=

−

−== n

ii

n

ii

yy

yyR

1

2

1

2

2

)(

)ˆ(

TSSESS

This is a measure of goodness-of-fit.

Econometrics

17

Note 10 2 ≤≤ R For the extreme values

XbynieeR in

ii =⇔==⇔=⇔= ∑

=,,1,001

1

22 …

i.e. a perfect fit.

0,

ˆ0)ˆ(0

21

1

222

====⇔

⇔=⇔=−⇔= ∑=

K

in

ii

bbyb

yyyyR

i.e. varying regressors Kxx ,,2 … do not help to explain y .

Econometrics

18

Because ∑∑==

−−=−n

iii

n

ii yyyyyy

11

2 ))(ˆ()ˆ(

2

1

2

1

2

1

1

2

1

2

2

1

2

1

2

1

2

2

)ˆ()(

))(ˆ(

)ˆ()(

)ˆ(

)(

)ˆ(

∑∑

∑

∑∑

∑

∑

∑

==

=

==

=

=

=

−−

⎟⎟⎠

⎞⎜⎜⎝

⎛−−

=

=−−

⎟⎟⎠

⎞⎜⎜⎝

⎛−

=−

−=

n

ii

n

ii

n

iii

n

ii

n

ii

n

ii

n

ii

n

ii

yyyy

yyyy

yyyy

yy

yy

yyR

The final expression is the sample correlation of y and y .

Econometrics

19

Note in 2R we use deviations from sample average, because

∑

∑

=

=n

ii

n

ii

y

y

1

2

1

2ˆ

can be made close to 1 by adding a constant to y .

Econometrics

20

Partitioned regression Consider uXy += β Partition X as

[ ]

21

21

KnKnKnXXX

×××=

2

1

2

1

KK

⎥⎦

⎤⎢⎣

⎡=

ββ

β

with KKK =+ 21 . Hence we can write εββ ++= 2211 XXy

Econometrics

21

What is the formula for the OLS estimator of 1β ? Define (compare with M defined previously) 2

12222 )( XXXXIM ′′−= −

2M is symmetric and idempotent ( 2

22 MM = ).

Further define **

222 ybXyyM =−= with yXXXb 2

122

*2 )( ′′= −

In words: *y is vector of OLS residuals of regression of y on 2X .

Econometrics

22

In the same way define 12

*1 XMX =

In words: the −k th column of *

1X is the vectors of OLS residuals of the −k th column of 1X on 2X .

Hence

**

11*

1*11 ')'( yXXXb −=

In words: the OLS estimator in a regression on 1X and 2X , 1b , is the OLS estimator of the OLS residuals of y (in regression on 2X ) on the OLS residuals of 1X (in regression on 2X ).

Econometrics

23

Note *1

*, Xy are purged of the effect of 2X , because OLS residuals are orthogonal to /uncorrelated with 2X . This is how least squares implements the ceteris paribus condition, i.e. 1b is the effect of 1X ‘holding 2X constant’. More like randomized assignment of 1X : *

1X is uncorrelated with the ‘omitted’ 2X , which is relegated to random error if only 1X is included.

Econometrics

24

Special cases

• Columns of 1X and those of 2X are orthogonal/uncorrelated 012 =′ XX , so that 112 XXM = and

yXXXb 1

1111 )( ′′= −

Conclusion: if we omit 2X we get the same estimate.

• ι=2X with ι an −n vector of 1’s. Then

nIIM ')( 1

2ιιιιιι −=′′−= −

yynyyyM ιιι

−=′

−=2

In words: 2M takes y in deviation from its sample average.

Econometrics

25

Conclusion: if we take both y and X in deviation from the sample means we obtain OLS estimators of y on varying regressors.

Econometrics

26

Estimation of 2σ The other parameter in CLR model is the variance of the random error 2σ . Obvious idea: use sample variance of the OLS residuals e . Instead we use (in model with K regressors including constant)

eeKn

eKn

sn

ii ′

−=

−= ∑

=1

22 11

Econometrics

27

The sampling distribution of 2s can be derived from εεβ MXMMye =+== )( so that

εεMKn

s−

=12

It can be shown that 22 )|( σ=XsE so that ( ) 222 )|()( σ== XsEEsE X In words: 2s is an unbiased estimator of 2σ .