part 7: estimating the variance of b 7-1/53 econometrics i professor william greene stern school of...

Part 7: Estimating the Variance of b7-1/53

Econometrics IProfessor William Greene

Stern School of Business

Department of Economics


Econometrics I

Part 7 – Estimating the Variance of b


Context The true variance of b|X is 2(XX)-1 . We

consider how to use the sample data to estimate this matrix. The ultimate objectives are to form interval estimates for regression slopes and to test hypotheses about them. Both require estimates of the variability of the distribution. We then examine a factor which affects how "large" this variance is, multicollinearity.


Estimating 2

Using the residuals instead of the disturbances:

The natural estimator: ee/N as a sample surrogate for /n

Imperfect observation of i, ei = i - ( - b)xi

Downward bias of ee/N. We obtain the result E[ee|X] = (N-K)2


Expectation of ee

1

1

( ' ) '

[ ( ' ) ']

( )

( '(

e y - Xb

y X X X X y

I X X X X y

My M X MX M M

e'e M M

'M'M 'MM 'M


Method 1:E[ ] E

E[ trace ( ) ] scalar = its trace

E[ trace ( ) ] permute in trace

[ trace E ( ) ] linear operators

e'e| X 'M | X

'M | X

M '| X

M '| X

2

2

2

[ trace E ( ) ] conditioned on X

[ trace ] model assumption

[trace ] scalar multiplication and matrix

trace [ - ( ) ]-1

M '| X

M I

M I

I X X'X X'

2

2

2

2

{trace [ ] - trace[ ( ) ]}

{N - trace[( ) ]} permute in trace

{N - trace[ ]}

{N - K}

Notice that E[ | ] is not a

-1

-1

I

e

X X'X X'

X'X X'X

e

I

X

function of .X


Estimating σ2

The unbiased estimator is s2 = ee/(N-K).

“Degrees of freedom correction”

Therefore, the unbiased estimator of 2 is

s2 = ee/(N-K)


Method 2: Some Matrix Algebra2E[ ] trace

What is the trace of ? is idempotent, so its

trace equals its rank. Its rank equals the number

of nonzero characeristic roots.

Characteric Roots :

Signature of a Matrix = Spectral

e'e| X M

M M

Decomposition

= Eigen (own) value Decomposition

= ' where

= a matrix of columns such that ' = ' =

= a diagonal matrix of the characteristic roots

element

A C C

C CC CC I

s of may be zero


Decomposing M

2 2

2 2

Useful Result: If = ' is the spectral

decomposition, then ' (just multiply)

= , so . All of the characteristic

roots of are 1 or 0. How many of each?

trace( ) = trace( ')=trace(

A C C

A C C

M M

M

A C C

' )=trace( )

Trace of a matrix equals the sum of its characteristic

roots. Since the roots of are all 1 or 0, its trace is

just the number of ones, which is N-K as we saw.

CC

M


Example: Characteristic Roots of a Correlation Matrix


6

1 ii

i iR = CΛC c c


Gasoline Data


X’X and its Roots


Var[b|X]

Estimating the Covariance Matrix for b|X

The true covariance matrix is 2 (X’X)-1

The natural estimator is s2(X’X)-1

“Standard errors” of the individual coefficients are the square roots of the diagonal elements.


X’X

(X’X)-1

s2(X’X)-1


Standard Regression Results----------------------------------------------------------------------Ordinary least squares regression ........LHS=G Mean = 226.09444 Standard deviation = 50.59182 Number of observs. = 36Model size Parameters = 7 Degrees of freedom = 29Residuals Sum of squares = 778.70227 Standard error of e = 5.18187 <= sqr[778.70227/(36 – 7)]Fit R-squared = .99131 Adjusted R-squared = .98951--------+-------------------------------------------------------------Variable| Coefficient Standard Error t-ratio P[|T|>t] Mean of X--------+-------------------------------------------------------------Constant| -7.73975 49.95915 -.155 .8780 PG| -15.3008*** 2.42171 -6.318 .0000 2.31661 Y| .02365*** .00779 3.037 .0050 9232.86 TREND| 4.14359** 1.91513 2.164 .0389 17.5000 PNC| 15.4387 15.21899 1.014 .3188 1.67078 PUC| -5.63438 5.02666 -1.121 .2715 2.34364 PPT| -12.4378** 5.20697 -2.389 .0236 2.74486--------+-------------------------------------------------------------


Bootstrapping

Some assumptions that underlie it - the sampling mechanism

Method:

1. Estimate using full sample: --> b

2. Repeat R times:

Draw N observations from the n, with replacement

Estimate with b(r).

3. Estimate variance with

V = (1/R)r [b(r) - b][b(r) - b]’


Bootstrap Applicationmatr;bboot=init(3,21,0.)$ Store results herename;x=one,y,pg$ Define Xregr;lhs=g;rhs=x$ Compute bcalc;i=0$ CounterProc Define procedureregr;lhs=g;rhs=x;quietly$ … Regressionmatr;{i=i+1};bboot(*,i)=b$... Store b(r)Endproc Ends procedureexec;n=20;bootstrap=b$ 20 bootstrap repsmatr;list;bboot' $ Display results


--------+-------------------------------------------------------------Variable| Coefficient Standard Error t-ratio P[|T|>t] Mean of X--------+-------------------------------------------------------------Constant| -79.7535*** 8.67255 -9.196 .0000 Y| .03692*** .00132 28.022 .0000 9232.86 PG| -15.1224*** 1.88034 -8.042 .0000 2.31661--------+-------------------------------------------------------------Completed 20 bootstrap iterations.----------------------------------------------------------------------Results of bootstrap estimation of model.Model has been reestimated 20 times.Means shown below are the means of thebootstrap estimates. Coefficients shownbelow are the original estimates basedon the full sample.bootstrap samples have 36 observations.--------+-------------------------------------------------------------Variable| Coefficient Standard Error b/St.Er. P[|Z|>z] Mean of X--------+------------------------------------------------------------- B001| -79.7535*** 8.35512 -9.545 .0000 -79.5329 B002| .03692*** .00133 27.773 .0000 .03682 B003| -15.1224*** 2.03503 -7.431 .0000 -14.7654--------+-------------------------------------------------------------

Results of Bootstrap Procedure


Bootstrap Replications

Full sample result

Bootstrapped sample results


OLS vs. Least Absolute Deviations----------------------------------------------------------------------Least absolute deviations estimator...............Residuals Sum of squares = 1537.58603 Standard error of e = 6.82594Fit R-squared = .98284--------+-------------------------------------------------------------Variable| Coefficient Standard Error b/St.Er. P[|Z|>z] Mean of X--------+------------------------------------------------------------- |Covariance matrix based on 50 replications.Constant| -84.0258*** 16.08614 -5.223 .0000 Y| .03784*** .00271 13.952 .0000 9232.86 PG| -17.0990*** 4.37160 -3.911 .0001 2.31661--------+-------------------------------------------------------------Ordinary least squares regression ............Residuals Sum of squares = 1472.79834 Standard error of e = 6.68059 Standard errors are based onFit R-squared = .98356 50 bootstrap replications--------+-------------------------------------------------------------Variable| Coefficient Standard Error t-ratio P[|T|>t] Mean of X--------+-------------------------------------------------------------Constant| -79.7535*** 8.67255 -9.196 .0000 Y| .03692*** .00132 28.022 .0000 9232.86 PG| -15.1224*** 1.88034 -8.042 .0000 2.31661--------+-------------------------------------------------------------


Quantile Regression:

Application of Bootstrap Estimation


Quantile Regression

Q(y|x,) = x, = quantile Estimated by linear programming Q(y|x,.50) = x, .50 median regression Median regression estimated by LAD (estimates

same parameters as mean regression if symmetric conditional distribution)

Why use quantile (median) regression? Semiparametric Robust to some extensions (heteroscedasticity?) Complete characterization of conditional distribution


Estimated Variance for Quantile Regression

Asymptotic Theory

Bootstrap – an ideal application


1 1

Model : , ( | , ) , [ , ] 0

ˆˆResiduals: u

1Asymptotic Variance:

= E[f (0) ] Estimated by

Asymptotic Theory Based Estimator of Variance of Q - REG

x | x

A C A

A xx

i i i i i i i i

i i i

u

y u Q y Q u

y

N

βx βx

-βx

1

.2

1 1 1ˆ1 | | B

B 2 Bandwidth B can be Silverman's Rule of Thumb:

ˆ ˆ( | .75) ( | .25)1.06 ,

1.349

(1- )(1- ) [ ] Estimated by

x x

C = xx

N

i i ii

i iu

uN

Q u Q uMin s

N

EN

12For =.5 and normally distributed u, this all simplifies to .2

But, this is an ideal application for bootstrapping

X

X

.

X

Xus


= .25

= .50

= .75


Multicollinearity

Not “short rank,” which is a deficiency in the model. A characteristic of the data set which affects the covariance matrix.

Regardless, is unbiased. Consider one of the unbiased coefficient estimators of k. E[bk] = k

Var[b] = 2(X’X)-1 . The variance of bk is the kth diagonal element of 2(X’X)-1 .We can isolate this with the result in your text. Let [X,z] be [Other xs, xk] = [X1,x2] (a convenient notation for the results in the text). We need the residual maker, MX.The general result is that the diagonal element we seek is [zM1z]-1 , which we know is the reciprocal of the sum of squared residuals

in the regression of z on X.


I have a sample of 24025 observations in a logit model. Two predictors are highly collinear (pairwaise corr .96; p<.001); vif are about 12 for eachof them; average vif is 2.63; condition number is 10.26; determinant of correlation matrix is 0.0211; the two lowest eigen vales are 0.0792 and 0.0427. Centering/standardizing variables does not change the story. Note: most obs are zeros for these two variables; I only have approx 600 non-zero obs for these two variables on a total of 24.025 obs.

Both variable coefficients are significant and must be included in the model (as per specification).

-- Do I have a problem of multicollinearity?? -- Does the large sample size attenuate this concern, even if I have a correlation of .96? -- What could I look at to ascertain that the consequences of multi-collinearity are not a problem?-- Is there any reference I might cite, to say that given the sample size, it is not a problem?

I hope you might help, because I am really in trouble!!!


Variance of Least Squares

1

2

22 1

Model : = + +

Variance of c

Variance of c is the lower right element of this matrix.

Var[c] = [ ] * *

where * = the vector of residuals from the

X

y Xβ z

b X X X z

z X z z

z M zz z

z

2 2n 2

ii 1

n2 2ii 1

22 1

n2 2ii 1

regression of on .

* *The R in that regression is R = 1 - , so

(z z)

* * 1 R (z z) . Therefore,

Var[c] = [ ] 1 R (z z)

z|X

X

z|X

z X

z z

z z

z M z

z|X


Multicollinearity

2

2 1n2 2

ii 1

Var[c] = [ ] 1 R (z z)

All else constant, the variance of the coefficient on rises as the fit

in the regression of on the other variables goes up. If the fit is

perfect, the

X

z|X

z M z

z

z

2

variance becomes infinite.

"Detecting" multicollinearity?

1Variance inflation factor: VIF(z) = .

1 R z|X


Gasoline Market

Regression Analysis: logG versus logIncome, logPG The regression equation islogG = - 0.468 + 0.966 logIncome - 0.169 logPGPredictor Coef SE Coef T PConstant -0.46772 0.08649 -5.41 0.000logIncome 0.96595 0.07529 12.83 0.000logPG -0.16949 0.03865 -4.38 0.000S = 0.0614287 R-Sq = 93.6% R-Sq(adj) = 93.4%Analysis of VarianceSource DF SS MS F PRegression 2 2.7237 1.3618 360.90 0.000Residual Error 49 0.1849 0.0038Total 51 2.9086


Gasoline MarketRegression Analysis: logG versus logIncome, logPG, ...

The regression equation islogG = - 0.558 + 1.29 logIncome - 0.0280 logPG - 0.156 logPNC + 0.029 logPUC - 0.183 logPPTPredictor Coef SE Coef T PConstant -0.5579 0.5808 -0.96 0.342logIncome 1.2861 0.1457 8.83 0.000logPG -0.02797 0.04338 -0.64 0.522logPNC -0.1558 0.2100 -0.74 0.462logPUC 0.0285 0.1020 0.28 0.781logPPT -0.1828 0.1191 -1.54 0.132S = 0.0499953 R-Sq = 96.0% R-Sq(adj) = 95.6%Analysis of VarianceSource DF SS MS F PRegression 5 2.79360 0.55872 223.53 0.000Residual Error 46 0.11498 0.00250Total 51 2.90858

The standard error on logIncome doubles when the three variables are added to the equation.


Condition Number and Variance Inflation Factors

Condition number larger than 30 is ‘large.’

What does this mean?


The Longley Data


NIST Longley Solution


Excel Longley Solution


The NIST Filipelli Problem


Certified Filipelli Results


Minitab Filipelli Results


Stata Filipelli Results


Even after dropping two (random columns), results are only correct to 1 or 2 digits.


Regression of x2 on all other variables


Using QR Decomposition


MulticollinearityThere is no “cure” for collinearity. Estimating something else is not helpful

(principal components, for example).

There are “measures” of multicollinearity, such as the condition number of X and the variance inflation factor.

Best approach: Be cognizant of it. Understand its implications for estimation.

What is better: Include a variable that causes collinearity, or drop the variable and suffer from a biased estimator?

Mean squared error would be the basis for comparison. Some generalities. Assuming X has full rank, regardless of the condition,

b is still unbiasedGauss-Markov still holds


Specification and Functional Form:Nonlinearity

2 21 2 3 4 1 2 3 4

2 3 2 3

22

Population Estimators

ˆ

[ | , ] ˆ2 2

ˆEstimator of the variance of

ˆ. [ ] [ ] 4

x x

x

x

y x x z y b b x b x b z

E y x zx b b x

x

EstVar Var b x Va

3 2 3[ ] 4 [ , ]r b xCov b b


Log Income Equation----------------------------------------------------------------------Ordinary least squares regression ............LHS=LOGY Mean = -1.15746 Estimated Cov[b1,b2] Standard deviation = .49149 Number of observs. = 27322Model size Parameters = 7 Degrees of freedom = 27315Residuals Sum of squares = 5462.03686 Standard error of e = .44717Fit R-squared = .17237--------+-------------------------------------------------------------Variable| Coefficient Standard Error b/St.Er. P[|Z|>z] Mean of X--------+------------------------------------------------------------- AGE| .06225*** .00213 29.189 .0000 43.5272 AGESQ| -.00074*** .242482D-04 -30.576 .0000 2022.99Constant| -3.19130*** .04567 -69.884 .0000 MARRIED| .32153*** .00703 45.767 .0000 .75869 HHKIDS| -.11134*** .00655 -17.002 .0000 .40272 FEMALE| -.00491 .00552 -.889 .3739 .47881 EDUC| .05542*** .00120 46.050 .0000 11.3202--------+-------------------------------------------------------------Average Age = 43.5272. Estimated Partial effect = .066225 – 2(.00074)43.5272 = .00018.Estimated Variance 4.54799e-6 + 4(43.5272)2(5.87973e-10) + 4(43.5272)(-5.1285e-8)= 7.4755086e-08. Estimated standard error = .00027341.


Specification and Functional Form:Interaction Effect

1 2 3 4 1 2 3 4

2 4 2 4

22

Population Estimators

ˆ

[ | , ] ˆz

ˆEstimator of the variance of

ˆ. [ ] [ ]

x x

x

x

y x z xz y b b x b z b xz

E y x zb b z

x

EstVar Var b z Va

4 2 4[ ] 2 [ , ]r b zCov b b


Interaction Effect----------------------------------------------------------------------Ordinary least squares regression ............LHS=LOGY Mean = -1.15746 Standard deviation = .49149 Number of observs. = 27322Model size Parameters = 4 Degrees of freedom = 27318Residuals Sum of squares = 6540.45988 Standard error of e = .48931Fit R-squared = .00896 Adjusted R-squared = .00885Model test F[ 3, 27318] (prob) = 82.4(.0000)--------+-------------------------------------------------------------Variable| Coefficient Standard Error b/St.Er. P[|Z|>z] Mean of X--------+-------------------------------------------------------------Constant| -1.22592*** .01605 -76.376 .0000 AGE| .00227*** .00036 6.240 .0000 43.5272 FEMALE| .21239*** .02363 8.987 .0000 .47881 AGE_FEM| -.00620*** .00052 -11.819 .0000 21.2960--------+-------------------------------------------------------------Do women earn more than men (in this sample?) The +.21239 coefficient on FEMALE wouldsuggest so. But, the female “difference” is +.21239 - .00620*Age. At average Age, theeffect is .21239 - .00620(43.5272) = -.05748.

part 7: estimating the variance of b 7-1/53 econometrics i professor william greene stern school of...

Documents

variance of b7

variance of bpart

true variance of bx

unbiased estimator

standard regression

natural estimator

correlation matrix

matrix algebra