chapter 4

Chapter 4

Multiple Regression

4.1 Introduction

• The errors are again due to measurement err

ors in y and errors in the specification of the relat

ionship between y and the x’s.

• We make the same assumptions about that

we made in Chapter 3. These are:

niuxxxy ikikiii ,....,2,1.......2211

iu

iu

4.1 Introduction

1.

2. for all i.

3. and are independent foe all .

4. and are independent foe all i and

j.

5. are normally distributed for all i .

0)( iuE

2)( iuV

iu ju ji

iu jx

iu

4.1 Introduction

6. There are no linear dependencies in the explanatory

variables, i.e., none of the explanatory variables can

be expressed as an exact linear function of the others.

(This assumption will be relaxed in Chapter 7.)

Also, it will be assumed that is a continuous

variables. (The case where it is observed as a dummy

variable or as a truncated variable will be discussed in

Chapter 8.)

iy

4.2 A Model with Two Explanatory Variables

•Consider the model

(4.1)

•The assumptions we have made about the error term u imply that

0),cov(0),cov(0)( 21 uxuxuE

niuxxy iiii ,....,2,12211


• Let , ,and be the estimators of , ,and ,

respectively.

• The sample counterpart of is the residual

• The three equations to determine , , and

are obtained by replacing the population

assumptions by their sample counterparts:

1 2 1 2

iu

iiii xxyu 2211ˆˆˆˆ

1 2


The Least Squares Method

• The least square method says that we should choose the estimators , , of , , so as to minimize

• Differentiate Q with respect to , , and and equate the derivatives to zero.

2

2211 )ˆˆˆ( iii xxyQ

1 2

21

1 2


• We get

)4.4(0)()ˆˆˆ(20ˆ

)3.4(0)()ˆˆˆ(20ˆ

)2.4(0)1()ˆˆˆ(20ˆ

22211

2

12211

1

2211

iiii

iiii

iii

xxxyQ

xxxyQ

xxyQ


• We can simplify this equation by the use of the following notation.

• Let us define

2222

2222

222212112

11121

2111

ynySxnxS

yxnyxSxxnxxS

yxnyxSxnxS

iyyi

iiyii

iiyi


•Now we can solve these two equations to get and . We get

(4.8)

Where .

•Once we obtain and we get from equation (4.5).We have

1 2

yy

yy

SSSS

SSSS

1122112

2121221

ˆ

ˆ

2122211 SSS

2211ˆˆˆ xxy

1 2


Thus the computational procedure is as follows:

1. Obtain all the means: , , .

2. Obtain all the sums of squares and sums of pr

oducts: , , ,and so on.

3. Obtain S11, S12 , S22 , S1y , S2y , and Syy.

4. Solve equations (4.7) and (4.8) to get and .

5. Substitute these in equation (4.5) to get .

y 1x 2x

21ix 2

2ix ii xx 21

1 2


)1()ˆvar(

)1()ˆvar(

21222

2

2

21211

2

1

rS

rS

)ˆ(var)ˆ,ˆcov(2)ˆ(var)ˆ(var 22221211

21

2

xxxxn


)1()ˆ,ˆcov(

21212

212

2

21 rS

r

)ˆ(var)ˆ,ˆ(cov)ˆ,ˆcov(

)ˆ,ˆ(cov)ˆ(var)ˆ,ˆ(cov

222112

212111

xx

xx


• If , then is an unbiased estimator for .

• If we substitute for in the expressions in result 2, we get the estimated variances and covariances.

• The square roots of the estimated variances are called the standard errors (denoted SE).

• Then

each have a t-distribution with d.f. ( n – 3 ).

An example

)ˆ(SE

ˆ

)ˆ(SE

ˆ

)ˆ(SE

ˆ

2

22

1

11

3)-(nRSSˆ 2 222 ˆor)ˆ( E2

2 2


• Note that the higher the value of (other things

staying the same), the higher the variances of

and .

• If is very high, we cannot estimate and

with much precision.

12r

1

2

12r 1 2


• In the case of simple regression we also defined

the following:

residual sum of squares =

explained sum of squares =

xyyy SS

xyS

yy

xyxy S

Sr

2


• The analogous expressions in multiple regression are

explained sum of squares =

yyyy SSS 2211ˆˆRSS

yy SS 2211ˆˆ

yy

yyy S

SSR 22112

12.

ˆˆ


• is called the coefficient of multiple determination and its positive square root is called the multiple correlation coefficient.

• The first subscript is the explained variable.

• The subscripts after the dot are the explanatory variables.

• To avoid cumbersome notation we have written 12 in

stead of x1x2.

• Since it is only x’s that have subscripts, there is no confusion in this notation.

212.yR

4.5 Partial Correlations and Multiple Correlation

• If we have explained variable y and three explanatory variables x1, x2, x3 and , , are the squares of

the simple correlations between y and x1, x2, x3,

respectively, then , , and measure the proportion of the variance in y that x1 alone, x2 alone, or

x3 alone explain.

• On the other hand, measures the proportion of the variance of y that x1, x2, x3 together explain.

• The relationship between simple and multiple correlations?

21yr 2

2yr 23yr

21yr 2

2yr 23yr

2123.yR


• We would also like to measure something else.

• For instance, how much does x2 explain after x1 is included in the regression equation?

• How much does x3 explain after x1 and x2 are included?

• These are measured by the partial coefficients of determination and , respectively.

• The variables after the dot are the variables already included.

21.2yr 2

12.3yr


• With three explanatory variables we have the following partial correlations:

These are called partial correlations of the first order.

• We also have three partial correlation coefficients of the second order:

• The variables after the dot are always the variables already included in the regress equation.

2.31.33.21.23.12.1 and,,,,, yyyyyy rrrrrr

12.313.223.1 and,, yyy rrr


• The order of partial correlation coefficient depends on the number of variables after the dot.

• The usual convention is to denote simple and partial correlations by a small r and multiple correlations by a capital R.

• For instance, are all coefficients of multiple determination (their positive square roots are multiple correlation coefficients.)

2123.

213.

212. and,, yyy RRR


• Partial correlations are very important in

deciding whether or not to include more

explanatory variables.

• For instance, suppose that we have two

explanatory variables x1 and x2 , and is very

high, say 0.95, but is very low, say 0.01.

• What this means is that if x2 alone is used to

explain y, it can do a good job.

22yr

21.2yr


• But after x1 is included, x2 does not help any more

explaining y; that is, x1 has done the job of x2 .

• In this case there is no use including x2.

• In face, we can have a situation where, for instance,

but96.0and95.0 2

221 yy rr

1.0and1.0 21.2

22.1 yy rr


• In this case each variable is highly correlated wit

h y but the partial correlations are both very low.

• This is called multicollinearity and we will discuss

this problem later in Chapter 7.

• In this example we can use x1 only or x2 only or s

ome combination of the two as an explanatory v

ariable.


• For instance, suppose that x1 is the amount of skilled

labor, x2 the amount of unskilled labor, and y the output.

• What the partial correlation coefficients suggest is that

the separation of total labor into two components --

skilled and unskilled -- does not help us much in

explaining output.

• So we might as well use x1 + x2 or total labor as the

explanatory variable.

Assignment

• The data from the teacher’s web site

• Calculate the following three types of correlation– Multiple correlation– Simple correlation– Partial correlation

4.9 Omission of Relevant Variables and Inclusion of Irrelevant Variables

• Until now we have assumed that the multiple regression equation we are estimating includes all the relevant explanatory variables.

• In practice, this is rarely the case.

• Sometimes some relevant variables are not included due to oversight or lack of measurements.

• At other times some irrelevant variables are included.

• What we would like to know is how our inferences change when these problems are present.


Omission of Relevant Variables

• Let us first consider the omission of relevant variables. Suppose that the true equation is

• Instead, we omit x2 and estimate the equation

• This will be referred to as the “misspecified model.”

• The estimate of we get is

)15.4(2211 uxxy

exy 11

1

21

11 x

yx


• Substituting the expression for y from equation (4.15) in this, we get

Since we get

Where is the regression coefficient from a regression of x2 on x1.

21

1

21

21212

1

221111

)(ˆx

ux

x

xx

x

uxxx

0)( 1 uxE

)16.4()ˆ( 22111 bE

212121 / xxxb


• Thus is a biased estimator for and the bias is given by

bias = ( coefficient of the excluded variable) ×

( regression coefficient in a regression of the excluded variable on the included variable)

• If we denote the estimator for from equation (4.15) by , the variance of is given by

where

11

1

1

~1

~

)1()~(var

21211

2

1 rS

2111 xS


• On the other hand,

• Thus is a biased estimator but has a smaller variance than .

• In fact, the variance would be considerably smaller if is high.

• However, the estimated standard error need not be smaller for than for .

111

2

1)ˆvar(S

1

212r

1 1

~


• This is because , the estimated variance of th

e error, can be higher in the misspecified model.

• It is given by the residual sum of squares divided

by degrees of freedom, and can be higher (or lo

wer) for the misspecified model.

2


Inclusion of Irrelevant Variables• Consider now the case of inclusion of irrelevant var

iables. Suppose that the true equation is

, but we estimate the equation

• The least squares estimators and from misspecified equation are given by

uxy 11

2211 xxy

1

~ 2

~

2122211

11221122

122211

2121221

~~

SSS

SSSS

SSS

SSSS yyyy


• The least squares estimators and from misspecified equation are given by

where ,and so on.

• Since we have

• Hence we get• Thus we get unbiased estimates for both the parameters.

1

~ 2

~

2122211

11221122

122211

2121221

~~

SSS

SSSS

SSS

SSSS yyyy

2112112111 ,, xxSyxSxS y

uxy 11

11111212 )(and)( SSESSE yy

0)~(and)

~( 211 EE


• This result, coupled with the earlier results regarding the bias introduced by the omission of relevant variables might lead us to believe that it is better to include variables (when in doubt) rather than exclude them.

• However, this is not so, because though the inclusion of irrelevant variables has no effect on the bias of the estimator, it does affect the variances.


• The variance of , the estimator of β1 from the correct equation is given by

• On the other hand, from the misspecified equation we have

where r12 is the correlation between x1 and x1 .

11

2

1)ˆ( SV

11212

2

1 )1()~(var

Sr

1


• Thus unless r12 =0.

• Hence we will be getting unbiased but inefficient estimates by including the irrelevant variable.

• An example: omit or Include Variables

)ˆvar()~

var( 11

chapter 4

Documents