© 2002 prentice-hall, inc.chap 14-1 introduction to multiple regression model

© 2002 Prentice-Hall, Inc. Chap 14-1

Introduction to Multiple Regression Model


Chapter Topics Multiple linear regression (MLR) model Residual analysis Influence analysis Testing for the significance of the

regression model Inferences on the population regression

coefficients Testing portions of the multiple

regression model


0 1 1 2 2i i i k ki iY b b X b X b X e

Population Y-intercept

Population slopesRandom Error

Multiple Linear Regression Model

A relationship between one dependent and two or more independent variables is a

linear function

Dependent (Response) variable for sample

Independent (Explanatory) variables for sample model

1 2i i i k ki iY X X X

Residual


Population Multiple Regression Model

X2

Y

X1YX = 0 + 1X 1i + 2X 2i

0

Y i = 0 + 1X 1i + 2X 2i + i

ResponsePlane

(X 1i,X 2i)

(O bserved Y )

i

X2

Y

X1YX = 0 + 1X 1i + 2X 2i

0

Y i = 0 + 1X 1i + 2X 2i + i

ResponsePlane

(X 1i,X 2i)

(O bserved Y )

i

Bivariate model


Sample Multiple Regression Model

X2

Y

X1

b0

Y i = b0 + b1X 1 i + b2X 2 i + e i

ResponsePlane

(X 1i, X 2i)

(O bserved Y)

^

e i

Y i = b0 + b1X 1 i + b2X 2 i

X2

Y

X1

b0

Y i = b0 + b1X 1 i + b2X 2 i + e i

ResponsePlane

(X 1i, X 2i)

(O bserved Y)

^

e i

Y i = b0 + b1X 1 i + b2X 2 i

Bivariate model

Sample Regression PlaneSample Regression Plane


Simple and Multiple LinearRegression Compared:

Example

Two simple regressions:

Multiple regression:

0 1

0 1

Oil Temp

Oil Insulation

0 1 2Oil Temp Insulation


Multiple Linear Regression Equation

Too complicated

by hand! Ouch!


Interpretation of Estimated Coefficients

Slope (bi) Estimated that the average value of Y changes

by bi for each one unit increase in Xi holding all other variables constant (ceterus paribus)

Example: if b1 = -2, then fuel oil usage (Y) is expected to decrease by an estimated two gallons for each one degree increase in temperature (X1) given the inches of insulation (X2)

Y-intercept (b0) The estimated average value of Y when all Xi = 0


Multiple Regression Model: Example

Oil (Gal) Temp Insulation275.30 40 3363.80 27 3164.30 40 1040.80 73 694.30 64 6

230.90 34 6366.70 9 6300.60 8 10237.80 23 10121.40 63 331.40 65 10

203.50 41 6441.10 21 3323.00 38 352.50 58 10

(0F)

Develop a model for estimating heating oil used for a single family home in the month of January based on average temperature and amount of insulation in inches.


1 2ˆ 562.151 5.437 20.012i i iY X X

Sample Multiple Regression Equation: Example

CoefficientsIntercept 562.1510092X Variable 1 -5.436580588X Variable 2 -20.01232067

Excel Output

For each degree increase in temperature, the estimated average amount of heating oil used is decreased by 5.437 gallons, holding insulation constant.

For each increase in one inch of insulation, the estimated average use of heating oil is decreased by 20.012 gallons, holding temperature constant.

0 1 1 2 2i i i k kiY b b X b X b X


Venn Diagrams and Explanatory Power of

Regression

Oil

Temp

Variations in oil explained by temp or variations in temp used in explaining variation in oil

Variations in oil explained by the error term

Variations in temp not used in explaining variation in Oil

SSE

SSR



Regression

Oil

Temp

2

r

SSR

SSR SSE

(continued)



Regression

Oil

TempInsulation

Overlapping Overlapping variation in both Temp and Insulation are used in explaining the variationvariation in Oil but NOTNOT in the estimationestimation of nor

12

Variation NOTNOT explained by Temp nor Insulation SSE


Coefficient of Multiple Determination

Proportion of total variation in Y explained by all X variables taken together

Never decreases when a new X variable is added to model Disadvantage when comparing models

212

Explained Variation

Total VariationY k

SSRr

SST



Regression

Oil

TempInsulation

212

Yr

SSR

SSR SSE


Adjusted Coefficient of Multiple Determination

Proportion of variation in Y explained by all X variables adjusted for the number of X variables used

Penalize excessive use of independent variables

Smaller than Useful in comparing among models

2 212

11 1

1adj Y k

nr r

n k

212Y kr


Coefficient of Multiple Determination

Regression StatisticsMultiple R 0.982654757R Square 0.965610371Adjusted R Square 0.959878766Standard Error 26.01378323Observations 15

Excel Output

SST

SSRr ,Y 2

12

Adjusted r2

reflects the number of explanatory variables and sample size

is smaller than r2


Interpretation of Coefficient of Multiple Determination

96.56% of the total variation in heating oil can be explained by difference in temperature and amount of insulation

95.99% of the total fluctuation in heating oil can be explained by difference in temperature and amount of insulation after adjusting for the number of explanatory variables and sample size

2,12 .9656Y

SSRr

SST

2adj .9599r


Using The Model to Make Predictions

Predict the amount of heating oil used for a home if the average temperature is 300 and the insulation is six inches.

The predicted heating oil used is 278.97 gallons

1 2

ˆ 562.151 5.437 20.012

562.151 5.437 30 20.012 6

278.969

i i iY X X


Residual Plots

Residuals vs. May need to transform Y variable

Residuals vs. May need to transform variable

Residuals vs. May need to transform variable

Residuals vs. time May have autocorrelation

Y

1X

2X1X

2X


Residual Plots: Example

Insulation Residual Plot

0 2 4 6 8 10 12

No discernable pattern

Temperature Residual Plot

-60

-40

-20

0

20

40

60

0 20 40 60 80

Re

sid

ua

ls

May be some non-linear relationship


Influence Analysis To determine observations that have

influential effect on the fitted model Potentially influential points become

candidates for removal from the model Criteria used are

The hat matrix elements hi

The Studentized deleted residuals ti*

Cook’s distance statistic Di

All three criteria are complementary Only when all three criteria provide

consistent results should an observation be removed


The Hat Matrix Element hi

If , Xi is an Influential Point Xi may be considered a candidate for

removal from the model

2

2

1

1 i

i n

ii

X Xh

n X X

2 1 /ih k n


The Hat Matrix Element hi :Heating Oil Example

Oil (Gal) Temp Insulation h i

275.30 40 3 0.1567363.80 27 3 0.1852164.30 40 10 0.175740.80 73 6 0.246794.30 64 6 0.1618

230.90 34 6 0.0741366.70 9 6 0.2306300.60 8 10 0.3521237.80 23 10 0.2268121.40 63 3 0.244631.40 65 10 0.2759

203.50 41 6 0.0676441.10 21 3 0.2174323.00 38 3 0.157452.50 58 10 0.2268

No hi > 0.4 No observation appears to be a candidate for removal from the model

15 2

2 1 / 0.4

n k

k n


The Studentized Deleted Residuals ti

*

: difference between the observed and predicted based on a model that includes all observations except observation i

: standard error of the estimate for a model that includes all observations except observation i

An observation is considered influential if is the critical value of a two-tail test at a

alpha level of significance

* 1

i

i

ii

et

S h

iY

iY

iS

ie

*2i n kt t

2n kt


The Studentized Deleted Residuals ti

* :ExampleOil (Gal) Temp Insulation t i

*

275.30 40 3 -0.3772363.80 27 3 0.3474164.30 40 10 0.824340.80 73 6 -0.187194.30 64 6 0.0066

230.90 34 6 -1.0571366.70 9 6 -1.1776300.60 8 10 -0.8464237.80 23 10 0.0341121.40 63 3 -1.853631.40 65 10 1.0304

203.50 41 6 -0.6075441.10 21 3 2.9674323.00 38 3 1.168152.50 58 10 0.2432

2 11

15 2

1.7957n k

n k

t t

t10* and t13

* are influential points for potential removal from the model

*10t

*13t


Cook’s Distance Statistic Di

is the Studentized residual

If , an observation is considered influential

is the critical value of the F distribution at a 50% level of significance

2

2 1i i

ii

SR hD

h

1i

i

YX i

eSR

S h

1, 1i k n kD F

1, 1k n kF


Cook’s Distance Statistic Di : Heating Oil Example

Oil (Gal) Temp Insulation D i

275.30 40 3 0.0094363.80 27 3 0.0098164.30 40 10 0.049640.80 73 6 0.004194.30 64 6 0.0001

230.90 34 6 0.0295366.70 9 6 0.1342300.60 8 10 0.1328237.80 23 10 0.0001121.40 63 3 0.308331.40 65 10 0.1342

203.50 41 6 0.0094441.10 21 3 0.4941323.00 38 3 0.082452.50 58 10 0.0062

No Di > 0.835 No observation appears to be candidate for removal from the modelUsing the three criteria, there is insufficient evidence for the removal of any observation from the model

1, 1 3,12

15 2

0.835k n k

n k

F F


Testing for Overall Significance

Show if there is a linear relationship between all of the X variables together and Y

Use F test statistic Hypotheses:

H0: …k = 0 (no linear relationship) H1: at least one i ( at least one independent

variable affects Y ) The null hypothesis is a very strong statement Almost always reject the null hypothesis


Testing for Overall Significance

Test statistic:

where F has p numerator and (n-p-1) denominator degrees of freedom

(continued)

all /

all

SSR pMSRF

MSE MSE


Test for Overall SignificanceExcel Output: Example

ANOVAdf SS MS F Significance F

Regression 2 228014.6 114007.3 168.4712 1.65411E-09Residual 12 8120.603 676.7169Total 14 236135.2

p = 2, the number of explanatory variables n - 1

p value

Test StatisticMSR

FMSE


Test for Overall SignificanceExample Solution

F0 3.89

H0: 1 = 2 = … = p = 0

H1: At least one i 0 = .05df = 2 and 12

Critical Value(s):

Test statistic:

Decision:

Conclusion:

Reject at = 0.05

There is evidence that at least one independent variable affects Y

= 0.05

F 168.47(Excel Output)


Test for Significance:Individual Variables

Show whether there is a linear relationship between the variable Xi and Y

Use t Test Statistic Hypotheses:

H0: i 0 (No linear relationship) H1: i 0 (Linear relationship between Xi and

Y)


t Test StatisticExcel Output: Example

Coefficients Standard Error t StatIntercept 562.1510092 21.09310433 26.65093769X Variable 1 -5.436580588 0.336216167 -16.16989642X Variable 2 -20.01232067 2.342505227 -8.543127434

t Test Statistic for X1 (Temperature)

t Test Statistic for X2 (Insulation)

i

i

b

bt

S


t Test : Example Solution

H0: 1 = 0

H1: 1 0

df = 12

Critical Value(s):

Test Statistic:

Decision:

Conclusion:

Reject H0 at = 0.05

There is evidence of a significant effect of temperature on oil consumption.

t0 2.1788-2.1788

.025

Reject H0 Reject H0

.025

Does temperature have a significant effect on monthly consumption of heating oil? Test at = 0.05.

t Test Statistic = -16.1699


Venn Diagrams and Estimation of Regression

Model

Oil

TempInsulation

Only this information is used in the estimation of 2

Only this information is used in the estimation of

1This information is NOT used in the estimation of nor1 2


Confidence Interval Estimate

for the Slope

Provide the 95% confidence interval for the population slope 1 (the effect of temperature on oil consumption).

11 1n p bb t S

Coefficients Lower 95% Upper 95%Intercept 562.151009 516.1930837 608.108935X Variable 1 -5.4365806 -6.169132673 -4.7040285X Variable 2 -20.012321 -25.11620102 -14.90844

-6.169 1 -4.704

The estimated average consumption of oil is reduced by between 4.7 gallons to 6.17 gallons per each increase of 10 F.


Contribution of a Single Independent Variable

Let Xk be the independent variable of interest

Measures the contribution of Xk in explaining the total variation in Y (SST)

kX

| all others except

all all others except

k k

k

SSR X X

SSR SSR X


Contribution of a Single Independent Variable kX

1 2 3

1 2 3 2 3

| and

, and and

SSR X X X

SSR X X X SSR X X

Measures the contribution of in explaining SST

1X

From ANOVA section of regression for


0 1 1 2 2 3 3i i i iY b b X b X b X 0 2 2 3 3i i iY b b X b X


Coefficient of Partial Determination of

Measures the proportion of variation in the dependent variable that is explained by Xk while controlling for (holding constant) the other independent variables

2 all others

| all others

all | all others

Yk

k

k

r

SSR X

SST SSR SSR X

kX


Coefficient of Partial Determination for kX

(continued)

1 221 2

1 2 1 2

|

, |Y

SSR X Xr

SST SSR X X SSR X X

Example: Two Independent Variable Model


Venn Diagrams and Coefficient of Partial Determination forkX

Oil

TempInsulation

1 2|SSR X X

21 2

1 2

1 2 1 2

|

, |

Yr

SSR X X

SST SSR X X SSR X X

=


Contribution of a Subset of Independent Variables

Let Xs be the subset of independent variables of interest

Measures the contribution of the subset xs in explaining SST

| all others except

all all others except

s s

s

SSR X X

SSR SSR X


Contribution of a Subset of Independent Variables:

Example

Let Xs be X1 and X3

1 3 2

1 2 3 2

and |

, and

SSR X X X

SSR X X X SSR X



0 1 1 2 2 3 3i i i iY b b X b X b X 0 2 2i iY b b X


Testing Portions of Model

Examines the contribution of a subset Xs of explanatory variables to the relationship with Y

Null hypothesis: Variables in the subset do not

significantly improve the model when all other variables are included

Alternative hypothesis: At least one variable is significant


Testing Portions of Model

Always one-tailed rejection region Requires comparison of two regressions

One regression includes everything Another regression includes everything

except the portion to be tested

(continued)


Partial F Test For Contribution of Subset of X

variables Hypotheses:

H0 : Variables Xs do not significantly improve the model given all others variables included

H1 : Variables Xs significantly improve the model given all others included

Test Statistic:

with df = m and (n-p-1) m = # of variables in the subset Xs

| all others /

allsSSR X m

FMSE


Partial F Test For Contribution of A Single

Hypotheses: H0 : Variable Xj does not significantly improve

the model given all others included H1 : Variable Xj significantly improves the

model given all others included Test Statistic:

With df = 1 and (n-p-1) m = 1 here

jX

| all others

alljSSR X

FMSE


Testing Portions of Model: Example

Test at the = .05 level to determine whether the variable of average temperature significantly improves the model given that insulation is included.


Testing Portions of Model: Example

H0: X1 (temperature) does not improve model with X2 (insulation) included

H1: X1 does improve model

= .05, df = 1 and 12

Critical Value = 4.75

ANOVASS

Regression 51076.47Residual 185058.8Total 236135.2

ANOVASS MS

Regression 228014.6263 114007.313Residual 8120.603016 676.716918Total 236135.2293

(For X1 and X2) (For X2)

Conclusion: Reject H0; X1 does improve model

1 2

1 2

| 228,015 51,076261.47

, 676.717

SSR X XF

MSE X X


When to Use the F test

The F test for the inclusion of a single variable after all other variables are included in the model is IDENTICAL to the t test of the slope for that variable

The only reason to do an F test is to test several variables together


Chapter Summary

Developed the multiple regression model

Discussed residual plots Presented influence analysis Addressed testing the significance

of the multiple regression model Discussed inferences on population

regression coefficients Addressed testing portion of the

multiple regression model


nieXXY iippii ,,2,1,110

Multiple Linear Regression

npnpn

p

p

n e

e

e

e

XX

XX

XX

X

Y

Y

Y

whereY

2

1

1

0

1

221

111

2

1

,,

1

1

1

,

Data Model:

Matrix Model: eXY


ppp

n

iippiip

p

XXXXwhereXYXXY

XXYQ

Q

FittingSquaresLeast

,,,,1

,,,

,,,,min.1

:

21

22

110

1

211010

10

ˆˆˆˆˆ

,,,minˆ,,ˆ.2

110

100

XXXY

valuefittedtheThen

QimizeLet

pp

pp

noningularisXXiff

YXXXXYXXYYSinceT

TTT

)(

)(ˆ0)ˆ(ˆ 1


YYXXY rRt ˆ,,,; 1

Multiple Correlation Coefficient:

Multiple Coefficient of Determination: may be interpreted as the proportion of variance explained by the regression of Y on X.

2ˆ,

2,,;

2

2ˆ

2ˆ

2

1

,

YYXXY

eYY

rRR

SSS

Moreover

t

2

2ˆ2

Y

Y

S

SR


)

0

0

..(,0~2

2

2..

eVareiINeassume

eXYModel

dii

YXXXEESolve TT 1ˆ:

12,~ˆ XXN TTheorem:

YEXXX TT 1

XXXX

XEXXX

eXEXXX

TT

TT

TT

1

1

1


12,~ˆ XXN T

ˆ,ˆˆ: CovVarSolve

YXXXYXXXCov TTTT 11,

TTTTT XXXYYCovXXX11

,

TTTTT XXXeXVarXXX11

TTTT XXXIXXX121

112 XXXXXX TTT

12 XX T

1

ˆ1

1

1

ˆˆ

1

21

2

2

pn

RSSYY

pnpn

ewhere

n

iii

n

ii


DATA;INPUT X1 X2 Y;CARDS;68 60 7549 94 6360 91 57.77 78 72;PROC PRINT; PROC REG;MODEL Y=X1 X2 / COVB CORRB R INFLUENCE; RUN;


Model: MODEL1 Dependent Variable: Y Analysis of Variance Sum of Mean Source DF Squares Square F Value Pr > F Model 2 1966.20840 983.10420 14.86 0.0002 Error 17 1124.79160 66.16421 Corrected Total 19 3091.00000 Root MSE 8.13414 R-Square 0.6361 Dependent Mean 74.50000 Adj R-Sq 0.5933 Coeff Var 10.91831 Parameter Estimates Parameter Standard Variable DF Estimate Error t Value Pr > |t| Intercept 1 14.49614 14.20435 1.02 0.3218 X1 1 0.56319 0.11801 4.77 0.0002 X2 1 0.26736 0.15704 1.70 0.1069


Covariance of Estimates

COVB Intercept X1 X2 Intercept 201.7635339 -0.635820247 -1.851491131 X1 -0.635820247 0.0139252459 -0.003440529 X2 -1.851491131 -0.003440529 0.0246625524

Correlation of Estimates

COVB Intercept X1 X2 Intercept 1.0000 -0.3793 -0.8300 X1 -0.3793 1.0000 -0.1857 X2 -0.8300 -0.1857 1.0000


Dep Var Predicted Std Error Std Error Student Cook's

Obs Y Value Predict Residual Residual Residual -2-1 0 1 2 D

1 75.0000 68.8346 4.2678 6.1654 6.925 0.890 | |* | 0.100

2 63.0000 67.2242 3.3214 -4.2242 7.425 -0.569 | *| | 0.022

3 57.0000 72.6172 2.2988 -15.6172 7.803 -2.002 | ****| | 0.116

4 88.0000 74.4491 1.9107 13.5509 7.907 1.714 | |*** | 0.057

5 88.0000 90.5143 4.2002 -2.5143 6.966 -0.361 | | | 0.016

6 79.0000 85.2747 2.6984 -6.2747 7.674 -0.818 | *| | 0.028

7 82.0000 67.5089 2.4898 14.4911 7.744 1.871 | |*** | 0.121

8 73.0000 66.4506 2.8567 6.5494 7.616 0.860 | |* | 0.035

9 90.0000 81.2755 2.5928 8.7245 7.710 1.132 | |** | 0.048

10 62.0000 59.7208 3.8097 2.2792 7.187 0.317 | | | 0.009

11 70.0000 77.4755 1.8990 -7.4755 7.909 -0.945 | *| | 0.017

12 96.0000 93.1309 3.8760 2.8691 7.151 0.401 | | | 0.016

13 76.0000 73.9825 2.5281 2.0175 7.731 0.261 | | | 0.002

14 75.0000 80.1776 2.3793 -5.1776 7.778 -0.666 | *| | 0.014

15 85.0000 84.6150 3.2590 0.3850 7.453 0.0517 | | | 0.000

16 40.0000 50.3917 5.9936 -10.3917 5.499 -1.890 | ***| | 1.414

17 74.0000 76.2637 2.1866 -2.2637 7.835 -0.289 | | | 0.002

18 70.0000 69.0846 2.0768 0.9154 7.865 0.116 | | | 0.000

19 75.0000 72.2929 2.6787 2.7071 7.680 0.352 | | | 0.005

20 72.0000 78.7158 2.5093 -6.7158 7.737 -0.868 | *| | 0.026

21 . 83.7560 3.0157 . . . .


Hat DiagObs Residual RStudent H 1 6.1654 0.8846 0.2753 2 -4.2242 -0.5572 0.1667 3 -15.6172 -2.2211 0.0799 4 13.5509 1.8281 0.0552 5 -2.5143 -0.3515 0.2666 6 -6.2747 -0.8094 0.1100 7 14.4911 2.0374 0.0937 8 6.5494 0.8530 0.1233 9 8.7245 1.1417 0.1016 10 2.2792 0.3086 0.2194 11 -7.4755 -0.9420 0.0545 12 2.8691 0.3911 0.2271 13 2.0175 0.2537 0.0966 14 -5.1776 -0.6543 0.0856 15 0.3850 0.0501 0.1605 16 -10.3917 -2.0627 0.5429 17 -2.2637 -0.2810 0.0723 18 0.9154 0.1130 0.0652 19 2.7071 0.3432 0.1084 20 -6.7158 -0.8613 0.0952


|

100 +

|

| o

| o o o o o

| o o o

90 + o

| o

| o

H |

O | o o

M 80 + o o

E | o

W |

O | o

R |

K 70 +

|

|

|

|

60 + o

|

|

|

|

50 + o

|

-+------------+------------+------------+------------+------------+------------+------------+

30 40 50 60 70 80 90 100

MIDTERM


0

00

11 ,.1

rrH

rrH

hypothesisthetesttowantweccrLet

a

pp

：：

rdtspntr

rforICaConstruct

cccrLet pp

ˆˆ1ˆ

..%95

.2

025.0

1100

TccVarcVarrVar

andpntrdts

rErtisstatistictesttheThen

ˆˆˆ

,1~ˆˆˆˆ

.ˆ

ˆ

ˆ),,,(,ˆˆˆ0

000

p

ppp cccwhereccrchooseWe


setdataaforelbetterachoosetoHow mod

Goodness of Fit

1,~1 0

pnqpFRSS

RSSRSS

qp

pnF

pqjoneleastatH

H

eldrestriceteatoeledunrestrictanreducetoHow

j

pq

,,1,0

0

?modmod

1

10

：

：

eXXY

elstricted

eXXY

eledUnrestrict

pqAssume

qq

pp

110

110

modRe

mod


?mod datathefittoelbetterachoosetoHow

eYelrestricted

eXYeledunrestrictEx

mod

mod：

00 0 ：H

2,1~1

2

0.0

20

10

nRSS

RSSRSSnF

HsvHSolve

：：：

.ˆˆ1

2

1

2

0

n

iii

n

ii

XYRSS

YYRSSwhere


Regression Effect

eY

elstricted

eXXY

eledUnrestrict

pp

0

110

modRe

mod

2ˆ

2ˆ0 11

:e

Y

S

S

p

pn

RSS

RSSRSS

p

pnFSolve

,0

0210

ja

p

oneleastatH

H

：：

2

2

1

1

R

R

p

pnFthen

22ˆ

22ˆ1

Ye

YY

SS

SS

p

pn

2

2

1

1

R

R

p

pn


i

ijiij

ijiij

njkifor

RSSeYeledUnrestrict

RSSeXYelstricted

,,2,1,,2,1

:mod

:modRe 0

；

k

iinn

RSS

RSSRSS

k

knF

1

0 ,2

jiif

jiifdkidddLet ijikkiii ,0

,1,,,2,1,2211

Goodness of Fit for using replicate observations

)(

mod

2211 RSSedddY

asrewrittenbecaneledunrestricttheThen

ijikkiiij

© 2002 prentice-hall, inc.chap 14-1 introduction to multiple regression model

Documents

regression modelinferences

multiple linearregression

average value of y changes

average temperature

independent variables

fuel oil usage y

adjusted r square0

temperature x1