simple and multiple regression analysis in matrix form

78
Simple and multiple regression analysis in matrix form Least square Beta estimation Simple linear regression Multiple regression with two predictors Multiple regression with three predictors Sum of square R 2 Test on parameters Covariance matrix of the Standard error of the

Upload: jerome-grimes

Post on 31-Dec-2015

105 views

Category:

Documents


1 download

DESCRIPTION

Simple and multiple regression analysis in matrix form. Least square Beta estimation Simple linear regression Multiple regression with two predictors Multiple regression with three predictors Sum of square R 2 Test on b parameters Covariance matrix of the b Standard error of the b. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Simple and multiple regression analysis in matrix form

Simple and multiple regression analysis in matrix form Least square Beta estimation Simple linear regression Multiple regression with two predictors Multiple regression with three predictors Sum of square R2

Test on parameters Covariance matrix of the Standard error of the

Page 2: Simple and multiple regression analysis in matrix form

Simple and multiple regression analysis in matrix form Tests on individual predictors Variance of individual predictors Correlation between predictors Standardized matrices Correlation matrices Sum of squares in Z R2 in Z

R2 between independent variables Standard error of in Z

Page 3: Simple and multiple regression analysis in matrix form

Least squareStarting from the general:

The method of least squares estimate of the beta parameter minimizing the sum of squares due to error.

In fact, if:

Page 4: Simple and multiple regression analysis in matrix form

You can estimate:

Least square

Page 5: Simple and multiple regression analysis in matrix form

Simple linear regression

Page 6: Simple and multiple regression analysis in matrix form

Simple linear regression

Page 7: Simple and multiple regression analysis in matrix form

Simple linear regression

Page 8: Simple and multiple regression analysis in matrix form

Simple linear regression

intercepts

slope

Page 9: Simple and multiple regression analysis in matrix form

Multiple regression Similar to the simple A single dependent variable (Y) Two or more independent

variables (X) Multiple correlation (rather than

simple) Estimation by least squares

Page 10: Simple and multiple regression analysis in matrix form

Simple linear regression (var.: 1 dep., 1 indep.)

Multiple linear regression (Var.:1 dep., 2 indep.)

intercepts errorIndependent variables

slope

Multiple regression

Page 11: Simple and multiple regression analysis in matrix form

Multiple regression matrix form

Page 12: Simple and multiple regression analysis in matrix form

Multiple regression matrix form

Page 13: Simple and multiple regression analysis in matrix form

Multiple regression matrix form

Page 14: Simple and multiple regression analysis in matrix form

X’X

inversa

Multiple regression matrix form

Page 15: Simple and multiple regression analysis in matrix form

Multiple regression matrix form

Page 16: Simple and multiple regression analysis in matrix form
Page 17: Simple and multiple regression analysis in matrix form

In matrix notation is briefly expressed :

Multiple regression with three predictors

Page 18: Simple and multiple regression analysis in matrix form

Multiple regression with three predictors

Page 19: Simple and multiple regression analysis in matrix form

Matrix form

Page 20: Simple and multiple regression analysis in matrix form

Matrix form

Page 21: Simple and multiple regression analysis in matrix form

Matrix form

Page 22: Simple and multiple regression analysis in matrix form

Matrix form

Page 23: Simple and multiple regression analysis in matrix form

General scheme

Page 24: Simple and multiple regression analysis in matrix form

General scheme

Page 25: Simple and multiple regression analysis in matrix form

The least squares method allows to check the following equality:

Sum squares

Page 26: Simple and multiple regression analysis in matrix form

Since in general:it's possible to derive that the sum of the squares of the distances of y from its average can be decomposed into the sum of squares due to regression and the sum of squares due to error, according to:

Sum squares

Page 27: Simple and multiple regression analysis in matrix form

Sum squares

It should be noted the equivalence of :

Page 28: Simple and multiple regression analysis in matrix form

Sum squares

Page 29: Simple and multiple regression analysis in matrix form

Sum squares

Page 30: Simple and multiple regression analysis in matrix form

In summary :

Sum squares

Page 31: Simple and multiple regression analysis in matrix form

R2

Page 32: Simple and multiple regression analysis in matrix form

Adjusted R2YY’

Because the coefficient of determination depends on both the number of observations (n) that the number of independent variables (k) it is convenient to correct by the degrees of freedom.

Adjusted R2YY’

In our example :

Page 33: Simple and multiple regression analysis in matrix form

Once a regression model has been constructed, it may be important to confirm the goodness of fit (R-squared )of the model and the statistical significance of the estimated parameters. Statistical significance can be checked by an F-test of the overall fit, followed by t-tests of individual parameters

Test on parameters

Page 34: Simple and multiple regression analysis in matrix form

You can test the hypothesis of differences with 0 of the parameters i taken together :

Test on parameters

Page 35: Simple and multiple regression analysis in matrix form

k= Number of columns of the matrix X excluding X0

n= Number of observations in y

Test on parameters

Page 36: Simple and multiple regression analysis in matrix form

Test on parameters

k= Number of columns of the matrix X excluding X0

n= Number of observations in y

Page 37: Simple and multiple regression analysis in matrix form

Covariance matrix of the

An estimate of the covariance matrix of the beta values result by:

We denote:

Page 38: Simple and multiple regression analysis in matrix form

Covariance matrix of the

Where the diagonal elements are an estimate of the variance of the single i

Page 39: Simple and multiple regression analysis in matrix form

Standard error of the

The standard error of the parameters can be calculated with the following formula:

where cii is the diagonal element inside the matrix(X’X)-1 corresponding to the parameter i .

Page 40: Simple and multiple regression analysis in matrix form

Standard error of the

Nota: quando il valore di cii è elevato il valore di sebi

cresce, indicando che la variabile Xi ha un alto coefficiente di correlazione multipla con le altre variabili X.

Page 41: Simple and multiple regression analysis in matrix form

Standard error of the

the increase in R2i led to a decreases of the

denominator of the ratio and, consequently, increases the value of the standard error of the parameter i.

The standard error of the i can also be calculated in the following way:

where

Page 42: Simple and multiple regression analysis in matrix form

With the standard error of measurement associated with each i you can make a t-test to verify:

Tests on individual predictors

Page 43: Simple and multiple regression analysis in matrix form

Tests on individual predictors

With the standard error of measurement associated with each i is also possible to estimate the confidence interval for each parameter:

Page 44: Simple and multiple regression analysis in matrix form

Tests on individual predictors

1. Calculate the SSreg for the model containing all the independent variables.

2. Calculate the SSreg for the model excluding the variable for which you want to test the significance (SS-i).

3. Perform an F-test with the numerator equal to the difference SSreg-SSi weighted for the difference between the degrees of freedom of the two models, and with denominator SSREs / (nk-1).

In order to conduct a statistical test on the regression coefficients is necessary:

Page 45: Simple and multiple regression analysis in matrix form

Tests on individual predictors

To test, for example, only the weight of the first predictor compared to the total model, it is necessary to calculate a new matrix i from the matrix Xi which was taken off the column belonging to the first predictor. From this follows immediately the calculation of SSi.

Page 46: Simple and multiple regression analysis in matrix form

Tests on individual predictors

Page 47: Simple and multiple regression analysis in matrix form

Tests on individual predictors

Same procedure is followed to test any subset of predictors.

Similarly we have:

Page 48: Simple and multiple regression analysis in matrix form

Tests on individual predictors

It is interesting to note that this test on a single predictor is equivalent to the t-test b1 = 0. When the numerator there is only one degree of freedom, that is in fact the equivalence:

Page 49: Simple and multiple regression analysis in matrix form

Summary table

On this occasion, none of the estimated parameters obtained statistical significance on the hypothesis i 0

Page 50: Simple and multiple regression analysis in matrix form

Variance of individual predictors Xi

Using the matrix X'X we can calculate the variance of each variable Xi .

Page 51: Simple and multiple regression analysis in matrix form

Variance of individual predictors Xi

Page 52: Simple and multiple regression analysis in matrix form

Covariance between predictors and the dependent variable

It is possible to calculate the covariance between the independent variables and the dependent variable according to:

Page 53: Simple and multiple regression analysis in matrix form

Covariance between predictors and the dependent variable

The correlation between the independent variables and the dependent variable is given by:

As we will see later the use of standardized matrices simplifies the calculation immediately.

Page 54: Simple and multiple regression analysis in matrix form

Test on multiple predictor You can perform a statistical test on a group

of predictors in order to verify the significance.

To do this, you use the formula specified above :

To test, for example, the weight of only the first and second predictors with respect to the total model, it is necessary to calculate a new matrix i from the matrix Xi which was taken off the column belonging to these predictors. From this follows immediately the calculation of SSi.

Page 55: Simple and multiple regression analysis in matrix form

Test on multiple predictor

Page 56: Simple and multiple regression analysis in matrix form

Correlation between predictors

Standard condition of independence between the variables Xi

Page 57: Simple and multiple regression analysis in matrix form

Correlation between predictors

Condition of dependence between variables Xi

Completely standardized solution.

Page 58: Simple and multiple regression analysis in matrix form

We denote by Ri. the multiple correlation of the variable Xi with the remaining variables, denoted by Xj

Correlation between predictors

The element cii represents the value of the diagonal of the matrix (X'X) -1 while S2

i is the variance of the variable Xi.

Page 59: Simple and multiple regression analysis in matrix form

In case you do not have the X'X matrix but you have the MSres and the standard error of the parameter i, the correlation between one X and the other one can be calculated in the following manner:

Correlation between predictors

Page 60: Simple and multiple regression analysis in matrix form

Correlation between predictors

Page 61: Simple and multiple regression analysis in matrix form

The X matrix and the y matrix can be converted into standardized scores by dividing the deviation of each element from the average for the appropriate standard deviation.

Standardized matrices

Page 62: Simple and multiple regression analysis in matrix form

In our example we have:

Standardized matrices

Page 63: Simple and multiple regression analysis in matrix form

Standardized matrices

With standardized variables is not necessary to include in the matrix Z the component 1 as the parameter 0 is equal to 0.

Page 64: Simple and multiple regression analysis in matrix form

The standardized coefficients can be obtained from those non-standardized using the formula:

The equation of the regression line becomes:

Standardized matrices

Page 65: Simple and multiple regression analysis in matrix form

Standardized matrices

In our example we have:

Page 66: Simple and multiple regression analysis in matrix form

Use standardized matrices allows to set the parameter 0 = 0. In fact, if the variables are standardized the intercept value for Y is 0, since all the means are equal to 0;Inoltre, essendo

the correlation between any two standardized variables is:

with i, j between 1 and k.

Standardized matrices

Page 67: Simple and multiple regression analysis in matrix form

Correlation matrices

If we multiply the matrix (Z'Z) for the scalar [1 / (n-1)] we obtain the correlation matrix R between the independent variables

Page 68: Simple and multiple regression analysis in matrix form

In our example we have:

Correlation matrices

Page 69: Simple and multiple regression analysis in matrix form

Correlation of Y with individual predictors

Similarly if the variable Y is also standardized and multiply the product by the scalar Z'Yz [1 / (n-1)] we obtain the correlation matrix ryi of the variable Y with its predictors Xi.

Page 70: Simple and multiple regression analysis in matrix form

Correlation of Y with individual predictors

Page 71: Simple and multiple regression analysis in matrix form

The solution of the system of normal equations of the line leads to the following equation:

The estimated values can be obtained using the equation:

Correlation of Y with individual predictors

Page 72: Simple and multiple regression analysis in matrix form

With standardized variables we have:

Starting from the general formulas it's possible to have the following simplified formulas:

Sum of squares in Z

Page 73: Simple and multiple regression analysis in matrix form

Sum of squares in Z

Page 74: Simple and multiple regression analysis in matrix form

Calculation of R2y.123

Having decomposed the variance component due to the regression and the component due to the residuals, it is immediate to calculate:

Page 75: Simple and multiple regression analysis in matrix form

Multiple correlation between the Xi.yz

If in general the squared multiple correlation of a variable with the other independent Xi is:

in the presence of standardized variables, it becomes:

where the element aii belongs to the diagonal of the matrix R-1.

Page 76: Simple and multiple regression analysis in matrix form

If you want to calculate the other two coefficients now you will have to do the following:

For example, the squared multiple correlation between the first variable X1 and the other two can be calculated in the following way:

Multiple correlation between the Xi.yz

Page 77: Simple and multiple regression analysis in matrix form

Standard error of z

The standard error of the standardized parameters is obtainable by the general formula:

Page 78: Simple and multiple regression analysis in matrix form

Standard error of z

You now have all the elements to test the differences of individual predictors from 0, obtaining the same results obtained with the non-standardized variables.