class 5 multiple regression ceram february-march-april 2008 lionel nesta observatoire français des...
DESCRIPTION
Introduction to Regression Typically, the social scientist is dealing with multiple and complex webs of interactions between variables. An immediate and appealing extension to simple linear regression is to extend the set of explanatory variable to other variables. Multiple regressions include several explanatory variables in the empirical modelTRANSCRIPT
Class 5Multiple Regression
CERAM February-March-April 2008
Lionel NestaObservatoire Français des Conjonctures Economiques
Introduction to Regression Typically, the social scientist is dealing with multiple
and complex webs of interactions between variables. An immediate and appealing extension to simple linear regression is to extend the set of explanatory variable to other variables.
Multiple regressions include several explanatory variables in the empirical model
1 21 2
pi i i p i iy x x x u
Introduction to Regression Typically, the social scientist is dealing with multiple
and complex webs of interactions between variables. An immediate and appealing extension to simple linear regression is to extend the set of explanatory variable to other variables.
Multiple regressions include several explanatory variables in the empirical model
1
k Kk
i k i ik
y x u
22
1 1
21
1
2
1
220 , ,
ˆˆmin min
0
, ,
ˆ
,
n n k K
n
j k
ki i i iki i k
i
Kik
n
y y y x
To minimize the sum of squared errors
1
12
ˆ
ˆcov( )
i i iy x u
β XX
y = Xβ +
y
u
X
β XX
Multivariate Least Square Estimator
Usually, the multivariate is described by matrix notation:
With the following least square solution:
Assumption OLS 1
20 1 1y x u
It is possible to operate non linear transformation of the variables (e.g. log of x) but not of the parameters like the following :
0 1 1 2 2 k ky x x x u
LinearityThe model is linear in its parameters
OLS can not estimate this
Assumption OLS 2
There is no selection bias in the sample. The results pertain to the whole population
All observations are independent from one another (no serial nor cross-sectional correlation)
Random SamplingThe n observations are a random sample of
the whole population
Assumption OLS 3
No independent variable is constant. Each variable has variance which can be used with the variance of the dependent variable to compute the parameters.
No exact linear relationships amongst independent variables
No perfect Collinearity There is no collinearity between independent
variables
Assumption OLS 4
Given any values of the independent variables (IV), the error term must have an expected value of zero.
In this case, all independent variables are exogenous. Otherwise, at least one IV suffers from an endogeneity problem.
Zero Conditional Mean The error term u has an expected value of zero
1 2 kE u x ,x , ,x 0
Sources of endogeneity
Wrong specification of the model
Omitted variable correlated with one RHS.
Measurement errors of RHS
Mutual causation between LHS and RHS
Simultaneity
Assumption OLS 5
21 2 k uVar u x ,x , ,x
Homoskedasticity The variance of the error term, u, conditional on RHS, is the same for all values of RHS.
Otherwise we speak of heteroskedasticity.
Assumption OLS 6
Normality of error termThe error term is independent of all RHS and follows a normal distribution with zero mean
and variance
2u Normal(0, )
2
Assumptions OLS
OLS1 Linearity
OLS2 Random Sampling
OLS3 No perfect Collinearity
OLS4 Zero Conditional Mean
OLS5 Homoskedasticity
OLS6 Normality of error term
Theorem 1
j jˆE , j 0,1,2, ,k
OLS1 - OLS4 : Unbiasedness of OLS. The set of estimated parameters is equal to the true unknown values of j
j
Theorem 2OLS1 – OLS5 : Variance of OLS estimate. The variance of the OLS estimator is
2u
j n 2 2ij j j
i 1
ˆVarx x 1 R
… where R²j is the R-squared from regressing xj on all other independent variables. But how can we measure ?
2u
Theorem 3OLS1 – OLS5 : The standard error of the regression is defined as
22
i ii2 2 i iu u
ˆy y uˆE
n k 1n k 1
This is also called the standard error of the estimate or the root mean squared errors (RMSE)
Standard Error of Each Parameter Combining theorems 2 and 3 yields:
uj n 2 2
ij j ji 1
ˆˆsex x 1 R
Theorem 4Under assumptions OLS1 – OLS5, estimators are the best linear unbiased estimators (BLUE) of
0 1 kˆ ˆ ˆ, , ,
0 1 k, , ,
Assumptions OLS1 – OLS5 are known as the Gauss-Markov Theorem, which stipulates that under OLS1-5, the OLS are the best estimation methodThe estimates are unbiased (OLS1-4)The estimates have the smallest variance (OLS5)
Theorem 5Under assumptions OLS1 – OLS6, the OLS estimates follows a t distribution:
j jn k 1
j
ˆtˆse( )
Extension of theorem 5: Inference We can define de confidence interval of β, at 95% :
.025
2 2
1
ˆt
1
ujj n
ij j ji
x x R
If the 95% CI does not include 0, then β is significantly different than 0.
Student t Test for H0: βj=0 We are also in the position to infer on βj
H0: βj = 0
H1: βj ≠ 0
Rule of decision
Accept H0 is | t | < tα/2
Reject H0 is | t | ≥ tα/2
ˆ ˆ
tse se
Summary
OLS1 Linearity
OLS2 Random Sampling
OLS3 No perfect Collinearity
OLS4 Zero Conditional Mean
OLS5 Homoskedasticity
OLS6 Normality of error term
T1UnbiasednessT2-T4
BLUET5β ~ t
The knowledge production function
Application 1: Seminal model
1 2
1 2
PAT f (RD,SIZE)
PAT A RD SIZE exp u
pat rd size u
The knowledge production function
Application 2: Changing specification
1
2
1 2
PAT f (RD,SIZE)
RDPAP A SIZE exp uSIZE
RDy log size uSIZE
The knowledge production function
Application 3: Adding variables
1
23
1 2 3
PAT f (RD,SIZE,SPE)
RDPAT A SIZE exp SPE uSIZErdpat size SPE usize
The knowledge production function
Application 4: Dummy variables
1
23 4
1 2 3 4
PAT f (RD,SIZE,SPE,BIO)
RDPAT A SIZE exp SPE BIO uSIZErdpat size SPE BIO usize
Application 4: Dummy variables
Coefficientsa
-5.465 .616 -8.864 .000 -6.676 -4.253.556 .042 .909 13.326 .000 .474 .638.492 .080 .313 6.123 .000 .334 .650.421 .145 .118 2.912 .004 .137 .706
1.657 .168 .665 9.835 .000 1.326 1.988
(constante)lnassetslnrd_assetsspebio
Modèle1
BErreur
standard
Coefficients nonstandardisés
Bêta
Coefficientsstandardisés
t SignificationBorne
inférieureBorne
supérieure
Intervalle de confiance à95% de B
Variable dépendante : lnpatenta.
Application 4: Dummy variables
Patent(lnpatent)
Size(lnasset)
4
42DBF: size
2LDF: size
2Slope
2Slope
The knowledge production function
Application 5: Interacting Variables
1
2
4 5
4
3
1 2 3
5
PAT f (RD,SIZE,SPE,BIO)
RDPAT A SIZESIZE
exp SPE BIO BIO size u
rdpat size SPEsize
BIO BIO size u
Application 5: Interacting Variables
Coefficientsa
-6.483 .843 -7.693 .000 -8.139 -4.827.620 .055 1.013 11.241 .000 .511 .728.474 .081 .301 5.863 .000 .315 .633.413 .144 .115 2.862 .004 .129 .697
3.592 1.108 1.441 3.242 .001 1.415 5.770-.144 .081 -.693 -1.767 .078 -.303 .016
(constante)lnassetslnrd_assetsspebiosizebio
Modèle1
BErreur
standard
Coefficients nonstandardisés
Bêta
Coefficientsstandardisés
t SignificationBorne
inférieureBorne
supérieure
Intervalle de confiance à95% de B
Variable dépendante : lnpatenta.
Application 5: Interacting variables
Patent(lnpatent)
Size(lnasset)
4
2 4 5DBF: size size bio
2LDF: size
2 5Slope size bio
2Slope