ols geometry

30
Vector Spaces OLS and Projections The FWL Theorem Applications OLS Geometry Walter Sosa-Escudero Econ 507. Econometric Analysis. Spring 2009 February 3, 2009 Walter Sosa-Escudero OLS Geometry

Upload: roger-asencios-nunez

Post on 03-Oct-2015

261 views

Category:

Documents


10 download

DESCRIPTION

Econometria

TRANSCRIPT

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    OLS Geometry

    Walter Sosa-Escudero

    Econ 507. Econometric Analysis. Spring 2009

    February 3, 2009

    Walter Sosa-Escudero OLS Geometry

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    Vector Space Geometry

    A vector space S is a set along with an addition and a scalarmultiplication on S that satisfies some properties:conmutativity, associativity, etc.

    The euclidean space

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    Some Definitions and Notation

    Inner product: < x, y > xyNorm: ||x|| (xx)1/2 = (ni=1 x2i )1/2.Orthogonality: x and y are orthogonal iff < x, y >= xy = 0Linear dependence: x1, . . . , xk are linearly dependent if thereexists xj , 1 j k and coefficients ci such thatxj =

    i 6=j cixi

    Walter Sosa-Escudero OLS Geometry

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    Vector geometry in

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    A vector in

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    Vector addition: parallelograms rule

    Walter Sosa-Escudero OLS Geometry

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    Subspaces of the Euclidean Space

    A vector subspace is any subset of a vector space that is itselfa vector space.

    Span: S(x1, . . . , xk) {z En | z = ki=1 bixi, bi

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    Orthogonal complement:S(X) {w En | wz = 0 for all z S(x)}. All vectorsthat are orthogonal to the columns of X.

    Basis: a basis of V is a list of linearly independent vectorsthat spans V .

    Dimension: # of vectors of any basis.Note dimS(X) (X)Result: Xnk with dimS(X) = k dimS(X) = n k

    Walter Sosa-Escudero OLS Geometry

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    X is a vector in

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    Variables and observations in the axis

    The goal is to represent the data and the OLS estimator.

    We need to change our notion of point. A scatter plot takesevery observation as a point.

    Now we need to think of Y and the columns of X as K + 1points in

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    Source: Bring, J., 1996, A Geometric Approach to Compare Variables in a Regression Model, The AmericanStatistician, 50,1, pp. 57-62.

    What do you expect to happen with this picture if we add a third person? A

    fourth?

    Walter Sosa-Escudero OLS Geometry

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    OLS Geometry

    By definition, any point in S(X) can be expressed as X,

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    The problem: min ||y x|| min ||y x||2.

    Define: (solution to the problem), Y = X , e = Y Y

    Some properties:

    e is orthogonal to any point in S(X), in particular, to X orX.

    = (X X)1X Y .From the orthogonality condition X (Y ) = 0.

    Walter Sosa-Escudero OLS Geometry

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    Walter Sosa-Escudero OLS Geometry

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    Walter Sosa-Escudero OLS Geometry

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    Projections

    A projection is a mapping that takes any point in En into apoint in a subspace of En.

    An orthogonal projection maps any point into the point of thesubspace that is closest to it.

    Y = X = X(X X)1X Y = PXY is the orthogonalprojection of Y on S(X). PX = X(X X)1X is theprojection matrix that projects Y orthogonally on to S(X).e = Y Y = Y X = (IX(X X)1X )Y = MXY is theprojection of Y on to the orthogonal complement of S(X),that is, S(X). MX I PX = I X(X X)1X . is theprojecton matrix that projects Y orthogonally on to S(X).

    Walter Sosa-Escudero OLS Geometry

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    Properties: easy to check algebraically, better to understand themgeometrically

    MX and PX are symmetric matrices.

    MX + PX = I. This suggests the orthogonal decompositionY = MXY + PXY

    Walter Sosa-Escudero OLS Geometry

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    PX and MX are idempotent: PXPX = PX , MXMX = MX .Intuition: if a vector is already in S(X), further projecting itin S(X) has no effect.PXMX = 0. Think about what you get of doing fisrt oneprojection and then the other (in any order). PX and MXanihilate each other. 0 is the only point that belongs to bothS(X) and S(X).MX anihilates any point in S(X), that is MXX = 0PX anihilates any point in S

    (X) : PXX = 0 CHECKIf A is a non-singular matrix K K, PXA = PX .(X) = (PX)

    Walter Sosa-Escudero OLS Geometry

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    Goodness of fit

    From the orthogonal decomposition

    Y = PY +MY

    Then

    Y Y = Y PY + Y MY (1)= Y P PY + Y M MY (2)

    ||Y ||2 = ||PY ||2 + ||MY ||2 (3)In

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    Walter Sosa-Escudero OLS Geometry

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    The Frisch-Waugh-Lovell Theorem

    Consider the linear model: Y = X + u

    And partition it as follows: Y = X11 +X22 + u

    X1, X2 matrices of k1 and k2 explanatory variables. Then,X = [X1 X2], = (1 2) and k = k1 + k2.

    M1 I X1(X 1X1)1X 1, projects any vector in Rn in theorthogonal complement of the span of X1.

    Y M1Y , X2 M1X2, respectively, OLS residuals of regressingY on X1, and all columns of X2 on X1.

    Walter Sosa-Escudero OLS Geometry

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    Suppose that we are interested in estimating 2, and consider thefollowing alternative methods:

    Method 1: Proceed as usual and regress Y on X obtainingthe OLS estimator = (1 2) = (X X)1X Y . 2 wouldbe the desired estimate.

    Method 2: Regress Y on X2 and obtain as estimate2 = (X2 X2 )1X2 Y

    Let e1 and e2 be the residuals vectors of the regressions in Method1 and 2, respectively.

    Theorem (Frisch and Waugh, 1933, Lovell, 1963): 2 = 2 (firstpart) and e1 = e2 (second part).

    Walter Sosa-Escudero OLS Geometry

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    Proof (boring): Start point with the orthogonal decomposition:

    Y = PY +MY = X11 +X22 +MY

    To prove the first part, multiply by X 2M1 to get:

    X 2M1Y = X2M1X11 +X

    2M1X22 +X

    2M1MY

    M1X1 = 0, why?X 2M1M = X 2M X 2P1M = 0 (same reasons as before)

    Then: X 2M1Y = X 2M1X22

    So: 2 = (X 2M1X2)1 X 2M1Y

    Walter Sosa-Escudero OLS Geometry

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    To prove the second part multiply the orthogonal decomposition byM1 and obtain:

    M1Y = M1X11 +M1X22 +M1MY

    Again, M1X1 = 0MY belongs to the orthogonal complement of [X1 X2], sofurther projecting it on the orthogonal complement of X1(which is what premultiplying by M1 would do) has no effect,hence M1MY = MY .

    This leaves:

    M1Y M1X22 = MYY X2 2 = MY

    e2 = e1

    Walter Sosa-Escudero OLS Geometry

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    Geometric Illustration of FWLT

    Walter Sosa-Escudero OLS Geometry

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    Geometric Illustration of FWLT

    Walter Sosa-Escudero OLS Geometry

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    Comments and Intuitions

    Idea of controling for X1: either put it in the model, or firstget rid of it by extracting its effect.

    What if X1 and X2 are orthogonal?

    Walter Sosa-Escudero OLS Geometry

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    Applications of the FWLT

    Deviations from means.

    Detrending

    Seasonal effects

    Later on: multicolinearity, omitted variable bias, panel-datafixed-effects estimation, instrumental variables.

    Walter Sosa-Escudero OLS Geometry

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    Deviation from means

    Simple model with intercept

    Y = X + u = 1 1 + [X2 X3 XK ] 1,

    1 (1, 1, . . . , 1), 1 = (2, 3, . . . , K), and Xk, k = 2, . . . ,Kare the corresponding columns of X.

    Two methods of estimating 1

    Method 1: Regress Y on X = [1 X2 XK ].

    Method 2: Get residuals of projecting Xk, k = 2, . . . ,K on 1, callthem Xk . Do the same with Y , and call them Y

    .

    Walter Sosa-Escudero OLS Geometry

  • Vector SpacesOLS and ProjectionsThe FWL Theorem

    Applications

    Note P1 = 1(11)11 = n1J , J is an n n matrix of 1s. Then

    P1Xk =1nJXk = (Xk, Xk, . . . , Xk)

    so Xk = M1Xk = (I P1)Xk = Xk (Xk, Xk, . . . , Xk), ann 1 vector with typical element

    Xik = Xik XkSo the second method consists in:

    1 Reexpress all varaibles as deviations from their sample means.

    2 Run the standard regression of these residuals withoutintercept.

    Question: what happens if we forget to reexpress Y as deviationsfrom its means. Generalize this result

    Walter Sosa-Escudero OLS Geometry

    Vector SpacesOLS and ProjectionsThe FWL TheoremApplications