linear models. kachman, s. d. 1999

Biometry 970Linear Models

S. D. KachmanDepartment of Biometry

University of NebraskaLincoln

http://www.ianr.unl.edu/ianr/biometry/faculty/steve/970/1999

Distributions

Outline

Linear Transformations, Quadratic Forms, Bilinear Forms Positive (semi-)definite matrices Distributions

Multivariate distributions Linear Transformation

Normal Distribution Central 2, F , and t Non-Central

Distribution of Quadratic Forms Moments Independence

Biometry 970 Linear Models Fall 1999 22

Linear Transformations, Quadratic Forms,Bilinear Forms

y (y,V yy)x (x,V xx)

cov(x,y) = V xy

Linear Transformation,Ay Quadratic Form, yQy

Without loss of generality we can assumeQ issymmetric. If it is not we can replaceQ with 12(Q+Q

) Bilinear Form, xQy

A bilinear form can be written as a quadratic form

(x y

)( 0 12Q12Q 0

)(x

y

)where (

x

y

)((xy

),

(V xx V xyV yx V yy

))


Direct Products

AB =

a11B a12B . . . a1nB

a21B a22B . . . a2nB...

aijB...

am1B am2B . . . amnB

A : m nB : r c

AB : mr nctr(AB) = tr(A) tr(B)

rank(AB) = rank(A) rank(B)|AB| = |A|r|B|m

(AB) = A B

(AB) = A B

[AB][C D] = [AC] [BD]


Sum of squares and quadratic forms

2 factor experiment (A, B) with a = 2 levels of factor A and b = 2 levels of factor B n = 3 replications of each of ab = 4 treatment

combinations If we let yijk be the observed response, and y be the

vector of yijks ordered y111, y112, y113, y121, . . . . Then the total sum of squares isSST =

ai=1

bj=1

nk=1 y

2ijk y2.../(abn)

in matrix form this isyy yJy = yCy.

The sum of squares for A isSSA =

ai=1 y

2i../nb y2.../(abn)

in matrix form this isyIa Jb Jny yJa Jb Jny= yCa Jb Jny.

The sum of squares for B isSSB =

ai=1

bj=1 y

2.j./an y2.../(abn)

in matrix form this isyIa Ib Jny yJa Jb Jny= yJa Cb Jny.


The sum of squares for the AB interaction isSSAB =

ai=1

bj=1 y

2ij./n

ai=1 y

2i../nbb

j=1 y2.j./na+ y

2.../(abn)

in matrix form this isyIa Ib Jny yIa Jb JnyyJa Ib Jny + yJa Jb Jny= yCa Cb Jny.

The sum of squares for error isSSE =

ai=1

bj=1

nk=1 y

2ijk

ai=1

bj=1 y

2ij./n

in matrix form this isyIa Ib Cny.


Positive (semi-)definite matrices

Definition: A matrix P is said to be positive definite (p.d.) iffor all real x 6= 0; xPx > 0.

Definition: A matrix P is said to be positive semi-definite(p.s.d.) if for all real x 6= 0 xPx 0 and for some x 6= 0;xPx = 0.

Definition: A matrix P is said to be non-negative definite(n.n.d.) if for all real x; xPx 0.

IfQ is n.n.d., then SS = yQy will be non-negative.


Some useful results

1. A matrix is p.d. (n.n.d.) iff all its principal leading minorsare p.d. (n.n.d.).In fact all the principal minors are p.d. (n.n.d.).

2. For P non-singular P AP is or is not p.(s.)d. iffA is or isnot p.(s.)d..

3. Eigen values of p.(s.)d. are all positive (non-negative).4. A sum of n.n.d. matrices is n.n.d. and is p.d. if at least one

of the matrices is p.d..5. A symmetric matrixA is p.(s.)d. iff it can be written asP P for a non-singular (singular) P .

6. P P is p.d. if P has column rank and it is p.s.d.otherwise.

7. If V is n.n.d.(), then P V P is also n.n.d.().8. A symmetric matrixA, of order n and rank r, can be

written using L a n r matrix of rank r andD anon-singular diagonal matrix of order r

(a) LL(b) LL with L real iffA is n.n.d.(c) LDL with L andD being real matrices.Because L has full column rank the matrix LL isnon-singular.


9. A symmetric matrix having eigen values equal to zero andone is idempotent.idempotent matrices are common when we calculate SSs.

10. IfA and V are symmetric and V is positive definite, thenAV having eigen values 0 and 1 impliesAV isidempotent.


A look ahead

Why do we care?

Consider the problem of testing HO : K = 0, what aresome of the properties we would want a test statistic to have?

We would want to separate signal from noise.Noise: (I X(X X)X )y = (I MX)ySignal: X(X X)X y = MXy We would want to separate from the signal the part

associated with the hypothesis.Hyp: K(X X)X y = T MXy We would want a scalar test statistic quadratic form

SSH = yMXT var(T MXy)1T MXy. var(T MXy) = T MXT2


Multivariate distributions

CDF = F (x) = Pr(Xi xii = 1 . . . n) See B&Ech. 1,2,5 Pr(X R) =

R

f(x)dx

k = E(Xk) = xif(x)dx

E(g(X)) = g(x)f(x)dx = E(X) and V = E [(X )(X )] mgf(t) = E

[etX]


Linear Transformation

E(TX) = E(tiX)i

E(tiX) =

j

tijxjf(x)dx

=j

tij

xjf(x)dx

=j

tijxj

= tix

E(TX)= Tx

Similarly,

E(AXB) = AxB


var(TX) = E[(TX Tx)(TX Tx)

]= E

[T (X X)(X X)T

]= T E

[(X X)(X X)

]T

= TV xxT


Normal Distribution

Standard Normal f(z) = (z) = 12pie

z2

2

Normal B&E 118ff

f(x;, 2) = 12pie(x)

2

22

mgf(t) = et+12t

22

Multivariate Normal B&E185,520

Density

f(x;,V ) = (2pi)n2 |V |12e12(x)V

1(x)


Moment generating function

mgf(t) = E(tx)

=

(2pi)n2 |V |12

exp

[1

2(x )V 1(x ) + tx

]dx

inside = 12[(x V t)V 1(. . . )]

+t+

1

2tV t

mgf(t) = et+12t

V t

Marginal Distribution

x1 N(1,V 11)

Use mgf of x = (x1 x2) and set t2 = 0 and noticethat it is the mgf for N(1,V 11). B&E 185

Conditional Distribution

x1|x2 N(1 + V 12V 122 (x2 2),V 11.2)

V 11.2 = V 11 V 12V 122 V 21


Show by using f(x|y) = f(x, y)/f(y), B&E4.5,185|V | = |V22||V 11.2| and the inverse of a partitioned

matrix (1.48)

V1

=

(0 00 V 122

)+(

I

V 122 V 21

)(V 111.2

) (I V 12V 122

) Independence: V ij = 0 i 6= j

Again use the mgf.


Central 2, F , and t

If x N(0, In), then u = xx 2n. B&E 271 E(u) = n

var(u) = 2n

Scaled SSs (SS/EMS) often have 2 distributions If u1 2n1, u2

2n2

, and u1 and u2 are independent,then v = u1n1/

u2n2 Fn1,n2. B&E 275ff

F-test 1) Show SSs are scaled 2 random variables, 2) Show

that they are independent, 3) Ratio of the MSs will thenbe an F random variable

If x N(0, 1), u 2n, and x and u are independent,then t = x/

u/n tn.

t-test 1) Standardize x, 2) Scaled standard error, 3) Show

independence.


Non-Central

If x N(, In), then u = xx 2(n, = 12) If u1 2(n1, ), u2 2n2, and u1 and u2 are

independent, then v = u1n1/u2n2 F (n1, n2, ).

Power!


Distribution of Quadratic Forms

mgf of xAx when x N(,V )

mgf(t) = |I 2tAV |1/2

exp

{1

2 [I (I 2tAV )1

]V1

}

1. mgfXAX(t) = E(etXAX) B&E 2.5

2. Use E[g(x)] = g(x)f(x;,V )dx

3. Complete the square (Lemma 10)

(x )V 1(x ) 2txAx =(x )V 1(x )

+ [I (I 2tAV )1]V 1

V = V (I 2tAV )1 = (I 2tV A)1

IfA = V = I the mgf of xAx reduces to

(1 2t)n/2e12(1 112t)


which is the mgf of 2(n, = 12). Furthermore if = 0

this reduces to

(1 2t)n/2

Theorem 2: When x N(,V ) thenxAx 2[r = rank(A), 12A] if and only ifAV isidempotent.

Proof. In the book they do it by equating the mgfs Sufficiency, that isAV being idempotent implies 2

1. A = LL where L is a n rank(A) non-singularmatrix. NoteA is n.n.d. becauseAV is idempotentand V is p.d..

2. xAx = (Lx)(Lx) with Lx N(L,LV L)3. LV L = Ir


Moments

Mean E(xAx) = tr(AV ) + A var(xAx) = 2 tr(AV AV ) + 4AV A

ln[mgf(t)] = 12

ln |I 2tAV |

12 [I (I 2tAV )1

]V1


Independence

If x N(,V ) then,

1. Ax andBx are independent iffAV B = 02. xAx andBx are independent iffAV B = 03. xAx and xBx are independent iffAV B = AV B = 0


Summary

Linear transformations

E(Ay) = A

var(Ay) = AV A

Quadratic Form

E(yQy) = tr(QV ) +

Q

var(yQy) = 2 tr(QV QV ) + 4

QV Q

Normal Distribution Ay Normal Conditional Distribution Density yQy 2 ifQV idempotent

F and t distributions Completing a square Independence (Normal)

AV B = 0


DistributionsLinear Transformations, Quadratic Forms, Bilinear FormsDirect ProductsSum of squares and quadratic forms

Positive (semi-)definite matricesSome useful results

Multivariate distributionsLinear TransformationNormal Distribution

Central chi2, F, and tNon-CentralDistribution of Quadratic FormsMomentsIndependence

Summary

linear models. kachman, s. d. 1999

Documents