ignou - lecture notes -linear algebra (1)

7/23/2019 IGNOU - Lecture Notes -Linear Algebra (1)

1/29

Block 5 Basis of Multivariate Normal (MVN).

0


2/29

UNIT 1 RECA !" #INEAR A#$EBRA

%tructure

1.1 Introduction

Objectives

1.2 Real Symmetric Matrices

Spectral decomposition theorem

1.3 Positive deinite and nonne!ative deinite matrices

S"uare root method

1.# Idempotent matrices

1.$ %ochran&s 'heorem

1.( Sin!ular )alue *ecomposition

1.+ Summary

1., Reerence

1.- Solutions /nsers

1.1 INTR!&UCTI!N

In this unit e recapitulate certain concepts and results hich ill be useul in the

study o multivariate statistical analysis. e start ith the study o real symmetric

matrices and the associated "uadratic orms. e deine a classiication or the

"uadratic orms and develop a method or determinin! the class to hich a !iven

"uadratic orm belon!s. Positive deinite and nonne!ative deinite matrices hich

e shall notice in unit 2 as the dispersion matrices4 are very important or the study o

multivariate distributions and in particular the multivariate normal distribution. In

the section 1.3 e obtain some characteri5ations o positive deinite and nonne!ativedeinite matrices and study some o their useul properties. e !ive a method o

computin! a s"uare root o a matrices hich play an important role in transormin!

correlated random variables to uncorrelated random variables as e shall see in the

unit 2. Idempotent matrices and %ochran&s theorem play a 6ey role in the distribution

o "uadratic orms in independent standard normal variables particularly in

connection ith the distribution o "uadratic orms to be independent chi7s"uares. In

sections 1.# and 1.$ e study the properties o idempotent matrices and prove the

al!ebraic version o %ochran&s theorem respectively. Sin!ular value decompositions

plays a very important role in developin! the theory and studyin! the properties o

canonical correlations beteen to random vectors. In section 1.( e study the

sin!ular value decomposition.

1


3/29

e use the olloin! notations. Matrices are denoted by boldace capital letters li6e

A' B' C. )ectors are denoted by boldace italic loer case letters li6ex'y, z. Scalars

are denoted by loer case letters li6e a b . 'he transpose ran6 and trace o amatri8 A are denoted by At ran6 A and trA4 respectively. Rndenotes the n7

dimensional 9uclidean space.

!ectives *

/ter completin! this unit you should be able to

*etermine the deiniteness o a !iven "uadratic orm /pply the spectral decomposition in the study o principal components %ompute a trian!ular s"uare root o a positive deinite matri8 /pply the properties o positive and nonne!ative deinite matrices to the

problems that ill be studied in unit 2 on multivariate normal distribution.

/pply %ochran&s theorem to the distribution o "uadratic orms in normal

variables /pply the sin!ular value decomposition in the development o canonical

correlations.

1.+REA# %,MMETRIC MATRICE%

Real symmetric matrices play a very important role in the study o Multivariate

statistical analysis. :or e8ample the variance7covariance matrices are real symmetric

matrices. 'hey also play crucial role in the distribution o "uadratic orms in

correlated normal random variables. e shall denote the i, j4th

element o a matri8Aby aij. 'hen e rite A; aij44.

*einition< / s"uare matri8 Ao order nx nis said to be a real symmetric matri8

i i4 all the elements o A are real and ii4 aij; aji or i j; 1 = n.

98ample 1. hich o the olloin! are real symmetric matrices>

i4

3

2

i

iii4

321

321iii4

12

31iv4

$#

#1

Solution 'he

olloin! results ill be useul toards that end.

#


6/29

'heorem 1. %onsider a "uadratic orm @x4 ;xtAxhere A is symmetric. Ma6e a

nonsin!ular linear transormation o the variables< y; Tx here T is nonsin!ular4

call the transormed "uadratic orm as 4.4 11 yyyt

t = ATT 'hen the ran!es o

@x4 and (y)are the same.

Proo< Aet belon! to the ran!e o @x4. So there is a vectorx0such the ; @x04 ;

00 xxtA . ritey0; Tx0. Ho ; 00 xx

tA ; 0

11

0 xxttt

TATTT ; 0

11

0 yyt

t ATT ;

(y0). Eence belon!s to the ran!e o (y). 'hus the ran!e o @x4 is a subset o theran!e o (y). Since Tis nonsin!ular by reversin! the ar!uments e can sho that

the ran!e o (y)is a subset o the ran!e o @x4.

hat e are sayin! throu!h the 'heorem 1 is that the ran!e o a "uadratic orm is

invariant under nonsin!ular linear transormations. 'hus the deiniteness o a

"uadratic orm is invariant under nonsin!ular linear transormations. Ma6in! a

nonsin!ular linear transormation can also be interpreted as chan!in! the basis as as

done in section 1#.# o M'9702.4

Recall that a real s"uare matri8 %is called an ortho!onal matri8 i %t; %71. It is easy

to see that i % and T are ortho!onal matrices o the same order then so is %Tor

Tt%t% T ; TtIT; I. Similarly % T Tt%t; I. Eence %T4t; Tt%tis the inverse o

S'. /lso it is easy to veriy that

T!

!1 is an ortho!onal matri8 i T is an

ortho!onal matri8.

Our object is to determine the deiniteness o a "uadratic orm @x4 the matri8 o

hich is not necessarily dia!onal. e shall no sho that e can ma6e anortho!onal transormation o the variables i.e. e can ma6e a transormation y; x

here is an ortho!onal matri84 such that under this transormation the "uadratic

orm is transormed into a "uadratic orm 2iiy . Since e 6no ho todetermine the deiniteness o 2iiy and since the deiniteness o 2iiy is thesame as that o @x4 e have the deiniteness o @x4.

I A is a real matri8 then it is not necessary that its ei!en values are real. :or

e8ample i A;

01

10 then the ei!en values are iand ?i. Eoever i Ais real

and symmetric then all its ei!en values are real as shon belo.

'heorem 2. Aet Abe a real symmetric matri8. /ll the ei!en values o Aare real and

all the ei!en vectors o Acan be chosen to be real.

Proo< Aet +ibe an ei!en value o Aand let the correspondin! ei!en vector be

x+iyhere and are real numbers andxandyare real vectors. %learly at least oneoxandyis nonnull asx+iy bein! an ei!en vector is nonnull. Ho

Ax+iy4 ; +i4x+iy4

9"uatin! the real parts on both sides and the ima!inery parts on both sides o theabove e"uality e !et

$


7/29

Ax; x-y = = = = 2.14Ay; y+x = = = = 2.24

Premultiplyin! 2.14 byytand 2.24 byxt e !et

ytAx; ytx-yty = = = = 2.34xtAy; xty+xtx = = = = 2.#4

Since Ais symmetric andytAxis a scalar e haveytAx; ytAx4t;xtAty;xtAy.

Similarlyytx;xty. Ho subtractin! 2.34 rom 2.#4 e !et

xtxyty4 ; 0. Since at least one ox andyis non7nullxtxyty0.

Eence all the ei!en values o Aare real. :urther Ax+iy4 ; x+iy4 yields Ax; x

and Ay; y. Since at least one oxandyis non null andxandyare real e canchoose the non7null vector beteenxandyas an ei!en vector o Acorrespondin! to

.

Ho e are ready to prove an important result concernin! the real symmetric

matrices namely the spectral decomposition theorem.

'heorem 3. Aet Abe a real symmetric matri8 o order n x n. 'hen there e8ists a real

ortho!onal matri8 o order n x nsuch that A ; there is a real dia!onal

matri8.

Proo< e shall prove the theorem by induction on n. Aet Abe a 1 x 1 real symmetric

matri8 i.e. A; ahere a is a real number. %learly 1t.a.1 ; 1.a.1 ; a. /lso 1 is an

ortho!onal matri8 o order 1 x 1 since 1t.1 ; 1.1 ; 1. So the theorem is true or n ; 1.

Aet the theorem be true n ; ma possible inte!er 14. Aet Abe a matri8 o orderm14 x m14. Aet x1 be a normali5ed ei!en vector o Acorrespondin! to ei!en

value 1. 'hen Ax1; 1x1. 'henx1can be e8tended to an orthonormal basisx1=xn o R

m+1. See unit 12 o M'97024. rite R ; x1


8/29

here is a real dia!onal matri8. ritin! % ;

1

1

%-

- e notice that % is an

ortho!onal matri8 and

RtAR; t%0-

-%

or A; ttR%0-

-R%

rite ; R%. Since R and % are ortho!onal matrices so is as noticed earlier.

ritin!

=0-

-

e observe that is a real dia!onal matri8.

'hus the theorem is true or n ; m1.

Eence the theorem ollos by induction on n.

'he beauty o the theorem 3 lies in its interpretation. Aet Abe a real symmetric

matri8 and let A; there is ortho!onal and is a real dia!onal matri8. e

then have

A ; or Ap1


9/29

98ample #. Obtain a spectral decomposition o the matri8 A 4

21

1#.

Solution< 'he characteristic e"uation o the matri8 is

21

1#; 0

or # 7 42 7 4 ? 1 ; 0

or 27 ( + ; 0

'he roots are 232

2,3((=

So the ei!en values are 23+ and 23 .

Aet

=

2

1

u

uu be the ei!en vector o the !iven matri8 correspondin! to the ei!en value

23+ .

'hen -IA =+ uK423J or -=

2

1

211

121

u

u

Hotice that the second column o ( )IA 423 + is ( 21 times the irst column.So 1and21 21 =+= uu satisy the e"uation ( ) -IA =+ u423

'o normali5e u e divide it by its norm namely ( ) 22#121 22 +=++ . 'hus thenormali5ed ei!en vector correspondin! to the ei!en value 23+ is

+

+ 1

21

22#

1. It can be shon similarly that the normali5ed ei!en vector

correspondin! to 23 is ( )

++ 211

22#

1. Eence A ; t here

( )

++

+=

211

121

22#

1 and

+=

230

023 .

sin! theorem 3 e can determine the deiniteness o a "uadratic orm. %onsider the

"uadratic orm @x4 ;xtAx. Aet A; tbe a spectral decomposition o A. 'hen

@x4 ;xtx;ytyherey; tx. Since is nonsin!ular in act ortho!onal4 thedeiniteness o @x4 is the same as the deiniteness oyty. 'he deiniteness oyty

is determined by the dia!onal elements 1=. nare the ei!en values o A.

'hus

,


10/29

xtAxis

=>=

ji

ji

i

ji

i

ji

ji

i

ji

i

someor0andsomeor0iindeinite

someor0andallor0itesemideinine!ative

allor0ideinitene!ative

someor0and0itesemideinipositive

allor0ideinitepositive

Lecause o the one7one correspondence beteen real symmetric matrices and the

"uadratic orms e call a real symmetric matri8 A as positive deinite positive

semideinite ne!ative deinite ne!ative semideinite or indeinite accordin! as the

"uadratic orm xtAx is positive deinite positive semideinite ne!ative deinite

ne!ative semideinite or indeinite respectively.

98ample #. *etermine the deiniteness o the "uadratic orms i42

221

2

12 xxxx + andii4 323121

2

3

2

2

2

1 333 xxxxxxxxx ++ .

Solution< i4 'he matri8 o the "uadratic orm is A;

1$.0

$.02.

e see that the characteristic e"uation A7 I ; 0 is274174 0.2$ ; 0 or 27 3 2.2$ ; 0

Eence the ei!en values hich are the roots o the above e"uation are 2

--3

or32 and 32.

Eence the "uadratic orm is positive deinite.

ii4 'he matri8 o the "uadratic orm is A;

1$.1$.1

$.11$.1

$.1$.11

It is easy to notice that the sum o each ro in Ais 72.

Eence A

=

11

1

211

1

. 'hus 72 is an ei!en value o A. :urther the sum o the ei!en

values hich is the same as the trace o A is 3. Eence there must be at least one

positive ei!en value o A. So the "uadratic orm is in deinite.

92. Aet Abe a real symmetric matri8 a dia!onal element o hich is ne!ative.

Sho that Acannot be positive deinite or positive semideinite.

93. *etermine the deiniteness o the olloin! "uadratic orms

--

-I ris an

e8ample o an idempotent matri8 o ran6 r.

:urther i Ais an idempotent matri8 and is a nonsin!ular matri8 o the same order

then A71. A71; A271; A71.

'hus A71is an idempotent matri8.

Eence

--

-Ir71is an idempotent matri8 o ran6 ror every nonsin!ular matri8 .

e shall no sho that every idempotent matri8 o ran6 r is o the orm

--

-Ir71or some nonsin!ular matri8 .

'heorem -. Aet / be an n xnmatri8 o ran6 r1 rn714. 'hen Ais idempotent i

and only i A;

--

-Ir71or some nonsin!ular matri8 .

Proo< I& part has already been proved above. :or the only i& part let Abe an

n xnidempotent matri8 o ran6 r1 rn714. Aet A; a1


18/29

Akk ll

ae ; 0 $; 1= n-r. 1.#.24

%onsider ;rnrnr llllii

aeaeaa


19/29

Solution< 'he i,i4th element o AB is !iven by =

n

j

jiijba

1

. Eence trAB4 ;

= =

m

i

n

j

jiijba

1 1

; = =

n

j

m

i

jiijba

1 1

; = =

n

j

m

i

ijjiab

1 1

; trBA4 since =

m

i

ijjiab

1

is the j,j4th

element o BA.

e are no ready to prove

'heorem 10. Aet Abe an idempotent matri8 o order n xn. 'hen ran6 A; trA4.

Proo< 'he proo is trivial i ran6 Ais 0 or n. Aet ran6 A; rhen 1 rn71. 'hen

by theorem - there e8ists a nonsin!ular matri8 such that A;

--

-Ir71.

Ho trA4 ;

17

--

-I

rtr ;

--

-I17rtr ; rtr

r =

--

-I; ran6 A.

e state belo another result on idempotent matrices.

'heorem 11. / s"uare matri8 Ao order n xn is idempotent i and only i ran6 I7A4

; n7 ran6 A.

:or proo one may reer to Rao and Lhimasan6aram 20004 Pa!e 13#.

'heorem 12. Aet Abe a real symmetric and idempotent matri8 o ran6 r. 'hen there

e8ists an ortho!onal matri8 % such that A ; %

--

-I r%t. Eence A is nonne!ative

deinite.

Proo< Aet as an e8ercise.

98ample #. Aet Aand Bbe idempotent matrices o the same order. 'hen sho that

ABis idempotent i and only i AB; BA; -.

Solution< I& part is trivial. :or the only i& part let A Band ABbe idempotent.

'hen AB; AB4AB4 ; A2B2AB BA; A B AB BA. So ABBA;

-. Premultiplyin! by A e !et ABABA; -. Ho post multiplyin! the previous

e"uality by A e !et ABAABA ; - or ABA ; -. Eence AB ; - and as a

conse"uence BA ;-.

913. Aet Abe a 2 x2 idempotent matri8. %an a11be e"ual to 2>

91#. Aet A and B be idempotent matrices. 'hen sho that

B-

-A is also

idempotent.

1,


20/29

91$. Sho that i Aand Bare idempotent and the column space o Ais contained

in the column space o B the BA; A.

1.5 C!C8RAN9% T8E!REM

%ochran&s theorem concerns the distributions o "uadratic orms in independent

standard normal variables. Aetx;

nx

x

1

be a vector o independent standard normal

variables. Aet A1 A2=A$ be real symmetric nonrandom4 matrices such that

xtx;xtA1xxtA2x = x

tA$x. e 6no thatxtxis distributed as chi7s"uare ith

nde!rees o reedom. %ochran&s theorem asserts thatxtAix i; 1= $are distributed

as independent chi7s"uares i and only i nran$n

i

i=

=1

A . In this section e prove an

al!ebraic version o this result. In the ne8t unit e shall prove the statistical version.

'heorem 13. Aet A1 A2=A$be real symmetric matrices such that A1 A2 = A$; I. 'he olloin! are e"uivalentIdentiy the correspondin! ei!en vectors. hat is the ran6 o A>

Solution< A; U

--

-:Vt

here U;

122

212

221

3

1 and V;

1111

1111

1111

1111

2

1

are ortho!onal matrices and ;

10

02

'he ei!en values o AAW are2

1 ; #2

2 ; 1 and 0. 'he correspondin! ei!en vectors

are the irst second and third columns respectively o namely

2

2

1

3

1

2

1

2

3

1

and

1

2

2

3

1respectively.

e leave it as an e8ercise to identiy the ei!en values and ei!en vectors o AtA.

22


24/29

'he ran6 o Ais the same as the number o ros o namely 2.

e remar6 here that the non5ero ei!en values o AAt and AtA are the same. In act

or any to matrices Aand Bo order m x nand n x mrespectively the non5ero ei!en

values o ABand BAare the same. 'he proo is beyon! the scope o this notes. :or

a proo e reer the reader to Rao and Lhimasan6aram 20004 pa!e 2,2.4

1.< %UMMAR,

In this unit e have covered the olloin! points %!#UTI!N% T! E?ERCI%E%

91. i4 %oeicient o2

1x 2

2x andx1x2are respectively 1 71 and 0. So the matri8

o the "uadratic orm 2221 xx is 10

01 .

ii4 %oeicients o2

1x 2

2x andx1x2 are respectively 2 $ and 3. So the

matri8 o the "uadratic orm is

$$.1

$.12.

iii4 %oeicients o2

1x 2

2x 2

3x x1x2x1x3andx2x3are respectively 0 0 0 3

7# $. So the matri8 o the "uadratic orm *x1x + xx* x1x* is

0$.22

$.20$.1

2$.10

.

iv4 %oeicients o2

1x 2

2x 2

3x 2

#x x1x2 x1x3 x1x# x2x3 x2x# andx3x# are

respectively 1 1 0 1 0 0 0 0 0 0. So the matri8 o the "uadratic orm2

1x

2

2x 2

#x is

1000

0100

0010

0001

.

92. Suppose aii D 0. Aet ei denote the ith column o the identity matri8. 'hen

i

t

i ee A ; aiiD 0. Eence Acannot be pd or psd.

93. i4 'he matri8 o the "uadratic orm2

221

2

1 +$ xxxx + is A;

+$.2

$.21.

'he ei!en values o / are the roots o the characteristic e"uation A7I;0 or174 +74 ? (.2$ ; 0.

'he %haracteristic e"uation can be reritten as 27 , 0.+$ ; 0.

Eence the roots are2

3(#, + and

2

3(#, hich are both positive.

Eence the "uadratic orm2

221

2

1 +$ xxxx + is positive deinite.

ii4 :orx1; 1 andx2;x3; 0 the value o the "uadratic orm is 1. /!ain orx2; 1x1; x3; 0 the value o the "uadratic orm is 71. Eence the "uadratic

orm is indeinite.

2#


26/29

iii4 'he matri8 o the "uadratic orm is A;

#00

033

032

. 'he characteristic

e"uation o / is #74274374 7 -4 ; 0. So # is a root o the above e"uation.

'he remainin! to roots are the roots o the e"uation 274374 7 - ; 0 or2 7 $ 7 3 ; 0 So the roots are

2

122$$ . 'hus all the three roots are

positive. Eence the !iven "uadratic orm is positive deinite.

9#. 'he ei!en values o / ;

21

12are the roots o the characteristic e"uation

274271 ; 0 or 27 # 3 ; 0 or 734714 ; 0. So the ei!en values are 1

; 3 and 2; 1. Aet

2

1

x

xbe an ei!en vector correspondin! to 1. 'hen

2x1x2 ; 3x1x1 2x2 ; 3x2

or 7x1x2 ; 0

x17x2 ; 0

'husx1;x2. So the normali5ed ei!en vector correspondin! to the ei!en value

3 is

1

1

2

1.

It can similarly be shon that

1

12

1 is the normali5ed ei!en vector

correspondin! to the ei!en value 1. So the spectral decomposition o Ais

A; ( ) ( )111

1

2

1.111

1

1

2

1.3

+

or

11

11

2

1

10

03

11

11

2

1

I Anxn; tis a spectral decomposition o A then A2; tt; 2t.

Ly induction it can be shon that A$; $t. or $; 12=..

'hus i 1= nare the ei!en values o A then $n$

1 are the ei!en values

o A$. 'he ei!en vectors o A$ can be ta6en to be the same as the ei!en

vectors o A.

Eence A100;

11

11

2

1

10

03

11

11

2

1 100

.

2$


27/29

9$. i4 Aet

=

22

2111

2221

11

0

0

21

1#

b

bb

bb

b

So211b ; # or b11; 2

b21b12 ; 1 or2

11

11

12 ==b

b

22

22

2

21 =+bb or

#

+

#

12

2

22 ==b

so2

+22=b

'hus the re"uired loer trian!ular s"uare root is

2

+

2

1

02.

ii4 Aet

=

33

3222

312111

333231

2221

11

00

00

00

(13

1$3

33-

b

bb

bbb

bbb

bb

b

so211b ;- or b11; 3

b11b21 ; 3 or b21; 1

b11b31 ; 3 or b31; 1

$2

22

2

21 =+bb or2

22b ; $71 ; # or b22; 2

b21b31 b22b32 ; 1 or b22b32 ; 171 ; 0 or b32 ; 02

33

2

32

2

31 bbb ++ ; ( or233b ; ( 7 0 7 1 ; $ or b33 ; $ .

'hus the re"uired trian!ular s"uare root is

$01

021

003

9(. I& part< let Cand &be positive deinite. 'hen

( ) 0< >+=

yyxx

y

xyx &C

&-

-Ctttt

henever at least one o x and y is

non7null. Eence

&-

-Cis pd.

Only i& part< Aet

&-

-Cbe pd consider

( ) xxx

x C-&-

-C-

tt =

<


28/29

9+. e 6no that ( ) yyy

y22


29/29

in act choose bany non5ero number and ;b

24

'hus

11

22is idempotent. Eence there is an idempotent matri8 ith a11;

2.

91#.

=

=

B-

-A

B-

-A

B-

-A

B-

-A

+

+

So

B-

-Ais idempotent i Aand Bare idempotent.

91$. Since the column space o Ais contained in the column space o B e !et

A; B&or some &.

Ho BA; B.B&; B&; A

re@are /Prof. P Bhimasankaram, Iia %tatistical Istitute' 8/eraa.

ignou - lecture notes -linear algebra (1)

Documents