11

Matrix Algebra and Matrix Algebra and Random VectorsRandom Vectors

Shyh-Kang JengShyh-Kang JengDepartment of Electrical Engineering/Department of Electrical Engineering/Graduate Institute of Communication/Graduate Institute of Communication/

Graduate Institute of Networking and MultiGraduate Institute of Networking and Multimediamedia

22

General Statistical DistanceGeneral Statistical Distance

)])((2

))((2))((2

)(

)()([

),(

]222

[),(

),,,(),0,,0,0(),,,,(

11,1

331113221112

2

22222

21111

1,131132112

22222

2111

2121

pppppp

pppp

pppp

ppp

pp

yxyxa

yxyxayxyxa

yxa

yxayxa

QPd

xxaxxaxxa

xaxaxaPOd

yyyQOxxxP

33

General Statistical DistanceGeneral Statistical Distance

Axx'

]222

[),(

2

1

21

22212

11211

21

1,131132112

22222

2111

2

ppppp

p

p

p

pppp

ppp

x

x

x

aaa

aaa

aaa

xxx

xxaxxaxxa

xaxaxaPOd

44

Quadratic FormQuadratic Form

2332

2221

21

3

2

1

321

2221

21

2

121

1 1

246

220

213

031

)(

211

11)(

')(

xxxxxxx

x

x

x

xxxxQ

xxxxx

xxxxQ

xxaQk

i

k

jjiij

Axxx

55

Statistical Distance under Rotated Statistical Distance under Rotated Coordinate SystemCoordinate System

22222112

2111

212

211

22

22

11

21

21

2),(

cossin~sincos~

~

~

~

~),(

)~,~(),0,0(

xaxxaxaPOd

xxx

xxx

s

x

s

xPOd

xxPO

66

Rotated Coordinate SystemRotated Coordinate System

2

1

2

1

212

211

cossin

sincos~

~cossin~

sincos~

x

x

x

x

xxx

xxx

77

Coordinate TransformationCoordinate Transformation

x

b

b

b

y

Bxy

'

'2

'1

2

1

21

22221

11211

2

1

k

kkkkk

k

k

k x

x

x

bbb

bbb

bbb

y

y

y

88

Quadratic Form in Transformed Quadratic Form in Transformed Coordinate SystemsCoordinate Systems

ΛBAB

ΛABB

yΛyy'

yABBy

yBAyB

Axxx

'

)(

'

)()'(

')(

1

1'1

1'1

11

Q

Q

99

Diagonalized Quadratic FormDiagonalized Quadratic Form

k

iii

k

yQ1

2

2

1

')(

00

00

00

Λyyy

Λ

1010

Orthogonal MatrixOrthogonal Matrix

cossin

sincos

cossin

sincos

10

01

cossin

sincos

cossin

sincos

'

''

1

1 AA

IAAAA

1111

Diagonalization Diagonalization

iii

k

k

k

kkkk

k

k

aaa

aaa

aaa

bAb

bbb

bbb

ΛBAB

BB

00

00

00

''

'

2

1

21

21

21

22221

11211

1

1212

Concept of MappingConcept of Mapping

Axy x

1313

EigenvaluesEigenvalues

3,1

0)3)(1(31

01

31

01

0)(

21

IA

A

xIA

xAx

1414

EigenvectorsEigenvectors

1

0,

0

0

01

02

3

1

2,

0

0

21

00

1

0)(

22

1

2

12

1

1

x

x

xIA

x

x

x

x

ii

1515

Spectral DecompositionSpectral Decomposition

k

iiii

kk

k

1

'

'

'2

'1

2

1

21

00

00

00

'

''

ee

e

e

e

eeeA

ΛBBA

ΛBAB

1616

Positive Definite MatrixPositive Definite MatrixMatrix Matrix AA is non-negative definite if is non-negative definite if

for all for all xx’=[’=[xx11, , xx22, …, , …, xxkk]]Matrix A is positive definite if Matrix A is positive definite if

for all non-zero for all non-zero xx

0' Axx

0' Axx

1717

Positive Definite MatrixPositive Definite Matrix

definite positive is

' allfor 0'

' and orthogonal ,

','

04''4'

4

1,4

22

232223

1-

'222

'111

22

21

'22

'11

'22

'11

21

2

12121

22

21

A

0xAxx

EEEExy

xeexxeex

xeexxeexAxx

eeeeA

xAx

yy

yy

x

xxxxxxx

1818

Inverse and Square-Root MatrixInverse and Square-Root Matrix

12/12/1

2/12/1

2/12/1

2/1

1

'2/1

11

1

'

'

'

,,1,0

)'()')('(

'

,'

AA

AA

AAA

PPΛeeA

PPΛPΛΛPΛΛPΛΛA

PPΛA

k

iiii

i

nn

k

iiii

ki

PPeeA

1919

Random Vectors and Random Vectors and Random MatricesRandom Matrices

Random vectorRandom vector– Vector whose elements are random Vector whose elements are random

variablesvariables

Random matrixRandom matrix– Matrix whose elements are random Matrix whose elements are random

variablesvariables

2020

Expected Value of a Expected Value of a Random MatrixRandom Matrix

BXAAXB

X

)()(

)(

)()(

)()()(

)()()(

)()()(

)(

21

22221

11211

EE

xpx

dxxfxXE

XEXEXE

XEXEXE

XEXEXE

E

ijxallijijij

ijijijij

ij

pppp

p

p

2121

Population Mean VectorsPopulation Mean Vectors

piii

ii

i

p

p

E

XE

XE

xf

xxxff

XXX

21

22

21

21

)(

)(

)(

)(on distributiy probabilit Marginal

),,,()(

function density y probabilitJoint

' vector Random

Xμ

x

X

2222

CovarianceCovariance

ik

xall xallkiikkkii

kikiikkkii

kkiiki

i k

xxpxx

dxdxxxfxx

XXEXX

),())((

),())((

))((),Cov(

2323

Statistically IndependentStatistically Independent

general)in not true is converse (the

tindependen are , if 0),Cov(

)()()(),,,(

)()()(

][][] and[

22112112

,

kiki

pppp

kkiikiik

kkiikkii

XXXX

xfxfxfxxxf

xfxfxxf

xXPxXPxXxXP

2424

Population Variance-Covariance Population Variance-Covariance MatricesMatrices

pppp

p

p

pp

pp

XXX

X

X

X

E

E

21

22221

11211

221122

11

)Cov(

)')((

X

μXμXΣ

2525

Population Correlation CoefficientsPopulation Correlation Coefficients

1

21

22221

11211

ii

kkii

ikik

pppp

p

p

ρ

2626

Standard Deviation MatrixStandard Deviation Matrix

2/12/1

2/12/1

22

11

2/1

00

00

00

ΣVVρ

ρVVΣ

V

pp

2727

Correlation Matrix from Correlation Matrix from Covariance MatrixCovariance Matrix

15/15/1

5/116/1

5/16/11

5/100

03/10

002/1

500

030

002

00

00

00

2532

391

214

2/12/1

2/1

33

22

112/1

333231

232221

131211

ΣVVρ

V

V

Σ

2828

Partitioning Covariance MatrixPartitioning Covariance Matrix

)')(()')((

)')(()')((

)')((

,

)2()2()2()2()1()1()2()2(

)2()2()1()1()1()1()1()1(

)2(

)1(

)2(

)1(

1

1

μXμXμXμX

μXμXμXμX

μXμX

μ

μ

μ

X

X

X

p

q

q

X

X

X

X

2929

Partitioning Covariance MatrixPartitioning Covariance Matrix

'1221

1,1

,11,1,11,1

1,1

11,1111

2221

1211

|

|

|

|

|

|

|

|

|

|

)')((

ΣΣ

ΣΣ

ΣΣ

μXμXΣ

ppqppqp

pqqqqqq

qpqqqqq

pqq

E

3030

Linear Combinations of Linear Combinations of Random VariablesRandom Variables

)Cov( and )( where

')'Var( ariance v

')'(mean

has

'n combinatioLinear 11

XΣXμ

ΣccXc

μcXc

Xc

E

E

XcXc pp

3131

Example of Linear Combinations of Example of Linear Combinations of Random VariablesRandom Variables

ΣccXcVar

μcXc

Xc

'][)'(

')'(

],['],,['

2

)]()([

)]()[()Var(

)()()(

2212

1211

21

12222

112

22211

2212121

212121

b

aba

E

XXba

abba

XbXaE

babXaXEbXaX

baXbEXaEbXaXE

3232

Linear Combinations of Linear Combinations of Random VariablesRandom Variables

')Cov(

)()(

21

22221

11211

CCΣCXΣ

CμCXZμ

CXXZ

XZ

XZ

EE

ccc

ccc

ccc

pppp

p

p

3333

Sample Mean Vector and Sample Mean Vector and Covariance MatrixCovariance Matrix

n

jpjp

n

jpjpj

n

jpjpj

n

jj

ppp

p

n

p

xxn

xxxxn

xxxxn

xxn

ss

ss

xxx

1

2

111

111

1

211

1

111

21

11

11

],,['

S

x

3434

Partitioning Sample Mean VectorPartitioning Sample Mean Vector

)2(

)1(

1

1

x

x

x

p

q

q

x

x

x

x

3535

Partitioning Sample Partitioning Sample Covariance MatrixCovariance Matrix

'1221

1,1

,11,1,11,1

1,1

11,1111

2221

1211

|

|

|

|

|

|

|

|

|

|

SS

SS

SS

S

ppqppqp

pqqqqqq

qpqqqqq

pqq

n

ssss

ssss

ssss

ssss

3636

Cauchy-Schwarz InequalityCauchy-Schwarz Inequality

dd

dbbb

dd

dbdd

dd

dbbb

dddbbbdbdb

db

dbddbbdb

'

''0

'

')'(

'

''0

)'()'(2')()'(0

0

:Proof

hen equality w with ),')('()'(

2

22

2

2

x

xxxx

-x

c

3737

Extended Cauchy-Schwarz Extended Cauchy-Schwarz InequalityInequality

'

1

2/1

12

2/12/1'2/12/122/12/1

2/1'2/12/12/1

112

)')('()'(

''')''(

'''

:Proof

hen equality w with )')('('

definite positive

ii

p

ii

c

eeB

dBdBbbdb

dBBdbBBbdBbB

dBbBdBBbbIddb

dBbdBdBbbdb

B

3838

Maximization LemmaMaximization Lemma

dBd

Bxx

dx

Bxx

dBdBxxdx

dBx

dBdBxx

dx

dB

x

12

12

1

12

0

''

'

0'

)')('('

:Proof

0for when attained maximum

''

'max

orgiven vect matrix, definite positive

cc

3939

Maximization of Quadratic Forms Maximization of Quadratic Forms for Points on the Unit Spherefor Points on the Unit Sphere

) when (attained'

'max

) when (attained'

'min

) when (attained'

'max

,,,

rseigenvecto associated and 0

eseigen valuh matrix wit definite positive

11,,

0

110

21

21

1

kk

pp

p

p

k

exxx

Bxx

exxx

Bxx

exxx

Bxx

eee

B

eex

x

x

4040

Maximization of Quadratic Forms foMaximization of Quadratic Forms for Points on the Unit Spherer Points on the Unit Sphere

11

1111

1

1

2

1

2

1

1

2

1

2

2/12/12/12/1

2/12/1

'

'

'

'],001[)'(',

'

'

'

'''

''

'

'

'

','

:Proof

ee

Bee

yy

ΛyyPeyex

yy

Λyy

yy

xPPΛPPΛx

xPPx

xBBx

xx

Bxx

xPyPPΛB

p

ii

p

ii

p

ii

p

iii

y

y

y

y

4141

Maximization of Quadratic Forms foMaximization of Quadratic Forms for Points on the Unit Spherer Points on the Unit Sphere

maximum asserted

thegives0,1 Taking

'

'

,0

implies,,

21

1

1

2

1

2

'2

'21

'1

'

1

2211

pkk

kp

kii

p

kiii

ipipiii

k

pp

yyy

y

y

kiyyyy

yyy

xx

Bxx

eeeeeexe

eex

eeePyx

4242

Maximization of Quadratic Forms foMaximization of Quadratic Forms for Points on the Unit Spherer Points on the Unit Sphere

10

'0

0'0

,,,0,,,1'

0'0

0'0

01'

10

'0

0'0

01'

0'0

0'0

0'0

0

1001

0

0

max'max

min'min

max'max

',1'

k

p

kk

xx

BxxBxx

xx

BxxBxx

xx

BxxBxx

xx

BxxBxxxx

xx

xx

eexxeexxx

xxx

xxx