SVD: Singular Value Decomposition

2

Motivation

yUx

bLy

bLUx

LUA

A

matrixany :

bSSx

SSA

SSA

e-vectorsete set ofwith compl

A

11

111

1

matrixany :

bQQx

QQA

QQA

symmetricA

T

T

T

1

11

matrix :

bUVx

UVA

VUA

A

T

T

T

1

11

matrixany :

bUVx

UVA

VUA

A

T

T

T

1

11

matrixany :

Clearly the winner

Assume A full rank

3

Ideas Behind SVD

Goal: for Am×n

find orthonormal bases for C(AT) and C(A)

rowspace

Ax=0

columnspace

ATy=0

orthonormal basis in C(AT)

orthonormal basis in C(A)

ARn Rm

There are many choices of basis in C(AT) and C(A), but we want the orthonormal ones

There are many choices of basis in C(AT) and C(A), but we want the orthonormal ones

4

SVD (2X2)

2

2

2

2

1

1

1

1

21

21

21

,

:)(in imagesr want thei we

:)(in rsunit vecto

2 Assume

Av

Av

AvAv

Av

Av

T

uu

AvAvACAND

vvAC

r

: represent the length of images; hence non-negative

: represent the length of images; hence non-negative

UAV

uuuuvvA2

121221121

I haven’t told you how to find

vi’s (p.9)

5

SVD 2x2 (cont)

TVUVUA

UAV

1

When A has complete set of e-vectors, we have

AS=S , A=SS

but S in general is not orthogonal

When A has complete set of e-vectors, we have

AS=S , A=SS

but S in general is not orthogonal

Another diagonalization using 2 sets of orthogonal bases

Another diagonalization using 2 sets of orthogonal bases

When A is symmetric, we have

A=QQT

When A is symmetric, we have

A=QQT

Compare

6

Why are orthonormal bases good? ( )-1=( )T

Implication: Matrix inversion

Ax=b TT

T

UVVUA

VUA111

system diagonal ... bUxV

bxVU

bAx

VUA

TT

T

T

7

More on U and V

TTTTT

TTTTT

UUUVVUAA

VVVUUVAA

22

21

22

21

Similarly,

V: eigenvectors of ATA

U: eigenvectors of AAT

V: eigenvectors of ATA

U: eigenvectors of AAT

[Find vi first, then use Avi to find ui

This is the key to solve SVD

8

SVD: A=UV

The singular values are the diagonal entries of the matrix and are arranged in descending order

The singular values are always real (non-negative) numbers

If A is real matrix, U and V are also real

9

Example (2x2, full rank)

2121

2121

20

022

10

01

1

0,

1

0

2

0

0

1,

0

1

0

22

21

21,

21

21,

53

35

11

22

222

111

21

T

T

VUA

uAv

uAv

vvAA

A

STEPS:1. Find e-vectors of ATA;

normalize the basis2. Compute Avi, get i

If i 0, get ui

Else find ui from N(AT)

STEPS:1. Find e-vectors of ATA;

normalize the basis2. Compute Avi, get i

If i 0, get ui

Else find ui from N(AT)

10

SVD Theory

If j=0, Avj=0vj is in N(A) The corresponding uj in N(AT)

[UTA=VT=0]

Else, vj in C(AT) The corresponding uj in C(A)

#of nonzero j = rank

rjuAv

UAV

jjj ,,2,1,

11

11

11

2

1

00

010

21

12

5

1

11

22

2

1

5

100)(

1

1

2

10)(

101

2

5

1

1

1

2

1

11

22

1

1

2

1;

1

1:basis )(,1,

11

22

122212

22212

11

111

1

T

TTT

T

VU

uAuuAANuuu

vAvANvvv

uAv

vACrA

Example (2x2, rank deficient)

Can also be obtained from

e-vectors of ATA

12

Example (cont)

11

11

2

1

00

010

21

12

5

1

11

22 TVU

TT

T

TT

vuv

uu

v

vuuVUA

11111

21

2

1121

0

00

0

Bases of N(A) and N(AT) (u2 and v2 here) do not contribute the final result. The are computed to make U and V orthogonal.

Bases of N(A) and N(AT) (u2 and v2 here) do not contribute the final result. The are computed to make U and V orthogonal.

13

Extend to Amxn

0

,,1,

1

1111r

mrrnrr

iii

uuuuvvvvA

riuAv

Basis of N(A) Basis of N(AT)

T

mmmmT

mm

TTTmm

T

Tnnnn

Tnn

TTTnn

T

Tnnnmmmnm

VUVUVUAA

VVVUVUAA

VUA

UAV

T

mmmmT

mm

TTTmm

T

Tnnnn

Tnn

TTTnn

T

Tnnnmmmnm

VUVUVUAA

VVVUVUAA

VUA

UAV

Dimension Check

14

Extend to Amxn (cont)

Trrr

T

Tm

Tr

Tr

T

rr

rmrrnrr

vuvu

v

vv

v

uuA

uuuuvvvvA

1111

1

11

1

1111

00

0

Bases of N(A) and N(AT) do not contribute

They are useful only for nullspace solutions

Summation of r rank-one matrices!

15

110

121

011

:

,2,110

01132223332

AAV

UVArA

T

)(nullspace

1

1

1

3

1,0

1

0

1

2

1,1

1

2

1

6

1,3

33

22

11

v

v

v

1,1

1

2

1

3,1

1

6

23

3

3

6

1

22

121

1

Av

Av

31

31

31

21

21

21

62

61

31

21

61

31

62

31

21

61

01

3

11

11

2

1

110

011

1

3

11

11

2

10

110

011

C(AT) N(A) C(A)

16

2

1

332211

of Expansion

707.0

0

707.0

542.0

643.0

542.0

454.0

776.0

454.0

, of vectors Eigen

A of rank values singular zero-non of Number,

0

14.0

86.28

, of values Eigen

6

10

6

106

1710

106

,

1

2

1

21

32

21

:Example

i

Tiii

TT

TT

TT

vuA

A

vuvuvu

AAAA

AAAA

AAAAA

17

Summary

SVD chooses the right basis for the 4 subspaces AV=U

v1…vr: orthonormal basis in Rn for C(AT) vr+1…vn: N(A) u1…ur: in Rm C(A) ur+1…um: N(AT)

These bases are not only , but also Avi=iui

High points of Linear Algebra Dimension, rank, orthogonality, basis, diagonalization, …

18

SVD Applications

Using SVD in computation, rather than A, has the advantage of being more robust to numerical error

Many applications: Inverse of matrix A Conditions of matrix Image compression Solve Ax=b for all cases (unique, many, no solutions; least

square solutions) rank determination, matrix approximation, …

SVD usually found by iterative methods (see Numerical Recipe, Chap.2)

19

otherwise

tifUVVUA

A

diagUVAorVUA

iA

iiTT

nTT

i

0

/1 where

dconditione-ill or singular is

/1,....,/1,/1 where

all for0 iff rnonsingula is

Inverse Matrix

10

10

11

21111

nσσA

xb

bAx

/ =:y singularit of Degree

in changes large produces in changes small when dconditione-Ill

=

equationslinear of System

matrix a ofy singularit of Degree

1

20

SVD and Ax=b (mn)

Check for existence of solution

existnot doessolution

,0but 0 If

ii

d

T

z

TT

d

dz

bUxVbxVU

21

Ax=b (inconsistent)

52

520

3

8

21

12

5

1

00

010

11

11

2

1

00

010

21

12

5

1

11

22

3

8,

11

22

2

1 bUz

z

VU

bA

T

T

No solution!

22

Ax=b (underdetermined)

21

21

2

2

2

2

0

22

11

11

2

1

0

22

0

520

4

8

21

12

5

1

00

010

11

11

2

1

00

010

21

12

5

1

11

22

4

8,

11

22

2

1

2

1

cxxx

Vzxz

z

bUz

z

VU

bA

nullparticularcomplete

particular

T

T

23

Pseudo Inverse (Sec7.4, p.395) The role of A:

Takes a vector vi from row space to iui in the column space

The role of A-1 (if it exists): Does the opposite: takes a vector ui from column

space to row space vi

ii

iii

iii

vuA

uAv

uAv

i

11

1

ii

iii

iii

vuA

uAv

uAv

i

11

1

24

Pseudo Inverse (cont)

While A-1 may not exist, a matrix that takes ui back to vi/i does exist. It is denoted as A+, the pseudo inverse

A+: dimension n by m

T

mr

r

nr

Tmmmnnn

iii

i

uuuvvv

UVA

riuArivuA

11

11

1

for 0 and for 1

25

Pseudo Inverse and Ax=b

Overdetermined case: find the solution that minimize the error r=|Ax–b|, the least square solution

A panacea for Ax=bA panacea for Ax=b

bUVbAx

bAxT

bUVbAx

bAxT

bAxAA TT ˆ Compare

26

Ex: full rank

1

1

2

0

10

01

0

0

2121

2121

)/1(diag

2121

2121

20

022

10

01

2

0,

11

22

21

221

x

bUVxbxVUbAx

VUA

bA

Ti

T

T

27

Ex: over-determined

T

T

UVA

VUA

UVArbA

11

11

2

11

3

0

10

11

01

,2,

1

2

1

,

10

11

01

31

21

61

31

62

31

21

61

23332223

Will show this need not be computed…

28

Over-determined (cont)

31

35

322

226

4

31

21

21

61

62

61

31

211

11

2

1

2111

11

2

1

1

2

1

0111

11

2

1

zyx

zyx

bUVx T

31

35

2

1

2

1

1

3

21

12

ˆ Compare

x

x

x

x

bAxAA TT

31

35

2

1

2

1

1

3

21

12

ˆ Compare

x

x

x

x

bAxAA TT

Same result!!

29

Ex: general case, no solution

1019

1019

101

51

101

51

51

52

521

521

51

52

101

212

1

3

8

3

8

0

0

3

8

00

0

11

11

2

1

00

010

21

12

5

1

11

22

3

8,

11

22

yx

yxy

x

bUVx

VU

bA

T

T

3

822

yx

yx

30

Matrix Approximation

nm

nm

vuvuvuA

AiA

imi

VUA

Tmmm

TT

i

i

Tii

:saveOperation

numbers )(matrix onerank :save Storage

distanceEuclidean of sence in the ion toapproximat rank best the:

zero) tos' last setting(by of version rank the:

222111

making small ’s to zero and back substitute(see next page for application in image compression)

31

Image Compression

As described in text p.352 For grey scale images: mn bytes

Only need to store r(m+n+1)

Trrr

TT vuvuvuA

rSVD

222111

: termst significanmost the taking,After

r = 1,3,5,10,16 (no perceivable difference afterwards)Original

6464

32

33