D. van Alphen 1

ECE 455 – Lecture 12

• Orthogonal Matrices

• Singular Value Decomposition (SVD)

• SVD for Image Compression

Recall: If vectors a and b have the same orientation (both row vectors or both column vectors, then a’b = b’a = dot(a,b)and

dot(a, a) = length2(a)

D. van Alphen 2

Orthogonal Matrices

• Definition: A square matrix is orthogonal () if its columns are orthonormal (i.e., orthogonal and of unit-length).

• Example: Q =

• Let Q be an matrix with columns q1, q2, …, qn.

• Then Q’ Q =

• Note (1) qi’ qi = |qi|2 = 1 (since vectors are unit-length);

and (2) qi’ qj = 0 (since vectors are orthogonal)

2

1

2

12

1

2

1

n

n

n

I

qqq

q

q

q

21

'

'2

'1

Diagonal elements are 1 by note (1) below; off-diagonal elements are 0, by note (2) below)

D. van Alphen 3

Orthogonal Matrices, continued

• So far: Q Q = In

• Similarly, Q Q = In

• Thus, Q-1 = Q

• More properties of orthogonal matrices:

– If Q is orthogonal, so is Q.– The rows of any orthogonal matrix are also orthogonal.

• Example 1: Q =

Q-1 = Q =

cossin

sincos

cossin

sincos

Note: This particular Q is called a rotation matrix.

Multiplying a (2, 1) vector x by Q will rotate x by angle .

D. van Alphen 4

Orthogonal Matrices, continuedRotation Matrix Example

• Let x = and let = 90 in the rotation matrix.

• Then

2

1

1

2

2

1

01

10

2

1

)90cos()90sin(

)90sin()90cos(

xQ

xQ x 90 Note: the length of

the vector was not changed.

D. van Alphen 5

Claim: Orthogonal Matrices Preserve Length

• Multiplying x by an orthogonal matrix Q does not change the length:

| Q x | = | x |

• Proof: | Q x | 2 = (Q x) (Q x) = x Q Q x = x x = | x | 2

• Example: Multiply [x y]’ by “plane rotation” matrix Q:

z = Q

| z | 2 = (x cos - y sin )2 + (x sin + y cos )2

= (x2 cos2 + y2 sin2 – 2xy sin cos

(x2 sin2 + y2 cos2 + 2xy sin cos

= (x2 + y2)(sin2 + cos2 x2 + y2 = |[ x y]’ |2

I, since Q-1 = Q

cosysinx

sinycosx

cossin

sincos

y

x

D. van Alphen 6

Orthogonal Matrices, ContinuedPermutation Matrix Example

• Recall permutation matrix Pij was used in Gaussian Elimination to exchange (or swap) 2 rows.

• Example: to find the 3-by-3 permutation matrix that swaps rows 1 and 2 by pre-multiplying a matrix, we started with identity matrix I3 and swapped rows 1 and 2:

I3 =

• Note that the columns of P12 are unit-length, and orthogonal to each other; hence P12 is an orthogonal matrix.

• Claim: All permutation matrices are orthogonal.

12P

100

001

010

100

010

001

R1 R2

D. van Alphen 7

Singular Value Decomposition (SVD)

• A way to factor matrices: A = U V

– where A is any matrix of size (m, n);

– U and V are orthogonal matrices; • U contains the “left singular vectors”• V contains the “right singular vectors”; and

– is a diagonal matrix, containing the “singular values” on the diagonal

• Closely related to Principal Component Analysis

• Mostly used for data compression – particularly for image compression

D. van Alphen 8

SVD Format

• Suppose that A is a matrix of size (m, n) and rank r. Then the SVD

• A = U V = [u1 … ur … um]

'

'

'11

0

0

n

rr

v

v

v

(m, m) (m, n) (n, n)

D. van Alphen 9

SVD Factorization: Finding and V

• Start with: A = U V (1)

• Note that U U = I, since U is orthogonal. Hence left-multiply both sides of equation (1) by A :

A A = (U V) U V

= V U U V = V V

• Note that is a diagonal matrix, with the elements i2 on the diagonal

• The eigenvalues of (A A) are the i2’s, and the eigenvectors are the

columns of matrix V.

I

D. van Alphen 10

SVD Example

• Let A =

• A’ A =

• MATLAB Code to find eigenvalues and eigenvectors of A’A:

>> [v d] = eig([5 -3; -3 5]) v =

-0.7071 -0.7071 -0.7071 0.7071

d = 2 0 0 8

11

22

53

35

11

22

12

12

sqrt

sqrt(

7071.

7071.,

7071.

7071.11 vv

D. van Alphen 11

SVD Factorization: Finding U

• Start with: A = U V (1)

• Right-multiply both sides of equation (1) by A :

A A = U V (U V)

= U V V U = U U

• As before, is a diagonal matrix, with the elements i2 on the

diagonal

• The eigenvectors of (A A) are the columns of matrix U.

I

D. van Alphen 12

SVD Example, continued

• Recall A =

• A A’ =

• MATLAB Code to find eigenvalues and eigenvectors of AA’:

>> [U d] = eig([8 0; 0 2])U =

0 1 1 0

d = 2 0 0 8

11

22

20

08

12

12

11

22

1

0,

0

121 uu

D. van Alphen 13

SVD Example, continued

• Recall A = = U V

– where U =

• Hence, A = U V

11

22

20

022

10

01

7071.7071.

7071.7071.V

10

01

20

022

7071.7071.

7071.7071.

D. van Alphen 14

SVD for Data Compression

• Recall A = U V = [u1 … ur … um]

A = 1 u1 v1 + 2 u2 v2 + … + r ur vr (2)

• If A has rank r, then only r terms are required in the above formula for an exact representation of A.

• To approximate A, we use fewer than r terms in equation (2).

– Since the i’s are sorted from largest to smallest value in , eliminating the last “few” terms in (2) have a small effect on the image.

'n

'r

'1

r

1

v0

0

v

v

D. van Alphen 15

Example: Clown Image

50 100 150 200 250 300

20

40

60

80

100

120

140

160

180

200

50 100 150 200 250 300

20

40

60

80

100

120

140

160

180

200

(200, 320) (200, 320)

Original Image, Uncompressed;

Full rank: r = 200

Compressed Image using only the first 20 singular values

D. van Alphen 16

Clown Image Compression: MATLAB Code

% program clown_svdload clown % uncompressed image, stored in Xfigure(1); colormap(map);size(X), r = rank(X), image(X)[U D V] = svd(X); % svd of the clown imaged = diag(D); % creates vector with singular valuesfigure(2); semilogy(d); % plot singular valuesk = 20; % # of terms to be used in approx.dd = d(1:k); % pick off 20 largest singular valuesUU = U(:,1:k);VV = V(:,1:k);XX = UU*diag(dd)*VV';figure(3); colormap(map)size(XX), image(XX) % compressed image

D. van Alphen 17

Singular Values for Clown Image

0 20 40 60 80 100 120 140 160 180 20010

0

101

102

103

104

Index of the 200 singular values

Singular

values

For this image: 20th singular value is 4% of 1st singular value

A = 1 u1 v1 + … + 20 u20 v20

+ 21 u21 v21 + … + 200 u200 v200

Negligible due to small singular values, i

D. van Alphen 18

Clown Image: How Much Compression?

• Original Image: 200 x 320 = 64,000 pixel values to store

• Compressed Image: A = 1 u1 v1 + 2 u2 v2 + … + 20 u20 v20

– Each ui has m = 200 components;– Each vi has n = 320 components;– Each i is a scalar 1 component

• Fraction of storage required for compressed image – 10,420/64,000 = 16.3%

• General formula for fraction of storage required:k * (m + n + 1)/(m*n)

where k = # of terms included in approximation

Total = 20 * (200+320 + 1)

= 10,420 values to store

D. van Alphen 19

MATLAB Commands for SVD

>> [U S V] = svd(A)

– Returns all the singular values in S, and the corresponding vectors in U and V;

>> [U S V] = svds(A, k)

– Returns the k largest singular values in S, and the corresponding vectors in U and V;

>> [U S V] = svds(A)

– Returns the 6 largest singular values in S, and the corresponding vectors in U and V;

>> Z = U*S*V’

– Regenerates A or A (the approximation to A), depending upon whether or not you used all of the singular values