linear algebra - 國立交通大學資訊工程學系yi/courses/linearalgebra... · 2010-12-08 ·...

LA

Linear Algebra

Chih-Wei Yi

Dept. of Computer ScienceNational Chiao Tung University

December 8, 2010

LA

Orthogonality

Section 1 The Scalar Product in Rn


LA

Orthogonality


Length and Distance

In what follows, u = (u1, u2, � � � , un), v = (v1, v2, � � � , vn),w = (w1,w2, � � � ,wn) are vectors in Rn, and c , d 2 R are scalars.

De�nition

1 The length of v, denoted by kvk, is de�ned as

kvk =�v21 + v

22 + � � �+ v2n

� 12 .

It is also called as magnitude or norm.

2 If kvk = 1, then the vector v is called a unit vector.3 The Euclidean distance between two vectors u and v is

d (u, v) = ku� vk .

LA

Orthogonality


Theorem

Assume v is a vector and c is a scalar. We have

1 kcvk = jc j kvk.2 If v is a nonzero vector, then 1

kvkv is a unit vector and has thesame direction as v.

LA

Orthogonality


Dot (Inner) Products

De�nition

The dot (inner) product of u and v is denoted by u � v and givenby u � v = u1v1 + u2v2 + � � �+ unvn. If u and v are written ascolumn vectors, then u � v = uT v.

Theorem

The following properties are true.

1 u � v = v � u.2 u � (v+w) = u � v+ u �w.3 c (u � v) = (cu) � v.4 u � u = kuk2.5 u � u � 0. u � u = 0 i¤ u = 0.

LA

Orthogonality


The Cauchy-Schwarz Inequality

Theorem

ju � vj � kuk kvk. In other words,�� n∑i=1 uivi��

n

∑i=1u2i

! 12

n

∑i=1v2i

! 12

.

De�nition

The angle between two nonzero vectors u and v is given by

cos θ =u � v

kuk kvk , 0 � θ � π.

Two vectors u and v are orthogonal if u � v = 0.

LA

Orthogonality


Properties of Triangles

Theorem (The Triangle Inequality)

For any two vectors u and v, we have ku+ vk � kuk+ kvk. Notethat the equality stands if and only if two vectors have the samedirection.

u+v

vu

Theorem (The Pythagorean Theorem)

Two vectors u and v are orthogonal if and only if

ku+ vk2 = kuk2 + kvk2 .

LA

Orthogonality


Projection

De�nition (Projection)

If u and v are vectors, we have v = v1 + v2 for some unique v1and v2 such that v2 � u = 0 and v1 = cu for some scalar c . Herev1 is called the vector projection of v onto u, and

v1 =u � vkuk2

u.

In addition, the scalar projection of v onto u is

α =u � vkuk .

LA

Orthogonality

Section 4 Inner Product Spaces


LA

Orthogonality


Inner Product Spaces

De�nition

Let u, v, and w be vectors in a vector space V, and c be anyscalar. An inner product on V is a function that associate a realnumber hu, vi with each pair of vectors u and v and satis�es thefollowing axioms.

1 hu, vi = hv,ui.2 hu, v+wi = hu, vi+ hu,wi.3 c hu, vi = hcu, vi.4 hu,ui � 0, and hu,ui = 0 i¤ u = 0.

LA

Orthogonality


Examples of Inner Product Spaces

Example

The dot product of vectors in Rn.

Example

Let f , g 2 C [a, b]. Then, hf , gi =R ba f (x) g (x) dx is an inner

product on C [a, b].

Example

Let A,B 2M2�2. Then,hA,Bi = a11b11 + a12b12 + a21b21 + a22b22 is an inner product onM2�2.

LA

Orthogonality


Properties of Inner Products

Theorem

Let u, v, and w be vectors in an inner product space V, and c beany scalar.

1 h0, vi = hv, 0i = 0.2 hu+ v,wi = hu,wi+ hv,wi.3 hu, cvi = c hu, vi.

LA

Orthogonality


Cauchy-Schwarz inequality

Theorem

Let u and v be vectors in an inner product space V. We have

hu, vi � kuk kvk ,

and the equality stands if and only if u = λv for some scalar λ.

Proof.

Consider the norm of vector u� λv

0 � hu� λv,u� λvi = kuk2 � 2λ hu, vi+ λ2 kvk2 .

Let λ = hu,vikvk2 . Then, we have 0 � kuk

2 � hu,vi2

kvk2 . So,

hu, vi2 � (kuk kvk)2. In other words, hu, vi � kuk kvk. Inaddition, the equality stands if and only if u� λv = 0.

LA

Orthogonality


Geometric Elements

De�nition

Let u and v be vectors in an inner product space V.

The norm of u is kuk = hu,ui12 .

The distance between u and v is d (u, v) = ku� vk.The angle between two nonzero vectors is given by

cos θ =hu, vikuk kvk , 0 � θ � π.

u and v are orthogonal if hu, vi = 0.If kuk = 1, the u is called a unit vector. If u 6= 0, 1

kuku is aunit vector in the direction of u.

LA

Orthogonality


Example

Let A,B 2M2�2 as follows A =�1 20 �1

�and

B =�3 �12 1

�. Find

1 hA,Bi.2 kAk and kBk.3 d (A,B).4 1

kAkA.

5 The angle between A and B.

Problem

If u 6= 0, prove that 1kuku is a unit vector.

LA

Orthogonality

Section 5 Orthonormal Sets


LA

Orthogonality


Orthogonal and Orthonormal

De�nition (Orthogonal Sets and Orthonormal Sets)

A set S of vectors in an inner product space V is called orthogonalif every pair of vectors in S is orthogonal. In addition, if everyvector in the set is a unit vector, then S is called orthonormal.

LA

Orthogonality


Example

S =n�

1p2, 1p

2, 0�,��p26 ,

p26 ,

2p23

�,� 23 ,�

23 ,13

�ois an

orthonormal set in R3.

Example

In P3, with the inner product hp,qi = a0b0 + a1b1 + a2b2 + a3b3,the standard basis B =

�1, x , x2, x3

is orthonormal.

Example

In C [0, 2π], with the inner product hf , gi =R 2π0 f (x) g (x) dx ,

the set S = f1, sin x , cos x , sin 2x , cos 2x , � � � , sin nx , cos nxg forany n 2 Z+ is orthogonal.

LA

Orthogonality


Orthogonal Implies Linearly Independent

Theorem

If S = fv1, v2, � � � , vng is an orthogonal set of nonzero vectors inan inner product space V, then S is linearly independent.

Proof.

Assume c1v1 + c2v2 + � � �+ cnvn = 0. Then,

hc1v1 + c2v2 + � � �+ cnvn, v1i = h0, v1i = 0, and

hc1v1 + c2v2 + � � �+ cnvn, v1i =n

∑i=1ci hvi , v1i = c1 kv1k2 .

Therefore, c1 = 0. Similarly, we can prove c2 = 0, � � � , cn = 0.Thus, S is an linearly independent.

LA

Orthogonality


Orthogonal Implies Linearly Independent

Corollary

If V is an inner product space of dimension n, then any orthogonalset of n nonzero vectors is a basis for V.

LA

Orthogonality


Coordinates Relative to an Orthonormal Basis

Theorem

If B = fv1, v2, � � � , vng is an orthonormal basis for an innerproduct vector space V, then

v = hv, v1i v1 + hv, v2i v2 + � � �+ hv, vni vn,

i.e. the coordinate representation of a vector v w.r.t. B is

[v]B = (hv, v1i , hv, v2i , � � � , hv, vni)T .

LA

Orthogonality


Proof.

Assume v = c1v1 + c2v2 + � � �+ cnvn. Then,

hv, v1i = hc1v1 + c2v2 + � � �+ cnvn, v1i= c1 hv1, v1i+ c2 hv2, v1i+ � � �+ cn hvn, v1i= c1 kv1k2 = c1.

Therefore, c1 = hv, v1i. Similarly, we can provec2 = hv, v2i , � � � , cn = hv, vni.

LA

Orthogonality


Example

B =�v1 =

� 35 ,45 , 0�, v2 =

�� 45 ,35 , 0�, v3 = (0, 0, 1)

is an

orthonormal basis for R3. Find the coordinate of w = (5,�5, 2)w.r.t. B.

Solution

hw, v1i = (5,�5, 2) �� 35 ,45 , 0�= �1;

hw, v2i = (5,�5, 2) �� 45 ,35 , 0�= �7;

hw, v3i = (5,�5, 2) � (0, 0, 1) = 2. Thus, w = �v1 � 7v2 + 2v3and [v]B = (�1,�7, 2)

T .

Problem (Quizzes)

What happens if B is orthogonal instead of orthonormal? Can youdevelop a formula for the projection?

LA

Orthogonality

Section 2 Orthogonal Subspaces


LA

Orthogonality


Direct Sum

De�nition

Let S1 and S2 be two subspaces of V. If each vector v 2 V can beuniquely represented as v = s1 + s2 where s1 2 S1 and s2 2 S2,then V is the direct sum of S1 and S2, and you can writeV = S1 � S2.

Example

B = f(1, 0, 1) , (1, 1, 0) , (1, 0, 0)g is a basis for R3. IfS1 = span f(1, 0, 1)g and S2 = span f(1, 1, 0) , (1, 0, 0)g, thenR3 = S1 � S2.

Problem

If V is an n dimension vector space and V = S1 � S2, prove thatdim (S1) + dim (S2) = n.

LA

Orthogonality


Orthogonal Subspaces

De�nition

The subspace S1 and S2 of Rn are orthogonal if v1 � v2 = 0 for allv1 2 S1 and v2 2 S2.

Example

In R3, let S1 = span fe1, e2g and S2 = span fe3g. Then, S1 andS2 are orthogonal.

LA

Orthogonality


Orthogonal Complement

De�nition

If S is a subspace of Rn, then the orthogonal complement of S isthe set S? = fu 2 Rn : u � v = 0 for all v 2 Sg.

Problem

If S is a subspace, prove that S? is a subspace.

Example

Find the orthogonal complement of the column space of the matrix

A =

26641 02 01 00 1

3775.

LA

Orthogonality


Some Properties

Theorem

Let S be a subspace of Rn. Then,

1 Rn = S� S?.2 dim (S) + dim (S?) = n.3 (S?)? = S.

LA

Orthogonality


Proof.

Assume fw1,w2, � � � ,wkg is an orthogonal basis of S. For anyv 2 Rn, let

v1 =hv,w1ihw1,w1i

w1 +hv,w2ihw2,w2i

w2 + � � �+hv,wk ihwk ,wk i

wk ,

v2 = v� v1.

We have v = v1 + v2, v1 2 S and v2 2 S?. So Rn is the directsum of S and S?. Then, dim (S) + dim (S?) = n follows. It canbe proved that fw1,w2, � � � ,wkg is also a basis of (S?)?.

LA

Orthogonality


Project onto a Subspace

De�nition

Let S be a subspace of Rn. Since Rn = S� S?, for any v 2 Rn,we have v = v1 + v2 for some unique v1 2 S and v2 2 S?. Thevector v1 is called the project of v onto the subspace S, and isdenoted as v1 = projSv.

Theorem

If fv1, v2, � � � , vkg be an orthonormal basis for the subspace S ofRn, then projSv = hv, v1i v1 + hv, v2i v2 + . . .+ hv, vk i vk .

LA

Orthogonality


Distance to a Subspace

Theorem

Let S be a subspace of Rn and v 2 Rn. Then, for any u 2 S andu 6=projSv, we have kv� projSvk � kv� uk.

Example

Let v = (1, 1, 3) and S = span f(0, 3, 1) , (2, 0, 0)g.1 Find the project of v onto S.2 Find the (shortest) distance between v and S.

LA

Orthogonality


Another De�nition of Projection

De�nition (Projection)

S is a subspace of an inner product space V, and v is a vector inV. projSv, called the projection of v onto S, is a vector in S suchthat kv� projSvk is minimum.

If B = fw1,w2, � � � ,wkg is an orthogonal basis of S,

projSv =hv,w1ihw1,w1i

w1 +hv,w2ihw2,w2i

w2 + � � �+hv,wk ihwk ,wk i

wk .

If B = fu1,u2, � � � ,ukg is an orthonormal basis of S,

projSv = hv,u1i u1 + hv,u2i u2 + � � �+ hv,uk i uk .

LA

Orthogonality

Section 6 The Gram-Schmidt Orthogonalization Process

Section 6 The Gram-SchmidtOrthogonalization Process

LA

Orthogonality


Theorem (Gram-Schmidt Orthonormalization Process)

Let B = fv1, v2, � � � , vng be a basis for an inner product vectorspace V. Let B 0 = fw1,w2, . . . ,wng, where wi is given by

w1 = v1,

w2 = v2 �hv2,w1ihw1,w1i

w1,

w3 = v3 ��hv3,w1ihw1,w1i

w1 +hv3,w2ihw2,w2i

w2

�,

...

wn = vn ��hvn,w1ihw1,w1i

w1 +hvn,w2ihw2,w2i

w2 + � � �+hvn,wn�1ihwn�1,wn�1i

wn�1

�.

Then, B 0 is an orthogonal basis for V.

LA

Orthogonality


Gram-Schmidt Orthonormalization Process (Cont.)

Let ui = wikwi k for i = 1, 2, � � � , n. Then,

B" = fu1,u2, � � � ,ung is an orthonormal basis for V.For any k = 1, 2, � � � , n,span (v1, v2, � � � , vk ) = span (u1,u2, � � � ,uk ).For any i = 1, 2, � � � , n� 1,

projspan(v1,v2,�� ,vi�1)vi= projspan(w1,w2,�� ,wi�1)vi

=hvi ,w1ihw1,w1i

w1 +hvi ,w2ihw2,w2i

w2 + � � �+hvi ,wi�1ihwi�1,wi�1i

wi�1.

LA

Orthogonality


Example

Let x1 = �1, x2 = 0, x3 = 1. Find an orthonormal basis for P2 ifthe inner product on P2 is de�ne by

hp,qi =3

∑i=1p (xi ) q (xi ) .

Solution

Consider the basis�v1 = 1, v2 = x , v3 = x2

.

w1 = 1; kw1k =p12 + 12 + 12 =

p3;u1 =

w1kw1k

=1p3.

LA

Orthogonality


Solution ((Cont.))

w2 = x � hx , 1ih1, 1i1 = x ; kw2k =q(�1)2 + 02 + 12 =

p2;

u2 =w2kw2k

=1p2x .

and

w3 = x2 �x2, 1

�h1, 1i 1�

x2, x

�hx , xi x = x

2 � 23; kw3k =

r23;

u3 =w3kw3k

=

p62

�x2 � 2

3

�.

Then,n

1p3, 1p

2x ,p62

�x2 � 2

3

�ois an orthonormal basis.

LA

Orthogonality


Revised Gram-Schmidt Orthonormalization Process

Theorem

Let B = fv1, v2, � � � , vng be a basis for an inner product vectorspace V. Let B 0 = fu1,u2, . . . ,ung is an orthonormal basis for V,where ui is given by

w1 = v1,u1 =w1kw1k

;

w2 = v2 � hv2,u1i u1,u2 =w2kw2k

;

...

wn = vn � (hvn,u1i u1 + hvn,u2i u2 + � � �+ hvn,un�1i un�1) ;un =

wnkwnk

.

LA

Orthogonality


Gram-Schmidt QR Factorization (Improved)

Theorem

If A is an m� n matrix of rank n, then A can be factored into aproduct QR, where Q is an m� n matrix with orthonormal columnvectors and R is an upper triangle n� n matrix whose diagonalentries are all positive.

Proof.

Let fq1,q2, . . . ,qng be the orthonormal basis of Col (A) derivedfrom the Gram-Schmidt process.

rij = hqi , aj i = qTi aj for 1 � i , j � n.

We have�

a1 = r11q1ak = r1kq1 + r2kq2 + � � �+ rkkqk for k = 2, � � � , n

.

Let Q =�q1 q2 � � � qn

�and R = [rij ].

What is the rij in the Gram-Schmidt orthonormal process.

LA

Orthogonality


Example

Compute the Gram-Schmidt QR factorization of the matrix

A =

26641 �2 �12 0 12 �4 24 0 0

3775 .

Solution26641 �2 �12 0 12 �4 24 0 0

3775 =266415 � 2

5 � 45

25

15

25

25 � 4

525

45

25 � 1

5

377524 5 �2 10 4 �10 0 2

35

LA

Orthogonality

Section 3 Least Squares Problems


LA

Orthogonality


Problem (Least Square Problems)

Let A be an m� n matrix and b be a vector in Rm .

1 If b is not in the column space of A, the system Ax = b iswith no solution.

2 Find x 2 Rn such that kAx� bk is minimal.

Solution

Let S denote the column space of A. The minimum of kAx� bkis given if Ax = projSb. Assume Ax

0 = projSb. Then,Ax0 � b = (projSb)� b is orthogonal to S. Then,

AT�Ax0 � b

�= 0 =) ATAx0 = AT b.

Therefore, the problem is reduced to solving the n� n linearsystem of equations ATAx0 = AT b, called the normal equation ofthe least square problem Ax = b.

LA

Orthogonality


Example

Let A =

24 0 23 01 0

35 and b =24 113

35. Find the orthogonal projectof b onto the column space of A.

Solution

ATA =

�0 3 12 0 0

� 24 0 23 01 0

35 = � 10 00 4

�,

AT b =

�0 3 12 0 0

� 24 113

35 = � 62

�.

LA

Orthogonality


Solution ((Cont.))

The normal equation is�10 00 4

� �x1x2

�=

�62

�with solution

x =�x1x2

�=

� 3512

�. So the projection is given by

Ax =

24 0 23 01 0

35 � 3512

�=

24 19535

35.

LA

Orthogonality


Problem

Find the least square regression line y = c0 + c1x for (1, 0), (2, 1),and (3, 3).

Solution

From (1, 0), we have c0 + c1 = 0. From (2, 1), we havec0 + 2c1 = 1. From (3, 3), we have c0 + 3c1 = 3.

Let A =

24 1 11 21 3

35, x = � c0c1

�, and b =

24 013

35. The problem is

equivalent to the LSP Ax = b.

LA

Orthogonality


Problem

Find the least square regression quadratic polynomialy = c0 + c1x + c2x2 for the data in the following table, and usethe model to estimate the value of y for x = 30.

x 0 5 10 15 20y 4.5 4.9 5.3 5.7 6.1

LA

Orthogonality


Solution

c0 = 4.5c0 + 5c1 + 25c2 = 4.9c0 + 10c1 + 100c2 = 5.3c0 + 15c1 + 225c2 = 5.7c0 + 20c1 + 400c2 = 6.1

=)

2666641 0 01 5 251 10 1001 15 2251 20 400

37777524 c0c1c2

35 =2666644.54.95.35.76.1

377775

LA

Orthogonality


Solution ((Cont.))

The normal equation ATAx = AT b are24 5 50 7550 750 12500750 12500 221250

3524 c0c1c2

35 =24 26.52854375

35

and the solution is x =

24 c0c1c2

35 �24 4.50.080

35. Therefore,y = 4.5+ 0.08x. If x = 30, we have y = 4.5+ 0.08� 30 = 6.9.

LA

Orthogonality

Section 7 Orthogonal Polynomials

Section 7 Orthogonal PolynomialsNOT COVERED IN THIS CLASS

linear algebra - 國立交通大學資訊工程學系yi/courses/linearalgebra... · 2010-12-08 ·...

Documents