ch6 inner product spacesocw.nctu.edu.tw/course/linear_algebra_ii/linear-algebra... · 2018. 1....

線性代數 II─應用數學系莊重老師

1

Ch6 Inner Product Spaces

HW 1,2,8,10,12,15,16,23,24,26

在這一章, Inner Product 將被引入向量空間, 這個概念可引出定義在一般抽

象的向量空間之角度, 長度和垂直的觀念. 因此 inner product 不但令 vector

spaces 有更豐富, 直觀的結構, 也讓定義在這些抽象空間的線性算子更豐富, 更

有應用價值.

除了§6.8 外, 這章的 field F = R or C.

§6.1 Inner Products and Norms F = R or C.

Def. (a) Let V be a vector space over F. An inner product on V is a function

, :V V F s.t., for all x, y, zV and all cF,

i) , , , .x z y x y z y

ii) , , .cx y c x y

iii) , , , : complex conjugate.x y y x ─

iv) , 0, if 0.x x x

(b) A vector space V over F endowed with an inner product is called an inner

product space. If F = C( R), then we call V a complex (real) inner product

space.


2

Remarks：

(a)(i)(ii) simply says that for fixed , ,y y is linear.

(b) if F = R, then (iii) reduce to , ,x y y x .

EX1. 1

,n

ni i

iV F x y x y

is the standard inner product on nF .

Thm6.1 Let v be an inner product space. Then for all , , and ,x y z V c F we

have that

(a) , , , .x y z x y x z

(b) , , .x cy c x y

(c) ,0 0, 0.x x

(d) , 0x x if and only if x = 0.

(e) if , ,x y x z for all xV, then y = z.

Pf：(c) ,0 ,00 0 ,0 0 ,0 0.x x x x

0, ,0 0 0.x x

EX2. Let 0,1V C the vector space of real-valued continuous functions on

[0,1].


3

Define 1

0, ( ) ( ) , , 0,1 .f g f t g t dt f g C

Claim： is an inner product.

Pf：(d) claim： , 0, if 0, ( ) 0 a 0,1 s.t. ( ) 0.f f f f x f a

0 1

0

2 21

0 0

(i) 0 0 s.t. ( ) 0 for x 0, min ( ) 0.

, ( ) ( ) 0.

(ii) (0,1) or 1, .

xa f x f x m

f f f t dt f t dt m

a a

同理可證

Def：AMn×n(F). We define A*, the conjugate transpose or adjoint of A to be the

matrix such that * =A for all , .jiijA i j

*

1 2Ex: .

2 3 4

2 .

1 2 3 4

i iA

i

iA

i i

EX4. V = Mn×n (F). Define , * .A B tr B A This inner product is called the

Frobenius inner product.

Def. Let V be an inner product space. For xV, we define the norm or length of x by

, .x x x


4

Thm6.2. V：an inner porduct space over F. Then for all x, yV and cF, then

(a) .cx c x

(b) 0 0. and 0.x x x

(c) , ( ) exercise 15x y x y 柯西不等式 . [等號成立情形見 ].

22 2 2 2 21 2 1 2 1 1 2 2

21 1 12 2

0 0 0

, (c) .

0,1 , (c) ( ) ( ) ( ) ( ) .

V R a a b b a b a b

V c f t g t dt f t dt g t dt

若則表示

若則表示

(d) ( ) exercise 15x y x y 三角不等式 . [等號成立情形見 ].

Pf：(c) If y = 0, then the result holds. Let 0y . For any 0c , we have

2 20 , , , , , .x cy x cy x cy x x c y x c x y c y y

Let,,

x yc

y y . Then the above inequality becomes

22

2

,0,

x yx

y

from which (c) follows

2

2 2

(d) ,. , , , , .

2 Re , .

x y x y x y x x x y y x y y

x x y y

2 2

22 2

2 , .

2 .

x x y y

x x y y x y


5

Def: : an inner product space.

i) , are orthogonal (perpendicular) ( ) if , 0.

ii) is orthogonal if , , , .

iii) is a unit vector if 1.

iv) is orthonormal if is

V

x y V x y x y

S V x y S x y x y

x V x

S V S

orthononal and consists of entirely of unit vectors.

Ex. i) V = F3,{(1,1,0), (1,-1,1), (-1,1,2)}: orthogonal set.

ii) 1 1 1{ (1,1,0), (1, 1,1), ( 1,1,2)}2 3 4

：orthonormal set.

Ex: H = the space of continuous complex-valued functions defined on [0,2 ] with

the inner product

2

0

1, ( ) ( ) 2

f g f t g t dt

.

int { } .

: orthonormal.

n ZS e H

S


6

§6.2 The Gram-Schmidt Orthogonalozation Process and Orthogonal Complements

H.W 1, 2(a), 2(h), 2(i), 2(m),3, 4, 6, 7, 8, 11, 13, 15, 18

在 3R 中若是標準基底, 則任一向量可由標準基底簡單表出. 此 3R 之標準

基底之向量彼此互相垂直且長度皆為 1. 在 inner product space 中因可定義垂直

和長度, 因此在這節我們要問一個 inner product space 是否具有類似 3R 中之標準

基底? 答案是肯定的(Thm6.5). 我們稱此種基底為 orthonormal. 事實上, 我們

可以將任一 basis 改造成一個 orthonormal basis. 這個改造過程我們稱之為

Gram-Schmidt process.

Def: V: inner product space. A subset of V is an orthonormal basis for V if it is basis +

orthonormal.

Ex. S = {(1,0,0), (0,1,0), (0,0,1)}is an orthonormal basis for F3.

Thm 6.3 S = {v1,v2…..vk}: orthogonal, 0 for all .iv i

If span( ), theny S

21

,.

ki

ii i

y vy v

v

12

,( ).i i

i

y vv

v在方向的組合係數分量 Pf:

Let 21 1

,, , , .

k ki

j i i j j i i i i ij j i

y vy a v y v a v v a v v a

v

Cor1. S: orthonormal If span( )y S , then 1

, .k

i ii

y y v v

iv

y


7

Cor2: S ={v1,...vk}: orthogonal + 0 for all .iv i

S is linearly indep.

1 1Pf: Let 0. Then 0 , , . 0 for all .

k k

i i i i j j j j ji i

a v a v v a v v a j

Gram-Schmidt Process (G.S.P): a process of constructing an orthonormal set from a

linearly independent subset of an inner product space.

先以兩個例子為例:給{w1,w2},造{v1,v2}: orthogonal.

‧

2v

1 1w v2w

2 12 2 12

1

,- .

w vv w v

v

‧{w1, w2, w3}→{v1,v2,v3}.

3 1 3 23 3 1 22 2

1 2

3

, ,- - .

.

w v w vv w v v

v v

w a b

2v1v3v

3w

a bc

將 w3 投影到 v1 得 a, 將 w3 投影到 v2 得 b, 組合起來得 c (= a+ b). 3 3.w c v

Thm6.4. V：inner space.

S = {w1,w2…wn}: linearly independent.


8

G.S.P.

S = {v1, v2,…..vn}.

where 1 1 andv w

1

21

,- . for 2 . (1)

, 1 1, .

kk j

k k jj j

k k j

w vv w v k n

v

w w v j k

分別減掉將投影到的投影向量

(1) :S orthogonal set of nonzero vectors.

(2) span( ) span( ).S S

Pf: Induction on n.

If n = 1, then 1 . the corresponding (1), (2) holdS w S v .

Let n = k -1 and the corresponding (1), (2) hold.

Wanted , ,...,1 2S v v vk k has the desired properties. For 1 ik-1,

1

21

1

21

,, - , .

, - , , , 0.

kk j

k i k j ij j

kk j

k i i k i k ij j

w vv v w v v

v

w vw v v w v w v

v


9

Note that from(1), if vk = 0, then wkspan{ 1kS }= span{Sk-1}.

a contradiction to the assumption that kS is linearly indep.

1 for all 1 -1. Since k i kv v i k S is orthogonal.

'kS is orthogonal.

By(1), and the induction assumption that span( 1kS )= span( 1kS ).

We have span (Sk) span( kS ).

Moreover, from(1) wk can be written as a linear combination of elements in'S .

Hence, span( kS ) span (Sk).

span(Sk) = span( kS ).

Thm 6.5 Let V: a nonzero finite –dimensional inner product space.

(i) V has an orthonormal basis .

(ii) If = {v1,v2,…..vn}, and ,x V then

1x, .

n

i ii

x v v

Cor. V and are given as in Thm 6.5. Let T be a linear operator on V , and

[ ] .A T Then


10

( ), .ij j iA T v v

1Pf: , .

, .

n

j j i ii

ij j i

T v T v v v

A T v v

1

1

2

Ex: ( ), , ( ) ( ) . If we continue applying the Graum

Schmidt process to the basis 1, , ,... for ( ). We obtain an orthogonal

basis whose elements are called the Legendro

V P R f g f t g t dt

x x P R

polynomials.

Cor: Let and be given as in Thm6.5. Let be a linear operator on , and

let . Then ( ), .ij j i

V T V

A T A T v v

Def: : an orthonormal subset of an inner product space V. Let x V . We

define the Fourier coefficients of x related to to be the scalars

, , where .x y y

Ex: V = H = the space of continuous complex-valued functions on 0, 2 .

2

0

int

2 int

0

1, ( ) ( ) .2

{ ( )} . ( ) .

1, ( ) ( ) = the fourier coefficients of relative to .2

n n Z n

n n

f g f t g t dt

f t f t e

c f f t f t e dt f


11

11

?

, , where is an orthonormal basis.

( ) , ( ). Fourier

Analysis .

nn

i i i ii

i ii

x x v v v

f t f f f t

在有限維之內積空間

上述的例子是無窮維空間,那麼這個問題是

中之一個基本問題

Def , ,S S V where V is an inner product space.

Define { : , 0 for all }.S x V x y y S

S is called the orthogonal complement of S.

Remark: (i) S is a subspace of V for any subset S of V.

(ii) Let S be a subspace of V. Then {0}.S S If

, x S S then , 0.x x Hence, x = 0. Of course 0 .S S

Remark: If S is not a subspace of V, then (ii) is not true.

EX: {0} and {0}.V V

EX: 3 3 and the plane.V R S e S xy

EX: 01,1 , and :eV C W W the subspace of V consisting of the even and odd

functions, respectively.

Then 0= .eW W It is easy to see that , 0f g for any 0f W and any .eg W

(Indeed , 1

1, ( ) ( ) 0 ( ( ) ( )f g f t g t dt f t g t

is an odd function on 1,1 .)


12

Hence, 0 .eW W To prove, 0 ,eW W

we need to use the continuity of a

function. (Why and how?)

Thm 6.6 Let W be a finite-dimentsional subspace of an inner product space V and let

.y V

(1) ! and s.t. .u W z W y u z

(2) If {v1, v2…vk}is an orthonormal basis for W, then

1, .

k

i ii

u y v v

Pf:

yz

u w

Let 1 2, ... kv v v be an orthormal basis and u be given as above. Let z = y - u.

Thenu W and y = u + z. We next show z W .

Now, 1

, - , , .k

j i i ji

z v y y v v v

1

, , , , , 0.k

j i i j j ji

y v y v v v y v y v

1, 0 for all . ( for some ).

k

j j ji

z w w W w c v c

To show uniqueness of u and z, suppose that , y u z u z where , u u W

and , .z z W

{0} , .u u z z W W u u z z


13

Cor. y x y u for any x W , where y and u are given as in Theorem 6.6.

u is “closest” to y.

22 2 2 2 2 2pf: .y x u z x u x z u x z z y u

Bessel’s Inequality: Let y and 1k

i iv

be given as in Thm6.6, then

22

1, .

k

ii

y y v

22 2 2 2

1Pf: = , .

k

ii

y u z u y v

由畢氏定理

int

1 2 2 222

1

2 22

2 2 21 1 1

Ex: Let ( ) : Let ( ) (

4 , ,1 , .3

1 1 1 = 2 .6 6

, B

n

k

n nn k n

k k

n n n

S f t e n z f t t f y

f f f f f f

n n n

. 也即將看成是上述不等式之 ). 則

上述第一個不等式即是代2

21

1essel's inequality. In fact, . 6

Parseval's Thm.

n n

這在

高微會提到這是

Thm6.7 Suppose that 1 2, ,..., ks v v v is an orthonormal set in an n-dimensional

inner product space V. Then

(a) S can be extended to an orthonormal basis 1 2 1, ,..., , ,...,k k nv v v v v for V.

(b) If W= span(S), then 1 1 2, ,...,k k kS v v v is an orthonormal basis for W .

(c) Let W be any subspace of V, then dim(V) = dim(W) + dim(W ).


14

Pf: Skip

Remark: In fact: V W W , where W is a subspace of a finite dimensional inner

product space.


15

§6.3 The Adjoint of a Linear Operator

H.W 1, 3, 7, 8, 9, 12, 15, 18, 19, 20(c), 22(d), 24

這節我們要定義 adjoint 的觀念.

Thm6.8 dim( ) , v V : inner product space over F, and let :g V F be a linear

transformation.

! s.t. ( ) , for all .y V g x x y x V

Pf: Let ={v1,..vn} be an orthonormal basis for V.

1For , then , .

n

j jj

x V x x v v

1 1

1

( ) , ( ) , ( ) .

Let ( ) . Then ( ) , .

If ( ) , , for all , then , 0 .

Thm6.9 is given an in Theroem6.8. : linear operator on .

(1) ! * :

n n

j j j jj j

n

j jj

g x x v g v x g v v

y g v v g x x y

g x x y x y x x y y y y

V T V

T V V

s.t.

( ), , * ( ) for all , .

(2) * is linear.

T x y x T y x y V

T


16

Pf: For any arbitarily fixed , let ( ) ( ), .

Thm6.8

! s.t. ( ) , ( ), .

y V g x T x y

y V g x x y T x y

Now, define *: as follows.

* ( ) . Hence, ( ), , *( ) .

Claim: * is unique and linear.

T V V

T y y T x y x T y

T

1 1

1 1

1 2 1 2

1 2

1 2 1 2

*1 2 1

(1) Let be that ( ), , ( ) , *( ) for any and .

( ) *( ) = *.

(2) , * ( ) ( ), .

( ), ( ), .

, * + , * , * * .

( ) *( )

T T x y x T y x T y x y V

T y T y T T

x T cy y T x cy y

c T x y T x y

c x T y x T y x cT y T y

T cy y cT y T

2*( ) ( is abritrary).y x

Def: (1)Let V be a finite dimensional inner product space. The adjoint T* of the

operator T is a unique linear operator on V satisfying ( ), , *( )T x y x T y

for all , .x y V

Remarks: (i) Owing to theorem 6.9, the definition above is well-defined.

(ii) If V is infinite-dimensional, such T* might not exist (見 Ex24).

(iii) , , , * * , .x T y T y x y T x T x y


17

Thm6.10 T: linear operator on V, where V is as before. Let be an orthonormal

basis for V. Then **T T .

Pf: Let ={v1...vn} be an orthormal basis for V.

Let * and [ ] . Then

*( ), , ( ) ( Remarks (iii)).

( ), * .

*.

ij j i j i

i j ji ij

T B T A

B T v v v T v

T v v A A

B A

由

Cor. *: matrix L L *.A AA n n In particular , , , *Ax y x A y

for all , .nx y F

Remark: 若是 basis 但不是 orthonormal basis, 則 Thm6.10 的結論是不對的.

Thm6.11

(a) * * *.

(b) * *, .

(c) * * *.

(d) ** .

(e) * .

T U T U

cT cT c F

TU U T

T T

I I

Pf of (d): Note that ** , **( ) , ( ) for all , .

Now, , **( ) * ( ), , and , ( ) ( ), , * ( ) * ( ), .

T T x T y x T y x y

x T y T x y x T y T x y x T y T y x


18

Cor. If T, U above are n n matrices, then (a - c) still hold.

以下為矩陣 A 和 A*之 range and null space 之間的關係的整理:

(i) Lemma1 rank (A*A) = rank (A).

(ii) ( *) ( ).R A N A

(iii) ( *) ( ) .R A N A

(iv) ( *) ( ).W R A N A

(ii),(iii)表示子空間 ( *)R A 和 ( )N A 互相垂直.

Pf of (i): By the Demension Thm, it suffices to showt that ( * ) ( ).

If ( ), then ( * ). Hence, ( * ) ( ). If ( * ),

then * 0. Hence, 0= * , , ** , 0.

Hence, ( *

N A A N A

x N A x N A A N A A N A x N A A

A A x A Ax x Ax A x Ax Ax Ax

N A A

) ( ).N A

Pf：(ii)‧ ( ) 0 , 0x N A Ax Ax y for any y.

, * 0. ( *) ( ) ( *) .x A y x R A N A R A

‧ ( *) , 0x R A x y for any ( *).y R A

s.t. * , * , 0.nz F A z y x A z Ax z

Since y is arbitrary and so is z, we have that

0 ( ) ( *) ( ).Ax x N A R A N A

‧ ( *) ( ).R A N A


19

Least square approximations (LSA)

( , ) : : times. : measurements.i i i it y t y

y

t 1 1,t y

,i it yy ct d

Find a ”best” linear relationship between y and t,

i.e., y = ct + d.

a ”best”：Let 21

( ) .m

i ii

E y ct d

Then E is a minimum.

LSA：‧Finding c and d so that E is a minimum.

‧y = ct + d is called the least square line.

‧可推廣問找一 2 次多項式 2y at bt c 來代表這些數據.

(Method I)：微積分方法.

(Method II)：矩陣方法.

1 1

2 2

11

Let , . and .: : :

1m m

t yt yc

A x yd

t y

Then 2 .E y Ax 若是找最佳二次多項式, 則相對應的問題可表成


20

211 1

222 2

2

11

, , and .:

1 mm m

yt ta

yt tA x b y

cyt t

推廣：

0 0 6 3 1 Let matrix. Find s.t. for all . ( . )n nA m n x F y Ax y Ax x F

註：這樣的 formulation 不只可解決 LSA 的問題, 也可以解決找一 k 次多項式來

最佳逼近這些數據.

Notation： , . , * .nn

x y F x y y x

Lemma 1. , , * .m n

Ax y x A y Here ( ) and .n mm nA M F x F y F

Pf: , *( ) ( * ) ( * )* , * .m n

Ax y y Ax y A x A y x x A y

Lemma 2. rank (A*A) = rank(A), ( ).m nA M F

Pf：It suffices to show that N (A*A) =N (A).

If ( * ), then * 0.x N A A A Ax

20 0, * , , .x A Ax x Ax Ax Ax

0 ( ). If is s.t. 0, then * 0. ( * ) ( ).Ax x N A x Ax A Ax N A A N A

Cor. : and rank( ) , then * is invertible.A m n A n A A

Minimization： 0 .y Ax y Ax


21

Theory： : . .mA m n y F Define { : }.nW Ax x F

0! x W that is closest to y. Then

0 for all .nAx y Ax y x F

y

0AxThe range of

The range of .AW L

A

0Ax y

0Ax

以上此圖說明了(6.3-1)之解, 也代表整個定理 6.12 的內容, 同時也提供 Practical

Method for finding 0x . 也即

0 0 0

10

, 0 , * 0. * * .

Rank( ) * * .

Ax Ax y x A x y A Ax A y

A n x A A A y

若

Thm6.12. ‧ ( ), .mm nA M F y F .

0 0

0

s.t. ( * ) * and

for all .n

x A A x A y

Ax y Ax y x F

‧If rank (A) = n, then x0=(A*A)-1A*y.

Pf： 0 0, 0 for all .n

mAx y W Ax Ax y x F

0, * ( ) 0 for all .n

nx A Ax y x F

0 0* ( ) 0. * * .A Ax y A Ax A y


22

If rank (A) = n, then x0 = (A*A)-1A*y.

Remark：

(i) For LSA, A is an m×2 matrix.

In real applications rank (A) = 2. Otherwise it t , that is the experimenter collects

all the data at exactly one time. That is for LSA, 10 * * .x A A A y

(ii) If 1

0m

ii

t

two columns of A would be orthogonal, so A*A would be a

diagonal matrix.

(iii) If one wants to find a best quadratic approximation to the data, then the model

can be set up as follows.

Let 2 .y ct dt e

1 21 1

2

2

3

1, , .

:1m m m

m

yc t t

yx d y A

e t ty

Minimal solutions to systems of Linear Equations.

Lemma3. For any matrix ( ), then * ( )m nA M F R A N A

and R A* ( ) . N A Furthermore, F = ( ) * and n N A R A

( ) * .mF N A R A


23

( ) 0 0 , * , for all .

* . ( )= * .

mn m

x N A Ax x A y Ax y y F

x R A N A R A

Since is finite dimensional and the fact that . nF W W (Ex. 13(c) of §6.2, we

have ( ) = * . N A R A

‧The remaining assertions of the theorem follows from Thm6.7.

Thm6.13 ( ). .mm nA M F b F Let Ax = b be consistent. Then,

(i) ! minimal solutions s of Ax = b and *( ).As R L

(ii) The vector s is the only solution to Ax = b that lies in *AR L ; that is,

if (AA*) u = b, then s = A*u. By minimal solution, we mean s u

for all other solution u. If rank(A) = m, then 1* * .S A AA b

x

( ) ( )AW N L N A

us

**R( ) R( )AW L A

v

W

u

s W

重點整理

(A): 理論部分

(i) x 是任一解, 即 Ax = b.


24

(ii) nF W W (Lemma3), where *W R A and ( )W N A .

(iii) 將 x 投影到 W 得 s.

(iv) ( , where ( ))As b x s u u N A , 也即 s 也為一解.

(v) 若 v為Ax = b之任一解, 則由Thm3.9, v s u , where ( )u N A . (即

一般解 v = 特殊解 s+在零空間的向量u ).

(vi) ( ) . minimal solution.v s s 畢氏定理即是一個

(vii) 若 = 0 .v s u u v s 則 =0

minimal solution. ( ).s s W 是唯一的一個且

(B): 實際上如何找 s.

在(A)部分中,其實也提供了一個找法, 但以下找法更簡潔, 更有系統.

想法: 找一個在 W 中之解.

(step1): 解 *AA u b , 得一解 u,

(step2): Let *s A u , 則此 s 和在理論部分得到的 s 是一樣的.

Why? 因為在 W 中找 Ax = b 解是唯一的. 因此, 找出來必定是理論部分得到的

s.

‧這個唯一是自然的, 因為 Lemma3. 若 v, s 為 Ax = b 之解, 且 ,v s W ,則

( ) 0 .v s W N A v s W W v s


25

Proof of (1)：Let W = R(LA*) = R(A*). Then W = N(LA) = N(A). (Why?)

Let x be any solution of Ax = b. Then from Thm6.6 + Lemma3, , x s y

where and .s W y W Moreover, such decomposition of x is unique.

( ) .b Ax A s y As Ay As

*( ) s also solution to .As R L i Ax b

We next show that s v for any v with Av = b.

By Thm3.9, v s u (也即一般解=特殊解+Ax = 0 之解) where .u W

Then 畢氏定理

vu

s *( )Aw R L

2 2 2 2 .v s u s 若 , 0 0 .v s u u v s 則

is the unique minimum solution of .s Ax b

(ii) To complete the proof, we need to show that if v W and Av b , then .v s

{0} .v s W W v s

Remark: Thm6.12 and Thm6.13 都是找

Ax b

之解. 但 Thm6.12 是找一個解 x, 使得 Ax b (=誤差)是最小.


26

Thm6.13 是找一個解 0 x , 得 0x x for any x satisfying Ax b .


27

§6.4 Normal And Self-Adjoint Operators

H.W 1, 2, 3, 7, 9, 11, 17, 19

一個定義在有限維空間上的線性算子是否可對角線化是第 5 章的主題. 如

果此有限維空間上有內積結構, 那又如何呢? 第五章的可對角線化是需要檢查

eigenvalues and eigenvectors. 有了內積結構是否能得到不需檢查 eigenvectors

and eigenvalues 的條件呢？這是這節主要討論的問題, 由此因而衍生出 Normal

and Self-Adjoint operators 的觀念. 又上述問題也引出了可三角線化的問題. 這

節有三個主要結果. 我們將以矩陣的語言表查此三個結果：

1. Schur Thm：Let . Then : uitary s.t. * , n nA M Q Q AQ U

where is upper triangularU .

2. Let and is normal. Then : uitary s.t. * , n nA M A Q Q AQ D where

1 2, ,..., nQ v v v , and , 1, 2,...,jv i n , are eigenvectors and D is a diagonal

matrix.

3. n nA M R and A is symmetric. Then : orthogonal s.t. , TQ Q AQ D

where 1 2, ,..., nQ v v v , and , 1, 2,...,iv i n , are eigenvectors and D is a

diagonal matrix.

Schur Theorem (即可否三角形化的條件)

Lemma. T: linear operator on a finite-dimensional inner product space V.

If T has an eigenvector, then so does its adjoint *T .


28

*

* *

* *

Pf: Let , 0. Then for any .

0 0, , , - .

- - is not onto.

- is not one to one N - 0

Any nonzero vector in the null space in an eigenvector of

Tv v v x V

x T I v x v T I x

v R T I T I

T I T I

* with

eigenvalue .

T

Thm 6.14 : : linear, dim , is an inner product space.

If C.P. of splits, then an orthonormal basis for s.t. is upper triangular.

T V V V n V

T V T

*

Pf: Induction on .

(i) 1 Of course.

(ii) If the assertion of the Thm holds true for dim( ) -1.

From lemma, a unit eigenvector s.t. T = .

n

n

V k n

z z

*

Let span . We next show that is invariant. To see this, let ,

and , then ( ), , ( ) , , 0.

( ) ( +Thm5.21), the C.P. of splits. Moreover,

W

W z W y W

x cz W T y x y T cz y c z c y z

T y W T

由已知

Thm6.7-c, dim( ) 1.

( Induction ), an orthnomal basis for s.t. is upper triangular.

W r

W n

r W T

由

由假設


29

Let is an orthormal basis for s.t.

is upper triangular.

r z V

T

Remarks: (i) , then the C.P. of any finite dimensional linear operator splits.

complex inner product space, ( ) .

(ii) A M ( ), A ,

n n

F T

T T

若

若定義於在一個則可上三角形化

若則可三角形化更正確

1 1 *

: unitary matrix s.t.

, where is an upper triangular matrix and .

(iii) Let be a finite-dimensional inner product space. If eigenvector

Q

U Q AQ U Q Q

V

的是,

s is an

orthnormal basis for .V

**

* * * *

is diagonal = is also diagonal. Because diagonal

matrices commute,

T T T

TT T T T T T T

* *

* * *

=

and commute. That is .

TT T T

T T TT T T

* *

* *

Def: : an inner product space. : linear operator on . is normal if .

An real or complex matrix is normal if .

Remark: is normal is normal.

V T V T TT T T

n n A AA A A

T T

2

Ex1: If is skew-symmetrix, that is . Then is normal,

because .

tn n

t t

A M R A A A

AA A A A


30

* *

*

2

Ex2 :

cos sin .

sin cos

is normal.

Ex3 : : a real symmetrix matrix (i.e. ).

. is normal.

n n

t

t t

A M F

AA I A A A

A A A A

AA A A A A

*

Thm6.15 : normal on . Here : an inner product space

(a) ( ) ( ) for all .

(b) - is normal for any .

T V V

T x T x x V

T cI c F

*

*

(c) : an eigenvector of : an eigenvector of .

In fact, if ( ) , then ( ) .

x T x T

T x x T x x

1 2 1 1 2 2 1 2

1 2

(d) If , and and , 0, 0.

.

Tx x Tx x x x

x x

2 * *

2* * *

Pf:

(a) ( ) = ( ), ( ) , ( ) , ( ) .

= ( ), ( ) ( ) .

T x T x T x x T T x x TT x

T x T x T x


31

* *

* * * *

*

(b) - - - - .

= .

= - - .

T cI T cI T cI T cI

TT cT cT ccI T T cT cT ccT

T cI T cI

( ) ( )* *

*

*1 1 2 1 1 2 1 2 1 2

1 2 2 2 1 2

1 2 1 2 1 2 1 2

(c) 0 - - - .

.

(d) , , , , .

, , .

, 0. , 0. .

b cT I x T I x T x x

T x x

x x x x Tx x x T x

x x x x

x x x x

Thm6.16 Let be a linear operator on a finite-dimersional complex inner product

space . Then is normal an orthonormal basis for consisting

of eigenvector of .

Pf

T

V T V

T

: ( ) proved (in Remark (iii))

( ) the C.P. of splits.F T

Thm6.14

1 2 an orthonormal basis , ,..., for s.t. is upper triangular.

Claim: consists of eigenvectors.

Prove by the math induction on .

nv v v V T A

n


32

1

1 1

1 is an eigenvector.

Suppose ... are eigenvectors. We next prove that is also an eigenvector of .

k k

n v

v v v T

Thm6.15*

1

*

Let 1 1.

.

Since is upper triangular, .

For , ( ), , ( ) , , 0.

( ) . By the induction the vector

j j j

j j j

k

k ik ii

jk k j k j k j j j k j

k kk k

Tv v j k

T v v

A A v A v

j k A T v v v T v v v v v

A v A v

s in are all eigenvectors.

Remarks : (iv) Thm6.16 Any normal linear operator on a finite-dimensional complex

inner product space is diagonalizable.

(v) It also implies that for any

n

T

V

A M

1

( ), and is normal, then is diagonalizable.

In fact, : orthonormal s.t.

.

(vi) The assertion of Thm 6.16 does not hold

n A A

Q

D Q AQ

if is an infinite-dimensional

complex inner product space. (see next example).

V

int

1 1

1 1

1 ( 1), , ( 1)

Ex: : ( ) span ( ).

, : by ( ) and ( ) .

Then ( ) , ( ) for all .

( ), , .

n n

n n n n

m n m n m n m n

S f f t e n Z V S

T U V V T f f f U f f f

T f f U f f n Z

T f f f f


33

*1

*

*

* * *

* *

= , , ( ) ( ), .

( ), = ( ), for any .

( )= ( ) for any .

( ) ( ) for any .

is normal.

In fact, is self-adjoint.

m n m n m n

m m

m m

f f f U f U f f

T f f U f f f V

T f U f m Z

T f U f f V T U U T

TT TU I T T T

T

+1

1 1

We next show that has no eigenvectors.

If ( ) , where , 0, , and .

( ) .

Since 0, we can write as a linear combination of , ,...

m

i i mi n

m m

i i i ii n i n

m m n n m

T

T f f f a f a n m Z m n

a f T f f a f

a f f f f

, a contraction

to the fact that is linearly independent.S

Remarks : (vi) real inner product space eigenvectors

orthonormal basis. Hence, , ,

.

1 (v) Let A

定義在的線性算子不能保證存在

之不能保證可對角線化我們需要更強條件

來保證可對角線化

5. Since 6,3 are eigenvalues; it follows from Theorem 5.9

4 2

that is diagonalizable. However, is not normal. Does this example

contradict the

A A

assertion of Thm 6.16 Why not？？


34

*

*

Definitions : self-adjoint (or Hermitian) if .

: self-adjoint (or Hermitian) if .

Lemma: : self-adjoint

(a) Every eigenvalue of is real.

(b) Suppose that

T T T

A A A

T

T

V

is a real inner product space. Then the C.P. of splits.T

*

Pf: (a) Let , 0. Since is also normal,

( ) ( ) is real.

(b) Let dim( ). be an orthnormal basis for , and . Then is

self-adjoint. Def

Tx x x T

x T x T x x

n V V A T A

** *

ine : .

by

( ) for all .

Note that is self-adjoint because , where is

the standard ordered (orthornomal) basis for .

n nA

nA

A A A A

n

T

T x Ax x

T T A A T T

C.P. split over . self-adjoint all those

eigenvalues are real. C.P. also splits over .

C.P. of C.P. of .

C.P. of splits over .

nA A A

A

A

T T T

T R

T T

T R

定義在之可以但是

之

但


35

Thm6.17 : a real inner product space.

is self-adjoint an orthonormal basis for consisting of

eigenvectors of .

Pf: ( ) Applying Schur Thm, we have that an o

V

T V

T

** *

rthonormal basis for s.t.

is upper triangular. Let . Then .

is diagonal.

consists of eigenvectors of .

V

T A T A T T T A

A

T

( ) Let be an orthonormal basis for consisting of eigencectors of . ,

where is diagonal. Furthermore, diagonal elements of are eigenvalues. Since is

V T T D

D D V

assumed to be real inner product spce, the existence of eigenvalues implies their

corresponding eigenvalues are real. Hence, is a real diagonal matrix.

is self-adjoint

D

D .

is self-adjoint. T


36

§6.5 Unitary and Orthogonal Operators and Their Matrices

H.W 1, 2(e), 4, 5,(a) 7, 9, 10, 12, 16, 17, 27(a), 28, 31

課本在這節一開始, 即提到複數(complex numbers) 和線性算子(linear

operators) 的對比. 其中, 算子的 adjoint 運作上好比一個複數冠上共軛. 更詳

細的部分可讀此節的前兩段說明. 當然這些說明也引出了討論具有下列性質的

算子, * *TT T T I , 這些算子具有保長的特性. 又這種性質的矩陣, 在討論

Normal or Self-adjoint 矩陣的可對角線化的條件也出現. 當 an orthonormal

basis = 1 2eigenvectors , ,..., .nv v v Let 1 2, ,..., , nQ v v v 則 * * .Q Q QQ I

Definitions T : V→V, linear, dim( )V , V: an inner product space over F.

If ( )T x x (保長) for all ,x V we call T a unitary operator, if F and an

orthogonal operator if F = R.

Ex1. 2 ;V R T: rotation or reflection in 2R preserves length and hence is an

orthogonal operator.

* *

Thm6.18 : dim( ) , an inner product space.

The following are equivalent.

(a) .

(b) ( ), ( ) , for all , .

(c) : orthonormal basis for , then ( ) is an orthonormal basis for .

(d) a

V V

TT T T I

T x T y x y x y V

V T V

n orthonormal basis s.t. ( ) is an orthonormal basis for .

(e) ( ) for all .

T V

T x x x V


37

以下若沒特別提, V 將是 a finite-dimensional inner product space.

*0

0

Lemma Let . If , ( ) 0 for all , then (0 ).

Pf: an orthnormal basis = eigenvectors . Hence, if , then . Thus,

0 , ( ) , , ;

So, 0 0 for all

U U x U x x V U T

x Ux x

x U x x x x x

Ux x U T

函數

.

*

1

Pf of Thm6.18:

(1) ( )

( ), ( ) , ( ) , .

(2) ( )

Let ,..., be an orthnormal basis for . Then

1 ( ), ( ) , .

0

n

i j i j ij

a b

T x T y x T T y x y

b c

v v V

i jT v T v v v

i j

1 1

( ) is an orthnormal basis.

(3) ( ) Obvious.

(4) ( )

Let ,..., be an orthnormal basis, so that ( ) ( )... ( )

is an orthonormal basis for . Let

n n

T

c d

d e

v v T T v T v

V x

1

.

n

i ii

a v


38

1 1

2

1 1 1 1

( ), ( ) ( ), ( ) .

( ), ( ) , .

= , ( ) .

n n

i i i ii i

n n n n

i i i i i i i i ii i i i

T x T x a T v a T v

a T v a T v a a v a v

x x T x x

* * * *

* * *

Lemma* *

(5) ( ) Note that ( - ) . - is self-adjoint.

, ( - ) ( ) , , ( ) .

= , ( ), ( ) 0.

, Similarly, .

e a I T T I T T I T T

x I T T x x x x T T x

x x T x T x

I T T T T I

1 2

Cor1. : dim ( ) , a real inner product space.

Then = , ,..., an orthonormal basis for ,

where ( ) , 1. is both self-adjoint and orthogonal.

Pf: ( ) Since eig

n

i i i i

V V

v v v V

T v v T

Thm 6.17

* * *

envectors orthonormal basis.

is self-adjoint.

Now, ( ) ( ) ( ) ( ) for each . i i i i i i i i i i i

T

T T v T v T v T v v v i

且是一個

T 6.18 (a)* *

T 6.17

1

. Similarly, is orthogonal.

( ) ,..., = an orthonormal basis and ( ) .

= ( ) = ; =1 for all .

hm

hm

n i i

i i i i i i i

T T I T T I T

v v T v v

v v T v v i

保長


39

Cor2. dim ( ) , : a complex inner product space.

Then has an orthogonal basis consisting of eigenvectors of with corresponding

eigenvalues of absolute value 1 is unitary.

Ex

V V

V T

T

2 22: : : rotation by , where - .

: orthogonal.

T R R

T

2 2Ex3: a reflection of about , where is a 1-d subspace of . We may view

as a line in the plane through the origin.

: orthogonal.

Definitions: Orthogonal: if .

t t

R L L R

L

T

A A AA I

* * Unitary: if . A A AA I

1 2

1 2

Ex4: , ... an orthonomal basis for .

Then , ... is unitary.

: rotation

cos sin .

sin cos

nn

n

v v v F

Q v v v

A

如

*

Definitoins: A and B are unitarily [orthogonally] equivalent if a unitary

[orthogonal] matrix s.t. .

P A P BP


40

Thm6.19 ( ). Then is normal. is untarily equivalent to a diagonal matrix.

( Thm6.16 ).

n nA M A A

這個定理即是的矩陣版

Thm6.20 ( ). Then is symmetric. is orthogonally equivalent to a real diagonal

matrix. ( Thm6.17 ).

n nA M F A A

這個定理即是的矩陣版

*

Thm6.21 (Schur) Let ( ).

(a) If , then : unitary matrix s.t. , where is an upper triangular matrix.

(b) If and the C.P. of splits, then : orthogonal s.t.

n nA M F

F C Q U Q AQ U

F R A Q U Q* , is a real

upper triangular matrix. ( Thm6.14 ).

AQ U

以上即是的矩陣版

2 2Thm6.23 : orthogonal operator on . Let , where is the standard basis for .

Then exactly one of the following conditions is satisfied.

(a) is rotation, and det( ) 1.

T R A T R

T A

(b) is reflection about a line through the origin, and det( ) 1. T A

Thm6.18(c)

1 2 1 2

1 1 2 2

2

Pf: Let , ( ) ( ), ( ) is an orthogomal basis

( ) cos ,sin . Since ( ) ( ), ( ) sin ,cos or sin , cos .

cos sin If ( ) sin ,cos .

sin cos

det

e e T T e T e

T e T e T e T e

T e A

( ) 1 and is rotation.A A


41

2

2

cos sin If ( ) sin , cos .

sin sin

is the reflection of about a line with = . Here is the angle 2

from the positive axis to . Moreover, det( ) 1 .

T e A

T R L

L A

2 2

Conic sections

2 , , , , .a b x

ax bxy cy x y a b c Rb c y

1 2

Let . Let , a symmetric matrix. Hence, an orthogonal matrix

and a diagonal matrix with real diagonal entries and s.t.

.t

x a bX A P

y b c

D

P AP D

2 21 2

Let , or equivalently .

( ) ( ) .

The transformation ( , ) ( , ) allows us to elimmate the -term.

From Thm6.23 could be rotatio

t

t t

xx P x Px x

y

x Ax x P APx x Dx x y

x y x y xy

P

2

1

n or reflection. reflection,

0columns , rotatoin, .

0

t

P P

Q Q AQ Q xy

若是則將的

交作換得 ,則而是且一樣可消去項

In summary, 利用旋轉矩陣(也卽引進(旋轉之後)的新座標)可將 xy 項消去.

從矩陣的觀點問的是 symmetric 矩陣 A 之對角線化. 又此方法可推廣到高維度

(也卽有 n 個變數)二次曲線的分類.


42

§6.6 Orthogonal Projections and The Spectral Theorem

對 normal 算子 T (if F = C) or a self-adjoint 算子(if F = R) T on a finite

dimensional inner product space V, T 可作 spectral 的分解, 也卽1

, k

i i ii

T T

are the distinct eigenvalues of T and iT : are orthogoral projections.

1 2 1 2

1 2 1

Def: Let and , then the linear operator on is the projection

on along , if ( ) .

V W W x x x T V

W W T x x

2W

1x

1 2,x x

1W

2x

2W

1x

1 2x x x

1W

2x

1 2

1 2

2

Remarks: (i) ( ) and ( ) .

(ii) = ( ) ( ).

(iii) is a projection .

Pf of (iii) ( ) Trivial.

( ) Since ( ( )) ( ) for any , ( ( ) ) 0. H

R T W N T W

V W W R T N T

T T T

T T x T x x V T T x x

1 2

1 2 1 2

ence, ( ) ,

where ( ) ( ) . Let ( ) and ( ).

. We next prove that 0 .

T x x y

y N T x T x y W R T W N T

V W W W W


43

1 2

22

1 2

Let . s.t. ( ) .

( ) ( ) ( ) ( ) ( ).

Since ( ), ( ), 0 ( ) .

Hence ( ) ( ).

x W W y V T y x

T y T x T y T x x y W N T

x N T y N T T y x

V W W R T N T

1 2 1 2

1 2 1 1

1 1

1 2 1 3 2 3 1

Let , where x ( ) and x ( ).

( ) ( ) ( ) ( ( )) ( ) .

( ( ) s.t. ( ) ).

Remark (iv) Because does not imply does not

uniquel

x x x R T N T

T x T x x T x T T y T y x

x R T y V T y x

V W W W W W W W

y determine . For an orthogonal projection, is uniquely determined

by its range.

T T

Def. Let : be a projection. Here is an inner product space. We say that is

an orthogonal projection, if ( ) ( ) and ( ) ( ).

Remark (vi) If dim( ) , then ( ) ( ) ( ).

T V V V T

R T N T N T R T

V N T R T R T Hence, one of the proceeding

2 0 conditions suffices to define an orthogonal projection. Let , then

0 0

( ) the x-axis and ( ) the y-axis. But is not

A

R A N T A a projection on ( )

along ( ).

(vii) 6.6. 6.2 , ( ) . is an orthogonal projection.

orthogonal projection on .

R A

N T

T y u T

W

在定理圖中則又僅有一個


44

(viii)

( )T v

( )U v

v

2W

2

0

( ) is the " best approximation in to ". That is, if , then - ( ) .

Ex: The set of continuous complex-valved function on 0,2 .

1 , ( ) ( ) .2

T v W v w W w v T v v

H

f g f t g t dt

int ( ) : ; 0 2 orthonormal set.

Let ( ) ( ) : 0 span ( ) .

n

n n

j j n n n j j nj n

S f t e n Z t

g t a f t a or a H f t

nH

f

( )T ff H

The best approximation of in ( ) , . Here , are the

Fourier coefficients of relative to the orthonormal set .

n

n j j jj n

f H T f f f f f f

f S

* 2 *

2 * *

Thm6.24 is an orthogonal projection has an adjoint and .

Pf: ( ) is trivial. We need only show that exists and . Now,

( ) ( ) and ( ) ( ). For

T T T T T T

T T T T T

V R T N T R T N T

any , , ( ) and

y ( ) s.t.

i

i

x y V x R T

N T

T: an orthogonal projection on W. U: a projection on W along 2W .


45

1 1 2 2

1 1 2 1 2

1 2 2 1 2

, and y Hence,

, ( ) , , .

( ), , , .

( ), , ( ) for all , .

x x y x y

x T y x y x x x

T x y x x y x x

T x y x T y x y V

T * * exists and .T T

2 *

*

( ) Now suppose that . Then this is projection by Remark (iii).

It remains to show that ( ) ( ) and ( ) = ( ).

Let ( ) and y ( ). Then ( ) ( ), and so

,

T T T

R T N T R T N T

x R T N T x T x T x

x y T

*

2

( ), , ( ) ,0 0.

( ) ( ) ( ) .

Let y ( ) . We must show that ( ), that is ( ) . Now,

( ) - ( ) , ( ) - ( ), ( ) - , ( ) - .

x y x T y x

x N T R T N T

N T y R T T y y

T y y T y y T y y T y T y y y T y y

2

* 2

Since ( ) ( ). ( ), , ( ) - 0. But also

( ), ( ) - , ( ( ) - ) , ( ) ( ) ,0 0.

( ) ( ). Hence ( ) ( ) . ( ) = ( ) ( ).

T y y N T T T y T y y

T y T y y y T T y y y T y T y y

T y y y R T R T N T R T N T N T

(See Ex. 13(b) of §6.2). Now suppose that ( )x R T . Then for any

y V , we have *( ), , ( ) , ( ) 0.T x y x T y x T y So, T (x) = 0, and thus

( ). ( ) ( ).x N T R T N T


46

Thm6.25 (The Spectral Thm): dim ( ) .

Let , 1, 2,..., , are distinct eigenvalues of .

Assume that

is normal if .

i

V

i k T

T F C

and that

is self-adjoint if .T F R

Let W and be the orthogonal projection of on W .ii i i

E T V

1 2

1 2

1 1 2 2

(a) ... .

(b) If the direct sum of the subspaces for , then .

(c) for 1 , .

(d) ... .

(e) T= ... .

k

i j i i

i j ij i

k

k k

V W W W

W W j i W W

TT T i j k

I T T T

T T T

1 2

1

Pf (a): is diagonalizable.

so ... by Thm5.11.

(b) If and y for some , then , 0. (by Thm6.15d).

If , where . , 0.

k

i j

k

i l l l ill i

T

V W W W

x W W i j x y

y W y y y W x y W W

已知

.

From (a), dim( ) dim( ) dim( ). On the other hand,

dim( ) dim( ) dim( ) .

i

i i

i i i i

W V W

W V W W W


47

1

11 1

1 2

(c) Let , ( ) ( ( )) ( ) . For .

( ) ( ) 0.

(d) Let . Then ( ) ( ... )( ) .

... .

(e) ( )

k

i i i i i i i i i ii

i j i i

k k

i i i k ii i

k

x y y W T x y T T x T y y i j

TT x T y

x V x y T x y T T x y x

I T T T

T x T1 1 1 1

( ) ( ) ( ).

k k k k

i i i i i ii i i i

y T y y T x

1 2

1 2

1 1 2 2

Remark (i): , ,... , the set of eigenvalues of , is called the spectrum of .

(ii) ... is called the resolution of the identity.

(iii) ... is calle

k

k

k k

T T

T T T

T T T

d the spectral decomposition of .T

1

(iv) Let the orthonormal basis for .

Let =

i i

k

ii

m W

m

1

2

1

2

1 1 2 2

1 1 2 2

0

.

0

(v) For any polynomial , ( ) ( ) ( ) ... ( ) .

Pf: For any , ... .

k

m

m

k m

k k

n n n nk k

I

IT

I

g g T g T g T g T

n N T T T T

1 1 2 2 Hence, ( ) ( ) ( ) ... ( ) .k kg T g T g T g T


48

*

1

* *

1

Cor1. If , then is normal ( ) for soeme polynormal .

( ) Let be its spectral decomposition.

, since . Choose a polynomial such that ( ) .

T

k

i ii

k

i i i i ii

F C T T g T g

T T

T T T T g

*

1 1

* *

hen ( ) ( ) .

( ) ( ) ( ) .

k k

i i i ii i

g T g T T T

TT Tg T g T T T T

1

*1 1

Cor2. If , then is unitary is normal and =1 for every eigenvalue of .

Pf: ( ) Done already.

( ) Let be its spectral decomposition of . Then

( ...

k

i ii

F C T T T

T T T

TT T

1 1 1 2)( ... ) ... .k k k k kT T T T T T I

*

1

Cor3. If and is normal, then is self-adjoint eigenvalue of is real.

( ) Done already.

( ) Let be its spectral decomposition of . Since are real, .k

i i ii

F C T T T

T T T T T

1

1 1 2 2

Cor4. Let as in the spectral theorem with spectral decomposition . Then

for each a polynomial such that ( ) .

Pf: Pick ( ) . Then ( ) ( ) ( ) ...

k

i ii

j j j

i i ij j j j

T T T

T g g T T

g g T g T g T

( ) .j k k jg T T


49

§6.7 The Singular Value Decomposition

H.W 1, 3(f), 6(f), 11

我們此處只談 Singular Value Decomposition (SVD) 之矩陣版本. 至於算子

的版本,精神上是一樣的, 但需要先定義. 若T : V → W. 什麼是T*? 在矩陣的

分解中, 若矩陣是 normal on complex spaces 或 self-adjoint on real spaces, 則其

eigenvectors 可構成一組 orthonormal basis. 利用此 basis 可將矩陣對角線化, 但

一般的矩陣則無此良好的分解性質, 因此我們問是否可以利用兩組 orthonormal

bases 來分解一個矩陣, 答案不但是肯定的, 而且對任一矩陣 (非方陣也可) 都

可如此作. 這樣的分解稱為 Singular Value Decomposition. 如此分解得到對角

線的元素稱為 Singular values, 這些 values 皆是非負的.

Thm6.26 Singular Value Theorem for Matrices

Let ( )m nA M F . Assume rank(A) = r. Then, there exist unitary matrices

1 2, ,... nU u u u and 1 2, ,... mV v v v , i.e., 1 2, ,... nu u u orthornormal basis for

nF and 1 2, ,... mv v v orthornormal basis for mF such that

1

2

*

0

,0

0 0

rA rL V AU F

where the diagonal entries are eigenvalues of *A A and iu are their corresponding

eigenvectors.


50

Remark: From * , we get . V AU F AU UF

, 1 .

Hence, 0, .

i ii

v i rAu

r i n

Pf: Consider the square matrix A*A of size n×n. It is a self-adjoint (Hermition)

matrix. Hence, there is an orthonormal basis 1 2, ,... nu u u consisting of

eigenvectors of A*A with corresponding eigenvalues 1 2 ... 0r and

0 for .i i r

To see this, we have that

2*

2, , , , 0 for all . i

i i i i i i i i i i ii

Avv v v v A Av v Av Av i

v

Note also that rank (A*A) = rank(A) = r. 0 for i i r and

0 for 1, 2,..., ix i r .

Therefore,

1 0

* * * .0

0 0

rU A AU AU AU

Let 1 2, ,... . nAU w w w W Then

* , 1 .,0 . 0 for .

ii i i i

i

i rw w w w

i r w i r


51

That is 1 2 1, ,... ,... ,0,0,...0 . n rW w w w w w

* 10 for ,... is linearly indep. i j rw w i j w w

Define , 1, 2,... . i iiii

w wv i r

Let 1 2, ,.... rv v v be extended to 1 2 1, ,.... , ,... . r r mv v v v v

So that is an orthonormal basis for mF .

Let 1 2, ,.... . mV v v v Then

* VF W AU F V AU .

2 2

2

1 2 1 2

EX1. Let be an invertible linear operation on and { : 1},

the unit circle in . Describe ' ( ).

Sol: By the Singular Value Decomposition, { , } and { , },

orth

T R S x R x

R S T S

u u v v

2

2

*

onormal basis for .

so that

( ) , 1, 2, where 0 and are the eigenvalues of

(i.e, are the singular values of T).

i i i i i

i

R

T u v i

T T


52

x

y x

1u1v2u

1

y

2v

1 2 21 2

2

1 1 2 2 1 1 1 2 2 2

1 1

2 2

2 2

1 1 2 21 2

1 2

Then [ ] . Moreover, 1.

Now, ( ) ( ) ( ) .

' [ ( )] .

'

, 1.

If , then

ax a a

a

T x a T u a T u a v a v

a xT x

a y

x yx a y a

1

1 2

( ) is a circle with radius .

If , then ( ) is a llipse.

T S

T S

Applications:

SVD 有許多應用例用求 Pseudo inverse, 利用 Pseudo inverse 可一齊解決 LSP

(Thm6.12) and the Minimum solution (Thm6.13). 解決 homogeneous linear

equations, 找 explicit representation of the range and null space of a matrix,

Low-rank matrix approximations, ….

此處我們簡單看 Pseudo inverse.

1 1 2 2

1 1 2 2

For any , then , , .

Let , and , .

x S x x u u x u u

a x u a x u


53

Pseudo inverse:

註 Pseudo: Not real, but pretending to be real. 一般而言, 一個線性 transformation

T : V →W 不一定有反函數. 這種函數如何定其”假裝”的反函數.

想法: (1) Let dim( ), dim( ) . V W

Let ( ) ( ) V N T N T and ( ) ( ) W R T R T .

Then N(T)

1 T 是有反函數的.

(2) Let : ( ) ( )L N T R T be defined by ( ) ( )L x T x . 即( )

1

N TL T .

(3) Define the pseudo inverse of T, denoted by : T W V ; such that

1 ( ) ( )( ) .

0 ( )

L y y R TT y

y R T

Remark: 1. Such *T is unique, and hence, well-defined.

2. If ( ) 0R T , i.e., T is the zero transformation. T is the zero

transformation from W to V.

3. If ( ) R T V , then -1 T T .

如何利用 SVD, 找 T .

(1) 1 2 , ... nu u u an orthonormal basis for V.

1 2, .... mv v v an orthonormal basis for W.


54

s.t.

, 1( ) ,

0,

i i

i

v i kT u

i k

where i are the singular values of T and rank (T ) = k.

1

1

1 2 k

1

(a) ( ) ,... .

( ) ,... .

(b) ( ) ,v ...v .

( ) ,... .

k n

k

k m

N T u u

N T u u

R T v

R T v v

(2) Hence, 1 1( ) ( ) ,1 i i i i ii

L u v L v u i k

.

1 , 1 .( )

0, .

iii

u i kT v

k i m

如何利用 SVD, 找 A . (Thm 6.29).

(1) Let 1 2 1 2, ,... , , ,... n mU u u u V v v v .

1 , 1 .

0, .

iii

u i kA v

k i n


55

1

*

*

1 0

.1

0 0

. Here is a singular value decomposition of .

k

n m

U A V

A U V A

定理 6.30 是結合 Thms 6.12 and 6.13 的結果. 作法是利用 pseudo inverse.

Thm 6.30 Consider Ax b , where ( ) m nA M F . Let z A b . Then

(i) If Ax b is consistent, then z is the minimal solution (See also Thm 6.13).

(ii) If Ax b is inconsistent, then z is the unique best approximation having

minimum norm (See also Thm 6.12).


56

§6.8 Bilinear and Quadratic Forms

The Second Derivative Test for Functions of Several Variables.

1 2: ( , ,... ), : smooth n

nf R R z f t t t f

Goal: Finding local extrema of f.

Critical point: for which ( ) 0 for 1, 2... .

n

i

fP R P i nt

Facts: (i) P is a local extremum P is a critical point.

(ii) 反之不成立.

(iii) The second partial derivatives of f at a critical point can often be used to

test for a local extremum at P.

Hessian matrix of f at P = 2 ( ) .

( )( )

i j

i j

f P A At t

Thm 6.37 (The Second Derivative Test). Let 1 2, ,... nf t t t be a real-valued function

is n real variables. Assume all third-order derivatives exist and are continuous.

Let P be a critical points of f, and let A(P) be the Hessian of f at P.

(a) If all eigenvalues of A(P) are positive, then f has a local minimum at P.

(b) If all eigenvalues of A(P) are negative, then f has a local maximum at P.

(c) If at least one positive and at least one negative eigenvalue, then f has no local

extremum at P (P is called a saddle-point of f ).

(d) If rank(A(P)) < n and A(P)does not have both positive and negative eigenvalues,


57

then the second derivative is inconclusive.

Method I: Let 1 2, ,... , where 1. nu u u u u Let ( )uf P be the directional

derivative of f along the direction u at P. Then

1 2( ) ( ) , where ( ) ( ), ( ),... ( ) . u nf P f P u f x f x f x f x Here

1 2, .... and ( ) ( )

n i i

fx x x x f x x fx

對第 i 個變數 ix 之偏微後, 作用

在 x . Hence, 1

( ) ( )

n

u i ii

f P f P u . Then

1 21 1 1

( ( )) | ( ) , ( ) ,..., ( ) .

nn n n

u u x p i i i i i ii i i

f f x f P u f P u f P u u

Here, 2

( ) ( ). ij j i

ff P Px x

( ( )) | , where ( ) . tu u x p ij n nf f x uHu H f P Note that

( ) ( )ij jif P f P and so H is a real symmetric matrix. Therefore, : Q

orthogonal s.t.

1

21 2

0

diag , ,...

0

t

n

n

Q HQ D

.

Hence, . tt t t t t t tuHu uQDQ u Q u DQ u Let

21 21

, ,...

ntt t t

n i ii

Q u u u u uHu u , where

2

11 and so 1.

n

t t ti

iQ u u u

Thus, if 0i for all i, then ( ( )) | 0 t

u u x pf f x uHu for all , 1. u u f 在 P


58

點沿任何方向看都是凹上方. ( ) f P is a local minimum.

On the other hand, if 0i for all i, then f 在 P 點沿任何方向看都是凹向下.

( ) f P is a local maximum. If , i j s.t. , 0 , ( ) i j P f P is a saddle point.

Pf: Method II:

If p 0, define

1 2 1 1 2 2, ,... , ,..., ( ). n n ng t t t f t p t p p t f p

Then

1. f has a local max(min) at p. g has a local max(min) at 0.

2. The partial derivatives of g at 0 = the corresponding partial derivatives of f at p.

3. 0 is a critical point of g.

4. 2 ( )( )

( )( )

i j

i j

g vA pt t

for all i at p.

WLOG, we may assume that p = 0 and f (p) = 0.

Applying Taylor’s Theorem to f to obtain the second-order approximation of f

around 0. We have

2

1 2 1 21 , 1

(0) 1 (0)( , ,.... ) (0) ( , ,.... ).2 ( )( )

n n

n i i j nt i j i j

f ff t t t f t t t S t t tt t t


59

2

1 2, 1

1 (0) ( , ,.... ),2 ( )( )

n

i j ni j i j

f t t S t t tt t

1 2 1 21 (0) ( , ,.... ) , where ( , ,.... ).2

t tn nx A x S t t t x t t t

Where S is a real-valued function on nR s.t.

1 2

1 22 2 2 20 { , ... } 0

1 2

( , ..., )( )lim lim 0 (1)...

n

n

x t t tn

S t t tS xt t tx

Since A(p) is symmetric, 2 2(0) (0) , orthogonal matrix such that

( )( ) ( )( )

i j j i

f f Qt t t t

1

2

0

(0) .

0

(0) .

t

n

t

Q A Q D

A QDQ

Hence,

1̀ 2 1 21, ,... ( , ..., ), where .2

t tn nf t t t y Dy S t t t y Q x

2

1

2 2

(a) Let 0 for all , and let min . Then ( ) ( ).

From (1), whe have that given 0 , 0 s.t.

if , then ( ) . Thus ( ) ( - ) 0 (0).

i ii ni f x x S x

a

x S x a x f x a x f


60

(b) Similarly, if 0 for all , then 0 is a local maximum point.

(c) If 0 and 0, ( ), then

i

i j

i p

i j

Then let 0,0,0, ,0,...0 ix t and let it be small enough. Then ( ) 0f x . On

the other hand, if 0,0,...0, ,0,...0 jx t and let jt be small enough , then

( ) 0f x . That is p = 0 is a saddle point.

(e) Let 2 4 2 41 2 1 2 1 2 1 2( , ) and ( , ) . f t t t t g t t t t Both functions have a critical

point at p = 0.

and

11 12 11 12

21 22 21 22

(0) (0) (0) (0)2 0( ) .

(0) (0) (0) (0)0 0

f f g gA p

f f g g

However, f does not have a local extremum at 0, where as g has a local minimum at 0.

ch6 inner product spacesocw.nctu.edu.tw/course/linear_algebra_ii/linear-algebra... · 2018. 1....

Documents