qr factorization –direct method to solve linear systems problems that generate singular matrices...
Post on 19-Dec-2015
222 views
TRANSCRIPT
![Page 1: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/1.jpg)
• QR Factorization– Direct Method to solve linear systems
• Problems that generate Singular matrices
– Modified Gram-Schmidt Algorithm– QR Pivoting
• Matrix must be singular, move zero column to end.
– Minimization view point Link to Iterative Non stationary Methods (Krylov Subspace)
Outline
![Page 2: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/2.jpg)
1
1
v1 v2 v3 v4
The resulting nodal matrix is SINGULAR, but a solution exists!
LU Factorization fails Singular Example
0
1
1
1
2100
1100
0011
0011
4
3
2
1
v
v
v
v
![Page 3: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/3.jpg)
The resulting nodal matrix is SINGULAR, but a solution exists!
Solution (from picture):v4 = -1v3 = -2v2 = anything you want solutionsv1 = v2 - 1
LU Factorization fails Singular Example
2100
1100
0011
0011
2100
1100
0000
0011One step GE
![Page 4: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/4.jpg)
1 1
2 21 2 N
N N
x b
x bM M M
x b
1 1 2 2 N Nx M x M x M b
Recall weighted sum of columns view of systems of equations
M is singular but b is in the span of the columns of M
QR Factorization Singular Example
![Page 5: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/5.jpg)
1 1 2 2i N N iM x M x M x M M b
0i jM M i j
Orthogonal columns implies:
Multiplying the weighted columns equation by i-th column:
Simplifying using orthogonality:
i
i i i i i
i i
M bx M M M b x
M M
QR Factorization – Key ideaIf M has orthogonal columns
![Page 6: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/6.jpg)
Picture for the two-dimensional case
1M
2Mb
Non-orthogonal Case
1M
2M
b
Orthogonal Case
2x1x
QR Factorization - M orthonormal
0 and 1i j i iM M i j M M M is orthonormal if:
![Page 7: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/7.jpg)
1 1
2 21 2 N
N N
x b
x bM M M
x bOriginal Matrix
1 1
2 21 2 N
N N
y b
y bQ Q Q
y bMatrix withOrthonormal
Columns
TQy b y Q b
How to perform the conversion?
QR Factorization – Key idea
![Page 8: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/8.jpg)
1 2 2 2 12 1Given , find so that, =M M Q M r M
1 2 1 2 12 1 0 M Q M M r M
1 212
1 1
M Mr
M M
2M
2Q
1M
12r
QR Factorization Projection formula
![Page 9: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/9.jpg)
1 2122 2 1=Now find so that 0rQ M Q Q Q
1 1 1 1
1 111
1
111Q M M Q Q
M M r
12 1 2r Q M
Formulas simplify if we normalize
2 2 2
222
2
Fi1
al1
n ly r
Q Q QQ Q
QR Factorization – Normalization
![Page 10: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/10.jpg)
1 11 2 1 1 2 2 1 2 1 1 2 2
2 2
x yM M x M x M Q Q y Q y Q
x y
1 1 2 2 111 22 12M Q M Q Qr r r
Mx=b Qy=b Mx=Qy
11 12 1 1
2 2220
x y
x
r r
r y
QR Factorization – 2x2 case
![Page 11: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/11.jpg)
1 1 11 2 1 2
2 2 2
11 12
220Upper
Triangular
x x bM M Q Q
x x b
Orthonormal
r r
r
Step 1) TQRx b Rx Q b b Two Step Solve Given QR
Step 2) Backsolve Rx b
QR Factorization – 2x2 case
![Page 12: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/12.jpg)
12 11 2 3 1 2 1 3 3 231 2rM M M M M M M rMr M
13 21 3 1 23 0M M M Mr r
13 22 3 1 23 0M M M Mr r
To Insure the third column is orthogonal
QR Factorization – General case
![Page 13: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/13.jpg)
13 21 3 1 23 0M M M Mr r
13 22 3 1 23 0M M M Mr r
1 31 1 1 2
2 32
1
1 2
3
232
M MM M M M
M MM M M
r
M r
QR Factorization – General case
In general, must solve NxN dense linear system for coefficients
![Page 14: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/14.jpg)
1,1 1 1 1 1
1 1 ,1 1 11
N N
N N N N N
N
N N
M M M M M M
M M M M M M
r
r
To Orthogonalize the Nth Vector
2 3 inner products or workN N
QR Factorization – General case
![Page 15: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/15.jpg)
11 2 3 2 13 21 3 1 232 1M M M M M Q M Q Qr r r
1 3 1 213 23 13 1 30Q M Q Q Mr Qr r
To Insure the third column is orthogonal
2 3 1 213 23 23 2 30Q M Q Q Mr Qr r
QR Factorization – General caseModified Gram-Schmidt Algorithm
![Page 16: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/16.jpg)
For i = 1 to N “For each Source Column”
For j = i+1 to N { “For each target Column right of source”
endend
1
ii i
irQ M
jij iMr Q
iii iMr M
Normalize
j jj i iM M Qr
2
1
2 2 operationsN
i
N N
3
1
( )2 operationsN
i
N i N N
QR FactorizationModified Gram-Schmidt Algorithm
(Source-column oriented approach)
![Page 17: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/17.jpg)
4M
3M
2M
1M
12 13 14r r r
1Q
2Q
3Q
4Q
11r
22r 23 24r r
33r
44r
34r
QR Factorization – By picture
![Page 18: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/18.jpg)
1 2 1 1 2 2
1 0 0
0 1
0 0
0 1
N NNx x xx e x e x ex
1 11 1 2 2 22 N N NNx Me x Me x x M x Me xM MMx
Suppose only matrix-vector products were available?
More convenient to use another approach
QR Factorization – Matrix-Vector Product View
![Page 19: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/19.jpg)
For i = 1 to N “For each Target Column”
For j = 1 to i-1 “For each Source Column left of target”
end
end
1
ii i
irQ M
jji iQr M
iii iMr M
Normalize
i ii j jM M Qr
2
1
2 2 operationsN
i
N N
3
1
( )2 operationsN
i
N i N N
"matrix-vector product"i iM Me
QR FactorizationModified Gram-Schmidt Algorithm
(Target-column oriented approach)
![Page 20: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/20.jpg)
4M
3M
2M
1M
1Q
2Q
3Q
4Q
QR Factorization
4M
3M
2M
1M
1Q
2Q
3Q
4Q
r11 r12
r22
r13
r23
r33
r14
r24
r34
r44
r11
r22
r12 r14r13
r23 r24
r33 r34
r44
![Page 21: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/21.jpg)
1 3
0
0
0NMQ M
What if a Column becomes Zero?
Matrix MUST BE Singular!1) Do not try to normalize the column.2) Do not use the column as a source for orthogonalization.3) Perform backward substitution as well as possible
QR Factorization – Zero Column
![Page 22: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/22.jpg)
1 3
0
0
0NQ Q Q
Resulting QR Factorization
11 12 13 1
33 3
0 0 0 0
0 0
0 0 0
0 0 0
N
N
NN
r r r r
r r
r
QR Factorization – Zero Column
![Page 23: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/23.jpg)
1 1
2 21 2 N
N N
x b
x bM M M
x b
1 1 2 2 N Nx M x M x M b
Recall weighted sum of columns view of systems of equations
M is singular but b is in the span of the columns of M
QR Factorization – Zero Column
![Page 24: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/24.jpg)
Reasons for QR Factorization
• QR factorization to solve Mx=b– Mx=b QRx=b Rx=QTb
where Q is orthogonal, R is upper trg
• O(N3) as GE
• Nice for singular matrices– Least-Squares problem
Mx=b where M: mxn and m>n
• Pointer to Krylov-Subspace Methods– through minimization point of view
![Page 25: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/25.jpg)
2
1
Minimize over all xN
T
ii
R x R x R x
Definition of the Residual R: R x b Mx
Find x which satisfies
Mx b
Equivalent if b span cols M
and min 0T
Mx b R x R xx
Minimization More General!
QR Factorization – Minimization View
![Page 26: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/26.jpg)
1 1 1 1 1 1Suppose and therefo rex x e Mx x Me x M
1 1 1 1
T TR x R x b x Me b x Me
One dimensional Minimization
21 1 1 1 12
TT Tb b x b Me x Me Me
1 1 1 12 2 0T TTd
b Me x MR x R Mxx
e ed
11
1 1
T
T T
b Mex
e M Me
Normalization
QR Factorization – Minimization ViewOne-Dimensional Minimization
![Page 27: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/27.jpg)
1 1Me M
b
1e
1x
One dimensional minimization yields same result as projection on the column!
11
1 1
T
T T
b Mex
e M Me
QR Factorization – Minimization ViewOne-Dimensional Minimization: Picture
![Page 28: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/28.jpg)
1 1 2 2 1 1 2 2Now and x x e x e Mx x Me x Me
1 1 2 2 1 1 2 2
T TR x R x b x Me x Me b x Me x Me
Residual Minimization
21 1 1 1 12
TT Tx b Me xb eb Me M
22 2 2 2 22
TTx b Me x Me Me
1 2 1 22T
x x Me MeCoupling
Term
QR Factorization – Minimization ViewTwo-Dimensional Minimization
![Page 29: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/29.jpg)
QR Factorization – Minimization ViewTwo-Dimensional Minimization: Residual Minimization
1 1 2 2 1 1 2 2
T TR x R x b x Me x Me b x Me x Me
22 2 2 2 22
TTx b Me x Me Me
1 2 1 22T
x x Me MeCoupling
Term
21 1 1 1 12
TT Tx b Me xb eb Me M
termcouplingeMeMxeMbdx
xRxdR TTT
)()(220)()(
1111
1
termcouplingeMeMxeMbdx
xRxdR TTT
)()(220)()(
2222
2
To eliminate coupling term: we change search directions !!!
![Page 30: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/30.jpg)
1 1 2 2 1 1 2 2and x v p v p Mx v Mp v Mp
21 1 1 1 12
TTT TR x R x b v b Mp v Mp Mb p
More General Search Directions
22 2 2 2 22
TTv b Mp v Mp Mp
1 2 1 22T
v v Mp MpCoupling
Term
1 2 1 2 span , = span ,p p e e
1 2If Minimization 0 s D ecouple!!T Tp M Mp
QR Factorization – Minimization ViewTwo-Dimensional Minimization
2211 exexx
![Page 31: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/31.jpg)
More General Search Directions
QR Factorization – Minimization ViewTwo-Dimensional Minimization
Goal: find a set of search directions such that
In this case minimization decouples !!!
pi and pj are called MTM orthogonal
ijpMMp jTT
i when 0
![Page 32: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/32.jpg)
1
1
0i
T Ti i ji j i j
j
p e r p p M Mp
i-th search direction equals orthogonalized unit vector
T
j i
ji T
j j
Mp Mer
Mp Mp
Use previous orthogonalized Search directions
QR Factorization – Minimization ViewForming MTM orthogonal Minimization Directions
![Page 33: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/33.jpg)
2Minimize: 2T T
i i i i iv Mp Mp v b Mp
Ti
i T
i i
b Mpv
Mp Mp
Differentiating 2 0 2:T T
i i i iv Mp Mp b Mp
QR Factorization – Minimization ViewMinimizing in the Search Direction
When search directions pj are MTM orthogonal, residual minimization becomes:
![Page 34: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/34.jpg)
For i = 1 to N “For each Target Column”
For j = 1 to i-1 “For each Source Column left of target”
end
end
1
ii i
irp p
iii iMpr Mp
Normalize
i ix x v p
i ip e
Tjj
Tiir p M Mp
i ji i jp p pr Orthogonalize Search Direction
QR Factorization – Minimization ViewMinimization Algorithm
![Page 35: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/35.jpg)
Intuitive summary
• QR factorization Minimization view
(Direct) (Iterative)• Compose vector x along search directions:
– Direct: composition along Qi (orthonormalized columns of M) need to factorize M
– Iterative: composition along certain search directions you can stop half way
• About the search directions:– Chosen so that it is easy to do the minimization
(decoupling) pj are MTM orthogonal
– Each step: try to minimize the residual
![Page 36: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/36.jpg)
1
1
11
1e
p
r
2 112
22
2
1e e
r
p
r
2
1i
N
N
iNN
rr
e e
p
1Q
M M
2Q
M
NQ
MTMOrthonormal
Orthonormal
Compare Minimization and QR
![Page 37: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/37.jpg)
Summary
• Iterative Methods Overview– Stationary– Non Stationary
• QR factorization to solve Mx=b– Modified Gram-Schmidt Algorithm– QR Pivoting– Minimization View of QR
• Basic Minimization approach• Orthogonalized Search Directions• Pointer to Krylov Subspace Methods
![Page 38: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/38.jpg)
Forward Difference Formula
Of order o(h2) forNumerical Differentiation
Y’ = e-x sin(x)
![Page 39: QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must](https://reader035.vdocument.in/reader035/viewer/2022062320/56649d3e5503460f94a16f1d/html5/thumbnails/39.jpg)
Terima kasihTerima kasih