qr factorization direct method to solve linear systems problems that generate singular matrices
DESCRIPTION
Pertemuan 2. Outline. QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices Modified Gram-Schmidt Algorithm QR Pivoting Matrix must be singular, move zero column to end. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/1.jpg)
• QR Factorization– Direct Method to solve linear systems
• Problems that generate Singular matrices
– Modified Gram-Schmidt Algorithm– QR Pivoting
• Matrix must be singular, move zero column to end.
– Minimization view point Link to Iterative Non stationary Methods (Krylov Subspace)
Outline
![Page 2: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/2.jpg)
1
1
v1 v2 v3 v4
The resulting nodal matrix is SINGULAR, but a solution exists!
LU Factorization fails Singular Example
0
1
1
1
2100
1100
0011
0011
4
3
2
1
v
v
v
v
![Page 3: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/3.jpg)
The resulting nodal matrix is SINGULAR, but a solution exists!
Solution (from picture):v4 = -1v3 = -2v2 = anything you want solutionsv1 = v2 - 1
LU Factorization fails Singular Example
2100
1100
0011
0011
2100
1100
0000
0011One step GE
![Page 4: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/4.jpg)
1 1
2 21 2 N
N N
x b
x bM M M
x b
1 1 2 2 N Nx M x M x M b
Recall weighted sum of columns view of systems of equations
M is singular but b is in the span of the columns of M
QR Factorization Singular Example
![Page 5: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/5.jpg)
1 1 2 2i N N iM x M x M x M M b
0i jM M i j
Orthogonal columns implies:
Multiplying the weighted columns equation by i-th column:
Simplifying using orthogonality:
i
i i i i i
i i
M bx M M M b x
M M
QR Factorization – Key ideaIf M has orthogonal columns
![Page 6: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/6.jpg)
Picture for the two-dimensional case
1M
2Mb
Non-orthogonal Case
1M
2M
b
Orthogonal Case
2x1x
QR Factorization - M orthonormal
0 and 1i j i iM M i j M M M is orthonormal if:
![Page 7: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/7.jpg)
1 1
2 21 2 N
N N
x b
x bM M M
x bOriginal Matrix
1 1
2 21 2 N
N N
y b
y bQ Q Q
y bMatrix withOrthonormal
Columns
TQy b y Q b
How to perform the conversion?
QR Factorization – Key idea
![Page 8: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/8.jpg)
1 2 2 2 12 1Given , find so that, =M M Q M r M
1 2 1 2 12 1 0 M Q M M r M
1 212
1 1
M Mr
M M
2M
2Q
1M
12r
QR Factorization Projection formula
![Page 9: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/9.jpg)
1 2122 2 1=Now find so that 0rQ M Q Q Q
1 1 1 1
1 111
1
111Q M M Q Q
M M r
12 1 2r Q M
Formulas simplify if we normalize
2 2 2
222
2
Fi1
al1
n ly r
Q Q QQ Q
QR Factorization – Normalization
![Page 10: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/10.jpg)
1 11 2 1 1 2 2 1 2 1 1 2 2
2 2
x yM M x M x M Q Q y Q y Q
x y
1 1 2 2 111 22 12M Q M Q Qr r r
Mx=b Qy=b Mx=Qy
11 12 1 1
2 2220
x y
x
r r
r y
QR Factorization – 2x2 case
![Page 11: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/11.jpg)
1 1 11 2 1 2
2 2 2
11 12
220Upper
Triangular
x x bM M Q Q
x x b
Orthonormal
r r
r
Step 1) TQRx b Rx Q b b Two Step Solve Given QR
Step 2) Backsolve Rx b
QR Factorization – 2x2 case
![Page 12: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/12.jpg)
12 11 2 3 1 2 1 3 3 231 2rM M M M M M M rMr M
13 21 3 1 23 0M M M Mr r
13 22 3 1 23 0M M M Mr r
To Insure the third column is orthogonal
QR Factorization – General case
![Page 13: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/13.jpg)
13 21 3 1 23 0M M M Mr r
13 22 3 1 23 0M M M Mr r
1 31 1 1 2
2 32
1
1 2
3
232
M MM M M M
M MM M M
r
M r
QR Factorization – General case
In general, must solve NxN dense linear system for coefficients
![Page 14: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/14.jpg)
1,1 1 1 1 1
1 1 ,1 1 11
N N
N N N N N
N
N N
M M M M M M
M M M M M M
r
r
To Orthogonalize the Nth Vector
2 3 inner products or workN N
QR Factorization – General case
![Page 15: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/15.jpg)
11 2 3 2 13 21 3 1 232 1M M M M M Q M Q Qr r r
1 3 1 213 23 13 1 30Q M Q Q Mr Qr r
To Insure the third column is orthogonal
2 3 1 213 23 23 2 30Q M Q Q Mr Qr r
QR Factorization – General caseModified Gram-Schmidt Algorithm
![Page 16: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/16.jpg)
For i = 1 to N “For each Source Column”
For j = i+1 to N { “For each target Column right of source”
endend
1
ii i
irQ M
jij iMr Q
iii iMr M
Normalize
j jj i iM M Qr
2
1
2 2 operationsN
i
N N
3
1
( )2 operationsN
i
N i N N
QR FactorizationModified Gram-Schmidt Algorithm
(Source-column oriented approach)
![Page 17: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/17.jpg)
4M
3M
2M
1M
12 13 14r r r
1Q
2Q
3Q
4Q
11r
22r 23 24r r
33r
44r
34r
QR Factorization – By picture
![Page 18: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/18.jpg)
1 2 1 1 2 2
1 0 0
0 1
0 0
0 1
N NNx x xx e x e x ex
1 11 1 2 2 22 N N NNx Me x Me x x M x Me xM MMx
Suppose only matrix-vector products were available?
More convenient to use another approach
QR Factorization – Matrix-Vector Product View
![Page 19: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/19.jpg)
For i = 1 to N “For each Target Column”
For j = 1 to i-1 “For each Source Column left of target”
end
end
1
ii i
irQ M
jji iQr M
iii iMr M
Normalize
i ii j jM M Qr
2
1
2 2 operationsN
i
N N
3
1
( )2 operationsN
i
N i N N
"matrix-vector product"i iM Me
QR FactorizationModified Gram-Schmidt Algorithm
(Target-column oriented approach)
![Page 20: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/20.jpg)
4M
3M
2M
1M
1Q
2Q
3Q
4Q
QR Factorization
4M
3M
2M
1M
1Q
2Q
3Q
4Q
r11 r12
r22
r13
r23
r33
r14
r24
r34
r44
r11
r22
r12 r14r13
r23 r24
r33 r34
r44
![Page 21: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/21.jpg)
1 3
0
0
0NMQ M
What if a Column becomes Zero?
Matrix MUST BE Singular!1) Do not try to normalize the column.2) Do not use the column as a source for orthogonalization.3) Perform backward substitution as well as possible
QR Factorization – Zero Column
![Page 22: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/22.jpg)
1 3
0
0
0NQ Q Q
Resulting QR Factorization
11 12 13 1
33 3
0 0 0 0
0 0
0 0 0
0 0 0
N
N
NN
r r r r
r r
r
QR Factorization – Zero Column
![Page 23: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/23.jpg)
1 1
2 21 2 N
N N
x b
x bM M M
x b
1 1 2 2 N Nx M x M x M b
Recall weighted sum of columns view of systems of equations
M is singular but b is in the span of the columns of M
QR Factorization – Zero Column
![Page 24: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/24.jpg)
Reasons for QR Factorization
• QR factorization to solve Mx=b– Mx=b QRx=b Rx=QTb
where Q is orthogonal, R is upper trg
• O(N3) as GE
• Nice for singular matrices– Least-Squares problem
Mx=b where M: mxn and m>n
• Pointer to Krylov-Subspace Methods– through minimization point of view
![Page 25: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/25.jpg)
2
1
Minimize over all xN
T
ii
R x R x R x
Definition of the Residual R: R x b Mx
Find x which satisfies
Mx b
Equivalent if b span cols M
and min 0T
Mx b R x R xx
Minimization More General!
QR Factorization – Minimization View
![Page 26: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/26.jpg)
1 1 1 1 1 1Suppose and therefo rex x e Mx x Me x M
1 1 1 1
T TR x R x b x Me b x Me
One dimensional Minimization
21 1 1 1 12
TT Tb b x b Me x Me Me
1 1 1 12 2 0T TTd
b Me x MR x R Mxx
e ed
11
1 1
T
T T
b Mex
e M Me
Normalization
QR Factorization – Minimization ViewOne-Dimensional Minimization
![Page 27: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/27.jpg)
1 1Me M
b
1e
1x
One dimensional minimization yields same result as projection on the column!
11
1 1
T
T T
b Mex
e M Me
QR Factorization – Minimization ViewOne-Dimensional Minimization: Picture
![Page 28: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/28.jpg)
1 1 2 2 1 1 2 2Now and x x e x e Mx x Me x Me
1 1 2 2 1 1 2 2
T TR x R x b x Me x Me b x Me x Me
Residual Minimization
21 1 1 1 12
TT Tx b Me xb eb Me M
22 2 2 2 22
TTx b Me x Me Me
1 2 1 22T
x x Me MeCoupling
Term
QR Factorization – Minimization ViewTwo-Dimensional Minimization
![Page 29: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/29.jpg)
QR Factorization – Minimization ViewTwo-Dimensional Minimization: Residual Minimization
1 1 2 2 1 1 2 2
T TR x R x b x Me x Me b x Me x Me
22 2 2 2 22
TTx b Me x Me Me
1 2 1 22T
x x Me MeCoupling
Term
21 1 1 1 12
TT Tx b Me xb eb Me M
termcouplingeMeMxeMbdx
xRxdR TTT
)()(220)()(
1111
1
termcouplingeMeMxeMbdx
xRxdR TTT
)()(220)()(
2222
2
To eliminate coupling term: we change search directions !!!
![Page 30: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/30.jpg)
1 1 2 2 1 1 2 2and x v p v p Mx v Mp v Mp
21 1 1 1 12
TTT TR x R x b v b Mp v Mp Mb p
More General Search Directions
22 2 2 2 22
TTv b Mp v Mp Mp
1 2 1 22T
v v Mp MpCoupling
Term
1 2 1 2 span , = span ,p p e e
1 2If Minimization 0 s D ecouple!!T Tp M Mp
QR Factorization – Minimization ViewTwo-Dimensional Minimization
2211 exexx
![Page 31: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/31.jpg)
More General Search Directions
QR Factorization – Minimization ViewTwo-Dimensional Minimization
Goal: find a set of search directions such that
In this case minimization decouples !!!
pi and pj are called MTM orthogonal
ijpMMp jTT
i when 0
![Page 32: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/32.jpg)
1
1
0i
T Ti i ji j i j
j
p e r p p M Mp
i-th search direction equals orthogonalized unit vector
T
j i
ji T
j j
Mp Mer
Mp Mp
Use previous orthogonalized Search directions
QR Factorization – Minimization ViewForming MTM orthogonal Minimization Directions
![Page 33: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/33.jpg)
2Minimize: 2T T
i i i i iv Mp Mp v b Mp
Ti
i T
i i
b Mpv
Mp Mp
Differentiating 2 0 2:T T
i i i iv Mp Mp b Mp
QR Factorization – Minimization ViewMinimizing in the Search Direction
When search directions pj are MTM orthogonal, residual minimization becomes:
![Page 34: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/34.jpg)
For i = 1 to N “For each Target Column”
For j = 1 to i-1 “For each Source Column left of target”
end
end
1
ii i
irp p
iii iMpr Mp
Normalize
i ix x v p
i ip e
Tjj
Tiir p M Mp
i ji i jp p pr Orthogonalize Search Direction
QR Factorization – Minimization ViewMinimization Algorithm
![Page 35: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/35.jpg)
Intuitive summary
• QR factorization Minimization view
(Direct) (Iterative)• Compose vector x along search directions:
– Direct: composition along Qi (orthonormalized columns of M) need to factorize M
– Iterative: composition along certain search directions you can stop half way
• About the search directions:– Chosen so that it is easy to do the minimization
(decoupling) pj are MTM orthogonal
– Each step: try to minimize the residual
![Page 36: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/36.jpg)
1
1
11
1e
p
r
2 112
22
2
1e e
r
p
r
2
1i
N
N
iNN
rr
e e
p
1Q
M M
2Q
M
NQ
MTMOrthonormal
Orthonormal
Compare Minimization and QR
![Page 37: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/37.jpg)
Summary
• Iterative Methods Overview– Stationary– Non Stationary
• QR factorization to solve Mx=b– Modified Gram-Schmidt Algorithm– QR Pivoting– Minimization View of QR
• Basic Minimization approach• Orthogonalized Search Directions• Pointer to Krylov Subspace Methods
![Page 38: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/38.jpg)
Forward Difference Formula
Of order o(h2) forNumerical Differentiation
Y’ = e-x sin(x)
![Page 39: QR Factorization Direct Method to solve linear systems Problems that generate Singular matrices](https://reader033.vdocument.in/reader033/viewer/2022051316/56814a96550346895db7a350/html5/thumbnails/39.jpg)
Terima kasihTerima kasih