Download - Krylov-Subspace Methods - II
Krylov-Subspace Methods - II
Lecture 7
Alessandra Nardi
Thanks to Prof. Jacob White, Deepak Ramaswamy, Michal Rewienski, and Karen Veroy
Last lectures review• Overview of Iterative Methods to solve Mx=b
– Stationary– Non Stationary
• QR factorization– Modified Gram-Schmidt Algorithm– Minimization View of QR
• General Subspace Minimization Algorithm• Generalized Conjugate Residual Algorithm
– Krylov-subspace – Simplification in the symmetric case– Convergence properties
• Eigenvalue and Eigenvector Review– Norms and Spectral Radius– Spectral Mapping Theorem
Approximately Solve Mx b
10 1Approximate as a weighted sum of , ,...,k
kx w w w
1 1 0
0
kk k
i ii
r b Mx r Mw
1
0
k
ki i
i
x w
Residual Minimizing idea: pick ' to minimizei s
2
21 1 1 0
22
kTk k k
i ii o
r r r r Mw
Arbitrary Subspace MethodsResidual Minimization
221 0
20 2
Minimizing is easy ifk
ki i
i
r r Mw
0 1 0 1, ,..., , ,...,k kspan p p p span w w w
0 1Create a set of vectors , ,..., such thatkp p p
0 or is orthogonal to T
i j i jMw Mw Mw Mw
and 0T
i jMp Mp
Use Gram-Schmidt on Mwi’s!
Arbitrary Subspace MethodsResidual Minimization
1 0 0
0
kk i
i ki
x M r M r
1 0 1 0 0
0
kk i
i ki
r r M r I M M r
kth order polynomial
Krylov Subspace Methods
k
iii
k wx0
1
},...,,{},..., 121 bMMbbspanw,wspan{w k
k
Krylov Subspace
Krylov Subspace MethodsSubspace Generation
0Note: for any 0
0 1 0 0 0 00span , = span ,r r r Mr r Mr
The set of residuals also can be used as a representation of the Krylov-Subspace
Generalized Conjugate Residual AlgorithmNice because the residuals generate next search directions
1
11
0
Tkkjk
k jTj
j j
Mr Mpp r p
Mp Mp
Tkk
k T
k k
r Mp
Mp Mp
1k kk kx x p
1k kk kr r Mp
Determine optimal stepsize in kth search direction
Update the solution (trying to
minimize residual) and the residual
Compute the new orthogonalized search direction (by using the most
recent residual)
Krylov-Subspace MethodsGeneralized Conjugate Residual Method
(k-th step)
1
11
0
Tkkjk
k jTj
j j
Mr Mpp r p
Mp Mp
Tkk
k T
k k
r Mp
Mp Mp
1k kk kx x p
1k kk kr r Mp
Vector inner products, O(n)Matrix-vector product, O(n) if sparse
Vector Adds, O(n)
O(k) inner products, total cost O(nk)
If M is sparse, as k (# of iters) approaches n,3total cost ( ) (2 ) .... ( ) ( )O n O n O kn O n
Better Converge Fast!
Krylov-Subspace MethodsGeneralized Conjugate Residual Method
(Computational Complexity for k-th step)
Summary
• What is an iterative non stationary method: x(k+1) =x(k)+akpk
• How search to calculate:– Search directions (pk)– Step along search directions (ak)
• Krylov Subspace GCR • GCR is O(k2n)
– Better converge fast!
Now look at convergence properties of GCR
0 jIf for all in GCR, thenj k 0 0
0 1span , ,..., span , ,...,1) kkp p p r Mr Mr
th1 02) is the k ( ) , o rderkk kx M r
21
2polynomial which minimizes kr
1 1 0 03) ( )k kkr b Mx r M M r
0 01( )k kI M M r M r
01where is the order poly 1
th
k M r k 21
12minimizing subject to 0 = 1 k
kr
Krylov Methods Convergence AnalysisBasic properties
GCR Optimality Property
k+1 polynomial such that 0 =1
ThereforeAny polynomial which satisfies the
constraints can be used to get an upper bound on
1
0
kr
r
1 0k+1 k+1( ) r where is any ordertk hkr M
Krylov Methods Convergence AnalysisOptimality of GCR poly
Theorem: Any induced norm is a bound on the spectral radius
max1 ll
l
Mxx
M
Proof:First pick , 1i i l
x u u
i i i i i il l lMu u u
Eigenvalues and eigenvectors reviewInduced norms
Given a polynomial
0 1p
pf x a a x a x
Apply the polynomial to a matrix
0 1p
pf M a a M a M
Then
spectrum f M f spectrum M
Useful Eigenproperties Spectral Mapping Theorem
Krylov Methods Convergence AnalysisOverview
where is any (k+1)-th order polynomial
subject to:
may be used to get an upper bound on
)(~1 Mk
0
1
r
r k
1)0(~1 k
01
01
101
1 )(~)()( rMrMrrMr kkk
kk
)(~1 Mk
Matrix norm property GCR optimality property
• Review on eigenvalues and eigenvectors– Induced norms: relate matrix eigenvalues to the
matrix norms– Spectral mapping theorem: relate matrix eigenvalues
to matrix polynomials
• Now ready to relate the convergence properties of Krylov Subspace methods to eigenvalues of M
Krylov Methods Convergence AnalysisOverview
1
1
1 1
condition number of M's eigen space
k
n n
k n
v v v v
Cond(V)
1
1
1 1
eigenvectors of M
( )k
n n
k n
k v v v vM
Krylov Methods Convergence AnalysisNorm of matrix polynomials
12
1
2
maxk
k i ixi
k n
x
max i k i
( ) max( ) i k ik cond VM
Krylov Methods Convergence AnalysisNorm of matrix polynomials
1) The GCR Algorithm converges to the exact solution in at most n steps
where . Then, max = 0, i i n iM
0 and therefore 0nn M r
2) If M has only q distinct eigenvalues, the GCR Algorithm converges in at most q steps
1 2Proof: Let = ... q qx x x x
1 2Proof: Let = ... n nx x x x
Krylov Methods Convergence AnalysisImportant observations
1
1 1 1( ) n nv v v vcond V
If M = MT then
2) M has real eigenvalues
( ) maxk i k iM
1) M has orthonormal eigenvectors
If M is postive definite, then > 0M
Krylov Methods Convergence AnalysisConvergence for MT=M - Residual Polynomial
* = evals(M)- = 5th order poly- = 8th order poly
1
Krylov Methods Convergence AnalysisResidual Polynomial Picture (n=10)
Keep as small as possible:k iStrategically place zeros of the poly
Krylov Methods Convergence AnalysisResidual Polynomial Picture (n=10)
Then a good polynomial ( is small) kp Acan be found by solving the min-max problem
min max,. .
0 1
min max
k
kth order kxpolys s tp
p x
The min-max problem is exactly solved by Chebyshev Polynomials
min max minConsider , , > 0 M
Krylov Methods Convergence AnalysisConvergence for MT=M – Polynomial min-max problem
1 cos cos 1,1kC x k x x
min max,. .
0 1
min max
k
kth order kxpolys s t
x
=
The Chebyshev Polynomial
min max
min
max min,
min
max min
1 2
max
1 2
k
x
k
xC
C
Krylov Methods Convergence AnalysisConvergence for MT=M – Chebyshev solves min-max
Chebychev Polynomials minimizing over [1,10]
min max,. .
0 1
min max
k
kth order kxpolys s t
x
max
min
max
min
1
2
1
k
max
max min
1
1 2kC
Krylov Methods Convergence AnalysisConvergence for MT=M – Chebyshev bounds
min max minIf , , > 0 M
max
0min
max
min
1
2
1
k
kr r
Krylov Methods Convergence AnalysisConvergence for MT=M – Chebyshev result
1 0 0
0 1
0
0 0 1
1 0 0
0 2
0
0 0 N
For which problem will GCR Converge Faster?
Krylov Methods Convergence AnalysisExamples
0
kr
r
Iteration
Which Convergence Curve is GCR?
GCR Algorithm can eliminate outlying
eigenvalues by placing polynomial
zeros directly on them.
Krylov Methods Convergence AnalysisChebyshev is a bound
Iterative Methods - CG
kkTk
kTkk
k
kT
k
kTk
k
kkkk
kkkk
drr
rrrd
Mdd
rr
Mdrr
dxx
)()(
)1()1()1(
1
)()(
)()1(
)()1(
)(
)(
)(
Convergence is related to:– Number of distinct eigenvalues– Ratio between max and min eigenvalue
Why ?How?
Now we know
• Reminder about GCR– Residual minimizing solution– Krylov Subspace– Polynomial Connection
• Review Eigenvalues– Induced Norms bound Spectral Radius– Spectral mapping theorem
• Estimating Convergence Rate– Chebyshev Polynomials
Summary