lsmr: an iterative algorithm for least-squares problemslsmr: an iterative algorithm for...
Post on 16-Mar-2020
14 Views
Preview:
TRANSCRIPT
LSMR:An iterative algorithm forleast-squares problems
David Fong Michael Saunders
Institute for Computational and Mathematical Engineering (iCME)Stanford University
Copper Mountain Conference on Iterative MethodsCopper Mountain, Colorado Apr 5–9, 2010
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 1/31
LSMR in one slide
solve Ax = b
min∥∥∥∥(AλI
)x−
(b0
)∥∥∥∥2
min ‖Ax− b‖2
LSQR ≡ CG on the normal equationLSMR ≡ MINRES on the normal equation
Almost same complexity as LSQRBetter convergence properties
for inexact solves
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 2/31
LSMR in one slide
solve Ax = bmin
∥∥∥∥(AλI)x−
(b0
)∥∥∥∥2
min ‖Ax− b‖2
LSQR ≡ CG on the normal equationLSMR ≡ MINRES on the normal equation
Almost same complexity as LSQRBetter convergence properties
for inexact solves
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 2/31
LSMR in one slide
solve Ax = bmin
∥∥∥∥(AλI)x−
(b0
)∥∥∥∥2
min ‖Ax− b‖2
LSQR ≡ CG on the normal equationLSMR ≡ MINRES on the normal equation
Almost same complexity as LSQRBetter convergence properties
for inexact solves
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 2/31
LSMR in one slide
solve Ax = bmin
∥∥∥∥(AλI)x−
(b0
)∥∥∥∥2
min ‖Ax− b‖2
LSQR ≡ CG on the normal equationLSMR ≡ MINRES on the normal equation
Almost same complexity as LSQRBetter convergence properties
for inexact solves
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 2/31
LSQR
Iterative algorithm for
min
∥∥∥∥(AλI)x−
(b0
)∥∥∥∥2
Properties
• A is rectangular (m× n) and often sparse• A can be an operator• CG on the normal equation (ATA+ λ2I)x = ATb
• Av, ATu plus O(m+ n) operations per iteration
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 3/31
LSQR
Iterative algorithm for
min
∥∥∥∥(AλI)x−
(b0
)∥∥∥∥2
Properties
• A is rectangular (m× n) and often sparse• A can be an operator• CG on the normal equation (ATA+ λ2I)x = ATb
• Av, ATu plus O(m+ n) operations per iteration
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 3/31
Monotone convergence of residualMeasure of Convergence• rk = b−Axk• ‖rk‖ → ‖r̂‖, ‖ATrk‖ → 0
— LSQR— LSMR
LSQR ‖rk‖
0 50 100 150 200 250 300 350 400 450 5001.69
1.695
1.7
1.705
1.71
1.715
1.72
1.725
1.73
1.735Name:lp fit1p, Dim:1677x627, nnz:9868, id=80
iteration count
||r||
LSQR log ‖ATrk‖
0 50 100 150 200 250 300 350 400 450 500−5
−4
−3
−2
−1
0
1
2Name:lp fit1p, Dim:1677x627, nnz:9868, id=80
iteration count
log|
|ATr|
|
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 4/31
Monotone convergence of residualMeasure of Convergence• rk = b−Axk• ‖rk‖ → ‖r̂‖, ‖ATrk‖ → 0
— LSQR— LSMR
LSQR ‖rk‖
0 50 100 150 200 250 300 350 400 450 5001.69
1.695
1.7
1.705
1.71
1.715
1.72
1.725
1.73
1.735Name:lp fit1p, Dim:1677x627, nnz:9868, id=80
iteration count
||r||
LSQR log ‖ATrk‖
0 50 100 150 200 250 300 350 400 450 500−5
−4
−3
−2
−1
0
1
2Name:lp fit1p, Dim:1677x627, nnz:9868, id=80
iteration count
log|
|ATr|
|
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 4/31
Monotone convergence of residualMeasure of Convergence• rk = b−Axk• ‖rk‖ → ‖r̂‖, ‖ATrk‖ → 0
— LSQR— LSMR
‖rk‖
0 50 100 150 200 250 300 350 400 450 5001.69
1.695
1.7
1.705
1.71
1.715
1.72
1.725
1.73
1.735
iteration count
||r||
Name:lp fit1p, Dim:1677x627, nnz:9868, id=80
lsqrlsmr
log ‖ATrk‖
0 50 100 150 200 250 300 350 400 450 500−5
−4
−3
−2
−1
0
1
2
iteration count
log|
|ATr|
|
Name:lp fit1p, Dim:1677x627, nnz:9868, id=80
lsqrlsmr
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 4/31
LSMR Algorithm
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 5/31
Golub-Kahan bidiagonalization
Given A (m× n) and b (m× 1)
Direct bidiagonalization
UTAV = B
Iterative bidiagonalization
1 β1u1 = b, α1v1 = ATu1
2 for k = 1, 2, . . . , setβk+1uk+1 = Avk − αkuk
αk+1vk+1 = ATuk+1 − βk+1vk
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 6/31
Golub-Kahan bidiagonalization
Given A (m× n) and b (m× 1)
Direct bidiagonalization
UTAV = B
Iterative bidiagonalization
1 β1u1 = b, α1v1 = ATu1
2 for k = 1, 2, . . . , setβk+1uk+1 = Avk − αkuk
αk+1vk+1 = ATuk+1 − βk+1vk
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 6/31
Golub-Kahan bidiagonalization (2)
The process can be summarized by
AVk = Uk+1Bk
ATUk+1 = Vk+1LTk+1
where
Lk =
α1
β2 α2
. . . . . .βk αk
, Bk =
α1
β2 α2
. . . . . .βk αk
βk+1
=
(Lk
βk+1eTk
)
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 7/31
Golub-Kahan bidiagonalization (3)
Vk spans the Krylov subspace:
span{v1, . . . , vk} = span{ATb, (ATA)ATb, . . . , (ATA)k−1ATb}
Define xk = Vkyk
Subproblem to solve
minyk‖rk‖ = min
yk‖β1e1 −Bkyk‖ (LSQR)
minyk‖AT rk‖ = min
yk
∥∥∥∥∥β̄1e1 −(BT
k Bk
β̄k+1eTk
)yk
∥∥∥∥∥ (LSMR)
where rk = b−Axk, β̄k = αkβk
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 8/31
Golub-Kahan bidiagonalization (3)
Vk spans the Krylov subspace:
span{v1, . . . , vk} = span{ATb, (ATA)ATb, . . . , (ATA)k−1ATb}
Define xk = Vkyk
Subproblem to solve
minyk‖rk‖ = min
yk‖β1e1 −Bkyk‖ (LSQR)
minyk‖AT rk‖ = min
yk
∥∥∥∥∥β̄1e1 −(BT
k Bk
β̄k+1eTk
)yk
∥∥∥∥∥ (LSMR)
where rk = b−Axk, β̄k = αkβk
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 8/31
Least squares subproblem
minyk
∥∥∥∥∥β̄1e1 −(BT
k Bk
β̄k+1eTk
)yk
∥∥∥∥∥
= minyk
∥∥∥∥∥β̄1e1 −(RT
kRk
qTk Rk
)yk
∥∥∥∥∥ Qk+1Bk =
(Rk
0
), RT
k qk = β̄k+1ek
= mintk
∥∥∥∥∥β̄1e1 −(RT
k
ϕkeTk
)tk
∥∥∥∥∥ tk = Rkyk, qk = (β̄k+1/(Rk)k,k)ek = ϕkek
= mintk
∥∥∥∥( zkζ̄k+1
)−(R̄k
0
)tk
∥∥∥∥ Q̄k+1
(RT
k β̄1e1
ϕkeTk 0
)=
(R̄k zk
0 ζ̃k+1
)
Things to notexk = Vkyk, tk = Rkyk, zk = R̄ktk, two cheap QRs
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 9/31
Least squares subproblem
minyk
∥∥∥∥∥β̄1e1 −(BT
k Bk
β̄k+1eTk
)yk
∥∥∥∥∥= min
yk
∥∥∥∥∥β̄1e1 −(RT
kRk
qTk Rk
)yk
∥∥∥∥∥ Qk+1Bk =
(Rk
0
), RT
k qk = β̄k+1ek
= mintk
∥∥∥∥∥β̄1e1 −(RT
k
ϕkeTk
)tk
∥∥∥∥∥ tk = Rkyk, qk = (β̄k+1/(Rk)k,k)ek = ϕkek
= mintk
∥∥∥∥( zkζ̄k+1
)−(R̄k
0
)tk
∥∥∥∥ Q̄k+1
(RT
k β̄1e1
ϕkeTk 0
)=
(R̄k zk
0 ζ̃k+1
)
Things to notexk = Vkyk, tk = Rkyk, zk = R̄ktk, two cheap QRs
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 9/31
Least squares subproblem
minyk
∥∥∥∥∥β̄1e1 −(BT
k Bk
β̄k+1eTk
)yk
∥∥∥∥∥= min
yk
∥∥∥∥∥β̄1e1 −(RT
kRk
qTk Rk
)yk
∥∥∥∥∥ Qk+1Bk =
(Rk
0
), RT
k qk = β̄k+1ek
= mintk
∥∥∥∥∥β̄1e1 −(RT
k
ϕkeTk
)tk
∥∥∥∥∥ tk = Rkyk, qk = (β̄k+1/(Rk)k,k)ek = ϕkek
= mintk
∥∥∥∥( zkζ̄k+1
)−(R̄k
0
)tk
∥∥∥∥ Q̄k+1
(RT
k β̄1e1
ϕkeTk 0
)=
(R̄k zk
0 ζ̃k+1
)
Things to notexk = Vkyk, tk = Rkyk, zk = R̄ktk, two cheap QRs
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 9/31
Least squares subproblem
minyk
∥∥∥∥∥β̄1e1 −(BT
k Bk
β̄k+1eTk
)yk
∥∥∥∥∥= min
yk
∥∥∥∥∥β̄1e1 −(RT
kRk
qTk Rk
)yk
∥∥∥∥∥ Qk+1Bk =
(Rk
0
), RT
k qk = β̄k+1ek
= mintk
∥∥∥∥∥β̄1e1 −(RT
k
ϕkeTk
)tk
∥∥∥∥∥ tk = Rkyk, qk = (β̄k+1/(Rk)k,k)ek = ϕkek
= mintk
∥∥∥∥( zkζ̄k+1
)−(R̄k
0
)tk
∥∥∥∥ Q̄k+1
(RT
k β̄1e1
ϕkeTk 0
)=
(R̄k zk
0 ζ̃k+1
)
Things to notexk = Vkyk, tk = Rkyk, zk = R̄ktk, two cheap QRs
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 9/31
Least squares subproblem
minyk
∥∥∥∥∥β̄1e1 −(BT
k Bk
β̄k+1eTk
)yk
∥∥∥∥∥= min
yk
∥∥∥∥∥β̄1e1 −(RT
kRk
qTk Rk
)yk
∥∥∥∥∥ Qk+1Bk =
(Rk
0
), RT
k qk = β̄k+1ek
= mintk
∥∥∥∥∥β̄1e1 −(RT
k
ϕkeTk
)tk
∥∥∥∥∥ tk = Rkyk, qk = (β̄k+1/(Rk)k,k)ek = ϕkek
= mintk
∥∥∥∥( zkζ̄k+1
)−(R̄k
0
)tk
∥∥∥∥ Q̄k+1
(RT
k β̄1e1
ϕkeTk 0
)=
(R̄k zk
0 ζ̃k+1
)
Things to notexk = Vkyk, tk = Rkyk, zk = R̄ktk, two cheap QRs
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 9/31
Least squares subproblem (2)
Remember xk = Vkyk, tk = Rkyk, zk = R̄ktk
Key steps to compute xkxk = Vkyk
= Wktk RTkW
Tk = V T
k
= W̄kzk R̄Tk W̄
Tk = W T
k
= xk−1 + ζkw̄k
where zk =(ζ1 ζ2 · · · ζk
)T
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 10/31
Least squares subproblem (2)
Remember xk = Vkyk, tk = Rkyk, zk = R̄ktk
Key steps to compute xkxk = Vkyk
= Wktk RTkW
Tk = V T
k
= W̄kzk R̄Tk W̄
Tk = W T
k
= xk−1 + ζkw̄k
where zk =(ζ1 ζ2 · · · ζk
)T
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 10/31
Least squares subproblem (2)
Remember xk = Vkyk, tk = Rkyk, zk = R̄ktk
Key steps to compute xkxk = Vkyk
= Wktk RTkW
Tk = V T
k
= W̄kzk R̄Tk W̄
Tk = W T
k
= xk−1 + ζkw̄k
where zk =(ζ1 ζ2 · · · ζk
)T
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 10/31
Least squares subproblem (2)
Remember xk = Vkyk, tk = Rkyk, zk = R̄ktk
Key steps to compute xkxk = Vkyk
= Wktk RTkW
Tk = V T
k
= W̄kzk R̄Tk W̄
Tk = W T
k
= xk−1 + ζkw̄k
where zk =(ζ1 ζ2 · · · ζk
)T
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 10/31
Least squares subproblem (2)
Remember xk = Vkyk, tk = Rkyk, zk = R̄ktk
Key steps to compute xkxk = Vkyk
= Wktk RTkW
Tk = V T
k
= W̄kzk R̄Tk W̄
Tk = W T
k
= xk−1 + ζkw̄k
where zk =(ζ1 ζ2 · · · ζk
)T
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 10/31
Flow chart of LSMR
Computational cost
Av,ATv, 3m+ 3n
O(1)
3n
O(1)
A,�b
Golub-Kahanbidiagonalization
QR decompositions
Estimate backward error
large
Solution x
small
Update x
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 11/31
Flow chart of LSMRComputational cost
Av,ATv, 3m+ 3n
O(1)
3n
O(1)
A,�b
Golub-Kahanbidiagonalization
QR decompositions
Estimate backward error
large
Solution x
small
Update x
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 11/31
Computational and storage requirement
Storage Workm n m n
MINRES on ATAx = ATb Av1 x, v1, v2, w1, w2 8LSQR Av, u x, v, w 3 5LSMR Av, u x, v, h, h̄ 3 6
where hk, h̄k are scalar multiples of wk, w̄k
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 12/31
Numerical experiments
Test Data• University of Florida Sparse Matrix Collection• LPnetlib: Linear Programming Problems• A = (Problem.A)’ b = Problem.c (127 problems)
Solve min ‖Ax− b‖2with LSQR and LSMR
• Backward error tests: nnz(A) ≤ 63220
• Reorthogonalization: nnz(A) ≤ 15977
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 13/31
Numerical experiments
Test Data• University of Florida Sparse Matrix Collection• LPnetlib: Linear Programming Problems• A = (Problem.A)’ b = Problem.c (127 problems)
Solve min ‖Ax− b‖2with LSQR and LSMR
• Backward error tests: nnz(A) ≤ 63220
• Reorthogonalization: nnz(A) ≤ 15977
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 13/31
Backward errors - optimalOptimal backward error
µ(x) ≡ minE‖E‖ s.t. (A+ E)T (A+ E)x = (A+ E)T b
Higham 1995:
µ(x) = σmin(C), C ≡[A ‖r‖
‖x‖
(I − rrT
‖r‖2
)].
Cheaper estimate (Grcar, Saunders, & Su 2007):
K =
(A‖r‖‖x‖
)v =
(r0
)y = argmin
y‖Ky − v‖
µ̃(x) =‖Ky‖‖x‖
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 14/31
Backward errors - optimalOptimal backward error
µ(x) ≡ minE‖E‖ s.t. (A+ E)T (A+ E)x = (A+ E)T b
Higham 1995:
µ(x) = σmin(C), C ≡[A ‖r‖
‖x‖
(I − rrT
‖r‖2
)].
Cheaper estimate (Grcar, Saunders, & Su 2007):
K =
(A‖r‖‖x‖
)v =
(r0
)y = argmin
y‖Ky − v‖
µ̃(x) =‖Ky‖‖x‖
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 14/31
Backward error estimates - E1, E2
Again, (A+ Ei)T (A+ Ei)x = (A+ Ei)
T b
Two estimates given by Stewart (1975 and 1977):
E1 =exT
‖x‖2, ‖E1‖ =
‖e‖‖x‖
, e = r̂ − r
E2 = −rrTA
‖r‖2, ‖E2‖ =
‖ATr‖‖r‖
where r̂ is the residual for the exact solution
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 15/31
‖rk‖ for LSQR and LSMR
0 50 100 150 200 250 3002.75
2.8
2.85
2.9
2.95
3
3.05
3.1
3.15
iteration count
||r||
Name:lp greenbeb, Dim:5598x2392, nnz:31070, id=100
lsqrlsmr
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 16/31
log10(‖E2‖) for LSQR and LSMR
0 200 400 600 800 1000 1200 1400 1600 1800−7
−6
−5
−4
−3
−2
−1
0
iteration count
log(
E2)
Name:lp pilot ja, Dim:2267x940, nnz:14977, id=88
E2 LSQRE2 LSMR
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 17/31
Backward errors for LSQR
0 200 400 600 800 1000 1200 1400 1600 1800−8
−7
−6
−5
−4
−3
−2
−1
0
1
iteration count
log(
Bac
kwar
d E
rror
)
Name:lp cre a, Dim:7248x3516, nnz:18168, id=93
E1 LSQRE2 LSQROptimal LSQR
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 18/31
Backward errors for LSMR
0 200 400 600 800 1000 1200 1400 1600 1800−8
−7
−6
−5
−4
−3
−2
−1
0
1
iteration count
log(
Bac
kwar
d E
rror
)
Name:lp cre a, Dim:7248x3516, nnz:18168, id=93
E1 LSMRE2 LSMROptimal LSMR
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 19/31
Optimal backward errors for LSQR and LSMR
0 100 200 300 400 500 600−7
−6
−5
−4
−3
−2
−1
0
iteration count
log(
||ATr|
|/||r
||)
Name:lp pilot, Dim:4860x1441, nnz:44375, id=107
Optimal LSQROptimal LSMR
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 20/31
Space-time trade-offs
LSMR is well-suited for limited memory computations.
What if we have• more memory• Av expensive
Can we speed things up?
Some ideas:• Reorthogonalization• Restarting• Local reorthogonalization
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 21/31
Space-time trade-offs
LSMR is well-suited for limited memory computations.
What if we have• more memory• Av expensive
Can we speed things up?
Some ideas:• Reorthogonalization• Restarting• Local reorthogonalization
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 21/31
Reorthogonalization
Golub-Kahan processInfinite precision Finite precisionUk, Vk orthonormal Lose orthogonalityAt most min(m,n) iterations Could take 10n or more
Apply modified Gram-Schmidt to uk+1 and/or vk+1:
u← u− (uTj u)uj j = k, k−1, k−2, . . .
(similarly for v)
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 22/31
Reorthogonalization
Golub-Kahan processInfinite precision Finite precisionUk, Vk orthonormal Lose orthogonalityAt most min(m,n) iterations Could take 10n or more
Apply modified Gram-Schmidt to uk+1 and/or vk+1:
u← u− (uTj u)uj j = k, k−1, k−2, . . .
(similarly for v)
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 22/31
Effects of reorthogonalization on various problems
0 10 20 30 40 50 60 70 80 90−8
−7
−6
−5
−4
−3
−2
−1
0
1
iteration count
log(
E2)
Name:lp ship12l, Dim:5533x1151, nnz:16276, id=91
NoOrthoOrthoUOrthoVOrthoUV
0 100 200 300 400 500 600 700 800 900−7
−6
−5
−4
−3
−2
−1
0
1
iteration count
log(
E2)
Name:lp scfxm2, Dim:1200x660, nnz:5469, id=62
NoOrthoOrthoUOrthoVOrthoUV
0 20 40 60 80 100 120 140 160 180−6
−5
−4
−3
−2
−1
0
1
iteration count
log(
E2)
Name:lp agg2, Dim:758x516, nnz:4740, id=58
NoOrthoOrthoUOrthoVOrthoUV
0 2000 4000 6000 8000 10000 12000−7
−6
−5
−4
−3
−2
−1
0
iteration count
log(
E2)
Name:lpi gran, Dim:2525x2658, nnz:20111, id=94
NoOrthoOrthoUOrthoVOrthoUV
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 23/31
Orthogonality of Uk
100
200
300
400
500
600
5
6
Original
100
200
300
400
500
600
5
6
7
8
910
11
1213
141516
MGS on U
100
200
300
400
500
600
5
6
7
8
910
11
1213
141516
MGS on V
100
200
300
400
500
600
5
6
7
8
910
11
1213
141516
MGS on U ,V
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 24/31
Orthogonality of Vk
100
200
300
400
500
600
5
6
Original
100
200
300
400
500
600
5
6
7
8
910
11
1213
141516
MGS on U
100
200
300
400
500
600
5
6
7
8
910
11
1213
141516
MGS on V
100
200
300
400
500
600
5
6
7
8
910
11
1213
141516
MGS on U ,V
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 25/31
What we learnt so far
• Reorthogonalizing Vk (only) is sufficient• Reorthogonalizing Uk (only) is nearly as good• xk converges the same for all options
What can be improved• May still use too much memory• Need more flexibility for space-time trade-off
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 26/31
Reorthogonalization with Restarting
Restarting LSMR
rk = b−Axk min ‖A∆x− rk‖
Restarting leads to stagnation
0 500 1000 1500 2000 2500 3000 3500 4000 4500−7
−6
−5
−4
−3
−2
−1
0
iteration count
Bac
kwar
d E
rror
Name:lp maros, Dim:1966x846, nnz:10137, id=81
NoOrthoRestart5Restart10Restart50NoRestart
0 2000 4000 6000 8000 10000 12000 14000 16000−7
−6
−5
−4
−3
−2
−1
0
1
iteration count
Bac
kwar
d E
rror
Name:lp cre c, Dim:6411x3068, nnz:15977, id=90
NoOrthoRestart5Restart10Restart50NoRestart
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 27/31
Reorthogonalization with Restarting
Restarting LSMR
rk = b−Axk min ‖A∆x− rk‖
Restarting leads to stagnation
0 500 1000 1500 2000 2500 3000 3500 4000 4500−7
−6
−5
−4
−3
−2
−1
0
iteration count
Bac
kwar
d E
rror
Name:lp maros, Dim:1966x846, nnz:10137, id=81
NoOrthoRestart5Restart10Restart50NoRestart
0 2000 4000 6000 8000 10000 12000 14000 16000−7
−6
−5
−4
−3
−2
−1
0
1
iteration count
Bac
kwar
d E
rror
Name:lp cre c, Dim:6411x3068, nnz:15977, id=90
NoOrthoRestart5Restart10Restart50NoRestart
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 27/31
Local reorthogonalization• Reorthogonalize wrto only the last l vectors• Partial speed-up• Less memory• Depends on efficiency of Av and ATu
Speed up with local reorthogonalization
0 50 100 150 200 250 300 350 400 450−7
−6.5
−6
−5.5
−5
−4.5
−4
−3.5
−3
−2.5
−2
iteration count
Bac
kwar
d E
rror
Name:lp fit1p, Dim:1677x627, nnz:9868, id=80
NoOrthoLocal5Local10Local50FullOrtho
0 200 400 600 800 1000 1200 1400−6
−5
−4
−3
−2
−1
0
1
2
iteration count
Bac
kwar
d E
rror
Name:lp bnl2, Dim:4486x2324, nnz:14996, id=89
NoOrthoLocal5Local10Local50FullOrtho
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 28/31
Local reorthogonalization• Reorthogonalize wrto only the last l vectors• Partial speed-up• Less memory• Depends on efficiency of Av and ATu
Speed up with local reorthogonalization
0 50 100 150 200 250 300 350 400 450−7
−6.5
−6
−5.5
−5
−4.5
−4
−3.5
−3
−2.5
−2
iteration count
Bac
kwar
d E
rror
Name:lp fit1p, Dim:1677x627, nnz:9868, id=80
NoOrthoLocal5Local10Local50FullOrtho
0 200 400 600 800 1000 1200 1400−6
−5
−4
−3
−2
−1
0
1
2
iteration count
Bac
kwar
d E
rror
Name:lp bnl2, Dim:4486x2324, nnz:14996, id=89
NoOrthoLocal5Local10Local50FullOrtho
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 28/31
Conclusions
Advantages of LSMR• All the good properties of LSQR• Monotone convergence of ‖ATrk‖• ‖rk‖ still monotonic• Cheap near-optimal backward error estimate⇒ reliable stopping rule
• Backward error almost surely monotonic
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 29/31
Acknowledgement
• Michael Saunders• Chris Paige• Stanford Graduate Fellowship
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 30/31
Questions
LSMR in MATLAB andslides for this talk are
downloadable athttp://zi.ma/lsmr
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 31/31
top related