orthogonality and stability in large matrix iterative ... · sparse days at cerfacs, toulouse,...

BREA Orthogonality MGS Lanczos process Applications

Orthogonality and stabilityin large matrix iterative algorithms

Chris Paige∗ & Wolfgang Wulling†

∗Computer Science, McGill University,with financial support from NSERC (Canada).

†W2 Actuarial & Math Services Ltd., Blackpool, Lancashire, England, FY4 5PN

Sparse Days at CERFACS, Toulouse, France, June 25–26, 2012

O(ε)‖A‖ ≈ 0, O(ε) ' 0, SUT=Strictly Upper Triangular Slide 1/35, June 26, 2012


Some Notation (reals only).

Singular Values: σi (A). Norms: ‖v‖ ≡ ‖v‖24=√vT v ,

‖A‖24= σmax(A), ‖A‖2

F4= trace(ATA).

sut(A) ≡ the STRICTLY UPPER TRIANGULAR part of A.

slt(A) ≡ the STRICTLY LOWER TRIANGULAR part of A.

ε the computer floating point precision.

O(ε)‖A‖ ≈ 0, O(ε) ' 0, SUT≡Strictly Upper Triangular



1 Backward Rounding Error Analysis (BREA)

2 Vector Orthogonalization Algorithms

3 MGS & Augmented Backward Stability

4 Orthogonality and the Lanczos process

5 Some possible Applications of the Analysis



Backward Rounding Error Analysis

(BREA)



Reliable Numerical Algorithms

E.g. Householder QR is a Backward Stable Algorithm (BSA),

so there exists an orthogonal Q such that for B = Q

[R0

]:

B + E = Q

[Rc

0

], ‖E‖ ≤ O(ε)‖B‖, ‖Qc−Q‖ ≤ O(ε),

Qc & Rc the computed Q & R,

Abbreviation:

B ≈ Q

[Rc

0

], Qc ' Q, QT Q = I .



Backward Rounding Error Analysis (BREA).

Because of Wilkinson’s BREA theory, we KNOW we haveBackward Stable orthogonal transformation algorithms,

e.g. the QR algorithm—while the matrix is not too big.

For many large sparse matrix problems we turn to

Vector orthogonalization algorithms, e.g. CG.

These are not BSAs, Wilkinson’s BREA theory doesn’t apply.

But many of these algorithms still ‘work’.

We seek a Matrix based Rounding Error Theoryto describe their subtle numerical behaviours∗.



Vector Orthogonalization Algorithms



‘Vector Orthogonalization’ Algorithms

orthogonalize each new vectoragainst previous supposedly orthogonal vectors.

For example:

Modified Gram–Schmidt (MGS),Arnoldi’s method for the unsymmetric eigenproblem,Saad & Schultz: MGS-GMRES for unsymmetric Ax = b,

Lanczos: Tridiagonalization of square A,Lanczos, & Hestenes & Stiefel: Conjugate gradients (CG) Ax =b,Golub & Kahan ‘Vector Orthogn.’ Bidiagonalization of general B.

ALL can lose orthogonality using finite precision computations!

Assume each “orthogonal” vector has unit length: ‖vj‖2 = 1.



An ideal measure of loss of orthogonality

Given V ∈ Rn×k with V TV = UT +I +U, U 4= sut(V TV ).

Instead of using ‖U‖2,

if S 4= (I + U)−1U , k × k , strictly upper triangular, then

‖S‖2 is a great measure of loss of orthogonality in V :

0 ≤ ‖S‖2 ≤ 1.

S = 0 ⇔ V TV = I ,

‖S‖2 = 1 ⇔ V rank deficient.

In fact

# of usvs of S = column rank deficiency of V .



Gram-Schmidt Orthogonalization&

Augmented Backward Stability



MGS

In theory MGS produces V and upper triangular R so that:

B = VR, V TV = I .

For many large sparse matrix problems:

MGS + Krylov sequences {b,Ab,A2b, . . .} leads to, e.g.:

Arnoldi’s method for unsymmetric Ax = xλ, which leads to:

Saad & Schultz’s MGS-GMRES for unsymmetric Ax = b.

Important to understand MGS.



Measuring loss of orthogonality in MGS

Remember:

k steps of MGS for Bk = VkRk gave computed Vk

where V Tk Vk = UT

k + I + Uk , Uk4= sut(V T

k Vk).

If Sk4= (I + Uk)−1Uk then

‖Sk‖2 is our measure of loss of orthogonality in Vk .



‖Sk‖2: Orthogy. Loss in Bk = VkRk in GS & MGSGS and MGS on Bm ∈ R15×15, singular values 1, 10−1, . . . , 10−14



Obtaining orthonormal Qk from n × k V ≡Vk

Given V ∈ Rn×k with V TV = UT +I +U, U 4= sut(V TV ),

if S 4= (I + U)−1U & Qk

4=

[S

V (I−S)

],

then QTk Qk = I .

(n+k)×k Qk is an “orthonormal augmentation” of n×k V ≡Vk .



Aside: Proof of Orthogonality.

Given V ∈ Rn×k with V TV = UT +I +U, U ≡ sut(V TV ),

if S ≡ (I + U)−1U & Qk ≡[

SV (I−S)

], then QT

k Qk = I .

Proof:

−(I + U)S = −U = I − (I + U), so (I + U)(I − S) = I .

QTk Qk = STS + (I−S)TV TV (I−S)

= STS + (I−S)T (−I + I + UT + I + U)(I−S)

= STS − (I − S)T (I − S) + I − S + I − ST

= STS − I + S + ST − STS + I − S + I − ST = I .

Clearly ‖S‖2 ≤ 1. ‖S‖2 = 1⇔ rank(V ) < k , (via the CSD).



Augmented BS of MGS for the QR of Bk ∈ Rn×k .

For Bk =VR, MGS gives computed V & R where from a REA:[0Bk

]+

[EF

]=QkR ≡

[S

V (I−S)

]R,

∥∥∥∥ EF

∥∥∥∥2

≤ O(ε)‖B‖2, QTk Qk = I ,

so that R is Backward Stable for the QR factorization of

[0Bk

],

an Augmented problem! Thus “ABS” of MGS.−−−−−−−−−−−−−−−−−−−−−−−−−−−−−How loss of orthogonality occurs:

σi (R) ≈ σi (Bk), E = SR, S = ER−1,

‖S‖2 ≤ ‖E‖2‖R−1‖2 ≤ O(ε)‖Bk‖2‖R−1‖2 ≈ κ2(Bk)O(ε).

Bjorck & Paige (1992); Paige, Rozloznık & Strakos (2006);following Charles Sheffield’s ∼1967 augmented matrix suggestion.



‖Sk‖2 ≤ κ2(Bk)ε for MGS example

On the graph in the middle of the next slide,

K2(Vk)∗eps

should beκ2(Bk)ε



‖Sk‖2 ≤ κ2(Bk)ε for MGS exampleon Bn ∈ R15×15, singular values 1, 10−1, . . . , 10−14



Un-augment the result:[0Bk

]≈ QkR ≡

[S

V (I−S)

]R, QT

k Qk = I .

By using the SVD of S , if rank(Bk) = k :

can show there exists n × k Qk such that:

Bk ≈ QkR, QTk Qk = I ,

so MGS is BS for computing R from Bk .

Bjorck & Paige (1992).

————————————-

Qk is the closest orthogonal matrix to V (I − S).

Nick Higham (2002).



MGS-GMRES for large sparse Ax = b



Convergence of Saad & Schultz MGS-GMRES

MGS-GMRES gives a BSS xk to Ax = b in k ≤ n steps if

σmin(A)� n2ε‖A‖F . Paige, Rozloznık & Strakos (2006).



Thoughts

We now know how MGS works numerically:

It provides a BSS R,It leads to BSS for LLS problems,It can be used as the basis for other algorithms.

We now know how full MGS-GMRES works numerically:

Full MGS-GMRES gives a BSS in ≤ n steps,Full MGS-GMRES does not need re-orthogonalization,Storage and operations per step increase with k.Use Truncated MGS-GMRES? Restarts? . . .

We have tools for examining Arnoldi’s algorithm.

This was the easy part—implicit orthogonalization algorithmsare much more difficult to analyze.[

SV (I−S)

]is very useful, S is amazing.

We can apply these ideas to many algorithms.



The Major Steps in the Analysis

Step 1: Vk4= [v1, . . . , vk ], the normalized, computed vectors.

Step 2: Uk4= sut(V T

k Vk), Sk4= (I + Uk)−1Uk ,

Sk → measures of loss of orthogonality & independence.

Step 3: Exists orthonormal Qk4=

[Sk

Vk(I − Sk)

], QT

k Qk = Ik .

Step 4: Find the augmented system. E.g. MGS of n × k Bk :

[0Bk

]≈ QkRc ≡

[Sk

Vk(I−Sk)

]Rc , QT

k Qk = Ik , REA.

Step 5: Un-augment this augmented system.

E.g. MGS: ∃ n × k Qk such that Bk ≈ QkRc , QTk Qk = Ik .∗



Orthogonality and the Lanczos process



Implicit orthogn.: the Lanczos process (1950)

Given A = AT ∈ Rn×n and v1 ∈ Rn, vT1 v1 = 1,compute an orthonormal sequence v1, v2, . . . , vk+1

via a 3 term recurrence:

vj+1βj+1 := Avj − vjαj − vj−1βj ,

implicit orthogonalization, makes life MUCH tougher!

Then with

Vk ≡ [v1, v2, . . . , vk ], Tk ≡

α1 β2

β2 α2 ·· · βk

βk αk

,AVk = VkTk +vk+1βk+1e

Tk = Vk+1Tk+1,k , V T

k+1Vk+1 = I .

Useful for large sparse Ax = xλ, Ax = b.



‘Recursive augmented stability’

of the Lanczos process



RAS of the computational Lanczos process

Ideally, given A = AT ∈ Rn×n and v1∈ Rn, vT1 v1= 1,

Vk4= [v1, v2, . . . , vk ], Tk

T = Tk4= tridiag(βi , αi , βi+1),

(1): AVk = VkTk + vk+1βk+1eTk , (2): Vk+1

TVk+1 = I .

Computed Tk & Vk from the Lanczos algorithm satisfy:

(1):

([Tk 00 A

]+ Hk

)Qk = QkTk + qk+1βk+1e

Tk

Hk = HTk , ‖Hk‖2 ≈ 0,

(2): [Qk | qk+1]T [Qk | qk+1] = Ik+1. RAS:

k steps of an exact Lanczos process for an AUGMENTED matrix.



The same old Qk .

Normalized computed Vk : V Tk Vk = UT

k + I + Uk ,

Uk4= sut(V T

k Vk), Sk4= (I +Uk)−1Uk ,

Qk4=

[Sk

Vk(I−Sk)

], QT

k Qk = Ik .



Development of Tk and Qk

The augmented problem seems weird:([Tk 00 A

]+Hk

)Qk =QkTk + qk+1βk+1e

Tk , Qk

4=

[Sk

Vk(I−Sk)

].

Let Qk be Qk less its zero kth row, then Qk+1 = [Qk , qk+1],

and the augmented problem becomes[Tk,k−1 0

0 A

]Qk≈ Qk+1Tk+1,k , QT

k Qk = Ik , QTk+1Qk+1 = Ik+1.

Showing howTk,k−1 → Tk+1,k & Qk → Qk+1.

“Recursive”, from implicit nature. Strange, but not quite so weird.



Augmented Biorthonormality

Suppose we want biorthonormal V & W , i.e. W TV = I .

With vector orthogonalization this might be far from true.

But for computed V & W with wTi vi = 1, i = 1, 2, . . .

U 4= sut(W TV ), S 4

= (I + U)−1U,

L 4= slt(W TV ), R 4= (I + LT )−1LT ,

Q 4=

[S

V (I − S)

], P 4

=

[R

W (I − R)

].

Then PTQ = I , biorthonormal augmented matrices!

Paige (2009), from a question by Ron Morgan, Zeuthen 2008.



The Unsymmetric Lanczos Process

Using this concept of augmented biorthonormalityshows that running k steps of the Lanczos processon unsymmetric A ∈ Rn×n in the presence of rounding errorsis equivalent to running k steps of an exact Lanczos processon a perturbation E of the augmented matrix diag(Tk ,A).

(Paige, Panayotov, & Zemke, in preparation, extending theanalysis by Zhaojun Bai, 1994).

The flaw in the unsymmetric Lanczos process:

‖E‖2 can be large if the process approaches breakdown.

Use “look ahead”? (Bill Gragg, Beresford Parlett, et al.).



Some possibleApplications of the Analysis



Lanczos Process: Symmetric A.

The eigenproblem: via the Lanczos process.

Solution of equations Ax = b, A = AT :

Positive definite A: CG, Lanczos process, etc.

Indefinite A: e.g. MINRES, SYMMLQ,(Mike Saunders & me), etc.

All above & even singular A: MINRES-QLP,(Sou-Cheng Choi, Mike Saunders & me).



Lanczos Process on Unsymmetric n × n A.

Use this to analyze:

The eigenproblem: unsym. Lanczos process & variants.

Solution of unsym. equations: Roland Freund’s QMR, etc.



General n ×m B.

Golub & Kahan’s “vector” bidiagonalization for the SVD.

(Theory: Lanczos process on symmetric

[0 BBT 0

],

[u1

0

].)

And with u1 = b/‖b‖2 for Bx ≈ b can solve:

Solution of equations & Least squaresCGLS (Hestenes & Stiefel), LSQR (Mike Saunders & me),LSMR (David Fong & Mike Saunders),Total Least Squares (Ake Bjorck),Scaled TLS (Zdenek Strakos & me),Core problems (Zdenek Strakos & me).



Background to this “Augmented” approach:

C. Sheffield, comment to Gene Golub, circa 1967.

C. C. Paige, Ph.D. thesis, London University 1971.

A. Greenbaum, Ph.D. thesis, UC, Berkeley 1981, LAIA 1989.

A. Bjorck and C.C. Paige, SIMAX 1992, BIT 1994.

J.L. Barlow, N. Bosner and Z. Drmac, LAIA 2005.

C. C. Paige, M. Rozloznık, and Z. Strakos, SIMAX 2006.——————————————–

C. C. Paige, A useful form of unitary matrix obtainedfrom any sequence of unit 2-norm n-vectors.SIMAX, 31 (2009), pp. 565–583.

C. C. Paige, An Augmented Stability Result for the LanczosHermitian Matrix Tridiagonalization Process.SIMAX, 31 (2010), pp. 2347–2359.


orthogonality and stability in large matrix iterative ... · sparse days at cerfacs, toulouse,...

Documents