Computational AstrophysicsLinear Equations

Peter Hauschildtyeti@hs.uni-hamburg.de

Hamburger SternwarteGojenbergsweg 112

21029 Hamburg

26. Juli 2018

1 / 43

Topics

I Why are they interesting?I general remarksI direct methodsI iterative methods

2 / 43

Why are they interesting?

I linear equations are everywhere:I Newton’s method for multiple non-linear equationsI discretization of differential equationsI fittingI finite-elementsI . . .

I can be large (> 106)

I full, sparse, special form

3 / 43

Literature

I examples and graphs from:I Press et al, ’Numerical Recipes in [Fortran, C]’, 2nd

Edition, Cambridge University Press, 1992I iterative methods:

I Engeln-Mullges, G. & Reutter, F.: ’Formelsammlungzur Numerischen Mathematik mit Standard-Fortran77 Programmen’, 6. Auflage, Wissenschaftsverlag,1988

4 / 43

general remarks

I matrices can be singular:I mathematicalI numerical due to roundoff error

I roundoff accumulation limits solving linear equationsby ’simple’ methods:

I IEEE single precision: N ' 20− 50I IEEE double precision: N ' several 100I sparse matrices: N > 10000− 106

I nearly mathematically singular: N ' 10

5 / 43

general remarks

I in general: do not write your own solverI use public domain packages instead:

I LaPack: most modern, uses BLAS→ fastI Linpack: now outdated, still very good, slowerI ScLaPack: MIMD parallel implementation of LaPackI SuperLU: modern, in particular sparse matrices,

parallelized

6 / 43

general remarks

I if available, commercial packages are OK:I IBM’s ESSLI NAG’s numerical librariesI Intel’s MKL

I in reality: use Lapack with vendor-supplied BLAS!

7 / 43

general remarks

I 2+1 classes of algorithms:I direct methods

I obtain solution in a finite number of steps/flopsI iterative methods

I use un-determined number of iterations to findsolution

I combinationI use low-accuracy direct method to approximate

solutionI then use iterative method to remove accumulated

roundoff error

8 / 43

direct methods

I Gaussian elimination type methodsI solve

A~x = ~b

I or

9 / 43

direct methods

I Gauss elimination method:I eliminate a21 to aM1 by subtracting ai1/a11 times the

first row for each row i > 1I repeat until matrix has the form

I Gauss-Jordan: reduce each column after rowreduction

10 / 43

Gaussian elimination

I Backsubstitution:I solve for

x4 = b′4/a′44

I solve forx3 =

1a′33

[b′3 − a′34x4

]I and so on→ recursive

11 / 43

pivoting

I it can happen that an aii is zero (or very close to zero)I → use row permutations to find an aii 6= 0I → pivotingI Gauss elimination is numerically unstable if no

pivoting is used (even if all aii 6= 0!)I partial pivoting→ exchange rows of unreduced

matrixI full pivoting→ exchange rows and columns of

unreduced matrix

12 / 43

pivoting

I partial pivoting is much easier (does not changesolution vector x)

I also is sufficient in most cases!I how to select the best pivot element?I mathematically not solved. . .

13 / 43

pivoting

I useful approach:1. equilibrate the matrix so that∑

k

|aik | = 1

for all i2. choose max(|aij |) of unreduced matrix as pivot

I usually done implicitly using partial pivotingI this applies to all direct methods!

14 / 43

Gaussian elimination

I uses 1/3N3 (reduction) and 1/2N2M(backsubstitution for M right hand sides (RHS)

I Gauss-Jordan→ approximate N3 flopsI disadvantage of both schemes: RHS’s must be known

before the process startsI not nice, e.g., if you like to keep the same Jacobian

for a number for Newton-Raphson iterations!

15 / 43

LU decomposition

I A as product of the form

A = L · U

I with

I then we can write

Ax = (L · U)x = L(Ux) = ~b

16 / 43

LU decomposition

I so that we can use a forward/backward substitution:

Ly = bUx = y

I once we have LU decomposition of A we can easilysolve for as many RHS as needed

I needs N2 flops for each RHS

17 / 43

Crout’s algorithm

I how to perform the LU decomposition?I each component has the equation

αi1β1j + · · · = aij

I number of terms depends on i and j :

18 / 43

Crout’s algorithm

I N2 equations for N2 + N unknownsI → over-determinedI → chose N variables arbitrarilyI → set αii = 1

19 / 43

Crout’s algorithm

I Crout’s algorithm:I solve the equations in the best order:

1. for each j :1.1 for all i ≤ j compute

βij = aij −i−1∑k=1

αikβkj

1.2 for all i > j compute

αij =1bjj

(aij −

j−1∑k=1

αikβkj

)

20 / 43

Crout’s algorithm

21 / 43

Crout’s algorithm

I this scheme produces

I in top to bottom columns from left to rightI partial pivoting: leave choice of βii open until all αij ’s

for a column are ’precomputed’I then chose largest one as bii and finish.I number of operations are identical to Gaussian

elimination

22 / 43

inverse & determinant

I LU decomposeI inverse:

I solve for ’unity matrix’I → inverse of matrix A!

I determinant:d =

∏βjj

23 / 43

tridiagonal matrices

I general form:

I arise from problems that generate diagonallydominant tridiagonal matrices with

|bj | > |aj |+ |cj |

I → no need for pivoting

24 / 43

tridiagonal matrices

I → solution is very compact:

25 / 43

band matrices

I Example form of band matrix:

I store in compact form→ rotate matrix 45◦ so that

26 / 43

band matrices

I problem: LU decomposition is not so compactI pivoting also requires extra storageI → band matrix LU decomposer’s need extra scratch

space

27 / 43

iterative methods

I roundoff errors accumulate in direct methods quicklyI especially for nearly (numerically) singular matricesI iterative improvement of approximate solutionI e.g., generated by LU decomposition with low

accuracy.

28 / 43

iterative methods

I perfect solution fulfills

Ax = b

I but we got a solution with an error

→ x + δx

withA(x + δx) = b + δb

I so that we haveAδx = δb

29 / 43

iterative methods

I we write δb so that

Aδx = A(x + δx)− b

I for this to work, A(x + δx)− b must be calculated withhigher precision

I → combat cancellations!I roundoff errors do not affect (stable) iterative methodsI → can be used to restore best possible solution

30 / 43

Jacobi method

I re-writeAx = a

I by solving the i th equation for xi

xi = −∑k 6=i

aik

aiixk +

ai

aii

or simplerx = Bx + c ≡ ϕ(x)

I this leads to the iteration sequence

x(n+1) = Bx(n) + c

31 / 43

Jacobi method

I convergence criteria?I if ϕ(x) fulfills a Lipschitz condition

||ϕ(x)− ϕ(x′)|| ≤ L||x− x′||

with 0 ≤ L < 1I then the iteration sequence started with arbitrary x

will converge to the unique solution xI the (a posteriori) truncation error is given by

||x(n) − x|| ≤ L1− L

||x(n) − x(n−1)||

32 / 43

Jacobi method

I simpler determinations of convergence:I row sum criterion:

maxi

∑k 6=i

∣∣∣∣aik

aii

∣∣∣∣ ≤ L∞ < 1

I column sum criterion:

maxk

∑i 6=k

∣∣∣∣aik

aii

∣∣∣∣ ≤ L1 < 1

33 / 43

Jacobi method

I simpler determinations of convergence:I Schmidt-Mises criterion:√√√√∑

i

∑k 6=i

∣∣∣∣aik

aii

∣∣∣∣2 ≤ L2 < 1

34 / 43

Jacobi method

I if convergence criteria check out, repeat iterationsuntil

1. absolute error

maxi|xi

(n) − xi(n−1)| < δ

2. relative error

maxi|xi

(n) − xi(n−1)| < max

i|xi

(n)|ε

I with given δ, ε > 0

35 / 43

Gauss-Seidel method

I iteration produces components x (n+1)i consecutively

I → use already known components x (n+1)i of step

(n + 1) to compute x (n+1)i+1

I formally write

x(n+1) = Bux(n) + Blx(n+1) + c

I where Bu and Bl are upper and lower triangularmatrices with zero on their diagonals

36 / 43

Gauss-Seidel method

I convergence criteria: row and column sums aboveI in addition, if A is symmetric and positive definite,

Gauss-Seidel will converge!

37 / 43

relaxation: Jacobi method

I write the iteration step

x(n+1) = Bx(n) + c

in terms of theI correction z(n):

x(n+1) = x(n) + z(n)

with

z(n) = B?x(n) + cB? = 1− B

38 / 43

relaxation: Jacobi method

I try to improve convergence rate by using

ωz(n)

instead of z(n)

I obvious problem: how to choose ω?

39 / 43

relaxation: Jacobi method

I one can show that

ω =2

2− λ1 − λn

gives better convergence ratesI where λ1 and λn are the largest and smallest

eigenvalues of BI sometimes these can be estimated from the physics

of the problemI ω < 1→ under-relaxationI ω > 1→ over-relaxation

40 / 43

relaxation: Gauss-Seidel

I similar approach for Gauss-Seidel:

x(n+1) = x(n) + ω(− (1− Bu)x(n) + Blx(n+1) + c

)I → method of successive relaxationI computation of optimal ω very hardI only methods with ω < 2 will, in general, converge

41 / 43

relaxation: Gauss-Seidel

I for many types of matrices generated by discretizingelliptical boundary value problems→

I successive over-relaxation (SOR) method with

ω =2

1 +√

1− λ2max

where λmax is the largest eigenvalue of B

42 / 43

Choices

I iterative methods are best used forI very large matricesI sparse matrices (tend to be large)I when you know it’ll convergeI to correct roundoff errors of a direct solution

I relaxation methods are great if you happen to knowthe largest eigenvectors

I or guess an ω

43 / 43