Pushing the boundaries:Fast integral methods for solving boundary element equations.

Bachelor Project Mathematics

july 2015

Student: T.W. Verstraaten

First supervisor: Dr. B. Carpentieri

Second supervisor: Prof. Dr. H. Waalkens.

faculty of mathematicsand natural sciences

Abstract

In this thesis various preconditioning techniques are discussed for solving boundary elementequations arising in electromagnetic scattering. Preconditioning is a crucial component foriterative Kylov methods in this context to accelerate their convergence, and there is a host ofpossible choices of preconditioners. These all attempt to increase the sparsity of the systemand give rise to a favourable eigenvalue distribution. First we discuss the finite difference andfinite element approaches to modelling differential equations and then we present the boundaryelement method (BEM) approach for the integral equation formulation. Afterwards, we discussKrylov subspace methods and we describe the fast multipole method, which is used to reducethe computational cost of the matrix vector product to O(n log n). Finally we discuss someefficient preconditioning techniques for this problem class and illustrate their performance onthe solution of realistic scattering problems.

Contents

1 Introduction 31.1 Some notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Some definitions and concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Numerical modelling of differential equations 42.1 Finite difference method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.2 The finite element method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.2.1 Weak formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.2.2 Galerkin method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3 Integral Equations 113.1 Integral equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.2 An integral equation for the acoustic problem . . . . . . . . . . . . . . . . . . . . . . 113.3 Discretizating our model problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

4 Projection methods 154.1 General projection method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174.2 Results about general projection methods . . . . . . . . . . . . . . . . . . . . . . . . 18

5 Krylov subspace methods 195.1 Krylov subspaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195.2 Arnoldi method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215.3 Full orthogonalization method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215.4 Restarted FOM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235.5 IOM and DIOM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235.6 GMRES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

5.6.1 Restarted GMRES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275.7 Notes on implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275.8 Comparing FOM and GMRES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305.9 Preconditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315.10 Deflation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

5.10.1 DEFLGMRES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

1

6 Fast multipole method 336.1 Basic concepts of the FMM method . . . . . . . . . . . . . . . . . . . . . . . . . . . 336.2 MLFMA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

7 Electric field integral equation 38

8 Incomplete factorization methods 418.1 Level of fill techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418.2 Problems with ILU preconditioners . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

9 Multilevel preconditioning 449.1 Independent set ordering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

10 Approximate inverse preconditioning 4610.1 Frobenius-norm minimization preconditioner . . . . . . . . . . . . . . . . . . . . . . 46

11 MLFMA-based preconditioners 50

12 Conclusion 52

1 Introduction

The first chapters of this thesis will be dedicated to techniques for numerically solving differentialequations, and discussing their strengths and weaknesses. The discretization of such systems resultsin linear systems of the form.

Ax = b,

We will outline some of the most common methods for solving such linear systems.

1.1 Some notation

We will adopt the following notation throughout the thesis:

• ∆ denotes the Laplacian operator ∆ =n∑k=1

∂∂xi

.

• 〈·, ·〉 denotes the standard inner product on Rn.

• For a matrix A, a?i and ai? denote the ith column and row respectively. Also, range(A) willdenote the column space of A, that is range(A) = spania?i.

• Vectors will be given in bold as x.

• ei will always denote the ith canonical basis vector.

• If D ⊂ Rn is some region in an n-dimensional space, we denote by ∂D its boundary.

2

• We suppress zeros in sparse matrix representation, for example we will write

B =

−4 11 −4 1

. . .

1 −4 11 −4

.

1.2 Some definitions and concepts

Some important definitions that are used in this manuscript are given here.

• The condition number κ2(A) of a matrix A is defined as

κ2(A) = ‖A‖2‖A−1‖.

It is a measure of how numerically stable a matrix is with respect to inversion. A numericallystable matrix A, will not amplify the small uncertainties of the data in the solution x.

• If x is an approximate solution to Ax = b, we define the residual vector r as

r = b−Ax.

2 Numerical modelling of differential equations

Differential equations are ubiquitous in computational science. Their numerical solution typicallyresults in linear systems of the form

Ax = b.

We will present some of the most common methods for solving such systems. In our discussion wewill refer to the model problem:

∆u+ k2u = 0. (2.0.1)

defined on some domain D.

2.1 Finite difference method

The finite difference method is the easiest method to numerically approximate solutions of ordinaryand partial differential equations. We present here only an introductory look at the method whichis extensively covered in many books. A more in depth analysis of the finite difference method canbe found for instance in [11].

We will assume that D is a square domain [a, b]× [c, d] and that u satisfies a homogeneous Dirichletboundary condition, i.e., that u|∂D = 0. In general these assumptions do not necessarily needto hold when using finite difference methods, but we are here concerned with an overview of thegeneral idea of the method.

3

The finite difference method relies on the Taylor expansion of a function

f(x+ h) = f(x) + hf ′(x) +h2

2!f ′′(x) + . . .+

hn

n!f (n)(x) + . . . (2.1.1)

where h is taken to be small. Upon truncating this series, we can obtain the approximation

f ′(x) ≈ f(x+ h)− f(x)

h. (2.1.2)

The truncation error for this approximation is given by

h2

2!f ′′(x) +

h3

3!f (3)(x) + . . . . (2.1.3)

Similarly to equation 2.1.1 we can set up an approximation for f(x − h). Then adding this toequation 2.1.1 gives

f(x+ h) + f(x− h) = 2

f(x) +

h2

2!f ′′(x) +

h4

4!f (4)(x) + . . .

.

Then we may approximate the second derivative of f at x by

f ′′(x) ≈ f(x+ h)− 2f(x) + f(x− h)

h2

with an error of

− 2

h4

4!f (4)(x) +

h6

6!f (6)(x) + . . .

.

The next step is to discretize our domain D into a rectangular grid. We subdivide the domain intonm rectangles with height h1 = d−c

n and width h2 = b−am . We denote

uij = u(a+ ih1, c+ jh2).

Now, using the previous results we can write

∆uij =∂2

∂x2uij +

∂2

∂y2uij (2.1.4)

≈ ui+1,j + ui−1,j

h22

+ui,j+1 + ui,j−1

h21

−(

2

h21

+2

h22

)ui,j . (2.1.5)

When we use this approximation into our model problem, we obtain the formulation

ui+1,j + ui−1,j

h22

+ui,j+1 + ui,j−1

h21

−(

2

h21

+2

h22

+ k2

)ui,j = 0,

which holds for all discrete points (xi, yi) on the domain D. On the boundary we have prescribed uto be equal to zero, so some of these terms will not appear in the final matrix. Now for simplicitywe take h := h1 = h2. If we now define the vector

u = [u1,1, . . . , u1,n, u2,1, . . . , u2,n, . . . . . . , un,n]T

4

u12 u13

u22 u23

u32 u33

u02 u03 u04

u14

u24

u34

x

y

a b

c

d

u11

u21

u31

u00 u01

u10

u20

u30

u40 u41 u42 u43 u44

Figure 2.1: The rectangle [a, b]×[c, d] dividedinto 16 subrectangles.

Then we can write the system in the form

(A+ k2I)u = 0, (2.1.6)

where A is a block matrix of the form

A =1

h2

B II B I

. . .

I B II B

with

B =

−4 11 −4 1

. . .

1 −4 11 −4

.

Equation 2.1.6 is an eigenvalue-eigenvector prob-lem. Solving it means finding the eigenvector cor-responding to the eigenvalue −k2. Note that our original problem is also an eigenvalue problem ifwe write is as

(∆ + k2)u = 0. (2.1.7)

Again the solution is the eigenvector of the operator ∆ with eigenvalue −k2. In general, if u doesnot satisfy a homogeneous Dirichlet boundary condition and the right hand side of equation 2.1.7 isnot zero, the right hand side of equation 2.1.6 would be non zero. Then we would have a standardlinear system

(A+ k2I)u = b, (2.1.8)

that may be solved by any method of choice. Note that the matrix involved is very sparse andthus iterative methods like the class of Krylov methods based on matrix-vector operations are moreefficient than direct methods. These methods will be discussed later on.

The finite difference method is very simple and easy to implement. One of the major issues withthis method, however, is that the domain D may have any arbitrary shape, and some highly ir-regular shapes can be approximated very poorly using rectangles. Another problem is the tuningof the value for h. Of course h needs to be sufficiently small so that the truncations made in theTaylor series are reasonably accurate. But in making h very small, we can see that entries in thematrix A become exceedingly large compared to the entries in the vector b in equation 2.1.8. Thispresents problems due to poor scaling where small errors may be enlarged. Therefore we cannotexpect that the accuracy of our result will always increase when taking a smaller value for h.

As a general example of the numerical problems that we may encounter, consider the followingexample.

5

0 0.2 0.4 0.6 0.8 10

5

10

15

x

u

n = 10

0 0.2 0.4 0.6 0.8 10

1

2

3

4

5

x

err

or

0 0.2 0.4 0.6 0.8 10

20

40

60

80

100

x

u

n = 100

0 0.2 0.4 0.6 0.8 10

10

20

30

40

x

err

or

0 0.2 0.4 0.6 0.8 10

200

400

600

800

1000

x

u

n = 1000

0 0.2 0.4 0.6 0.8 10

100

200

300

400

x

err

or

Figure 2.2: A simple finite difference approximation of u′ = 1(1−x)2

shows some of the problems

that may occur when using the finite difference method. Notice that as n increases, the error nearx = 1 grows while the error away from x = 1 seems to decrease.

6

Example 2.1. Consider the one-dimensional problem

u′ =1

(1− x)2, x ∈ [0, 1),

with u(0) = 1. The exact solution to this problem is u(x) = 11−x . A finite difference approximation

would divide the interval into n sub intervals of length h = 1n+1 and to use equation 2.1.2. Now we

need to be careful here and look at how accurate this approximation is. Notice that

dn

dxn

(1

1− x

)=

n!

(1− x)n

So, at xn = nh = nn+1 , we have

dn

dxn

(1

1− xn

)=n!

hn.

Equation 2.1.3 now gives a bound on the truncation error of

1 + 1 + 1 + 1 + . . .

This series diverges and based on this error estimate we may expect exceedingly large errors forthe approximation in equation 2.1.2. So in this case the implicit assumption that the truncationerrors are not too large is not valid. In figure 2.2 we show some output from a simple Matlab finitedifference code.

Next we move on to the finite element method which is a slightly more complex method.

2.2 The finite element method

Originally conceived as a technique used in civil engineering, the finite element method is now awidely used tool for numerically solving ordinary and partial differential equations. It has the ad-vantage that it can handle complex geometries and boundary conditions in a more natural way thanthe finite difference method. We will see that the finite element method is a projection method,these methods will be discussed in more detail later on. This overview of the finite element methodwill follow [19]

We consider again equation 2.0.1, this time with the more general boundary condition

u|∂D1 = gD, (2.2.1)

∂u

∂n

∣∣∣∣∂D2

= gN . (2.2.2)

So on one part of the boundary we have a Dirichlet condition and on the other part a Neumanncondition with ∂D = ∂D1 ∪ ∂D2 and ∂D1 ∩ ∂D2 = ∅.

There is some nomenclature that is important here. A solution to the DE with boundary condi-tions 2.2.1 and 2.2.2 will be known as a classical solution. Also, Dirichlet and Neumann boundaryconditions are known as essential and natural boundary conditions, respectively.

7

We will first derive what is known as the weak formulation of this problem. To do, this we will needto use Green’s first identity. This states that if u is a twice continuously differentiable functiondefined on D and v is a once continuously differentiable function defined on D, then∫

D

v∆u dA =

∮∂D

v∂u

∂ndS −

∫D

∇u · ∇vdA. (2.2.3)

2.2.1 Weak formulation

To derive the weak formulation we first multiply equation 2.0.1 by some test function v, and weintegrate over the entire domain D to obtain∫

D

(v∆u+ vk2u

)dA = 0.

Now we can employ equation 2.2.3 to obtain∮∂D

v∂u

∂ndS +

∫D

vk2u dA =

∫D

∇v · ∇udA. (2.2.4)

A solution to the original ODE must satisfy this equation for all v1. However, it is possible thatthere exist solutions u to equation 2.2.4 that are not sufficiently smooth to satisfy the originalproblem. For instance solutions u of shock wave problems can not be classical solutions as they arenot differentiable.

We proceed by demanding that this equation is satisfied for all v in a certain function space. Wewill need the notion of a Sobolev space H1. This space is defined as

H1 =

u : D → R :

∫D

(u2 +∇u · ∇u

)dA <∞

.

This choice of search space for v ensures that the integrals in 2.2.4 converge provided that u isbounded and the region D is bounded. We will now define two spaces.

Solution space: H1S =

w ∈ H1 : w|∂D1 = gD

, (2.2.5)

Test space: H10 =

w ∈ H1 : w|∂D1 = 0

. (2.2.6)

The solution space is the space in which we will look for u and the test space is the space fromwhich we will take our test function v. Note that the solution space is technically not a vectorspace as it is not closed under addition unless gD ≡ 0. It is clear now that any classical solution uis also a solution to the weak formulation given by

Find uh ∈ H1S such that

∫∂D2

vhgN dS + k2

∫D

vhuh dA =

∫D

∇vh · ∇uhdA, for all vh ∈ H10. (2.2.7)

1Provided that the integrals converge for this choice of v.

8

2.2.2 Galerkin method

We will now construct a finite subspace of the solution space KS ⊂ H1S . The idea is to take a finite

dimensional subspace K0 ⊂ H10 with a basis φ1, . . . , φn. In order to ensure that the Dirichlet

boundary condition on ∂D1 is satisfied, we add an additional set of vectors φn+1, . . . , φn+m, sothat for fixed coefficients un+1, . . . , un+m we have(

n+m∑k=n+1

ukφk

)∣∣∣∣∣∂D1

= gD,

and (n+m∑k=n+1

ukφk

)∣∣∣∣∣∂D2

= 0.

This ensures that we can reconstruct the Dirichlet boundary condition on ∂D1. Define now KS ⊂H1S as

KS =

n∑k=1

ukφk +

n+m∑k=n+1

ukφk

∣∣∣∣∣ (u1, . . . , un) ∈ Rn.

Notice that all functions in KS satisfy the essential boundary condition on ∂D1. Now we look forthe finite element approximation uh in the function space KS . The functions φ1, . . . , φn are oftencalled the trial functions or shape functions. In a similar way we construct a finite basis that formsa subspace that will be our test space; denote this L0. In the Galerkin method, test functions arechosen to be in the space K0, i.e., K0 = L0. In the Petrov-Galerkin method, the space of testfunctions is not equal to spanφ1, . . . , φn.

Now we demand that equation 2.2.7 holds for each of the basis functions φi. The correspondingweak formulation writes then:

Find (u1, . . . , un) such that

n∑k=1

uk

∫D

∇φj · ∇φkdA− k2

∫D

φjφk dA

=

∫∂D2

φjgN dS −n+m∑k=n+1

uk

∫D

∇φi · ∇φkdA.

for all j = 1, . . . , n. Now we are ready to write this in matrix form as

Au = f ,

with

Aij =

∫D

(∇φi · ∇φj − k2φiφj

)dA, (2.2.8)

eTi f =

∫∂D2

φigNdS +

n+m∑k=n+1

uk

∫D

∇φi · ∇φkdA. (2.2.9)

9

Historically, from development and use in engineering, the matrix A is often called the stiffnessmatrix. Clearly, A is symmetric. In the case that our original problem is the Laplace equation, i.e.,when k = 0, we have that A is also positive definite if ∂D = ∂D1. To see this, note that

vTAv =

n∑i=1

n∑j=1

viaijvj (2.2.10)

=

n∑i=1

n∑j=1

vi

∫D

φi · φjdAvj (2.2.11)

=

∫D

vh · vhdA (2.2.12)

≥ 0. (2.2.13)

If ∂D = ∂D1, then vh|∂D = 0. Then by the continuity of vh we have that v ≡ 0. Now as φ1, . . . , φnform a basis for K0 this implies that v = 0. In the case that ∂D 6= ∂D1 we have only that A ispositive semi definite.

This concludes the brief introductions into the two traditional methods of modelling differentialequations; these are probably the first two approaches that students encounter. The next sectionwill discuss integral equations and show how to derive an integral equation for the acoustic scatteringproblem.

3 Integral Equations

In many cases, differential equations have an equivalent integral equation representation on theboundary of the domain that is sometimes preferable to use. Here we will first introduce integralequations and then derive an integral equation for the acoustic problem.

3.1 Integral equations

One of the most basic kinds of integral equations is known as a Fredholm equation. This is expressedmathematically as

b∫a

K(x, t)y(t)dt = f(x). (3.1.1)

Here f and K are known functions and y(t) is the function that we want to determine. Integralequations are characterized by the following definitions:

• If the unknown function appears both inside and outside the integral, we call it an integralequation of the second kind. Otherwise, we call it an integral equation of the first kind.

• If the limits of integration are constant, as in equation 3.1.1, it is termed a Fredholm equation.If one of the limits of integration is variable, the equation is known as a Volterra equation.

• If the right hand side is equal to zero, the equation is homogeneous, analogously to the caseof differential equations. Otherwise, the integral equation is called inhomogeneous.

10

Now, an integral equation with two variable bounds can transformed by a suitable change of co-ordinates into a Volterra equation.

3.2 An integral equation for the acoustic problem

As an example of the derivation of an integral equation, we present here the case of the acousticscattering problem

∆u+ k2u = 0,

defined in some domain D. We will specify the boundary condition for this problem later. It willprove very useful to find an analytic solution for the 3D acoustic field in some bounded region Ddue to a point source at x0 ∈ D [6]. Green’s second theorem will be instrumental in this derivation.Green’s second theorem states that if f and g are twice continuously differentiable on D, then∫

∂D

(f∂g

∂n− g ∂f

∂n

)dS =

∫D

(g∆f − f∆g) dV.

Here ∂g∂n = n(x)·∇g(x) is the directional derivative in the direction normal to ∂D pointing outward.

Here we are assuming that the boundary of D, denoted ∂D, is piecewise smooth.

To derive an integral equation for u we make the following assumptions on u:

1. u must satisfy the Helmholtz equation ∆u+ k2u = 0 in D \ x0.

2. Near x0, the solution is ’close’ to the free field solution. If we define

us(x) := u(x)−G(x,x0),

then this condition implies that us and ∇us are continuous in some neighbourhood of x0.

3. u is twice continuously differentiable in D \ x0.

At this stage, fix a point x in D and define

φ(y) := u(y), (3.2.1)

ψ(y) := G(y,x). (3.2.2)

Here u is the solution we are looking for, and

G(y,x) :=eik|y−x|

|y − x| .

To avoid singularities at x and x0, we define

V = y ∈ D | y /∈ x,x0,

11

and notice that both φ and ψ satisfy the conditions for Green’s second theorem on V , as well asthe Helmholtz equation. This implies that

ψ∆φ− φ∆ψ = ψ(−k2φ)− φ(−k2ψ) = 0,

so applying Green’s second theorem yields∫∂V

(φ∂ψ

∂n− ψ∂φ

∂n

)dS = 0.

Notice that this holds for x in the interior of D as well as on the boundary. We will first considerthe case where x is contained in the interior of D. Taking ε sufficiently small and dividing ∂V intothree surfaces, namely, ∂D, the sphere γε around x and the sphere Γε around x0, we may write∫

∂D+

∫γε

+

∫Γε

= 0. (3.2.3)

We analyse what happens to these integrals as we let ε→ 0. We focus our attention first to∫γε

. Asimple computation shows that

∂ψ

∂n= n(y) · ∇ψ(y) = −e

ikε(ikε− 1)

4πε2.

Observing that on γε the function ψ takes the simple form

ψ(y)|γε = G(y,x)|γε = −eikε

4πε

it follows that ∫γε

= − 1

4π

eikε(ikε− 1)

ε2

∫γε

φdS − eikε

ε

∫γε

∂φ

∂ndS

.

We are interested in the case when ε→ 0. The function φ is defined at x, and thus we can use theresult ∫

γε

φdS ∼ φ(x)

∫γε

dS = 4πε2φ(x). (3.2.4)

Similarly, as ∂φ∂n is bounded, we see that∫

γε

∂φ

∂ndS = O(ε2).

Using both these results we conclude that∫γε

∼ −eikε(ikε− 1)φ(x) +O(ε), as ε→ 0,

and this implies that limε→0

∫γε

= φ(x).

12

Similarly we can show that limε→0

∫Γε

= −ψ(x0).

Taking the limit, as ε tends to zero, of equation 3.2.3, and using the previous two results andrecalling that φ(y) = u(y) and ψ(y) = G(y,x), we obtain

u(x) = G(x0,x)−∫∂D

(u(y)

∂G(y,x)

∂n(y)−G(y,x)

∂u(y)

∂n(y)

)dS. (3.2.5)

This is the integral equation that u must satisfy inside the domain D \ x0. Now, a similar resultcan be derived if we instead look at x on the boundary of D. First, we define the quantity

Ω(x) = limε→0

1

ε2

∫Sε

dS

where Sε is the part of the sphere around x of radius ε contained in D. If the boundary is smoothat x then Ω(x) = 2π, for instance. We then replace equation 3.2.4 by∫

γε

φdS ∼ φ(x)

∫γε

dS ∼ Ω(x)ε2φ(x).

Thus we only need to change one of the terms in equation 3.2.5 to finally obtain

Ω(x)

4πu(x) = G(x0,x)−

∫∂D

(u(y)

∂G(y,x)

∂n(y)−G(y,x)

∂u(y)

∂n(y)

)dS,

for x on the boundary of D. Now we have rewritten the Helmholtz equation to an integral equation,specifically an inhomogeneous Fredholm equation of the second kind. In deriving these equations,we have not used the boundary conditions and thus this integral representation holds irrespective ofthe boundary condition. In light of our particular problem, we can now use the boundary equation

∂u

∂n= −ikβu

in the derived integral equation. This leads to the integral equations

u(x) = G(x0,x)−∫∂D

∂G(y,x)

∂n(y)+ ikβ(y)G(y,x)

u(y)dS, (3.2.6)

Ω(x)

4πu(x) = G(x0,x)−

∫∂D

∂G(y,x)

∂n(y)+ ikβ(y)G(y,x)

u(y)dS, (3.2.7)

for x in D and ∂D, respectively. These equations are called boundary integral equations as theintegration occurs over the boundary of D. Now, one advantage of this boundary integral formula-tion of our original acoustic problem becomes apparent if we solve equation 3.2.7 on the boundary∂D and then can reconstruct u on the whole domain D using equation 3.2.6. We have effectivelyreduced a problem over D to a problem over its boundary only. However, we do have to evaluatethe Green’s functions and their integrals, which may be costly. Also, in our integral formulation,

13

the value of u at one point (of the boundary) depends on the value of u at all other points of theboundary due to the integration. Upon discretization, this will lead to linear systems with fullydense matrices in contrast to numerical schemes where approximate the Laplace operator directlyusing the values of u in some small neighbourhood.

In order to make an integral equation problem amenable to numerical approximation, we need todiscretize the problem in some way. A suitable method to use is the finite element method. Sub-sequently we have to solve the problem of numerically approximating the integrals in the equation.In the next section we will discuss one simple method to discretize the problem.

3.3 Discretizating our model problem

We apply the ideas of FEM to our integral equation. The boundary is divided into N controlsurfaces, usually triangles. We denote these surfaces as γj for j = 1, . . . , N . Now, in the simplestcase we can restrict u to be constant on each γi, i.e., u|γj = uj . Then equation 3.2.7 takes the form

Ω(x)

4πu(x) ≈ G(x0,x)−

N∑j=1

uj

∫γj

∂G(y,x)

∂n(y)+ ikβ(y)G(y,x)

dS. (3.3.1)

Using the collocation method, we pick a point xi in each γi and impose that equation 3.3.1 holdsexactly at these points. Usually xi is chosen as the centroid of γi. We can quite safely assume thatxi lies on a smooth part of ∂D, so Ω(xi) = 2π. We thus require ui to satisfy

1

2ui = G(x0,xi)−

N∑j=1

uj

∫γj

∂G(y,xi)

∂n(y)+ ikβ(y)G(y,xi)

dS

for i = 1, . . . , N . This is a set of N simultaneous equations in N variables and thus it writes as alinear system by setting

bi = G(x0,xi), (3.3.2)

cij =

∫γj

∂G(y,xi)

∂n(y)+ ikβ(y)G(y,xi)

dS. (3.3.3)

The linear system has the form1

2u = b− Cu.

Equivalently, we can cast this in the more familiar form

Au = b (3.3.4)

if we set A = 12I + C. An important question is how to evaluate the entries in b and C. Note

for instance that in the expression for cii we run into trouble due to the singularity of G(y,xi) aty = xi ∈ γi. Often some form of Gaussian quadrature is used. Now from the expression for cijwe can see that the problem mentioned earlier appears, namely that C (and thus A) is a densematrix. We can already see some possibilities for improvement, though. The elements cij describe

14

the influence on ui due to to a point source at γj . We can simplify the problem by consideringa collection of point sources far away from γi as a single point source. We will explore this ideafurther when we discuss the fast multipole method in section 6.

When the geometry of the problem is irregular, the size of the control surfaces may vary in size,leading to an ill-conditioned matrix A. This will make the solution of the linear system 3.3.4 poten-tially more difficult. In Figure 3.1 we plot the eigenvalue distribution for a small scattering problemfrom a typical geometry matrix. There is little clustering of the eigenvalues and many eigenvalueshave large negative real part. Such a distribution is unfavourable for rapid convergence of Krylovmethods for solving the linear system [5].

Figure 3.1: Eigenvalue distribution of the geometry matrix of a satellite. Source [5]

Now that we have introduced the notion of integral equations and BEM formulation of a differentialequation we to briefly discuss projection methods before diving into Krylov subspace methods whichare the focus of this thesis.

4 Projection methods

Projection methods are used when we want to extract some approximate solution to a problemfrom some space K. This space is called the search space, of dimension m. In this case we need mconstraints to ensure that we will get a unique solution. Often this is realized by demanding thatthe residual b−Ax is orthogonal to some other m dimensional space L, also called the constraintspace or left subspace [15]. These two conditions on the approximate solution are known as the

15

Petrov-Galerkin conditions.

L

K

b−Ax∗

K = L

b−Ax∗

Oblique Projection

Orthogonal projection

Figure 4.1: Oblique and orthogonal projec-tion.

There are roughly two classes of projectionmethods: orthogonal and oblique. In or-thogonal projection methods we set L = Kwhile in oblique projection methods this isnot the case and L will depend on the al-gorithm of choice. If the search space isthe same as the constraint space the con-dition is often called the Galerkin condi-tion.

Now consider again the problem2

Ax = b. (4.0.5)

If we want to impose the Petrov-Galerkin condi-tions this means we want to find x∗ ∈ K such thatb− Ax∗ ⊥ L. We sometimes wish to incorporatean initial guess x0, in which case we look for x∗

in the affine space x0 + K. The conditions thenare given by

find x∗ ∈ x0 +K such that b−Ax∗ ⊥ L.

The requirement that b − Ax∗ be orthogonal toL suggests that we can also adopt a minimiza-tion point of view on the projection technique.Namely, we can use the alternate but equivalentconditions

find x∗ ∈ K such that d(b−Ax∗,L) is minimized.

This is again equivalent to

find x∗ ∈ K such that 〈b−Ax∗, w〉 = 0 for all w ∈ L.

It is often the case that the spaces K and L change at each iteration, often they will be enlarged. Wewill see such methods later on when we describe Krylov subspace methods which are an importantexample of projection methods.

4.1 General projection method

Now suppose we want to find an approximation to equation 4.0.5. Consider V = [v1, . . . ,vm] andW = [w1, . . . ,wm] such that

spanv1, . . . ,vm = K, (4.1.1)

spanw1, . . . ,wm = L. (4.1.2)

2We present here the argument for a linear system, but these methods also exists for problems of the form Au = fwhere A is an operator and f and u are functions.

16

Then we can write the approximate solution as

x∗ = x0 + V y,

for some y ∈ Rm. Then multiplying by W TA yields

W TAx∗ = W TAx0 +W TAV y.

Noting that W TA(x∗ − x0) = W T (r − r0) = W T r0 gives

W T r0 = W TAV y.

In the case that W TAV is invertible we can solve this to obtain

y = (W TAV )−1W T r0.

A general algorithm for a projection method with Petrov-Galerkin conditions is given in algorithm4.1.

Algorithm 4.1: General projection algorithm

1 repeat2 Select two subspaces K and L3 Find bases V and W for K and L4 r = b−Ax5 y = (W TAV )−1W T r6 x = x0 + V y

7 until convergence;

4.2 Results about general projection methods

The following proposition guarantees that W TAV is invertible under some conditions.

Proposition 4.1. Let A, K and L satisfy one of the following two properties:

1. A is SPD and K = L, or

2. A is invertible and L = AK.

Then B = W TAV is invertible, regardless of the bases V and W of K and L respectively.

Proof. Consider the first case. Let V and W be bases of L and K respectively. Then, as L = K,we can write W = V G for some nonsingular matrix G. Now

B = W TAV = GTV TAV.

Because A is SPD we have that V TAV is SPD, and hence invertible. Thus B is invertible.

17

Consider now the second case. By the condition that L = AK, it now follows that we can writeW = AV G for some nonsingular matrix G. Thus

B = GT (AV )TAV.

Because A is nonsingular, the matrix AV has full rank3 and so (AV )TAV is nonsingular. Thus Bis nonsingular.

For the same two cases, two slightly more complicated results can be derived about the approxim-ation x∗ they deliver. consider the following two theorems.

Proposition 4.2. Let A be SPD and K = L. Then x∗ is result of the orthogonal projection methodonto K with initial vector x0 if and only if x∗ satisfies

E(x∗) = minx∈x0+K

E(x),

where E(x) = 〈x− x, x− x〉12A with x satisfying Ax = b.

Proof. By the properties of projections, x∗ minimizes E(x) if and only if

V TA(x− x∗) = 0

for any V whose columns span K. Then we write x∗ = x0 + V y and rearrange to get

V TA(x− x0) = V TAV y.

Here A(x− x0) = r0, so thaty = (V TAV )−1V T r0, (4.2.1)

and sox∗ = x0 + (V TAV )−1V T r0.

This is precisely the x∗ obtained from the orthogonal projection method as seen in section 4.1.Note that the inverse in equation 4.2.1 exists by theorem 4.1.

Now we derive a similar result for the case that A is arbitrary and L = AK.

Proposition 4.3. Let A be invertible and L = AK. Then x∗ is result of the oblique projectionmethod onto K orthogonal to L with initial vector x0 if and only if x∗ satisfies

R(x∗) = minx∈x0+K

R(x),

where R(x) = ‖b−Ax‖2.

Proof. See [15]

The two cases just discussed turn out to be very important and give rise to two widely usedalgorithms that will be discussed later on.

3Suppose not, in that case there would exist a non zero vector y such that AV y = 0. But that means that Az = 0with z = Ay, with z non-zero as V has full rank. This contradicts the nonsingularity of A.

18

5 Krylov subspace methods

In the last section we looked at projection techniques in a general setting. We will now look moreconcretely at a certain type of projection method where K is what is known as a Krylov subspace.

5.1 Krylov subspaces

Aleksey Nikolaevich Krylov, 1863-1945.

Given A ∈ Rn×n and v ∈ Rn×1, we define the corres-ponding m dimensional Krylov space as

Km(A,v) := spanv, Av, A2v, . . . , Am−1v.

Krylov subspace methods rely on finding an approx-imation to the exact solution A−1b in the spaceKm(A,b). We can find an m for which we can be surethat A−1b ∈ Km(A,b). Recall the following defini-tion.

The minimal polynomial corresponding to an n×n mat-rix A is the lowest degree monic polynomial q(·) such that

q(A) = 0.

The Cayley-Hamilton theorem tells us that

p(A) =n∑k=0

αkAk = 0,

where p(·) is the characteristic polynomial of A. Thus the degree of the minimum polynomial of Adoes not exceed n. Suppose that the minimal polynomial of A has degree m. Then we can write

q(A) =

m∑k=0

βkAk = 0.

Pre multiplying this equality with A−1 and rearranging shows us that we can solve for A−1 in termsof A,A2, . . . Am−1. So it follows that

A−1b ∈ Km(A,b).

In practice it is almost never useful to first find the minimal polynomial and then use this inform-ation to restrict the maximum size of the Krylov space as this would require knowledge of thespectrum of A, which is not available in general. Instead, the Krylov space is enlarged at eachiterations and a residual is calculated to be used in a stopping criterion.

As we increase the dimension of the Krylov space, the new vectors Akb become increasingly linearlydependent. To see this, decompose4

b =n∑i=1

βizi,

4Such a decomposition is not always possible if A is singular, i.e., when the eigenvectors of A do not span Rn.

19

where zi are the normalized eigenvectors of A. Then

limp→∞

Apb

‖Apb‖ → zm

where zm is the eigenvector of A corresponding to the largest eigenvalue. This property is actuallyexploited in the so-called power method to determine the largest eigenvalue and the correspondingeigenvector of a matrix A [2]. Therefore we may expect that, due to numerical errors and roundoff errors, after a certain amount of iterations the computed vectors will no longer be linearlyindependent. To avoid this problem, we generate an orthonormal basis v1,v2, . . . ,vm such that

spanv1,v2, . . . ,vm = Km(A,v).

A well known method for generating an orthonormal basis of a Krylov space is called the Arnoldimethod.

5.2 Arnoldi method

The discussion of the Arnoldi method follows [15]. In essence, the Arnoldi method generates an or-thonormal basis forKm(A,v) by using the Gram-Schmidt orthogonalization process on v, Av, . . . , Am−1v).

Algorithm 5.1: Arnoldi algorithm

1 v1 = v/‖v‖22 for j = 1,. . . ,m do3 for i = 1,. . . ,j do4 hij = vTi Avj5 end

6 wj = Avj −j∑i=1

hijvi

7 hj+1,j = ‖wj‖28 if hj+1,j = 0 then9 Stop

10 else11 vj+1 =

wjhj+1,j

12 end

13 end

Proposition 5.1. Let Vm be the n × m matrix with columns vectors v1, . . . ,vm and Hm be the(m+1)×m Hessenberg matrix whose non-zero entries are hij, as defined in algorithm 5.1. Further,let Hm = Hm(1 : m, 1 : m). Then the following relations hold:

AVm = VmHm + hm+1,mvm+1eTm (5.2.1)

= Vm+1Hm, (5.2.2)

V TmAVm = Hm. (5.2.3)

where em = (0, 0, . . . , 0, 1)T is the mth canonical basis vector for Rm.

20

Now that we have constructed a basis for Km(A,v) we can use this to find approximate solutionsto linear systems. We will describe here some iterative methods exploiting the idea of the Arnoldibasis.

5.3 Full orthogonalization method

The full orthogonalization method (FOM) seeks an approximate solution xm = x0 +Vmy such that

b−Axm ⊥ Km.

It is thus an orthogonal projection method. It is common to define β = ‖r0‖2 and let v1 = r0/β.Then proposition 5.1 states that

V TmAVm = Hm

and we have thatV Tmr0 = V T

m (βv1) = βe1,

by the orthonormality of Vm. Then we can state the following proposition:

Proposition 5.2. The solution to the computed by the full orthogonalization method is

xm = x0 + Vmy,

withy = H−1

m (v1).

Proof. It is clear that xm ∈ x0 +Km. It remains to show that b−Axm ⊥ Km, i.e., that

Vm(b−Axm) = 0.

Now,

V Tm (b−Axm) = V T

m (b−A(x0 + VmH−1m (βe1))) (5.3.1)

= V Tm (b−Ax0)− V T

mAVmH−1(βe1) (5.3.2)

= V Tmr0 −HmH

−1(βe1) (5.3.3)

= βe1 − βe1 (5.3.4)

= 0. (5.3.5)

Algorithm 5.2: Pseudo code FOM algorithm

1 Set r0 = b−Ax0, β = ‖r0‖2 and v1 = r0/b.2 Compute Hm using the Arnoldi algorithm3 Compute ym = H−1

m (βe1)4 Set xm = x0 + Vmym.

In practice is it not desirable to set a fixed m beforehand as in algorithm 5.2 but to choose itdynamically according to some criterion. The following useful result can be used.

21

Proposition 5.3. The residual b − Axm for the approximate solution xm found by algorithm 5.2satisfies

b−Axm = −hm+1eTmymvm+1.

Therefore‖b−Axm‖2 = hm+1|eTmym|. (5.3.6)

This theorem allows us to evaluate the residual at each step so that we may design a stoppingcriterion without having to explicitly determine xm at each iteration.

Proof. We have

b−Axm = b−A(x0 + Vmym) (5.3.7)

= r0 −AVmym (5.3.8)

= r0 − VmHmym − hm+1eTmymvm+1 (5.3.9)

= r0 − VmHmH−1m (βe1)− hm+1e

Tmymvm+1 (5.3.10)

= −hm+1eTmymvm+1 (5.3.11)

The second identity of proposition 5.3 follows by simply taking the norm.

Then we can use this result to calculate the residual at each step by simply performing one vector-vector product.

5.4 Restarted FOM

Due to the orthogonalization process required in the FOM method, after m iterations we needto store m vectors. Also, at the new iteration, orthogonalization with respect to the previouslycomputed m basis vectors will be performed. Thus the amount of flops required for each iterationincreases with m and may become prohibitive. There are several methods to alleviate this problem.The first is the restarted FOM method.

In the restarted FOM method, k iterations of the FOM algorithm are performed and if at thatpoint no sufficient convergence is achieved, the FOM method is restarted after setting x0 = xm.Using this method, no more than k vectors need to be stored in total and the orthogonalizationruns over at most k vectors. So the time and storage complexity cannot grow unbounded as thenumber of iterations increases.

Algorithm 5.3: Pseudo code of restarted FOM algorithm

1 while not converged do2 Set r0 = b−Ax0, β = ‖r0‖2 and v1 = r0/b3 Compute Hm using the Arnoldi algorithm4 Compute ym = H−1

m (βe1)5 Set xm = x0 + Vmym6 Set x0 = xm7 end

22

5.5 IOM and DIOM

Another method of controlling the time and storage complexity of the FOM method is by usingthe incomplete orthogonalization method (IOM). In this method we select an integer k > 0 and wereplace the orthogonalization process that uses all vector determined so far by an orthogonalizationprocess that uses only the k last determined vectors. To do this we simply modify the Arnoldiprocess in algorithm 5.1 to the algorithm 5.4.

Algorithm 5.4: Incomplete Arnoldi algorithm

1 v1 = v/‖v‖22 for j = 1,. . . ,m do3 for i = max(1, j − k + 1), . . . , j do4 hij = vTi Avj5 end

6 wj = Avj −j∑

i=max(1,j−k+1)

hijvi

7 hj+1,j = ‖wj‖28 if hj+1,j = 0 then9 Stop

10 else11 vj+1 =

wjhj+1,j

12 end

13 end

Now we need to store and orthogonalize at most k vectors so we may bound the time and storagecomplexity. One problem arises, however, in the construction of xm. To do this we require thematrix Vm which the algorithm did not fully store if the number of iterations exceeded k. Onesolution is to recompute the matrix Vm after the algorithm finishes but this essentially doubles thecomputation time required and is thus no good. An alternative approach consists of updating thesolution xm at each iteration so that we include the contribution due to each new vector vi.

The Hessenberg matrix Hm that we obtain in the IOM method is has upper bandwidth k − 1 andlower bandwidth 1 due to its Hessenberg structure. Then if LU = Hm is the LU factorization ofHm it is a well known result that U will have upper bandwidth of k − 1 as well. The direct IOMmethod (DIOM) exploits this property. The approximate solution xm is, as before, given by

xm = x0 + VmH−1m (βe1) = x0 + VmU

−1m L−1

m (βe1).

Now definePm = VmU

−1m , zm = L−1

m (βe1).

Thenxm = x0 + Pmzm. (5.5.1)

23

From PmUm = Vm we have that

pm =1

umm

[vm −

m−1∑i=m−k+1

uimpi

],

where pm is the last column of Pm. Also, if we let ζm = −lm,m−1ζm−1 we may write

zm =

(zm−1

ζm

),

with ζ1 = β. Now from equation 5.5.1 we get that

xm = x0 +(Pm−1,pm

)(zm−1

ζm

)= x0 + Pm−1zm−1 + ζmpm = xm−1 + ζpm.

So we can update xm using xm−1, the previous vectors pm−k+1, . . . ,pm−1 and ζm−1.

Algorithm 5.5: DIOM

1 Set r0 = b−Ax0, β = ‖r0‖2, v1 = r0/b and j = 1. while not converged do2 for i = max(1, j − k + 1), . . . , j do3 compute hij using the incomplete Arnoldi algorithm.4 end5 compute vj+1

6 find the LU factorization of Hj by updating the LU factorization of Hj−1

7 Set ζm =

β, if m = 1

−Lj,j−1ζj−1, else

8 Set pm = 1umm

[vm −

m−1∑i=m−k+1

uimpi

].

9 Set xm = xm−1 + ζmpm10 end

5.6 GMRES

The GMRES method was proposed by Saad and Schultz in 1986 in [16]. It is a projection method

where we choose K = Km(A, r0β

), and L = AK. It is thus an oblique projection method. As

discussed in section 4 the residual b−Ax is minimized over all vectors in x0 +Km, so the solutioncan be written as

x0 + Vmy.

DefineJ(y) = ‖b−Ax‖2 = ‖b−A(x0 + Vmy)‖2.

Then

b−Ax = b−Ax0 +AVmy (5.6.1)

= r0 −AVmy (5.6.2)

= βv1 − Vm+1Hmy (5.6.3)

= Vm+1(be1 − Hy). (5.6.4)

24

By construction, Vm+1 is an orthonormal matrix so

J(y) = ‖βe1 − Hmy‖2.

We want to minimize this function. The solution xm can be computed by first solving the minim-ization problem

ym = argmin ‖βe1 − Hmy‖2, (5.6.5)

and then recovering xm viaxm = x0 + Vmym.

The minimizer in equation 5.6.5 can be cheaply computed given that m is not too large [15]. Nowthe algorithm is built upon successively increasing the size of the Krylov space, determining ymand checking if some stopping criterion is met. The algorithm of the GMRES method is given inalgorithm 5.6.

Algorithm 5.6: GMRES

1 Set r0 = b−Ax0, β = ‖r0‖2, v1 = r0/b and j = 1.2 while not converged do3 for i = 1,. . . ,j do4 hij = vTi Avj5 end

6 wj = Avj −j∑i=1

hijvi

7 hj+1,j = ‖wj‖28 if hj+1,j = 0 then9 Stop

10 else11 vj+1 =

wjhj+1,j

12 end13 yj = argmin ‖βe1 − Hjyj‖214 j = j + 1

15 end16 Compute xm = xm + Vmym.

The algorithm differs from the Arnoldi algorithm only by the fact that we use the orthonormalbasis generated to explicitly find an approximate solution ym in each iteration. The algorithmbreaks down only if the Arnoldi process breaks down, i.e., when hj+1,j = 0.

Proposition 5.4. [15] Given that A is non-singular, then the GMRES method breaks down at stepj if and only if the solution xj is the exact solution to Ax = b.

This condition is sometimes also known as the optimality principle of GMRES. It simply meansthat we will be assured to converge to the exact solution assuming exact arithmetic in at most niterations. A proof can be found in [15].

25

Note that the GMRES method requires the storage of m orthonormal n × 1 vectors after m iter-ations. Furthermore, each iterations requires m inner products. This storage and computationalrequirement may become too large giving rise to the development of the so-called restarted GMRESmethod.

5.6.1 Restarted GMRES

The idea of the restarted GMRES method is to perform a certain amount of iterations, say kiterations, of the standard GMRES method, to compute the current approximation xm and restartthe algorithm setting x0 = xm. There are some advantages to this method. The major advantageof the restarting procedure is that the matrix Vm can be discarded from memory at every innercycle of k inner iterations. In this manner the memory storage is about k × n for Vk, and eachiteration requires at most k inner products.

Algorithm 5.7: Restarted GMRES.

1 while Not converged do2 Set r0 = b−Ax0, β = ‖r0‖2 and v1 = r0/b.3 Generate Hk using the Arnoldi algorithm starting with v1.4 Determine yk and xk.5 x0 = xk.

6 end

Every restart cycle is called an outer loop and the k loops performed at each restart are referredto as the inner loops. Note that the optimality principle is lost, that is, after n inner loops there isno guarantee the exact solution was computed even using exact arithmetic.

5.7 Notes on implementation

One of the problems with the GMRES algorithm as described above is that the true residual b−Axmis not known unless it is explicitly calculated. A possible remedy is to compute xm after a givennumber of iterations and evaluate the residual then to see if sufficient convergence is achieved. Wecan do better, however.

The least squares problem in equation 5.6.5 is often solved using plane rotations to transform theHessenberg matrix into an upper triangular form. This is done by using Givens rotation matricesof the form

Ωi =

1. . .

1ci si−si ci

1. . .

1

.

26

Here ci and si satisfy the condition c2i + s2

i = 1. After m iterations the matrix Hm has size(m+ 1)×m. We can premultiply this matrix by Givens rotation matrices of size (m+ 1)× (m+ 1).The subscript i means that the first ci is in the ith row.

Notational convention for this section: Throughout this section we use a bar to denote matriceshaving dimension (m+ 1)×m at step m. The same symbol without the bar is used to denote thefirst m×m submatrix. Similarly gm ∈ R(m+1)×1 and gm is the subvector of gm with the last entryremoved.

We will illustrate this method using the matrix Hm created after 3 iterations of GMRES which hasthe form

H3 =

h11 h12 h13

h21 h22 h23

h32 h33

h43

.

Also, we define

g0 =(β 0 0 0

)T= βeT1 .

Now we pre multiply H3 by

Ω1 =

c1 s1

−s1 c1

11

where we let

s1 =h21√

h211 + h2

21

, c1 =h11√

h211 + h2

21

.

Then we obtain

H(1)3 := Ω1H3 =

h

(1)11 h

(1)12 h

(1)13

h(1)22 h

(1)23

h32 h33

h43

, Ω3 =

1

1c′3 s′3−s′3 c′3

, g(1)0 =

cβ−sβ

00

Here the superscript denotes which entries have just been modified. Similarly, define

Ω2 =

1

c2 s2

−s2 c2

1

,

where

si =hi+1,i√(

h(i−1)ii

)2+ h2

i+1,i

, ci =hi−1ii√(

h(i−1)ii

)2+ h2

i+1,i

. (5.7.1)

Notice that si = sinϕ and ci cosϕ where ϕ is the rotation angle. Define Qm = Ω3Ω2Ω1 and

Rm = QmHm, (5.7.2)

gm = Qmg0. (5.7.3)

27

Then

H(3)3 = Rm =

h

(3)11 h

(3)12 h

(3)13

h(3)22 h

(3)23

h(3)33

0

, g3 =

γ1

γ2

γ3

γ4

.

Now by construction Ωi are unitary matrices and thus Qm is a unitary matrix. Therefore, in thegeneral case after m iterations we have

min ‖βe1 − Hmy‖2 = min ‖gm − Rmy‖2. (5.7.4)

The right hand side is the norm of a vector where the last element is fixed and equal to γm+1 =eTm+1gm. The vector y that minimizes the right hand side and thus the solution to the least squaressolution in equation 5.6.5 can be determined via

ym = R−1m gm,

where gm is equal to gm with its last element removed, and Rm is the matrix Rm with its last rowdeleted.

Note that the same method may be used to solve for ym in the FOM method in equation 5.2, inwhich case we need only to leave out the last Givens rotation matrix Ωm.

Now assume that after m iterations we have the matrix Rm. Then if the desired residual is notreached we must perform another iteration and the matrix Hm+1 is determined. We then take the

last column of this matrix, denoted hm+1, and determine Qmhm+1 = h(m)m+1 which is a vector of

length m+ 2. Then we append a row of zeros after the last row of R5 and write

H(m)m+1 =

(Rm h

(m)m+1

), g

(5)m+1 =

(γ1 . . . γm+1 0

)T.

The matrix Rm is already upper triangular and so this H(m)m+1 can be written as

H(m)m+1 =

(T

em+1h(m)m+1

).

Here T is an upper triangular matrix. Then we require one more Givens rotation to obtain

Rm+1 =

r11 r12 . . . r1,m+1

r22 . . . r2,m+1

. . ....

rm+1,m+1

0

, gm+1 =

γ1

γ2...

cm+1γm+1

−sm+1γm+1

.

Now using the expression ym+1 = R−1m gm in equation 5.7.4 we observe that the residual is given by

min ‖gm+1 − Rm+1ym+1‖2 = min ‖gm+1 − Rm+1R−1m gm+1‖2 (5.7.5)

= min ‖gm+1 − I+gm+1‖2 (5.7.6)

= ‖gm+1 − gm+1‖ (5.7.7)

= |sm+1γm+1|. (5.7.8)

28

Here Rm+1R−1m = I+ denotes the (m+ 1)× (m+ 1) identity matrix with a zero row appended at

the end.

So this approach allows us to cheaply keep track of the residual during the GMRES algorithm. TheGivens matrices require very little storage due to their structure.

5.8 Comparing FOM and GMRES

We will adopt here again the notation used in [15] and denote iterates for the FOM and GMRESmethods with an F and G superscript respectively. Furthermore, let ρFm and ρGm denote the residualnorms of the FOM and GMRES methods respectively. It follows from equation 5.7.8 that

ρGm = |sm|ρGm−1.

This immediately yieldsρGm = |s1s2 . . . sm|β.

We define again the matrices Hm and Hm as before and similarly gm and gm. Now by equation5.3.6 we have

‖rFm‖ = hm+1,m|eTmym|Now from ym = H−1

m (βe1) we get

Ωm−1 . . .Ω1Hmym = H(m−1)m ym = g(m−1)

m .

This then implies that eTmym = eTmg(m−1)m /h

(m−1)mm

5, so

ρFm = hm+1,m

∣∣∣∣∣eTmg(m−1)m

h(m−1)mm

∣∣∣∣∣ .Now recall from the previous discussion that eTmg

(m−1)m = ρGm−1 = |s1s2 . . . sm|β. Then the above

turns into

ρFm = |s1s2 . . . sm|β∣∣∣∣∣hm+1,m

h(m−1)mm

∣∣∣∣∣ .Inspecting equation 5.7.1 reveals that

hm+1,m

h(m−1)mm

= smcm

. Thus we can now state the following result:

ρFm = |s1s2 . . . sm|β∣∣∣∣smcm

∣∣∣∣ =1

|cm|ρGm ≥ ρGm.

That is, the GMRES method always does better or equally good when considering the residualcompared to the FOM method. It is for this reason that GMRES and its variants are widely used.

The Best of the 20th Century: Editors Name Top 10 Algorithms. In the May 2000 editionof SIAM news, the Society for Industrial and Applied Mathematics published an overview of theirchoice for the top 10 algorithms developed in the 20th century. Amongst this list, the discussedconcept of Kylov subspace iteration methods is included.

5H(m−1)m is an upper triangular matrix

29

5.9 Preconditioning

Preconditioning plays a vital role in finding an approximate solution to a linear system using iterat-ive methods. It is widely recognized as the determining factor in reducing the amount of iterationsneeded to converge to a desired accuracy. There are many preconditioning techniques and there isno clear best, rather the optimal preconditioner is highly problem dependent [15]. Generally theaim of preconditioners is to change a linear system to one that has a more favorable eigenvaluedistribution or is less dense than the original system, reducing time complexity.

Preconditioning a linear system means that we transform it into a new system with the same solutionbut better properties with regards to convergence of iterative methods. Let M be a nonsingularmatrix that in some yet undetermined sense approximates A, then we look at the two systems

M−1Ax = M−1b, (5.9.1)

AM−1y = b, x = M−1y. (5.9.2)

These systems are preconditioned from the left and from the right respectively. Equation 5.9.1has the same solution as Ax = b. In equation 5.9.2 we solve first for y and then recover x. InKrylov methods it is not necessary (and not recommended) to form the matrices M−1A and AM−1.Instead, when determining vi+1 = M−1Avi we solve

Mvi+1 = vi, with vi = Avi. (5.9.3)

So it is clear that our choice of M should allow for the efficient solving of systems of the formMx = b.

Finally there is also split preconditioning which turns a linear system Ax = b into

M−11 AM−1

2 y = M−11 b, x = M−1

2 y.

In this case the preconditioner matrix is the matrix M = M1M2. The choice of the preconditioneris problem dependent as mentioned and each of the three methods here shown have their ownadvantage. Right preconditioning for instance has the same residual as the original system, aproperty that the other methods do not share and makes it simpler to design stopping criteria.Note also that the matrices M−1A, AM−1 and M−1

1 AM−12 are all similar [1]. Split preconditioning

also preserves symmetry. We will present several preconditioning strategies in later sections.

5.10 Deflation

It is well known that the presence of eigenvalues close to the origin tends to cause stagnation in theconvergence of the restarted GMRES method [9]. A technique called deflation can be used in thissituation to shift a eigenvalues near the origin to a more desirable location. In the flexible GMRESmethod, the preconditioner is allowed to vary for each iteration. A similar idea is presented in [9]where in the restarted GMRES method the preconditioner is allowed to vary every outer loop. Thelatter variant will be described next.

30

5.10.1 DEFLGMRES

We will now look at the DEFLGMRES method as described in [9], which utilizes the restartedGMRES method where the preconditioner may change after each restart. As stated before, thegoal of the preconditioner in this setting is to push eigenvalues close to the origin to the positivereal number that is the magnitude of the largest eigenvalue of A, possibly close to one. Then thepreconditioner will contain the eigenvalues of A that were not removed and a number of eigenvaluesthat were shifted. Assuming that A is diagonalizable, denote the eigenvalues in increasing orderas |λ1| ≤ |λ2| ≤ . . . ≤ |λn|. Let P be an r-dimensional invariant subspace corresponding to the rsmallest eigenvalues, that is,

P = spanv1, . . . ,vr,where vi is the eigenvector corresponding to λi. Let U be an orthonormal basis for P .

Proposition 5.5. [9] Let T = UTAU and M = In + U(

1λnT − Ir

)UT , then

M−1 = In + U(|λn|T−1 − Ir)UT

and AM−1 has the spectrum

σAM−1 = λr+1, λr+2, . . . λn, |λn|,

where the eigenvalue |λn| has algebraic multiplicity of at least r.

Proof. The identities MM−1 = M−1M = I using the definitions for M and M−1 can be checkedby carefully expanding the matrix products. Now, let Z =

(U W

)be an orthonormal matrix

whose first r columns span U and for which range(Z) = Rn. Then by a similarity transformation,A is similar to a matrix

A = ZTAZ =

(T A12

0 A22

).

Also,

ZTMZ = M =

(|λn|T 0

0 In−r

),

Its inverse is clearly

M =

(|λn|T−1 0

0 In−r

).

Then

E = AM−1 = ZTAM−1Z

=

(|λn|Ir A12

0 A22

),

which has eigenvalues |λn| with multiplicity at least r. Since this matrix is similar to A they havethe same eigenvalues

31

This is a nice result but in practice we do not know the subspace P but instead we can find anapproximation P . Using an orthonormal basis (U , W ) we then get

E =

(|λn|Ir A12

|λn|A21T−1 A22

).

If the matrix |λn|A21T−1 is sufficiently small we may still expect the eigenvalues to be close to

those derived in the previous proposition and thus will see an improved convergence in the GMRESmethod [9].

Some numerical results concerning the convergence of the deflated GMRES are shown in figure 5.1.The problem to be solved is

SDS−1x =(1 1 . . . 1

)T.

Here S is a bidiagonal matrix with ones on the diagonal and β on the superdiagonal. The valuesof β and of the elements of D are given below the figures. These results are obtained from [9].

6 Fast multipole method

The fast multipole method, first developed by Rohklin and Greenhard in [10], is used for computingmatrix-vector products Au with boundary element matrices at each step of iterative algorithmslike the CG or GMRES method. The method has a computational time and storage complexityof O(n log n) for the matrix-vector product operation instead of standard O(n2). The underlyingprinciple is to determine the contributions of interactions between far away sources with less accur-acy than those between nearby sources [5]. The idea is based on using a truncated series expansionof the Green’s function as in equation 6.1.1.

6.1 Basic concepts of the FMM method

Let u be expanded in a set of basis functions vi as

u(x) =n∑i=1

uivi(x).

Now, if we want to evaluate u at some evaluation points xK , assume that we can write u(xK)as a sum of terms corresponding to each source points. Denote as uL(xK), the contribution tou(xK) from the source point xL. Furthermore, if G is a set containing some points, denote byuG(xK) =

∑L:xL∈G

uL(xK), the contribution to u(xK) due to all the points contained in G. The aim

of the fast multipole method is to combine this contribution of a set of source points in G into asingle source point, provided that xK and G are far away.

Suppose we have elements AKL in the simplified coefficient matrix resulting from a Galerkin methodof the form

AKL =

∫∫G(xi,xj)vK(xi)vL(xj)dxidxj .

32

(a) β = 1.1, D = diag(1, 2, . . . , n). (b) β = 0.9, D = diag(1, 2, . . . , n).

(c) β = 0.9, D = diag(1, 100, 200, . . . , 10000). (d) β = 0.9, D = diag(−10,−9, . . . , 90).

Figure 5.1: Convergence history for three different problems [9].

33

Typically for a BEM or FEM code we end up with approximating far away interactions via Gaussianquadratures with m evaluation points resulting in

AKL =

m∑i=1

m∑j=1

wiwjG(xKi ,xLj )vK(xKi)vL(xLj ).

Here wi and wj are the weights for the Gaussian quadrature. For the fast multipole method it isimportant that we can write G as a sum of products of terms, one depending on the source pointand one on the evaluation or target point when the points are far away [12]. In general this will bean infinite series that we truncate to P terms, i.e.,

G(xi,xj) =

P∑p=1

φp(xi)ψp(xj), when ‖xj − xi‖ is sufficiently large. (6.1.1)

Depending on the error term associated with this truncation we can choose an appropriate P tokeep the accuracy within the desired limits. Now we can compute the effect of a group of sourcesfar away from the evaluation point xk as follows

uG(xK) =∑L∈G

AKLuL

=∑L∈G

m∑i=1

m∑j=1

wiwjuLG(xKi ,xLj)vK(xKi)vL(xLj )

=∑L∈G

m∑i=1

m∑j=1

wiwjuL

P∑p=1

φp(xKi)ψp(xLj )vK(xKi)vL(xLj ). (6.1.2)

In the case that G contains only a single point xL we have

uL(xK) = AKLuL =

m∑i=1

m∑j=1

wiwjuLG(xi,xj)vK(xKi)vL(xLj ) (6.1.3)

We exchange the order of summation in 6.1.2 to obtain:

uG(xK) =m∑i=1

P∑p=1

wi

∑L∈G

m∑j=1

wjψp(xLj )uLvL(xLj )

vK(xKi)φp(xKi).

Now we can isolate the multipole coefficients Mp(G) as

Mp(G) =∑L∈G

m∑j=1

wjuLψp(xLj )vL(xLj )

which is independent of the evaluation point xK . We then have

uG(xK) =m∑i=1

P∑p=1

wiMp(G)vK(xKi)φp(xKi). (6.1.4)

34

The idea of the FMM is to subdivide the geometry of the problem into small subregions Gi eachcontaining some points xi1 , . . . ,xini . Then we pre-compute the multipole coefficients MP (G) foreach of these regions and use them when computing uG(xK) for xK and G sufficiently separated.We say that xK and G are well separated if xK has distance of at least diam(G) to G. This sufficesin practice to ensure that we can use the approximated truncated form for the Green’s function inequation 6.1.1.

Now if we have u and want to perform the operation u ← Au, this would have time complexityO(n2) using normal matrix-vector product operation. Consider algorithm 6.1 that performs anapproximate matrix vector product Au.

Algorithm 6.1: Pseudocode for FMM for performing u← Au

1 Divide the domain into subdomains, each containing a set of points G2 compute the multipole coefficients MP (G)3 for K=1,. . . ,n do4 u(xK) = 05 for G well separated from xK do6 u(xK)← u(xK) + uG(xK) using 6.1.47 end8 for G not well separated from xK do9 for y ∈ G do

10 u(xK)← u(xK) + uy(xK) using 6.1.311 end

12 end

13 end

Now if groups enclose points contained in sufficiently small neighbourhoods, then almost all groupswill be well separated. In this case we can neglect the cost of the second inner for loop. The orderthen comes out to be O(P (n+M)), where M is the number of groups used. The MP terms comesfrom the precomputation of the M multipole coefficients. So in the case that P n the FMM canreduce the cost of the ordinary matrix vector product operation significantly.

We can write the matrix A in the following manner:

A = Anear +Afar.

Here Anear represent the part of A that describes the interactions between nearby sources, andsimilarly for Afar. The fast multipole method can then be viewed as the approximation

A ≈ Anear + Afar

where Afar is the approximation of the distant mesh interactions due to the multipole expansions.In this light we can view the FMM method as a preconditioning technique which will be describedlater.

35

Figure 6.1: Hierarchical subdivision of a domain into smaller and smaller cubes.

One of the choices that we have is how to choose the grouping of source points. The most commonmethod will be the topic of the next subsection.

6.2 MLFMA

The MLFMA is based on encapsulating the domain of interest in a cube that is then subdivided intoeight cubes. Then each cube is recursively subdivided into eight cubes until the size of each cube issmall compared to the wavelength, generally half of the wavelength. The hierarchical subdivisionis stored in an octree, which is a data structure in which each internal node has eight children.After the subdivision some of the cubes may contain no mesh points and they are deleted from thetree. The smallest cube, that is the cubes that have no children are called the leaf boxes. Multipolecoefficients are then computed for all the lead boxes and then recursively for their parent cubes bysumming the multipole coefficients of their children [5].

When computing the interactions between a point x and all other points, first we compute allthe interactions of x with all points contained in boxes that are not well separated from the boxcontaining x. Then we proceed to move up one level continue the procedure, see figure 6.2.

The Best of the 20th Century: Editors Name Top 10 Algorithms. Just as the krylovsubspace iteration methods, the fast multipole method is also included in the Society for Industrialand Applied Mathematics’ top 10 algorithm list.

36

Figure 6.2: Interaction for the gray boxes are computed directly. Then the interactions representedby the dashed lines are computed using the multipole coefficients computed for the smallest boxes.The solid lines represent interactions between boxes one level higher.

7 Electric field integral equation

Many of the important ingredients for successfully implementing a boundary element method tosolve scattering problems have now been discussed. In practice, the BEM is often applied toelectromagnetic scattering problems, that satisfy the electric field integral equation (EFIE). Theequation reads∫

Γ

∫ΓG(|y − x|)

(j(x) · jt(y)− 1

k2(O · j(x)) · (O · jt(x))

)dxdy =

i

kZ0

∫Γ

Einc(x) · jt(x)dx (7.0.1)

Here [3]

• Γ is the boundary of the domain.

• k = 2πλ is the wave number of the incoming radiation.

• G(|y − x|) is the Green’s function as before.

• Z0 is the impedance of a vacuum.

The aim is to find the surface current j(x) for x ∈ Γ such that for all tangential test function jt(x),equation 7.0.1 holds, see figure 7.1. Since a closed form solution of this equation is not available,we approximate the solution numerically. We then construct a mesh of n edges, and expand j(x)into a set of basis function φ1, . . . , φn. Commonly, the Rao-Wilton-Glisson basis is used here.If we use this same set of functions as our test functions we can recast the problem into a linearsystem

Aλ = b, (7.0.2)

37

Figure 7.1: Computed current distribution over the surface of an airplane.

where

Aij =

∫Γ

∫ΓG(|y − x|)

(φi(x) · φj(y)− 1

k2(O · φi(x)) · (O · φj(x))

)dxdy, (7.0.3)

bj =i

kZ0

∫Γ

Einc(x) · φj(x)dx. (7.0.4)

Then

j(x) =n∑i=1

λiφi(x).

The matrix A will be a dense complex matrix which may or may not be symmetric [3], dependingon the particular integral formulation. For EFIE, it is complex symmetric non-Hermitian.

Problem arising from the discretised EFIE’s:

• Problems related to the size of the matrix:The size of the matrices arising from this method can be very large, although they aretypically smaller then those arising from finite element of finite volume formulations of thesame problems. It is however not preferable, or even impossible, to use direct methods(i.e., Gaussian elimination) to solve equation 7.0.2 due to the size of A. Therefore iterativemethods like GMRES are used, which means that preconditioning often becomes an importantcomputational aspect and will largely determine the convergence rate. As mentioned before,preconditioning fully dense matrices can be quite hard and an appropriate choice must bemade as to which type of preconditioning will be used. In some cases the size of the matrixA combined with the fact that it is fully dense means that the matrix cannot be fully storedin the memory and we need an ad-hoc method to carry out the matrix-vector products thatare required at each step of an iterative method. Such a method is FMM method.

• Problems related to the eigenvalues of the matrix:Typically the matrix A will have poorly clustered eigenvalues, leading to slow convergence.

38

(a) A mesh for an airplane. (b) A wing with a slit.

(c) An air intake.

Figure 7.2: Meshes for different geometries taken from [3].

39

According to the theory of Krylov methods, fast convergence is achieved if the residual vectorcan be expanded into a low order polynomial which takes on value 1 at the origin andapproximately zero at the eigenvalues of A. Clearly, if the spectrum of A is poorly clusteredor contains many eigenvalues near the origin such a low order polynomial cannot be found[3].

Next we will discuss some methods of preconditioning the linear system 7.0.2.

8 Incomplete factorization methods

Incomplete factorization methods are based on finding lower triangular matrix L and upper trian-gular matrix U such that

LU ≈ A.with stipulation that L and U are somehow easy to invert. Typically we would like these twomatrices to be sparse. Several such methods exist and they are based on choosing which values ofthe actual L and U factors to drop. The preconditioned system is then

L−1AU−1y = L−1x, x = U−1y.

8.1 Level of fill techniques

One possibility is demanding that Lij = 0 when Aij = 0 and likewise for U . To this end define

S = (i, j) ∈ 1, . . . , n × 1, . . . , n | aij 6= 0 or i = j.

Notice that besides all the non zero positions of A we also include the main diagonal. Then we dropall positions in L and U that are not in S. These types of preconditioners are easy to implementand inexpensive to compute. They were originally designed for and are well suited for M matricesand diagonally dominant matrices. When S is computed as described the resulting preconditioneris called the no-fill incomplete LU factorization, or ILU(0). Exactly the same concept applies tothe Cholesky factorization A = LLT .

More sophisticated methods based on the same idea exists where we allow a higher level of fill-inis supplied. The level of fill for the lower triangular factor L is initialized as [3]

lev(lki) =

0 if aki 6= 0 or k = i

∞ else,

and then updated during the factorization via

lev(lki) = minlev(lki), lev(lij) + lev(lkj) + 1.

The preconditioner that is obtained in called the ILU(p) factorization where the non-zero patternfor the triangular factors is defined as

F = (i, j) ∈ 1, . . . , n × 1, . . . , n | lev(lki ≤ p.

40

Density of S = 2%

ILU(level) Density of L κ∞(L) GMRES(30) GMRES(50)

ILU(0) 2.0% 2 · 103 378 245ILU(1) 5.1% 1 · 103 79 68ILU(2) 9.1% 9 · 102 58 48

Density of S = 4%

ILU(level) Density of L κ∞(L) GMRES(30) GMRES(50)

ILU(0) 4.0% 6 · 109 – –ILU(1) 11.7% 2 · 105 – –ILU(2) 19.0% 7 · 103 40 38

Density of S = 6%

ILU(level) Density of L κ∞(L) GMRES(30) GMRES(50)

ILU(0) 6.0% 8 · 1011 – –ILU(1) 18.8% 5 · 1011 – –ILU(2) 29.6% 7 · 104 – –

Table 1: Number of iterations of GMRES varying the sparsity level of S and the level of fill-in ofthe approximate factor L on the model problem of Figure 8.1 (n = 2430, κ∞(A) = ‖A‖∞‖A−1‖∞ ≈O(102)). The symbol ’-’ means that convergence was not obtained after 500 iterations.

Increasing the parameter p makes the ILU(p) preconditioner increasingly more dense and expensiveto compute. In Table 1 we compare the result obtained for different levels of fill. The problemsolved in this case is that of the sphere shown in Figure 8.1. The matrix is sparsified to differentdegrees before applying the preconditioner6. It can be seen from the results that increasing thelevel of fill while keeping the matrix sparse can result in much more robust preconditioners thanusing a denser matrix. Also, the density of the lower triangular factor remains lower in this case.It is also apparent that ill-conditioning of the L factor is not necessarily related to ill-conditioningof A. This problem may be detected in advance by evaluating

‖(LU)−1e‖ ≤ ‖(LU)−1‖∞,

where e =(1 1 . . . 1

)T. This can be done relatively cheaply [3].

Incomplete factorization can also use a drop by size criterion instead of a drop by position criterion,like the one discussed before. We then define a drop tolerance parameter. A drop tolerance is apositive number τ that is used to determine when to drop values in L and U , i.e., Lij = 0 if Lij < τ ,and similarly for U . One of the difficulties with this condition is that it is often not clear beforehandwhich entries in L and U will be large, and so we cannot beforehand determine the sparsity patternand thus the storage requirement. The drop tolerance may also be changed dynamically to try andachieve a certain desired density.

It turns out that in the case of BEM matrices, the sparsity pattern of A closely resembles thesparsity pattern of A−1. It appears however that a priori selection strategies based on the sparsitypattern of A or a sparsified version of A, are not very useful as the structure of A does not translateinto any noticable structure in the triangular factors, see figure 8.2.

6If we would not sparsify A then we would be computing the ILU(0) preconditioner of a fully dense matrix, i.e.,computing its exact L and U factors.

41

Figure 8.1: A sphere of length 1 meter, illuminated at 300MHz. It is discretized using 2430 edges.

8.2 Problems with ILU preconditioners

If L and U are the approximate factors of A, then we can write the factorization as

A = LU + E,

where E is the error incurred by the incomplete factorization. Then the preconditioned matrix willbe

L−1AU−1 = I + L−1EU−1. (8.2.1)

It is the preconditioned error matrix L−1EU−1 that plays a vital role in the quality of the precon-ditioner. In the case that L−1 and U−1 have very large values, equation 8.2.1 may not be close tothe identity matrix at all [7]. Here we list ([8]) some problems that might occur

1. Small pivots: Pivots encountered during the computation of the approximate L and U canbe very small. These pivots will give rise to very large values in these factors which will causeinaccuracy in the sense that L−1EU−1 is not sufficiently close to zero.

2. Unstable factors: Due to dropping, the L and U factors may be unstable, that is, theircondition number may be very high. The triangular solves are then ill-conditioned and thusthe preconditioner is useless.

3. Inaccuracy due to dropping: The idea of ILU preconditioners is to drop elements in theexact L and U factors to make the factors more economical to store and solve with, as well asmake their construction cheaper. This form of dropping due to a drop tolerance or an a priorisparsity pattern determines the error matrix E. Often these inaccuracies can be remediedby using a lower drop tolerance or enlarging the sparsity pattern. If the inaccuracy comesinstead from small pivots this will not help in general.

These problems may combine and mask together in complex ways, for instance reducing the droptolerance to increase the accuracy of the factorization may produce smaller pivots that deterioratethe stability of the factors.

The use of ILU preconditioners on dense matrices is still very much ongoing research work, andnot much literature is available yet.

42

(a) Sparsified A−1. (b) Sparsified L−1.

Figure 8.2: Sparsified A−1 and L−1 matrices, where a dropping tolerance of 5.0× 10−2 was used.

9 Multilevel preconditioning

The multilevel method is built upon the fact that unknowns that are not coupled may be eliminatedsimultaneously using Gaussian elimination, and are thus well suited for parallelism [13].

9.1 Independent set ordering

LetG = (V,K) be the adjacency graph of a matrix A. That is, V contains the n vertices representingthe n unknowns and there is an edge between node i and j if and only if aij 6= 0. So every edgerepresent the binary information equation i involves unknown j, as stated in [14]. This is a directedgraph unless A is symmetric. Let (x, y) denote an edge from vertex x to y. Then an independentset S ⊂ V is a set such that

if x ∈ S then [(x, y) ∈ K or (y, x) ∈ K ⇒ y /∈ S] .

So an independent set is a set for which all vertices are disconnected from each other. Such a setS is called maximal if we cannot add elements from V \ S to S in such a way that S remainsan independent set. Finding a maximal set is an NP-hard problem and so we often use greedyalgorithms to find independent sets that are not necessarily maximal. This idea can be generalizedto block independent sets, where There may be coupling between the unknowns of each group.More information on parallel implementations of independent set finding algorithms for multilevelILU factorizations can be found for instance in [17].

Independent set orderings are very useful in direct methods such as Gaussian elimination as theyallow for parallelism in the Gaussian elimination by simultaneously solving smaller systems for theindependent sets. They can however also be used in the design of preconditioners.

43

In most multilevel ILU factorizations the unknowns are first reordered so that the unknowns asso-ciated to an independent set are listed first. Then we can write

PjAjPTj =

(Dj FjEj Cj

). (9.1.1)

Here Dj is, in general, a block diagonal matrix. The original system now reads(Dj FjEj Cj

)(x1

x2

)=

(b1

b2

),

so we may solve for x1 to getx1 = D−1

j (b1 − Fjx2) .

The matrix D−1j can be computed cheaply as Dj has a block diagonal structure. Furthermore,

there is an opportunity for parallelism here as we can solve matrix products like D−1j Fj for each

entry in D−1j simultaneously [13]. It remains to compute x2 by solving the linear system

Cjx2 = b2 − Ejx1 = b2 = EjD−1j (b1 − Fjx2) . (9.1.2)

Denoting by Sj = (Cj − EjD−1j Fj), the Schur complement of Cj , equation 9.1.2 can be rewritten

asSjx2 = (b2 − EjD−1

j b1)

which is a smaller linear system that the original one. Usually this procedure is repeated untilsome criterion is met. So far the procedure is still exact, as there is no approximation made. Inthe context of preconditioning techniques, we would like to improve sparsity, and often droppingstrategies are used in conjunction with the reordering strategies. We can find an approximate blockLU factorization of 9.1.1 as

PjAjPTj =

(Dj FjEj Cj

)=

(I 0

EjD−1j I

)(Dj Fj0 Sj

).

If a drop tolerance of a priori sparsity pattern is used this is no longer exact.Here we may utilize static or adaptive sparsity patterns as well as a drop tolerance to retain asmuch sparsity in Sj as possible. See figure 9.1 for some example multilevel preconditioners. Be-sides the independent set ordering there are other orderings that are used. Some of these includethe approximate minimum degree (amd), multilevel nested dissection by nodes (metisn) and edges(metise) and the nonsymmetric ddPQ (pq) orderings. We will not go into further details aboutthese different orderings strategies here. When used in conjunction with a dropping tolerance,multilevel ILU preconditioners are denoted as MILUT(τ) where τ is the dropping threshold. Someresults obtained when using MILUT(τ) preconditioners can be seen in table 2. In each case, all thementioned orderings were used and the table shows the ordering that performed best.

It is clear that the use of the MILUT(τ) preconditioner significantly reduces the amount of iterationsneeded. For each of the iterative methods and all the sample problems a reduction of roughly oneorder of magnitude was achieved in the iterations needed to converge. The stopping criterion usedis a relative residual reduction of 1012 which means that the iteration stops successfully only whenthe residual is a factor 1012 smaller than the original residual. In actual industrial applicationsoften a residual reduction of 103 may be sufficient.

44

Figure 9.1: Three different multilevel ILU preconditioners for the top left matrix which a 9-pointmatrix for a 10 × 10 grid. Top right: using a minimum block size of 2. Bottom left: minimumblock size 12. Bottom right: minimum block size of 50 for a 20× 25 grid [13].

10 Approximate inverse preconditioning

Approximate inverse preconditioners (AINV) construct a matrix M such that either

MA or AM

are close to the identity matrix. This often means that we aim to minimize

‖I −MA‖

for some norm.

10.1 Frobenius-norm minimization preconditioner

The idea of the Frobenius-norm minimization preconditioner is to compute the sparse approximateinverse M ≈ A−1 that minimizes [18]

‖I −MA‖F , or ‖I −AM‖F ,

in the case of left and right preconditioning respectively. The Frobenius-norm can be expressed as

‖B‖F =

√√√√ n∑i=1

n∑j=1

|bij |2

45

Ex.1 - density(S) = 1.69% - ordering: METISE

prec mem timeGMRES(50) CORS SQMR

its time its time its time

none - - +3000 +101.99 742 44.99 809 24.14MILUT(10−2) 6.0 0.47 78 2.74 60 3.36 70 2.37

Ex.2 - density(S) = 0.47% - ordering: AMD

prec mem timeGMRES(50) CORS SQMR

its time its time its time

none - - +3000 +1862.99 1133 1032.76 1453 697.62MILUT(10−2) 4.6 7.81 430 236.57 145 164.69 155 115.66

Ex.3 - density(S) = 0.57% - ordering: AMD

prec mem timeGMRES(50) CORS SQMR

its time its time its time

none - - +3000 1361.71 1683 1299.71 1947 698.02MILUT(10−2) 4.2 4.33 926 369.83 176 138.03 168 62.05

Ex.4 - density(S) = 0.65% - ordering: AMD

prec mem timeGMRES(50) CORS SQMR

its time its time its time

none - - +3000 +4891.44 809 2570.93 999 1260.70MILUT(10−2) 5.2 64.12 274 461.49 128 469.02 185 472.48

Ex.5 - density(S) = 2.73% - ordering: PQ

prec mem timeGMRES(50) CORS SQMR

its time its time its time

none - - +3000 +15278.08 +3000 +27287.82 +3000 +13723.47MILUT(10−3) 6.8 753.50 98 906.59 69 802.67 70 480.46

Table 2: Experiments with inverse-based multilevel ILU preconditioner. Here S stands for Anear,which is the near field field FMM matrix. GMRES(n) is the restarted GMRES method as we haveseen before and the CORS and SQMR are two method that are, like the restarted GMRES method,also based on short-term vector recurrences [4].

The Frobenius norm is convenient because it allows us to decouple the minimization problem inton least square problems due to the relation

‖I −AM‖2F =n∑j=1

‖ej −Am?j‖22, (10.1.1)

or

‖I −MA‖2F = ‖I −AMT ‖2F =n∑j=1

‖ej −AmTj?‖22. (10.1.2)

This decoupling into n least squares problems gives possibilities for parallelism. The issue lies inselecting an appropriate sparsity pattern

S = (i, j) ⊂ [1, n]2 | mij = 0. (10.1.3)

A priori methods exist where the sparsity pattern is determined beforehand. Other methods aredynamic and start of with some initial sparsity pattern like a diagonal matrix. Then this sparsity

46

pattern is recomputed and refined until some condition of the form

‖ej −Am?j‖22 < ε, ∀j = 1, . . . , n

is met [5]. As is to be expected, static a priori methods require much smaller computing time andmemory storage but may not be as robust as dynamic methods. In the case of a static sparsitypattern O(1) nonzero locations are determined for each column of m?j of M . Then each of the nleast squares problems in equation 10.1.1 is solved using QR methods, a standard technique whichwe will not go into here. Each of these requires about O(n) flops and so we are left with a timecomplexity of about O(n2) for constructing the preconditioner M . There is room for improvementby choosing M as the matrix minimizing

‖I −MA‖Fwhere A is a sparse approximation of A and M fulfils some sparsity conditions as before [5].

Considering the matrix form of the EFIE, from a physical perspective, using a sparse approximateinverses makes sense. This is due to the fact that strong interaction exists only between nearbypoints. Therefore, dropping some small values intuitively might not be a bad idea.

Furthermore, it is observed that matrices arising from the Green’s function as is the case withelectromagnetic scattering have the property that the distribution of the large entries of A is verysimilar to the distribution of the large values in A−1, see figure 10.1. Based on this result, denoteby A a sparse approximation of A that is obtained after dropping all entries below some threshold.Then a suitable choice for the sparsity pattern, as in equation 10.1.3, is the sparsity pattern of A.This would mean that we are utilizing a static approximate inverse method. With this in mind wecan reduce the cost of the least squares problems in equation 10.1.1 and 10.1.2 arising from theSPAI preconditioner.

Other selection methods for the sparsity pattern exist. In particular, we may choose as a sparseapproximation of A, the near-field multipole matrix Anear described previously. Then the minim-ization problems 10.1.1 and 10.1.2 turn into the problem of finding a matrix M such that

‖I −MAnear‖2F , or ‖I −AnearM‖2Fare minimized in the case of left or right preconditioning respectively [3].

Typically Frobenius norm minimization techniques manage to effectively move most of the eigenval-ues to 1. In practice there are often a few eigenvalues that remain close to the origin however, andthese may present a problem for convergence. In this case deflation techniques can be applied whichattempt to move the remaining eigenvalues into the cluster. Deflation techniques were discussedin the section on Krylov subspaces, and are often used to speed up the convergence of GMRESmethods utilizing SPAI preconditioners.

From the result in table 3 it can be seen that the GMRES method scales very poorly with the sizeof the problem. Larger problem size is obtained by illuminating the object at a higher frequency.This is especially true at high frequency as the size of the smallest boxes used in the FMM methodare related to the size of the wavelength. As the size of the boxes decreases, the near field matrixbecomes very sparse, and subsequently we may lose all information of the distant interactions [3].

47

(a) A (b) A−1

Figure 10.1: Sparsity pattern of A and A−1 for a small sphere, a problem that is representative ofthe general trend. The magnitude of the entries is given by different colors, where the magnitudedecreases as the color goes from red to green to yellow to blue.

Figure 10.2: Eigenvalue distribution of the BEM formulation of scattering for a cylinder with aslit. Some clustering away from the origin and around 1 can be seen while some eigenvalues tendto cluster around the origin which is detrimental to the convergence.

48

Cetaf

Size Density SPAI Time SPAIGMRES(∞) GMRES(120)

Iter / Time Iter / Time

86256 0.18 4m 656 / 1h 25m 1546 / 1h 44m134775 0.11 6m 618 / 1h 45m 1125 / 1h 55m264156 0.06 13m 710 / 9h 1373 / 4h 46m531900 0.03 20m 844 / 1d 18m 1717 / 14h 8m

1056636 0.01 37m T.L.E. / >9h(32) - / > 1d

Aircraft

Size Density SPAI Time SPAIGMRES(∞) GMRES(120)

Iter / Time Iter / Time

94704 0.28 11m 746 / 2h 9m 1956 / 3h 13m213084 0.13 31m 973 / 7h 19m - / 7h 56m

591900 0.09 1h 30m 1461 / 16h 42m(64) - / 1d 57m

1160124 0.02 3h 24m M.L.E.(64) / N.A. - / > 4d

Cobra

Size Density SPAI Time SPAIGMRES(∞) GMRES(120)

Iter / Time Iter / Time

60695 0.24 2m 369 / 26m 516 / 23m179460 0.09 7m 353 / 1h 11m 406 / 1h 2m

Table 3: Number of matrix-vector products and elapsed time required to converge on two problemson 8 processors of the Compaq machine, except those marked with (k), that were run on k processors.Tolerance for the iterative solution was 10−3. The symbol ’-’ means that no convergence wasachieved in 2000 iterations. Acronyms: N.A. ≡ not available. M.L.E. ≡ memory limits exceeded,T.L.E. ≡ CPU time limits exceeded. The Time SPAI column shows the time taken to constructthe preconditioner.

11 MLFMA-based preconditioners

Scalability issues and high density requirements for reliable preconditioners have been observerwith the use of SPAI and ILU preconditioners respectively. Using a preconditioning strategy basedon the MLFMA algorithm as described before can alleviate these problems. These techniques mustbe used in conjunction with an iterative methods that allows us flexibly change the preconditionerat each iteration. An example of this is the DELFGMRES algorithm described earlier. Moregenerally, the FGMRES algorithm is simply the GMRES algorithm where each iteration a newpreconditioner is constructed and used.

Recall that in the case of left preconditioning we are solving the system

M−1Ax = M−1b,

where at each iteration we need to solve equation 5.9.3, repeated here:

Mvi+1 = vi.

This is again a linear system that may be solved by an iterative method. the idea of MLFMAbased preconditioners is use a variable preconditioner M at each outer iteration and then to again

49

precondition the system 5.9.3 and solve it using a method like the normal GMRES with only alimited small number of iterations, lets say the maximum number of iteration is denoted maxit. Inthe matrix-vector products in the outer loop we may choose an accurate MLFMA approach anduse a less accurate one for the inner loop by for instance increasing the box size.

Algorithm 11.1: MLFMA-based algorithm

1 while not converged do2 Outer loop: Outer solves with high accuracy FFM3 for i=1,. . . ,maxit do4 Inner solver: Use low accuracy FMM and SPAI preconditioning5 end

6 end

In table 4 we compared the restarted GMRES method to the FMGREM method for some of theexample problems discussed earlier. The results show that for some of the problems we can attaina near linear time scaling with the problem size when using the FGMRES method, namely forthe Cetaf problem. The FGMRES is in all cases faster than the normal GMRES and when theproblem becomes particularly large it is in fact much larger. Similar results hold when we comparethe normal GMRES (RGMRES(∞)) to the FGMRES method where we do not use a restart inthe outer. One of the most challenging problems seems to be the aircraft problem. Notice that atthe largest size RGMRES does not converge while the full GMRES method exceeds the memoryavailable.

The main advantage to this type of preconditioning is that we can in fact combine it when differenttypes of preconditioners previously discussed. For instance for the inner loop we could just aswell have chosen a MILUT(τ) preconditioner. In theory the idea of nested iterative method couldbe extended of course to three or more levels although there are not many result available aboutthis approach. It seems that we cannot however continue this process too far as we are losingsome information when we use low accuracy FMM matrix-vector products and at some point theaccuracy simply will not be enough and we need to use direct techniques.

50

Cetaf

Size GMRES(120) FGMRES(30,60)Iter Time Iter Time

86256 1546 1h 44m 17+ 960 55m134775 1125 1h 55m 15+ 840 1h 19m264156 1373 4h 46m 17+ 960 2h 22m531900 1717 14h 8m 19+1080 6h1056636 - > 1d 22+1260 14h

Aircraft

Size GMRES(120) FGMRES(30,60)Iter Time Iter Time

94704 1956 3h 13m 27+1560 2h 14m213084 - 7h 56m 34+1920 5h591900 - 1d 57m 57+3300 1d 9h 45m

1160124 - > 4d 51+2940 16h 41m(64)

Cobra

Size GMRES(120) FGMRES(30,60)Iter Time Iter Time

60695 708 29m 24+660 18m179460 433 48m 20+540 42m

Table 4: Number of matrix-vector products and elapsed time required to converge on eight pro-cessors of the Compaq machine. The tests were run on eight processors of the Compaq machine,except those marked with (k), which were run on k processors. The symbol ’-’ means that noconvergence was achieved in 2000 iterations. The values of m and n in the symbol FGMRES(m,n)refer to the restart of the outer and of the inner solver. For FGMRES we give the number of innerplus total outer iterations needed to achieve convergence. The tolerance for the iterative solutionwas 10−3.

12 Conclusion

Comparing the numerical results obtained it is clear that the MILUT(τ) and SPAI precondition-ers both perform very well. The SPAI preconditioner has the advantageous property that wecan a priori determine the storage size required by looking at the sparsity pattern of the sparsi-fied matrix A, for example Anear. The same is true for the MILUT(τ) preconditioner. Particularlythe MILUT(τ) preconditioner seems to increase perfomance for all the test cases in a dramatic way.

In contrast, the ILU(p) preconditioners seem to be less predictable. Their storage requirements cannot be determined a priori unless we choose some sparsity pattern in advance which is hard as thetriangular factors do not seem to inherit any particular structure from the sparsity pattern of A.It seems that higher levels of fill in increase the performance of the preconditioner but as seen intable 1, ILU(2) has a density of 29.6% using a sparsified A with density 6%.

The use of MLFMA techniques seems very promising and indeed very flexible. The parametersof the FMM matrix-vectors products such as size of the boxes may be fine tuned to achieve thedesired level of precision with the least amount of iterations.

51

Cetaf

SizeGMRES(∞) FGMRES(∞,60)

Iter Time Iter Time

86256 656 1h 25m 17+ 960 55m134775 618 1h 45m 15+ 840 1h 19m264156 710 9h 17+ 960 2h 22m531900 844 1d 18m 19+1080 6h

1056636 T.L.E. > 9h(32) 22+1260 14h

Aircraft

SizeGMRES(∞) FGMRES(∞,60)

Iter Time Iter Time

94704 746 2h 9m 23+1320 2h 30m213084 973 7h 19m 30+1740 6h 11m

591900 1461 16h 42m(64) 43+2520 12h(32)

1160124 M.L.E.(64) > 1d 43+2520 14 h 28m(64)

Cobra

Size GMRES(∞) FGMRES(∞,60)

Iter Time Iter Time

60695 369 26m 21+600 17m179460 353 1h 11m 18+510 38m

Table 5: Number of matrix-vector products and elapsed time required to converge on eight pro-cessors of the Compaq machine. The tests were run on eight processors of the Compaq machine,except those marked with (k), that were run on k processors. The values of m and n in the symbolFGMRES(m,n) refer to the restart of the outer and of the inner solver. For FGMRES we give thenumber of inner plus total outer iterations needed to achieve convergence. The tolerance for theiterative solution was 10−3. Acronyms: M.L.E. ≡ memory limits exceeded, T.L.E. ≡ CPU timelimits exceeded

Clearly the choice of preconditioner is very important and there is a large variety available based ondifferent properties of the system. It is also clear that for effective preconditioning we need to takeas much information as to the origin of the problem into account. Choosing a wrong preconditionercan in fact prohibit convergence all together and so spoil the solution. It is due to this that mostblack box solvers, i.e. solvers that are aimed at solving linear systems with no information of theirorigin, often use direct methods.

It is impossible to say if the BEM performs better than the FEM for scattering problems. One ofthe reasons is that there is no one single FEM approach. Just as the BEM formulation depends onthe iterative method used and the preconditioner used, the FEM formulation depends on the choiceof boundary conditions, the choice of control surfaces used and the basis functions. However, ingeneral FEM formulations always produce larger linear systems but they are always much sparserthan those arising from the BEM formulation.

52

Acknowledgements

I gratefully thank the EMC Team at CERFACS in Toulouse and the EADS-CCR (European Aero-nautic Defence and Space - Company Corporate Research Center) in Toulouse, for providing someimages of meshes included in this thesis.

References

[1] Michele Benzi. Preconditioning techniques for large linear systems: A survey. J. Comput.Phys., 182(2):418–477, November 2002.

[2] J. Burden, R. Faires. Numerical Analysis. Cengage Learning, 2011.

[3] B. Carpentieri. Algebraic preconditioners for the fast multipole method in electromagneticscattering analysis from large structures: trends and problems. Electronic Journal of BoundaryElements, 7:1, 2009.

[4] B. Carpentieri and M. Bollhofer. Symmetric inverse-based multilevel ilu preconditioning forsolving dense complex non-hermitian systems in electromagnetics. Progress in electromagneticsresearch-Pier, 128:55–74, 2012.

[5] Bruno Carpentieri, Iain S. Duff, Luc Giraud, and Guillaume Sylvand. Combining fast multipoletechniques and an approximate inverse preconditioner for large electromagnetism calculations.Siam journal on scientific computing, 27(3):774792, 2005.

[6] Simon Chandler-Wilde and Steve Langdon. Boundary element methods for acoustics. 2007.

[7] E. Chow and Y. Saad. Approximate inverse preconditioners via sparse-sparse iterations. SIAMJ. Sci. Comput., 19:995–1023, 1998.

[8] Edmond Chow and Yousef Saad. Experimental study of ilu preconditioners for indefinitematrices. J. COMPUT. APPL. MATH, 86:387–414, 1997.

[9] Jocelyne Erhel, Kevin Burrage, and Bert Pohl. Restarted gmres preconditioned by deflation.Journal of Computational and Applied Mathematics, 69:303–318, 1995.

[10] L. Greengard and V. Rokhlin. A fast algorithm for particle simulations. J. Comput. Phys.,73(2):325–348, December 1987.

[11] Randall LeVeque. Finite Difference Methods for Ordinary and Partial Differential Equations:Steady-State and Time-Dependent Problems (Classics in Applied Mathematics Classics in Ap-plied Mathemat). Society for Industrial and Applied Mathematics, Philadelphia, PA, USA,2007.

[12] Jussi Rahola. Experiments on iterative methods and the fast multipole method in electromag-netic scattering calculations, 1998.

[13] B. Saad, I. Suchomel. Arms: an algebraic recursive multilevel solver for general sparse linearsystems. NUMERICAL LINEAR ALGEBRA WITH APPLICATIONS, 9:359–378, 2002.

53

[14] I. Saad. Ilum: A multi-elimination ilu preconditioner for general sparse matrices. 1999.

[15] Y. Saad. Iterative Methods for Sparse Linear Systems: Second Edition. Society for Industrialand Applied Mathematics, 2003.

[16] Yousef Saad and Martin H Schultz. Gmres: A generalized minimal residual algorithm forsolving nonsymmetric linear systems. SIAM J. Sci. Stat. Comput., 7(3):856–869, July 1986.

[17] Chi Shen and Jun Zhang. A fully parallel block independent set algorithm for distributedsparse matrices. Paral. Comput, 2003.

[18] H.A. van der Vorst. Iterative Krylov Methods for Large Linear Systems. Cambridge mono-graphs on applied and computational mathematics. Cambridge University Press, 2003.

[19] Elman. Silverster. Wathen. Finite Elements and Fast Iterative Solvers: with Applications inIncompressible Fluid Dynamics. Oxford Scholarship Online, 2014.

54