introduction to multigrid methods - fau€¦ · multigrid methods, and their various multiscale...

Slide Universidade Federal do Paraná Introduction to Multigrid

Introduction toMultigrid Methods

an excerpt of the lectureSimulation and Scientific Computing

Ulrich RüdeLehrstuhl für SystemsimulationUniversität Erlangen-Nürnberg

Curitiba, October 29, 2014

1

2

This Lecture isBased on a Tutorial by

Irad Yavneh

Department of Computer Science

Technion – Israel Institute of Technology

Haifa 32000, Israel

[email protected]

3

Contents of Tutorial• Basic Concepts: The main ideas via the soldier

alignment problem (due to A. Bruckstein)• Multigrid smoothing and coarse grid correction• V-cycle• Full Multigrid• Full approximation storage• Tau-Extrapolation• Algebraic Multigrid

4

Some Relevant Books/Articles:

1. W.L. Briggs, V.E. Henson, and S.F. McCormick: “A Multigrid Tutorial”, 2nd ed., SIAM, 2000.

2. U. Trottenberg, C.W. Oosterlee, and A. Schueller: “Multigrid”, Academic Press, 2001.

3. Brandt, A., “1984 Guide with Applications to Fluid Dynamics”, reprinted in SIAM Classics in Applied Math. Series, 2011.

4. Hackbusch, W., “Multigrid Methods and Applications”, Springer, Berlin, 1985.

5. W. Hackbusch and U Trottenberg eds.: “Multigrid Methods”, Springer-Verlag, Berlin, 1982.

6. Wienands, R., and Joppich, W., “Practical Fourier Analysis for Multigrid Methods”, Chapman & Hall/CRC, 2004.

7. Computing in Science and Engineering (CiSE) Special Issue on Multigrid Computing, Comput. Sci. Eng. 8, 56 (2006)

8. Steve McCormick and Ulrich Rüde, "Multigrid Computing," Computing in Science & Engineering, vol. 8, no. 6, November/December 2006, pp. 10-11

5

M

Why Multigrid MethodsAre So Efficient

M U L T I G R I DC O M P U T I N G

Originally introduced as a way to numerically solve elliptic boundary-value problems,

multigrid methods, and their various multiscale descendants, have since been developed

and applied to various problems in many disciplines. This introductory article provides the

basic concepts and methods of analysis and outlines some of the difficulties of developing

efficient multigrid algorithms.

6

What is Multigrid about?

• A framework of efficient iterative methods for solving problems with many variables and many scales.

Slide Universidade Federal do Paraná Introduction to Multigrid 7

Framework: common concept, different methods.Efficient: usually O(N) or O(N log N) operations

The importance of efficient methods becomes greater as computers grow stronger! Iterative: most nontrivial problems in our field cannot be solved directly efficiently.Solving: approximately, subject to appropriate convergence criteria, constraints, etc.Many variables: the larger the number of variables, the greater the gain of efficient methodsMany scales: typical spatial and/or temporal sizes.

A framework of efficient iterative methods for solving problems with many variables and many scales.

8

Basic Concepts: Local vs. Global processing.

Imagine a large number of soldiers who need to be arranged in a straight line and at equal distances from each other. The two soldiers at the ends of the line are fixed. Suppose we number the soldiers 0 to N , and that the length of the entire line is L.

9

Initial Position

10

Final Position

11

Global processing. Let soldier number j stand on the line connecting soldier 0 to soldier N at a distance jL/N from soldier number 0.

13

Global processing. Let soldier number j stand on the line connecting soldier 0 to soldier N at a distance jL/N from soldier number 0.

This method solves the problem directly, but it requires a high degree of sophistication: recognition of the extreme soldiers and some pretty fancy arithmetic.

14

A step in the right direction

16

Slow convergence

18

Fast convergence

20

Slow convergence

21

Local solution: damping

22


23


24


25

The multiscale idea: Employ the local processing with simple arithmetic. But do this on all the different scales.

27

Large scale

28

Large scale

29

Intermediate scale

30

Intermediate scale

31

Small scale


How much work do we save?

Jacobi’s method requires about N2 iterations and N2 *N = N3 operations to improve the accuracy by an order of magnitude.

The multiscale approach solves the problem in about Log2(N) iterations (whistle blows) and only about N operations.

Example: for N = 1000 we require about:10 iterations and 1000 operations

instead of about 1,000,000 iterations and 1,000,000,000 operations

34

How important is computational efficiency?Suppose that we have three different algorithms for a given problem, with different computational complexities for input size N :

Algorithm 1: 106 N operationsAlgorithm 2: 103 N2 operationsAlgorithm 3: N3 operations

Suppose that the problem size, N, is such that Algorithm 1 requires one second. How long do the others require?

35

Algorithm 3O(N3)

Algorithm 2O(N2)

Algorithm 1 O(N)

NComputer

Speed(ops/sec)

0.000001 sec 0.001 sec 1 sec 1 1M (~106)(1980’s)

1 sec 1 sec 1 sec 1K 1G (~109)(1990’s)

12 days 17 min 1 sec 1M 1T (~1012)(2000’s)

31,710 years 12 days 1 sec 1G 1P (~1015)(2010’s)

Stronger Computers

Greater Advantage of Efficient Algorithms!

How would you solvea finite element system A u = f

with 1012 unknowns?

36

Choose one of:

• If the inverse matrix B = A-1 is already available, by computing u = A-1 f ?

• Apply a multigrid solver ?• Some other technique ?


A thought experimentThought Experiment

1012 elements/nodesassume that every entity must contribute to any other (as is typical for an elliptic problem)equivalent to either

multiplication with inverse matrixn × n Coulomb interaction

Results in 1024 data movements

each 1 NanoJoule (optimistic)

Together: 1015 Joule

37

!"#$%"&'$()*!+,&--."/012&"+

3"405/6++(+,)*+0&--."/012&"+

!"#%1"&'$(7)*+,&--."/012&"+

Figure 5: With new scaling rules and massive growth in parallelism, data locality is increasinglyimportant. This diagram shows the cost of a double-precision multiply-add at di↵erent levels ofthe memory hierarchy (from the cost of performing the flop to the movement of data operandsfrom registers, di↵erent distances on-chip, and distances o↵-chip.) The model or FPU and registeraccess is based on the Tensilica LX2 core energy model, the energy consumed for cross-chip wiresuses the Orion2 power model which is based on Balfour’s model [6], memory access is based on theJEDEC DDRx memory roadmap, and cross-system is based on projections for VCSEL-based opticaltransceivers.

2.4 New Communication and Synchronization Mechanisms

Increased use of non-conventional (non-cache-coherent) methods for interprocessor communicationon chip (e.g. GPU Warps and Intel SCC inter-processor hardware message queues) suggests newapproaches are needed for expressing data locality so that runtime systems can manage data move-ment instead of increasing the burden on the application programmer.

Configurable Caches: Increased use of configurable caches (automatic vs. user-managed) toexplicitly manage vertical data movement. Motivates new APIs for making e↵ective use ofthese new hardware locality management features in a cross-platform manner.

Coordinated Power Management: Local power management decisions (such as Intel/Sandybridgethermal-throttling of FPU clock rates) will result in system-wide performance inhomogeneity.This motivates systemwide coordinated power management.

Energy Aware Data Movement/Load-Balancing: New sources of transient load-imbalancesare increasing (such as software-based fault resilience mechanisms, and localized power man-agement), but rebalancing the load may be more costly than tolerating it. Balancing the costof fixing a load-imbalance vs. tolerating its continuation requires an API for expressing theenergy-costs of data movement to the application and runtime.

ASCR Exascale Programming Challenges Workshop 14

12

from: Exascale Programming Challenges, Report of the 2011 Workshop on Exascale Programming Challenges, Marina del Rey, July 27-29, 2011

277 GWh or240 Kilotons TNT

the picture shows the Badger-Explosionof 1953 with 23 Kilotons TNT

Source: Wikipedia

Büro

für G

esta

ltung

Wan

gler

& A

bele

04.

Apr

il 20

11

Optimal Algorithms for PDE - Ulrich Rüde 38

#Cores Coarse Grid Unkn (x 106) Crse Grd Its Tme to soln128 1536 535 15 5,64

64256 3072 1070 20 5,66512 6144 2142 25 5,69

1024 12288 4286 30 5,712028 24576 8577 45 5,754096 49152 17158 60 5,928192 98304 34326 70 5,86

16384 196608 68669 90 5,9132768 393216 137355 105 6,1765536 786432 274743 115 6,41

131072 1572864 549554 145 6,42262144 3145728 1099276 280 6,82294912 294912 824365 110 3,80

Parallel scalability of scalar elliptic problem in 3Ddiscretized by tetrahedral finite elements.

Times to solution on Jugene, Jülich

Largest problem solved:1.099 x 1012 DOFS (6 trillion tetrahedra) on 262000 processors.

B. Bergen, F. Hülsemann, U. Rüde, G. Wellein: ISC Award 2006: „Is 1.7× 1010 unknowns the largest finite element system that can be solved today?“, SuperComputing, Nov‘ 2005.Gmeiner, UR, Stengel, Waluga, Wohlmuth: Towards Textbook Efficiency for Parallel Multigrid, submitted.

Solution with Parallel Multigrid


What do we learn?Sometimes what looks like an obvious solution (multiplying with the inverse matrix) is a very bad idea.computational complexity is importantFor very large problems an algorithm with „optimal“ complexity (that is, the work grows only linearly with problem size) wins dramatically against an algorithm whose complexity is quadratic.Energy consumption will become a major concern for large scale computing

39

What can multigrid achieve?• Solve elliptic PDE in asymptotically optimal complexity• Poisson‘s eqn in 2D: <30 FLOPs per unknown

– In terms of flops cheaper than computing

• Solve FE problems (linear, scalar, elliptic PDE) with more than 1012 tetrahedral elements– on a massively parallel supercomputer– reminder: 1012 ≠ 2 ✕ 106

40

ui,j = sin(xi) sin(yj)


Multigrid is not the answer to everything!

+ Sparse, low dimension, large, stiff, elliptic PDE, geometric, smooth long-range effects, structured, isotropic, smoothly varying coefficients, symmetric positive definite.

~ Nonlinear, disordered, anisotropic, discontinuous coefficients, singular-perturbation and non-elliptic PDE, PDE systems, non-symmetric, indefinite, non-deterministic.

- Dense, high-dimensional, small, single-scale.

42

The catch: in less trivial problems, we cannot construct appropriate equations on the large scales without first propagating information from the small scales. Skill in developing efficient multilevel algorithms is required for:1. Choosing a good local iteration.2. Choosing appropriate coarse-scale variables. 3. Choosing inter-scale transfer operators. 4. Constructing coarse-scale approximations to the fine-scale problem.


Analysis of the local iterative process

We can show that the eigenfunctions of the iterative process are sine-shaped functions.

This means that we can analyze the processes by examining sine functions of different frequencies independently.

45

Analysis of the iterative processMatrix representation:

( ) ( )1−= ii Sxxwhere

⎥⎥⎥⎥⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢⎢⎢⎢⎢

⎣

⎡

=

01101

101

101101

10

21

………S

46

This matrix S has linearly independent eigenvectors, , and corresponding real eigenvalues, :kλ

Since span the space , any initial configuration of the soldiers can be written as a linear combination:

.k kkλ=S v v

( ) ∑−

=

=1

1

0N

k

kkc vx

with some coefficients, ck.

kv

kv 1−ℜN

1−N

47

Observation: the eigenvectors and eigenvalues of the matrix S are given by

with k =1, …, N –1.Proof: Using the trigonometric identity,

and the fact that , we obtain by substitution,

{ }

,cos

,1,,1,sin

⎟⎠

⎞⎜⎝

⎛=

−=⎟⎠

⎞⎜⎝

⎛==

Nk

NjNjkv

k

kj

k

πλ

π…v

( ) ( ) ,sincos1sin1sin21

Njk

Nk

Nkj

Nkj ππππ

=⎥⎦

⎤⎢⎣

⎡ ++

−

0sin0sin == πk

kk vvS λ=

48

Hence, we obtain after m iterations:( ) ( ) ( )

( ) ∑∑ ===

=== −−

k

kmkk

k

kk

mm

mmm

cc vvSxSxSSxx

λ0

221

…

Conclusion:

That is, the iteration converges if the spectral radius, , of the iteration matrix, S, is smaller than 1.

( ) 1,,1,10lim −=<→∞→

Nkif km

m…λx

ρ

49

Note: since , the method converges. But, for some eigenvectors, is close to 1, so convergence is slow. In particular, for , we have,

For k =1 we obtain

Hence, O(N 2) iterations are required to reduce such an error by an order of magnitude.

1<kλ

kλ

1/ <<Nkπ2

211cos ⎟

⎠

⎞⎜⎝

⎛−≈⎟⎠

⎞⎜⎝

⎛Nk

Nk ππ

2

21

1 211

⎟⎠

⎞⎜⎝

⎛−

≈⎥⎦

⎤⎢⎣

⎡⎟⎠

⎞⎜⎝

⎛−≈ Nm

mm e

N

ππ

λ

50

Note that convergence is also slow forThis can be overcome by damping:

where is a parameter. Then,

where

Note: are eigenvectors of . The corresponding eigenvalues are now

For we have convergence, .

.1/ ≈Nk

( ) ( ) ( ) ( )( ),21)1( 1

111

1 −+

−−

− ++−= ij

ij

ij

ij xxxx ωω

ω( ) ( ),1−= ii xSx ω

( ) .1 SIS ωωω +−=

kv ωS

( ) ( ).111 kkk λωωλωλω −−=+−=

10 ≤<ω( ) 1<ωλk

51

Definition: Eigenvectors with

are called smooth (low-frequency). Those for k large are called rough or oscillatory (high-frequency).

Note that for rough eigenvectors, .

Problem: Find which yields optimal convergence for the set of rough modes for arbitrary N, i.e.,

Note that can be any number in (-1, 0]. What is then the bound on the convergence factor, , of the rough modes?

kv 2/1 Nk <≤

NkN <≤2/0≤kλ

10 <<ω

( ) min!maxsup:2

=<≤

ωλω kNkNN

kλ

( )ωλk

52

For the smoothest eigenfunction, , we have shown that O(N 2) iterations are required to reduce the error by an order of magnitude.

On the other hand, highly-oscillatory eigenfunctions require only O(1) iterations (when damping is used). These are eigenfunctions whose wave-lengths are on a scale of just a few variables (“soldiers” in our example).

53

Err

or

Iterations


The mathematical background is the solution of (elliptic) PDEPoisson Equation in a simple domain (unit square)

with appropriate boundary conditionsmodels many different physical processes, e.g.

electrostatic potential, gravitation potentialheat distribution, diffusionporous media flow, ...

is a building block in many CS&E methodspressure correction for flow solverslong range interactions in molecular dynamics

Multigrid for Elliptic PDE in 2D

54

55

2D Model ProblemFind u which satisfies:

This is the 2D Poisson equation, with Dirichlet boundary conditions. It is an elliptic partial differential equation which appears is many models.

(4)

ui,j =

1

4

(ui�1,j + ui+1,j + ui,j�1 + ui,j+1) for i, j = 1...n


Grid of wires (as replacement of a 2D plate)

Temperature at each node = Average of neighboring temperatures

Boundary values given.

This is indirectly a discretization of the Laplace- or Poisson-equation.)

ui,j ui+1,jui-1,j

ui,j+1

ui,j-1

Example for Use of an Iterative Method

More about matrix representation later.

57


4

4

4

4

4

4

-1

-1-1

-1

-1-1

-1-1

-1

-1-1

-1

-1

-1

-1

-1

-1

-1

-1

-1

u11

u1n

u21

unn

u2n

=

Matrix representation

58


Please note: the n × n grid is not a matrix. The u-values in the grid are the unknowns. Use well-defined terms: Grid or Mesh (Gitter) versus MatrixThe matrix has size n2 × n2 .The right hand side contains

given „boundary values“ and „Heat sources“ at the nodes of the grid (if the problem is a bit generalized).

We could now assemble the matrix in a suitable sparse matrix data structure, but here it is more efficient (and simpler) to use a „matrix-free“ approachImplementation of an iterative method

Towards the Linear System

59


Matrix free implementation of the Gauss-Seidel Method

w=1: Gauss-Seidel; w>1: SOR

Iterative Methods

<(

60


Exact Solution (of PDE) Boundary values to start the iteration

Graphical Illustration (Visualization)

61


View of (approximate) solution after first and second Gauss-Seidel Iteration

Visualization of Iterations

62


view of appproximate solution after 10 and after 100 iterations.

Visualization of Iterations (cont‘d)

63


Visualization of Convergence

after 2 iterations

after 1 iterations

after 10 iterations

before any iteration

64


Visualization of Convergence

after 100 iterations after 1000 iterations

overlayed with true solution

overlayed with true solution

65


View of iterative solution after 1000 Gauss-Seidel-IterationsThe Gauss-Seidel-method needs for this problem O(N) iterations, where N(=n2) is the number of unknowns (grid points).

Example: Iterations -- Visualisation

66


Gauss-Seidel-Method:When N iterations are required, each of which needs O(N) operations then the total cost is O(N2) operationsN=n2, n number of grid points in one mesh dimensions

SOR-method: only n=N1/2 iterations necessary, if the relaxation parameter ω>1 is chosen optimally.

For the model problem this can be shown mathmatically, see e.g. Stoer/Bulirsch.

Be clear that there are different types of error:Rounding errors (here of secondary importance)Iteration error: stopping the iteration after finitely many stepsDiscretisation: even after ∞ many iterationenan error relative to the partial differential equation remains (discretization error: grid vs. plate)

Iterative Methods: Discussion of Example

67


Choose ordering of grid traversals more inteligently. However this unfortunately only helps substantially, when the problem at hand has a „preferred direction. (Physically this corresponds to convection rather than diffusion) Search systematically for equations/unknows which are far from equlibrium (large residual) and iterate preferably on those (search algorithm often too expensive)Search for better initial guess:

interpolate boundary values Start on coarser grid, compute an approximate solution there, and interpolate to finer grid. (Cascadic algorithm, Nested iteration)

Ideas for improvement (1)

68


Acceleration possible?:Store several iterates xi, xi+1, xi+2,... and search for better solution by taking (linear) combinations -> leads to the method of conjugate gradients with preconditioning or more generally to Krylov-space methods such as (GMRES, etc.)Use coarser grids to accelerate fine grid iteration process: Multigrid methods.....

Many books, e.g.: Wolfgang Hackbusch: Iterative Lösung großer schwachbesetzter Gleichungssysteme, Teubner, Stuttgart, 2. Auflage (1993)

Ideas for accelerating the algorithms

69

70

Practical conclusion:1. A smooth error can be approximated well on a coarser

grid.2. A coarser grid implies less variables, hence less

computation.3. On the coarser grid the error is no longer as smooth

relative to the grid, so relaxation may once again be efficient.

Key Observation re-worded: Relaxation cannot be generally efficient for reducing the error (i.e., the difference field ). But relaxation may be extremely efficient for smoothing the error relative to the grid.

71Hierarchical Grids

72

Aliasing(a)

(b)

Figure 5. Aliasing. Red disks correspond to the

coarse grid: (a) v (3) and (b) –v(13).

73

The solution, uh, depends only on the equation and the data, so it is not, of course, smoothed by relaxation. Only the error is smoothed. Hence, we reformulate our problem:Denote

Recall

Subtract from both sides, and use the linearity of Lh to obtain:

It is this equation that we shall approximate on the coarse grids.

.~hhh uuv −=

.hhh fuL =

hhuL ~

hhhhhh ruLfvL ≡−= ~

74

As we have seen, we need to smooth the error on the fine grid first, and only then solve the coarse-grid problem. Hence, we need two types of intergrid transfer operations:

1. A Restriction (fine-to-coarse) operator:2. A Prolongation (coarse-to-fine) operator:

For restriction we can often use simple injection, but full-weighted (local averaging) transfers are preferable.For prolongation, linear interpolation (bi-linear in 2D) is simple and usually effective.

.HhI.hHI

75

Two-grid Algorithm² Relax several times on grid h, obtaining

with a smooth corresponding error.² Calculate the residual:² Solve approximate error-equation on the

coarse grid:

² Interpolate and add correction:

² Relax again on grid h.

Multi-grid is obtained by recursion.

.~hhhh uLfr −=

.hHh

HHH rIfvL ≡=

.~~ HhH

hh vIuu +←

76

Multi-grid Cycle

( )

( )

( )

timesonRelaxCorrect

timesonRelaxCorrect

timesonRelaxCorrect

Solve

SettimesonRelax

SettimesonRelax

SettimesonRelax

2

22

2222

424

222

444

848

44

844484

81

444

422242

41

222

2221

0,

0,

0,

vfuLuIuu

vfuLuIuuvfuLuIuu

fuL

uuLfIfvfuLuuLfIf

vfuLuuLfIf

vfuL

hhh

hhh

hh

hhh

hhh

hh

hhh

hhh

hh

MhMhMh

hhhhhh

h

hhh

hhhhhh

h

hhh

hhhhhh

h

hhh

=

+←

=

+←

=

+←

=−

=−=

=

=−=

=

=−=

=

( )21,ννV

77

Remarks:1. Simple recursion yields a V cycle. More generally, we

can choose a cycle index , and define a –cycle recursively as follows: Relax; transfer to next coarser grid; perform cycles; interpolate and correct; Relax. (On the coarser grid define the cycle as an exact solution).

2. The best number of pre-relaxation + post-relaxation sweeps is normally 2 or 3.

3. The boundary conditions for all coarse-grid problems is zero (because the coarse-grid variable is the error). The initial guess for the coarse-grid solution must be zero.

ã

ã

ã

79

Multigrid vs. Relaxation

Iterations

Err

or

Solution of Poisson‘s eqn

80

Dirichlet boundary conditions

81

After one step of Gauss-Seidel

82

File: c/u.2

enorm = 4.47048252627281e-02

0

10

20

0

10

20

30

0

5

10

12

On the next coarser grid, after two Gauss-Seidel steps

83

On the next coarser grid, after V-cycle reursion

84

File: c/c.4/u.26

enorm = 1.38371516388512e-01

0

10

0

10

0

5

8

On finest grid, after coarse grid correction

85

File: c/u.29

enorm = 9.11869751280490e-02

0

10

20

0

10

20

30

0

5

10

12

87

The cost of the V cycle in terms of computation and storage is given by

Where d is the dimension and N is the number of variables on the finest grid. Here, c is some constant that depends on the discrete operators and the number of relaxation sweeps per level.Thus, for a 2D problem, the V-cycle with one pre-relaxation and one post-relaxation requires approximately the same number of operations as 3-5 relaxation sweeps.

The convergence rate of a V-cylce is <1 and bounded away from 1 for a wide class of elliptic PDE independent of the mesh size. In practice, we try (and often succeed) to achieve a convergence rate of µ ≈ 0.1.

88

ConclusionsMultigrid is a framework of efficient algorithms for large scale scientific computingUsing a hierarchy of resolutions is essentialInterplay between relaxation and coarse-grid-correctionasymptotically optimal

convergence rates (independent of system size)cost complexity (because of geometric series)

What remains to doextend the idea to real life problems„asymptotically optimal“ does not yet mean the program runs fastparallelization

Thank you for your attention!

introduction to multigrid methods - fau€¦ · multigrid methods, and their various multiscale...

Documents