1

Case Study in Computational Science & Engineering - Lecture 5

Solution of Sparse Linear Systems

• Direct Methods– Systematic transformation of system of equations

into equivalent systems, until the unknown variables are easily solved for.

• Iterative methods– Starting with an initial “guess” for the unknown

vector, successively “improve” the guess, until it is “sufficiently” close to the solution.

2

Case Study in Computational Science & Engineering - Lecture 5

Direct Solution of Linear SystemsGaussian Elimination

2 3 136

3 2 2 15

1 2 3

1 2 3

1 2 3

x x xx x xx x x

x x xx x xx x x

1 2 3

1 2 3

1 2 3

15 0 5 6 56

3 2 2 15

. . .

x x xx xx x

1 2 3

2 3

2 3

15 0 5 6 50 5 0 5 0 52 5 0 5 4 5

. . .

. . .

. . .

x x xx xx x

1 2 3

2 3

2 3

15 0 5 6 51

2 5 0 5 4 5

. . .

. . .

x x xx x

x

1 2 3

2 3

3

15 0 5 6 51

2 2

. . . x x xx x

x

1 2 3

2 3

3

15 0 5 6 511

. . .

div by 2

*(-1)

*(-3)

• Unknowns solved by back-substitution after Gaussian Elimination

3

Case Study in Computational Science & Engineering - Lecture 5

LU Decomposition• More efficient than Gaussian Eimination when solving

many systems with the same coefficient matrix.• First A is decomposed into product: A = LU

• To solve linear system Ax=b, we need to solve (LU)x=b• Let z=Ux; we have L(Ux)=b, or Lz=b. This can be solved

for z by forward-substitution. • Since Ux=z, and z is now known, we can solve for x by

back-substitution.

A A AA A AA A A

11 12 13

21 22 23

31 32 33

LL LL L L

11

21 22

31 32 33

0 00

10 10 0 1

12 13

23

U UU=

4

Case Study in Computational Science & Engineering - Lecture 5

Cholesky Factorization• If A is symmetric and positive definite , it can

be factored in the form

• Cholesky factorization requires only around half as many arithmetic operations as LU decomposition.

• The forward and back-substitution process is the same as with LU decomposition.

A A AA A AA A A

11 12 13

21 22 23

31 32 33

LL LL L L

11

21 22

31 32 33

0 00

10 10 0 1

21 31

32

L LL=

(x Ax > 0) T

A = LL T

5

Case Study in Computational Science & Engineering - Lecture 5

Sparse Linear Systems• A significant fraction of matrix elements are known to be

zero, e.g. matrix arising from a finite-difference discretization of a PDE:

• At most 5 non-zero elements in any row of the matrix, irrespective of the size of the matrix (number of grid points).

• Sparse matrix is represented in some compact form that keeps information about the non-zero elements.

1 2 3

4 5 6

1 2 3 4 5 61 4 -1 0 -1 0 02 -1 4 -1 0 -1 03 0 -1 4 0 -1 -14 -1 0 0 4 -1 05 0 -1 0 -1 4 -16 0 0 -1 0 -1 4

6

Case Study in Computational Science & Engineering - Lecture 5

Sparse Linear Systems• For a 100 by 100 grid, with a finite difference discretization

using a 5-point stencil, less than .05% of the matrix elements are non-zero.

n1

n2

1n2

n2

Physical nxn Grid

Resulting n2 n2x sparse matrix

7

Case Study in Computational Science & Engineering - Lecture 5

Compressed Sparse Row Format

• A commonly used representation for sparse matrices:

0 1 2 3 4 50 4 -1 0 -1 0 01 -1 4 -1 0 -1 02 0 -1 4 0 0 -13 -1 0 0 4 -1 04 0 -1 0 -1 4 -15 0 0 -1 0 -1 4

rb

a

col 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

0 3 7 10 13 17 20

0 1 3 0 1 2 4 1 2 5 0 3 4 1 3 4 5 2 4 5

4 -1 -1 -1 4 -1 -1 -1 4 -1 -1 4 -1 -1 -1 4 -1 -1 -1 4

for (i = 0; i<n; i++) for(j=0;j<n;j++) y[i] += a[i][j]*x[j];

Dense MV Multiply

for (i = 0; i<n; i++)

for(j=rb[i];j<rb[i+1];j++)

y[i] += a[j]*x[col[j]];

Sparse MV Multiply

8

Case Study in Computational Science & Engineering - Lecture 5

Fill-in Non-Zeros • During solution of sparse linear system (by GE or LU or

Cholesky), row-updates often result in creation of non-zero entries that were originally zero.

• Row updates using row-1 result in fill-in non-zeros (F).

X X 0 XX X 0 00 0 X XX 0 X X

X X 0 XX X 0 F0 0 X XX F X X

9

Case Study in Computational Science & Engineering - Lecture 5

Effect of reordering on fill-in • Re-ordering the equations (rows) or unknowns (columns)

can result in significant change in the number of fill-in non-zeros, and hence time for matrix factorization.

X X X XX X 0 0X 0 X 0X 0 0 X

X X X XX X F FX F X FX F F X

X 0 0 X0 X 0 X0 0 X XX X X X

X 0 0 X0 X 0 X0 0 X XX X X X

Fill-inwith GE

No fill-inwith GE

Reorder rows/cols

10

Case Study in Computational Science & Engineering - Lecture 5

Associated graph of matrix • A graph-based view of matrix’s sparsity structure is extremely

useful in generating low-fill re-orderings.

• The associated graph of a symmetric sparse matrix has a vertex corresponding to each row/col. of matrix, and an edge corresponding to each non-zero matrix entry.

1 2 3 4 5 61 X X X X2 X X X X3 X X X X4 X X5 X X6 X X

32

6 5

4

1

11

Case Study in Computational Science & Engineering - Lecture 5

Fill-in and graph transformation • Row-i updates row-j, j>i iff Aji is non-zero; in the

associated graph a matrix non-zero corresponds to an edge.• Row-update(i->j) could cause fill-in non-zero Ajk

corresponding to all non-zeros Aik.

• After all updates from row-i, all neighbors of vertex i in the associated graph form a clique.

i j k l

i X X X X

j X X F F

k X X

l X X

jiji

ll

kk

12

Case Study in Computational Science & Engineering - Lecture 5

Fill-in and graph transformation • Each row’s effect on fill-in generation is captured by the

“clique” transformation on the associated graph.• The graph view is valuable in suggesting matrix re-

ordering approaches.

32

6 5

4

1

1 2 3 4 5 61 X X X X2 X X X X3 X X X X4 X X5 X X6 X X

1 2 3 4 5 61 X X X X2 X X X F X3 X X X F X4 X F F X5 X X6 X X

1 2 3 4 5 612 X X F X3 X X F X F4 F F X F5 X X6 X F F X

1 2 3 4 5 6123 X F X F4 F X F F5 X F X F6 F F F X

32

6 5

4

1

32

6 5

4

13

2

6 5

4

1

13

Case Study in Computational Science & Engineering - Lecture 5

Matrix re-ordering: Minimum Degree • Graph-based algorithm for generating low-fill re-ordering.• Matrix permutation is viewed as node-numbering problem

in associated graph.• Low-degree nodes are numbered early - so that they are

removed without adding many fill-in edges.

• For example, minimum-degree finds a no-fill ordering.

d=1 d=1

d=1

d=3

d=3 d=3

d=1 d=1

1

d=2

d=3 d=3

2 d=1

1

d=2

d=2 d=3

2 3

1

d=2

d=2 d=2

2 3

1

4

d=1 d=1

14

Case Study in Computational Science & Engineering - Lecture 5

Re-ordered matrix

32

6 5

4

1

1 2 3 4 5 61 X X X X2 X X X F X3 X X X F X F4 X F F X F F5 X F X F6 X F F F X

32

65

4

1 Old # New #1 42 53 64 15 36 2

1 2 3 4 5 61 X X2 X X3 X X4 X X X X5 X X X X6 X X X X

15

Case Study in Computational Science & Engineering - Lecture 5

Matrix re-ordering: Nested Dissection • Find a minimal vertex-separator to bisect associated graph;

number those nodes last; recursively apply to both halves.• Property: Given a numbering of nodes, fill-in Aij exists, j>i,

iff there is a path from i to j in graph using only lower numbered vertices.

• No fill-in edges between one half and other half of partition.

43 49

1-21

22-42

19

21

40

42

16

Case Study in Computational Science & Engineering - Lecture 5

Comparison of Ordering Schemes

Grid => 4x4 8x8 16x16 32x32 64x64 128x128 256x256Nest Dis 120 768 4500 25072 131904 659880 3,180260

Min Deg 100 654 4020 23172 133278 771088 4,438460BW Min 108 792 5936 45664 357568 2,828670 ***Natural 118 974 7966 64574 520318 *** ***

Number of non-zeros after fill-in

Grid => 4x4 8x8 16x16 32x32 64x64 128x128 256x256Nest Dis 77.3 77.7 84.8 134.6 504.8 3063.1 22807.8Min Deg 74.4 76.1 89.5 154.7 674.8 5076.8 48664.7BW Min 76.9 78.4 91.3 160.3 1006.0 16604.6 ***Natural 75.5 80.0 96.3 226.1 1844.8 *** ***

Sparse matrix factorization time