Design of parallel algorithms

Linear equations

Jari Porras

Linear equations

a0,0x0 + ... + a0,n-1xn-1 = b0

...

an-1,0x0 + ... + an-1,n-1xn-1 = bn-1

• Ax = b• Usually solved in 2 stages

– reduce into upper triangular system Ux = y– back-substitution xn-1 ... x0

• Gaussian elimination

Gaussian elimination

Gaussian elimination

Gaussian elimination

• Gaussian elimination requires– n2/2 divisions (line 6)– (n3/3) – (n2/2) subtractions and multiplications

(line 12)

• Sequential run time 2n3/3

• How is the gausian elimination peformed in parallel ?

Parallel Gaussian elimination

• Row/column striping vs. chackerboarding ?

• Block vs cyclic striped ?

• Number of processors p < n, p = n, p > n

• Active processors ?

• Required steps ?

Analysis

• 1st step– kth iteration requires n – k – 1 divisions at

processor Pk

• 2nd step– (ts + tw(n – k – 1)) log n time on hypercube

• 3rd step– kth iteration requires n – k – 1 multiplications

and subtractions at all processors Pi

• Tp = 3/2 n(n-1) + tsnlog n + ½ twn(n-1)logn

Analysis

• Not cost-optimal since pTp = (n3logn)

• What is the main reason ?– Inefficient parallelization ?– What could be done ?

Analysis

• Pipelined operation– all n steps are executed in parallel– last step starts in nth step and is completed in

constant time (changes only the bottm right corner element)

(n) steps– Each step takes O(n) time– Thus parallel run time O(n2) and cost (n3)– Cost-optimal !!

p < n ?

• Block striping– several rows / processor

• Does the activity change ?– Block vs. cyclic striping

Analysis

• With block striping– processor with all rows belonging to the active

part performs (n – k – 1)n/p multiplications and subtractions

– if the pipelined version is used the number of arithmetic operations (2(n-k-1)n/p) is higher than number of words communicated (n-k-1)

– computation dominates– parallel run time n3/p

Checkeboard partitioning

• Use n x n mesh

• Same approach as before, but– requires two broadcasts (rowwise and

columnwise)– Analyse the cost-optimality

• How about the pipelining ?

Pipelined checkerboard

Pipelined checkerboard

p < n2

• Map matrix onto p x p mesh by usin block checkerboard partitioning

• Remember the effect of active processors !!

• Number of multiplications and subtractions n2/p and n/ p word communication– computation dominates !

Partial pivoting

• Basic algorithm fails if any elemnt on diagonal is zero

• Partial pivoting helps– select row that has the largest element on the

wanted column and exchange rows

• What is the effect to the partitioning strategy ?

• How about pipelining

Back-substitution

• The second stage of solving linear equations

• Back-substitution is used to determine vector x

• Complexity n2

– use partitioning scheme that is suitable for Gaussian elimination

Back substitution