parallel programming: speedups and amdahl’s...

mjb – March 15, 2020

1

Computer Graphics

Parallel Programming:Speedups and Amdahl’s law

speedups.and.amdahls.law.pptx

Mike [email protected]

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License


2

Computer Graphics

Definition of Speedup

1n

n

TSpeedupT

If you are using n processors, your Speedupn is:

nn

SpeedupEfficiencyn

And your Speedup Efficiencyn is:

where T1 is the execution time on one core and Tn is the execution time on n cores.Note that Speedupn should be > 1.

which could be as high as 1., but probably never will be.


3

Computer Graphics

However, Multicore is not a Free Lunch:Amdahl’s Law

1 1 1

(1 )n

parallel parallelnsequential parallel

TSpeedup F FT F Fn n

If you think of all the operations that a program needs to do as being divided between a fraction that is parallelizable and a fraction that isn’t (i.e., is stuck at being sequential), then Amdahl’s Law says:

If you put in n processors, you should get n times Speedup(and 100% Speedup Efficiency), right? Wrong!

There are always some fraction of the total operation that is inherently sequential and cannot be parallelized no matter what you do. This includes reading data, setting up calculations, control logic, storing results, etc.

This fraction can be reduced by deploying multiple processors. This fraction can’t.


4

Computer Graphics

A Visual Explanation of Amdahl’s Law

# of Cores

1 2 4

Exec

utio

n Ti

me

ParallelPortion

SequentialPortion

SequentialPortion

SequentialPortion

ParallelPortion Parallel

Portion

SequentialPortion

~∞

The Sequential Portion doesn’t go away, and it also doesn’t get any smaller. It just gets more and more dominant.

≈


5

Computer Graphics

Amdahl’s Law as a Function of Number of Processors and Fparallel

0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29# Processors

x Sp

eedu

p

60%

40%

90%

80%

20%

Fparallel :


6

Computer Graphics

Parallel Fraction

# processors

X Sp

eedu

p


7

Computer Graphics

Amdahl’s Law

1 1max lim 1n sequential parallel

Speedup SpeedupF F

Note that these fractions put an upper bound on how much benefit you will get from adding more processors:

Fparallel maxSpeedup

0.00 1.00

0.10 1.11

0.20 1.25

0.30 1.43

0.40 1.67

0.50 2.00

0.60 2.50

0.70 3.33

0.80 5.00

0.90 10.00

0.95 20.00

0.99 100.00


8

Computer Graphics

1 1

1 11 1 1

1 1

1 11 ( ) 111 1 1 1 ( 1) ( 1) ( 1)

n n n

n n

T T T T Tn T T T TT T T n nSF n n n n T n n T n Speedup

n n n n

Solving for F:

If you’ve got several (n,S) values, you can take the average (which is actually a least squares fit):

1 1

(1 )n

TS FT Fn

Amdahl’s law says:

1

1

2

, 2..( 1)

1

inii

iN

ii

T TnF i Nn T

FF

N

1 (1 ) 1F F nFFS n n

1 (1 )1 nFS n

1niT Tnote that when i=1,

You can also solve for Fparallel using Amdahl’s Law if youknow your speedup and the number of processors

Use this if you know the timing

Use this if you know the speedup


9

Computer Graphics

A More Optimistic Take on Amdahl’s Law:Gustafson’s Observation

Gustafson observed that as you increase the number of processors, you have a tendency to attack larger and larger versions of the problem. He also observed that when you use the same parallel program on larger datasets, the parallel fraction, Fp, increases.

Let P be the amount of time spent on the parallel portion of an original task and S spent on the serial portion. Then

orp

PFP S

p

p

P PFS

F

Without loss of generality, we can set P=1 so that, really, S is now a fraction of P. We now have:

1 p

p

FS

F

ParallelTime

SerialTime


10

Computer Graphics


' ''pP NPF

P S NP S

'

1pp

p

N NF FN S NF

We know that if we multiply the amount of data to process by N, then the amount of parallel work becomes NP. Surely the serial work must increase too, but we don’t know how much. Let’s say it doesn’t increase at all, so that we know we are getting an upper bound answer.

In that case, the new parallel fraction is:

And substituting for P (=1) and for S, we have:


11

Computer Graphics


If we tabulate this, we get a table of Fp’ values:


12

Computer Graphics


Or, graphing it:


13

Computer Graphics


We can also turn Fp’ into a Maximum Speedup:

parallel programming: speedups and amdahl’s...

Documents