Methodology

easy but important

1

ToC

1. What is performance evaluation about ?2. Metrics, Load and Goals

3. Hidden Factors4. The Scientific Method

5. Patterns

2

What is Performance Evaluation ?

LoadYou need to define the load under which your system operatesMake the difference between

Intensity of the load (e.g. nb jobs per second)Nature of the load

Statistical details that may matter: e.g. job sizes are heavy tailed or not

Benchmarks are artificial load generators; we will play with one of them

MetricDefine a metric; examples

Response timePower consumptionThroughput

Define operational conditions under which metric is measured (« Viewpoint », see Chapter 11)

Compare Windows vs Linux

6

7

Syscall Benchmark

8

Memory Access Time

9

Ghostscript

Metrics are often Multidimensional

10

A and D are non dominated

11

Know your goals

A1 and A3 are comparisons, A2 is an absolute statement

E2 is an engineering rule

3. Hidden FactorsFactor: an element that may impact the performance

(desired factors): intensity of load, number of servers(nuisance factors): time of the day, presence of denial of service attack

13

TCP Throughput Increases with Mobility

TCP Throughput Decreases with Mobility

Why were we fooled ?Hidden factor had a more important role than the factor we were interested in

We interpreted correlation as causality

Need to be aware of all factors and incorporate in the analysis

Or randomize experiment to reduce impact of hidden factors

Simpson’s ParadoxA well known phenomenon -- Special case of Hidden Factor paradox when metric is success rate and factors are discrete

Berkeley Sex Case 1973 (source: wikipedia)

18

Take Home MessagePitfall number 1 is the presence of hidden factorAny study is susceptible to itEasy for opponents to find

4. Be ScientificJoe measures performance of his Wireless Shop:

what would you conclude ?

Scientific MethodJoe buys 2 more Access Points

improvement ?

21

Before After

Scientific MethodA conclusion can only proven to be wrongDo not draw conclusions unless the experiment was designed to test the statement

Measurement 1 suggested that the wireless network was congested, but the experiment was not designed to test this statement

Joe should: design an experiment to validate:H1: “the wireless network is the bottleneck”

for example: measure the number of collisions / packet lossresult: collision · 1%; conclusion: H1 is not validhypothesis H2: the server is saturatedexperiment: measure memory utilization : result ¼ 100%

22

Performance After Doubling Server Memory

23

Example from Nitin Vaidya, Mobicom 2000 Tutorial, slides 298-299

24

25

26

27

Use of Scientific Method

Recognize a fact

Pose a hypothesis

Verify the hypothesis on simulations / measurements designed to test it

TCP throughput may increase with mobility(1) Duration of link failure period is impacted by speed(2) It has a negative impact on TCP throughputDo more simulationsmeasure distrib of link failure period

Verify (1) and (2)How ?

28

Is ATM-ABR better than ATM-UBR ?

29

Take Home MessageYou should not conclude from an experiment without trying to invalidate the conclusion

(Popper, 1934): you should alternate between the roles ofProponentAdversary

5. Patterns

These are common traits found in different situationsKnowing some of them may save a lot of time

Bottlenecks may be your enemyBottlenecks are like non invited people at a party – they may impose their agenda

Previous example: what we are measuring is the bottleneck, not the intended factor

33

Bottlenecks are Your Friends

Bottlenecks are Your FriendsSimplify your life, analyze bottlenecks !In many cases, you may ignore the rest

35

Behind a Bottleneck May Hide Another Bottleneck

37

i’

i”

i

i”

i’

i”

i

i”

38

39

Congestion CollapseDefinition: Offered load increases, work done decreasesFrequent in complex systemsMay be due to

cost per job increases with loadImpatienceRejection of jobs before completion

Designer must do something to avoid congestion collapseEg. Admission control in Apache serversEg. TCP congestion control

Analyst must look for congestion collapse

40

Sources use TCP (= fair scheduling). Increase capacity of link 5 to 100 kb/s; what happens ?

41

Competition Side EffectSystem balances resources according to some schedulingApparent paradox: put more resources, some get less

42

No TCP, users send as much canIncrease capacity of link 2 from 10 to 1000 kb/s

43

10 kb/s

Competition Side EffectApparent paradox: put more resources, all get less

44

Museum Audio Guide

45

Low speed USB connections at docking stationHigh speed

Latent Congestion CollapseSystem is susceptible to congestion collapseLow speed access prevents congestion collapseAdding resources reveals congestion collapse

46

Take Home MessageWatch for patterns, they are very frequent

BottlenecksCongestion collapseCompetition side effectsLatent Congestion collapse

47

Now it’s your turn…

48