systematic evaluation of complex systems · systematic evaluation of complex systems...

1

Systematic Evaluation of Complex Systems

Acknowledgement: Parts of these slides are based on [Jai91]

Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems

2


§ Motivation: Analysis of TCP Congestion Control§ 2k - Factorial Designs§ 2kr - Factorial Designs with Replications§ 2k-p – Fractional Factorial Designs§ One Factor Experiments§ Two Factor Experiments§ Two Factor Experiments with Replications§ General Full Factorial Designs


3

Quick Reminder: TCP Congestion Control


Slow start

Additive increase, congestion avoidance

New slow start to new threshold, then linear increase

20=40/2

Reset CW to 1, new threshold = CW/2

Initial congestion threshold

Initial congestion window size

4

TCP Congestion Control Parameters

§ Performance affected by• Initial Congestion Window• Initial Congestion Window Threshold• Timeout• Enable Duplicate Acknowledgements• Size of TCP-Buffers• Maximum Segment Size (MSS)• …• MacOS 10.7: sysctl -a | grep tcp | wc –l => 79 Parameters• MacOS 10.10: sysctl -a | grep tcp | wc –l => 116 Parameters• MacOS 10.11: sysctl -a | grep tcp | wc –l => 120 Parameters• Some Boolean other numeric

§ For different scenarios, e.g.• Internet, LAN, DSL, Satellite, Congestion etc.

§ For different runs


5

Choosing Optimal Parameters

§ Choosing good parameters is a multi-criteria optimization problem§ Finding optimal:

• 79 parameter (each assumed to have 2 legitimate values only)• 5 scenarios• 32 runs• Requires 279 x 5 x 32 = 9.67 x 1025 simulations!

§ Combinatorial Explosion § Care must be taken in choice of examined settings!

→ Analysis of Variance (ANOVA)


6

Some Terminology

§ Response Variables:• Outcome of experiment with regards to a performance index

§ Primary Factors:• Freely chosen parameter that may influence the response variable

§ Secondary Factors:• Parameters whose influence is not of interest (e.g. fixed ones)

§ Levels:• Possible values of factors (e.g. true/false, 100Mb/1Gb/10Gb, 5s, …)

§ Replications:• Number of runs performed


7

More Terminology

§ Interaction:• In which experiment do factors A & B influence each other?


A1 A2

B1 3 5B2 6 8

A1 A2

B1 3 5B2 6 9

8

Rules for the Planning of Experiments

§ Identify & Control all important parameters

§ Isolate effects of different parameters• Do not changes too many parameters at once

§ Aggregate parameters changes reasonably• Do not check each parameter combination

§ Treat interactions• Check for parameters that influence each other (positive and negative!)

§ Regard variation of responses with fixed parameters• Perform multiple runs• Use confidence intervals


9




10

2k - Factorial Designs

§ Examine effects of • k factors F• with two levels• No replications• Single scenario

§ Which of the factors has the largest impact?• To get a feeling of the factors in the beginning of a study• To find a point to start further optimization• To reduce number of primary factors


11

2k - Factorial Designs – Approximation of Responses

§ Observation: Impact of factors often follows strictly monotone functions

§ Idea: Approximate effects with a linear function


Level 1 Level 2

12

2k - Factorial Designs – Example for a 22 design (I)

§ 2 Factors at two levels

§ Response variable with• - Estimated average throughput (Buffer = 15Kb, MSS = 250 Bytes)• - Impact by changing MSS• - Impact of different buffer sizes• - Impact due to interaction of MSS and Buffers (0 iff no interaction)


TCP ThroughputMSS = 100 Bytes MSS = 400 Bytes

Buffer = 10Kb 15 45

Buffer = 20Kb 25 75

q0

y = q0 + qAxA + qBxB + qABxAxB

qAqBqAB

xA =

(�1 if MSS = 100

1 if MSS = 400xB =

(�1 if Bu↵er = 10

1 if Bu↵er = 20

13

2k - Factorial Designs – Example for a 22 design (II)

§ Calculation of by linear equation system:

§ Influence of factor A twice as high as B§ Positive correlation between both factors


q0, qA, qB , qAB

y = 40 + 20xA + 10xB + 5xAxB

15 = y0 = q0 � qA � qB + qAB

45 = y1 = q0 + qA � qB � qAB

25 = y2 = q0 � qA + qB � qAB

75 = y3 = q0 + qA + qB + qAB

14

2k - Factorial Designs – Example for a 22 design (III)

§ Sign Table Method

§ Column AB is calculated by Column A * Column B§ Total is calculated sum of yj values with sign of row


I A B AB yj

+1 -1 -1 +1 15+1 +1 -1 -1 45+1 -1 +1 -1 25+1 +1 +1 +1 75160 80 40 20 Total40 20 10 5 Total / 4

15

2k - Factorial Designs – Estimating the Variation

§ Variance may be better indicator to determine variables with high impact

§ Variance of response variable:

§ Simplification: compare changes of variation (without normalizing)

§ This is by definition ( , vectors x are orthogonal)


q0 = y

SST =X

f2P(F )

SSf =X

f2P(F )

2kq2f = 2kX

f2P(F )

q2f

s2 =

P2n

i=1(yi � y)2

2n � 1

SST =2nX

i=1

(yi � y)2

16

2k - Factorial Designs – Example for a 22 design (IV)

§ Total variation in the example:

§ Quadratic influence of the factors:

→ Conclusion for the study: evaluate A further!


SST = SSA + SSB + SSAB

= 22q2A + 22q2B + 22q2AB

= 22202 + 22102 + 2252

= 1600 + 400 + 100 = 2100

influence of A =SSASST

=1600

2100= 76.2%

influence of B =SSBSST

=400

2100= 19.0%

17

Exercise: Estimating the impact in a 23 design

§ Which of the factors in the following measurements require further analysis?


A1 A2

C1 C2 C1 C2

B1 100 15 120 10

B2 40 30 20 50

18

Exercise: Solution

A B C AB AC BC ABC y!1 !1 !1 1 1 1 !1 100!1 !1 1 1 !1 !1 1 15!1 1 !1 !1 1 !1 1 40!1 1 1 !1 !1 1 !1 301 !1 !1 !1 !1 1 1 1201 !1 1 !1 1 !1 !1 101 1 !1 1 !1 !1 !1 201 1 1 1 1 1 1 5015 '105 '175 '15 15 215 65 Total

1.875 '13.125 '21.875 '1.875 1.875 26.875 8.125 Total4/48

3.515625 172.265625 478.515625 3.515625 3.515625 722.265625 66.01562528.125 1378.125 3828.125 28.125 28.125 5778.125 528.125 11596.875

0.2% 11.9% 33.0% 0.2% 0.2% 49.8% 4.6%


19




20

2kr - Factorial Designs with Replications

§ Examine effects of • k factors• with two levels• r replications• Single scenario

§ How sure can we be that the factors are correctly estimated?


21

2kr - Factorial Designs with Replications

§ Response variables where i represents the factorial combination and j reflects the run

§ Measurements influenced by errors § Thus

§ The errors for each parameter set are expected to sum to zero, thus

§ Estimation of and the influence of the factors as in 2k design but by using


yi,j

ei,j

q0 qP(F )

yi

yi,j = q0 +X

f2P(F )

qfxf + ei,j

yi =1

r

rX

j=1

yi,j

22

Variation in 2kr Designs

§ If we additionally assume that errors are statistically independent:

§ The quadratic impact is again

§ Replications also allow for estimation of errors done during estimation!


Impact of f =SSfSST

SST =X

f2P(F )

SSf + SSE =X

f2P(F )

2krq2f +2kX

i=1

rX

j=1

e2i,j

23

Confidence Intervals in 2kr Designs

§ Replications allow for assurance checks of effects§ Variance for confidence intervals:

§ Q: Why not ?§ As errors are assumed to be distributed independently:

§ Confidence intervals for :

§ Similar for responses


s2e =SSE

2k(r � 1)

2kr � 1

s2q0 = s2qA = s2qf =s2e2kr

qf

qf ± t[1�↵/2,2k(r�1)] · sf

yi

24

Assumptions on the errors

§ Assumptions about the errors• Statistically independent• Additive• Normally distributed• Constant standard deviation • Effects of factors are additive

§ Must be verified for every response variable!• Take system knowledge into account• Or use statistical Tests: Chi-Squared-Test, F-Test• Or use visual verification


25

Example for visual error verification

§ Consider the following measurements


A1 A2

B1 (15, 18, 12) (25, 28,19)

B2 (45, 48, 51) (75, 81, 75)

26

Calculated Effects and Variations


I A B AB y yx1 yx2 yx3 ex1 ex2 ex31 "1 "1 1 15.00&&&&&& 15 18 12 0 3 "31 "1 1 "1 48.00&&&&&& 45 48 51 "3 0 31 1 "1 "1 24.00&&&&&& 25 28 19 1 4 "51 1 1 1 77.00&&&&&& 75 75 81 "2 "2 4

165.00 39.00 85.00 19.00 Total41.25 9.75 21.25 4.75 Total7/74

SSE= 102SE2= 12.75

Effect 380.25 1806.25 90.25 2378.75 SQ0= 3.092Percent7of7Variation 15.99% 75.93% 3.79% Conf= 7.131

Confidence 16.88 28.38 11.88Intervals 2.62 14.12 "2.38

"0.2940321"3.10624980.89337968"1.52987271.5298727"0.29403213.10624976"1.5298727"3.10624981.5298727"4.34256853.10624976

"6&

"5&

"4&

"3&

"2&

"1&

0&

1&

2&

3&

4&

5&

0& 20& 40& 60& 80& 100&

"4.00&

"2.00&

0.00&

2.00&

4.00&

6.00&

"6& "4& "2& 0& 2& 4&

27

Plot Errors

§ Plot errors against projected y-value

§ Make sure: no direct relationship visible

→ Looks good!


-6

-5

-4

-3

-2

-1

0

1

2

3

4

5

0 50 100

28

Normal-Quantile-Quantile-Plot

§ Plot measured errors vs. where they are expected in a perfect normal distribution

§ Should follow the linear function y = x

§ Ensures errors follow a normal distribution

§ In Excel Speak: =NORMINV(RANG(P21,P$18:P$29,1)/(12+1),0,STABW(P$18:P$29))

→ Looks good!


-6,00

-4,00

-2,00

0,00

2,00

4,00

6,00

-5 0 5

29

Exercise: Discussion of a scenario

§ Take the following (hypothetical) performance measurements to determine if one of the operating systems is particularly well suited one of the scenarios.


TCP ThroughputWindows Linux

Scenario 1 (9.55, 9.33, 11.22) (20.58, 19.51, 21.28)

Scenario 2 (96.48, 97.99, 102.67) (395.39, 407.22, 366.43)

30

Exercise: Solution (1st Try)


I A B AB y yx1 yx2 yx3 ex1 ex2 ex31 "1 "1 1 10.00 9.55 9.33 11.13 "0.45 "0.67 1.131 "1 1 "1 99.05 96.48 97.99 102.67 "2.57 "1.06 3.631 1 "1 "1 20.46 20.58 19.51 21.28 0.12 "0.95 0.821 1 1 1 389.68 395.40 407.22 366.43 5.71 17.54 "23.25

519.31 301.21 458.15 280.06 Total129.83 75.30 114.54 70.01 Total7/74

SSE= 905.348SE2= 113.168

Effect 22681.87 52474.68 19607.94 95669.83 SQ0= 9.21284Percent7of7Variation 23.71% 54.85% 20.50% Conf= 21.2448

Confidence 96.55 135.78 91.26Intervals 54.06 93.29 48.77

41.1653.5

65.17@@@@@@@@@@@50.08

"30.00@

"25.00@

"20.00@

"15.00@

"10.00@

"5.00@

0.00@

5.00@

10.00@

15.00@

20.00@@"@@@@ @50.00@@@100.00@@@150.00@@@200.00@@@250.00@@@300.00@@@350.00@@@400.00@@@450.00@@

Error7vs.7y7

"30.00@"25.00@"20.00@"15.00@"10.00@"[email protected]@[email protected]@[email protected]@

"15@ "10@ "5@ 0@ 5@ 10@ 15@

Normal7QuanHle7QuanHle7Plot7

31



-30,00

-25,00

-20,00

-15,00

-10,00

-5,00

0,00

5,00

10,00

15,00

20,00 - 100,00 200,00 300,00 400,00 500,00

Error vs. y

32



-30,00

-25,00

-20,00

-15,00

-10,00

-5,00

0,00

5,00

10,00

15,00

20,00-15 -10 -5 0 5 10 15

Normal Quantile Quantile Plot

33

Exercise: Solution (2nd Try – Using logarithms)


I A B AB y yx1 yx2 yx3 ex1 ex2 ex31 "1 "1 1 1.00 0.98 0.97 1.05 "0.02 "0.03 0.051 "1 1 "1 2.00 1.98 1.99 2.01 "0.01 0.00 0.021 1 "1 "1 1.31 1.31 1.29 1.33 0.00 "0.02 0.021 1 1 1 2.59 2.6 2.61 2.56 0.01 0.02 "0.03

6.90 0.95 2.29 0.28 Total1.72 0.24 0.57 0.07 Total7/74 SSE= 0.01

SE2= 0S0= 0.02

Effect 0.22 1.31 0.02 1.56 Conf= 0.05Percent7of7Variation 14.36% 84.02% 1.25%

Confidence 0.29 0.63 0.12Intervals 0.18 0.52 0.02

"0.02"0.010.000.01"0.030.00"0.020.020.050.020.02"0.03

"0.04?"0.03?"0.02?"0.01?

0?0.01?0.02?0.03?0.04?0.05?0.06?

0? 0.5? 1? 1.5? 2? 2.5? 3?

Error7vs.7y7

0?

0.01?

0.02?

0.03?

0.04?

0.05?

34



-0,04

-0,03

-0,02

-0,01

0

0,01

0,02

0,03

0,04

0,05

0,06

0 0,5 1 1,5 2 2,5 3

Error vs. y

35



-0,04

-0,03

-0,02

-0,01

0

0,01

0,02

0,03

0,04

0,05

-0,04 -0,02 0,00 0,02 0,04 0,06

36

2kr Designs – Lessons learned

§ False assumptions lead to false correlations

§ But replications allow for verification of experiments§ Always perform F-Test or visual verification otherwise confidence

intervals are useless!§ Be sure to understand your system!


37




38

2k-p – Fractional Factorial Designs

§ Examine effects of • k factors• with two levels• without replications• Single scenario• But less experiments…

§ How can we save work?§ Idea: Neglect relationship between factors (e.g. )


qABCD

39

Example for a 2k-p Design

§ Example for a 23-1 Design:

§ Factors AB, AC, BC, and ABC are neglected!


I A B C yj

+1 -1 -1 +1 15+1 +1 -1 -1 45+1 -1 +1 -1 25+1 +1 +1 +1 75160 80 40 20 Total40 20 10 5 Total / 4

40

Requirements on 2k-p Designs

§ Values of must form orthogonal vector space, s.t.

• Sum of any column j equals 0

• Sum of product of any two columns j and g equals 0

• Sum of the squares of any column j is 2k-p


xi,j

2k�pX

i=1

xi,j = 0

2k�pX

i=1

xi,jxi,g = 0

2k�pX

i=1

x2i,j = 2k�p

41

Example for a 2k-p Design

§ Effect calculated like before:

§ However, what about ?

§ Both effects are confounded!§ Which effects confound depends on the choice of § Within the example:


qC =y1 � y2 � y3 + y4

4

qC

qAB

qAB =y1 � y2 � y3 + y4

4

xi,j

I = ABC ) A = A2BC = BC,B = AC,C = AB

42

Calculating the Variation in a 2k-p Design

§ Variation was defined as

§ Where SST is the variation of all factors and combinations, but not all might be calculated…

§ However due to all x adding to 0

§ Thus


Impact of f =SSfSST

SSY = SS0 + SST

SST = SSY� SS0

=2k�pX

i=1

y2i � 2k�pq20

43

Exercise: A 24-1 Design

§ Calculate all effects and variations of effects. Which factors are negligible?

§ Which factors interact? How would you plan such an experiment?


A1 A2

C1 C2 C1 C2

B1D1 - 1.5 10 -D2 4 - - 3

B2D1 - 2 12 -D2 1 - - 5

44

Exercise: A 24-1 Design


I A B C D y1 "1 "1 "1 1 41 "1 "1 1 "1 1.51 "1 1 "1 1 11 "1 1 1 "1 21 1 "1 "1 "1 101 1 "1 1 1 31 1 1 "1 "1 121 1 1 1 1 5

38.5 21.5 1.5 -15.5 -12.5 Total4.8125 2.6875 0.1875 -1.9375 -1.5625 Total/8

57.78125 0.28125 30.03125 19.53125 115.9687549.8% 0.2% 25.9% 16.8%

SSY/= 301.25SS0/= 185.28125SST/=/ 115.96875

45

Exercise: Confoundings

§ I = ACD§ A = CD§ B = ABCD§ C = AD§ D = AC§ BA = BCD§ BC = BAD§ BD = BAC

§ Different design would be better....


46

2k-p Designs – Lessons learned

§ Choose vector space wisely!§ Do it only if interaction between factors is expected be negligible!

§ May safe quite a lot of runs….


47




48

One Factor Experiments

§ Examine effects of • one factors F• with a levels• r replications• Single scenario

§ Standard setup: • We can measure each level separately• Calculate a mean for each level• Calculate confidence intervals for each level

§ But, we can do better! (if certain presumptions are fulfilled)


49

One Factor Experiments

§ Model:

§ Average response without levels § Influence of a level j:§ Errors (independent of level)§ Estimating

§ Estimating


rX

i=1

aX

j=1

yi,j = arµ+ raX

j=1

↵j +rX

i=1

aX

j=1

ei,j

µ

↵j

ei,jµ

↵j

µ =

Pri=1

Paj=1 yi,j

ar

↵j =1

r

rX

i=1

yi,j � µ

50

One Factor Experiments – Estimating Variations

§ Total variation explained by the factor

§ Helps to compare factors vs. errors


SSA

SST=

SSA

SSY� SS0

=rPa

j=1 ↵2jPr

i=1

Paj=1 y

2i,j � arµ2

51

One Factor Experiments – Confidence Intervals

§ Error variance like before:

§ Variances of average and effects

§ Confidence intervals for response variables:

§ Again: Distribution & Independence must be verified for every response variable!


s2e =SSE

a(r � 1)

sµ =separ

yj = µ+ ↵j ± t[1�↵/2,a(r�1)]sµ+↵

s↵ =

rs2e(a� 1)

ar= se

ra� 1

arsµ+↵ =

qs2µ + s2↵

52

Exercise: Comparing the approach to the naïve one

§ Compare the average throughput on the three machines• by estimating a common µ• by estimating the parameters separately


TCP ThroughputWindows (206.9 144.7 95.3 198.1 103.9)

Linux (172.4 122.2 157.7 163.5 149.8)MacOS (157.3 204.2 172.6 203.2 189)

53

Exercise: Solution


Y Mean Alpha206.9 144.7 95.3 198.1 103.9 149.78 ,12.94 0.05172.4 122.2 157.7 163.5 149.8 153.12 ,9.60 0.03157.3 204.2 172.6 203.2 189 185.26 22.54 0.14

Total 162.72 Variation 22%Errors

57.12 ,5.08 ,54.48 48.32 ,45.8819.28 ,30.92 4.58 10.38 ,3.32,27.96 18.94 ,12.66 17.94 3.74

SSE 13800.47 SST= 17638.744

s t Var VarNaivee 33.91 1.78 76.32mu 8.76 1.78 15.61 28.27alpha 12.38 1.78 22.07 29.85mu_alpha 15.17 1.78 27.03

189

0@

50@

100@

150@

200@

250@

0@ 0.5@ 1@ 1.5@ 2@ 2.5@ 3@ 3.5@

0@

50@

100@

150@

200@

250@

0@ 1@ 2@ 3@ 4@

,40.00@

,30.00@

,20.00@

,10.00@

0.00@

10.00@

20.00@

30.00@

40.00@

50.00@

0@ 1@ 2@ 3@ 4@

54

Confidence Intervals: α-Values


-40,00

-30,00

-20,00

-10,00

0,00

10,00

20,00

30,00

40,00

50,00

0 0,5 1 1,5 2 2,5 3 3,5

55

Confidence Intervals: µ + α-Values


0

50

100

150

200

250

0 0,5 1 1,5 2 2,5 3 3,5

56

Confidence Intervals: Separate Estimation


0

50

100

150

200

250

0 0,5 1 1,5 2 2,5 3 3,5

57

One Factor Experiments - Lessons learned

§ Calculate effects based on common average estimation leads to cleaner results

§ Saves runs!

§ If and only if presumptions are met!


58




59

Two Factor Experiments without replications

§ Examine effects of • two factors F• with a and b levels• one replication• Single scenario

§ Full factorial design§ Better estimation of parameter importance (no longer limited to two

levels)


60

Two Factor Experiments - Model

§ Model: Additive effects & additive errors

§ Effects and errors sum up to 0, thus


yi,j = µ+ ↵i + �j + ei,j

µ =1

ab

aX

i=1

bX

j=1

yi,j

↵i =1

b

bX

j=1

yi,j � µ

�j =1

a

aX

i=1

yi,j � µ

61

Example: a 32 Design

§ Calculation of Effects in a 3x3 Design


A1 A2 A3 Mean Effect

B1 15.9 33.9 45.4 31.73 -7.8

B2 25.5 38.6 50.0 38.03 -1.5B3 33.5 47.2 65.8 48.83 9.3

Mean 24.97 39.90 53.73 39.53Effect -14.57 0.37 14.20

62

Two Factor Experiments - Variations

§ By design (again):

§ Variation of factor A

§ Variation of factor B analogous


SSY = SS0 + SST

SSA

SST=

SSA

SSY� SS0

=bPa

i=1 ↵2iPa

i=1

Pbj=1 y

2i,j � abµ2

63

Two Factor Experiments – Confidence Intervals

§ Error variance similar to designs before:

§ Variances of average and effects

§ Confidence intervals for response variables:

§ Yet again: Distribution & independence must be verified for every response variable!


s2e =SSE

(a� 1)(b� 1)

sµ =sepab

s↵ = se

ra� 1

ab

yi,j = µ+ ↵i + �j ± t[1�↵/2,(a�1)(b�1)]sµ+↵+�

sµ+↵+� =qs2µ + s2↵ + s2�

64

Exercise: A Two Factor Experiment

§ How large are the variations in the following experiment?


MSS

200 400 600 800 1000

Tim

eout 1s 153.5 139.8 184.7 231.1 266.5

2s 333.2 339.4 369.3 377.7 414.23s 194.7 239.3 221.1 281.9 306.5

65

Solution: A Two Factor Experiment

Values Mean Effect Variation153.5 139.8 184.7 231.1 266.5 195.12 775.07 27.9%333.2 339.4 369.3 377.7 414.2 366.76 96.57 46.2%194.7 239.3 221.1 281.9 306.5 248.7 721.49 2.3%

Mean 227.13 239.50 258.37 296.90 329.07 270.19 76.5%Effect 743.06 730.69 711.83 26.71 58.87Variation 5.5% 2.8% 0.4% 2.1% 10.3% 21.2%

SSY= 1195933.51SS0= 1095066.56SST= 100866.949

Errors1.44 724.63 1.41 9.27 12.519.50 3.33 14.37 715.77 711.43

710.94 21.29 715.77 6.49 71.07SSE= 2405.21067Error5Variation 2.38%

y error normAinv153.5 1.44 0333.2 9.50 8.84072532194.7 710.94 76.4065288139.8 724.63 720.108146339.4 3.33 2.06191503239.3 21.29 20.1081459184.7 1.41 72.061915369.3 14.37 15.0779502221.1 715.77 715.07795231.1 9.27 6.40652877377.7 715.77 711.628077281.9 6.49 4.17649503266.5 12.51 11.6280774414.2 711.43 78.8407253306.5 71.07 74.176495

730.00A

720.00A

710.00A

0.00A

10.00A

20.00A

30.00A

0A 50A 100A 150A 200A 250A 300A

725A

720A

715A

710A

75A

0A

5A

10A

15A

20A

25A

730.00A 720.00A 710.00A 0.00A 10.00A


66



-30,00

-20,00

-10,00

0,00

10,00

20,00

30,00

0 100 200 300 400 500

67



-25

-20

-15

-10

-5

0

5

10

15

20

25

-30,00 -20,00 -10,00 0,00 10,00 20,00 30,00

68



0,00

100,00

200,00

300,00

400,00

500,00

600,00

0 200 400 600 800 1000 1200

69

Two Factor Experiments – Lessons learned

§ Allows systematic evaluation of parameters§ May calculate confidence intervals without replication!

§ Experiment must (of course) qualify

§ More complex evaluations possible• Analysis of incomplete datasets (e.g. if your program crashes)• Testing relevance of parameter variations• Evaluating multiplicative effects, etc.• See [Jai91] for details


70




71

Two Factor Experiments with Replications

§ Examine effects of • two factors F• with a and b levels• r replications• Single scenario

§ Full factorial design§ Determination of influences between two parameters§ Even better confidence in correctness


72

Two Factor Experiments – Model (I)

§ Model: Additive effects & additive correlation between effects & additive errors


aX

i=1

bX

j=1

rX

k=1

yi,j,k = abrµ

+ braX

i=1

↵i + arbX

j=1

�j

+ raX

i=1

bX

j=1

�i,j

+aX

i=1

bX

j=1

rX

k=1

ei,j,k

73

Two Factor Experiments – Model (II)

§ Effects and errors sum up to 0, thus


µ =1

abr

aX

i=1

bX

j=1

rX

k=1

yi,j,k

�j =1

ar

aX

i=1

rX

k=1

yi,j,k � µ

�i,j =1

r

rX

k=1

yi,j,k � ↵i � �j � µ

↵i =1

br

bX

j=1

rX

k=1

yi,j,k � µ

74

Two Factor Experiments – Variations

§ Variations are calculated similarly:

§ Degrees of freedom

§ Variances:


SSY = SS0 + SST = SS0 + SSA + SSB + SSAB + SSE

abr = 1 + (a� 1) + (b� 1) + (a� 1)(b� 1) + ab(r � 1)

se =

sSSE

ab(r � 1)sµ =

sepabr

s↵ = se

ra� 1

abr

sµ+↵+�+� = se

r1 + (a� 1) + (b� 1) + (a� 1)(b� 1)

abr

75

Exercises: Two-Factor Experiments

§ Which formulas can be used to calculate confidence intervals?§ Which factor has more influence in the following design:


MSS

200 400 600 800 1000

Tim

eout

1s 320 315 325

470 474 466

320 322 318

510 520 500

680 676.5 683.5

2s 512 510 514

940 930 950

416 410 422

561 557.5 564.5

1224 1229 1219

3s 896 890 884

1974 1979 1969

736 730 742

2234 2244 2254

2856 2836 2876

76




77

General Full Factorial Designs

§ Examine effects of • k factors• with different levels• r replications

§ Full factorial design§ Determination of influences between k factors§ Even better confidence in correctness


78

Full Factorial Designs

§ Model:• Sum of response values is sum of µ, factors, two-factor-interactions, three-

factor-interactions, …, and errors• E.g. for three factors

• Calculations happen analogous to two factor design, but even more complex!

• E.g.


yijkl = µ+ ↵i + �j + ⇠k + �ABij + �ACik + �BCjk + �ABCijk + eijkl

Pai=1

Pbj=1

Pck=1

Prl=1 yijkl

abcr= µ

Pbj=1

Pck=1

Prl=1 yijkl

bcr� µ = ↵i

79

Exercises: Full-Factor Experiments

§ Which formula can be used to calculate variations?§ What is the variance of

• se

• sα?

§ How can confidence intervals be calculated?


systematic evaluation of complex systems · systematic evaluation of complex systems...

Documents