![Page 1: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/1.jpg)
1
Systematic Evaluation of Complex Systems
Acknowledgement: Parts of these slides are based on [Jai91]
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
2
Systematic Evaluation of Complex Systems
§ Motivation: Analysis of TCP Congestion Control§ 2k - Factorial Designs§ 2kr - Factorial Designs with Replications§ 2k-p – Fractional Factorial Designs§ One Factor Experiments§ Two Factor Experiments§ Two Factor Experiments with Replications§ General Full Factorial Designs
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
![Page 2: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/2.jpg)
3
Quick Reminder: TCP Congestion Control
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
Slow start
Additive increase, congestion avoidance
New slow start to new threshold, then linear increase
20=40/2
Reset CW to 1, new threshold = CW/2
Initial congestion threshold
Initial congestion window size
4
TCP Congestion Control Parameters
§ Performance affected by• Initial Congestion Window• Initial Congestion Window Threshold• Timeout• Enable Duplicate Acknowledgements• Size of TCP-Buffers• Maximum Segment Size (MSS)• …• MacOS 10.7: sysctl -a | grep tcp | wc –l => 79 Parameters• MacOS 10.10: sysctl -a | grep tcp | wc –l => 116 Parameters• MacOS 10.11: sysctl -a | grep tcp | wc –l => 120 Parameters• Some Boolean other numeric
§ For different scenarios, e.g.• Internet, LAN, DSL, Satellite, Congestion etc.
§ For different runs
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
![Page 3: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/3.jpg)
5
Choosing Optimal Parameters
§ Choosing good parameters is a multi-criteria optimization problem§ Finding optimal:
• 79 parameter (each assumed to have 2 legitimate values only)• 5 scenarios• 32 runs• Requires 279 x 5 x 32 = 9.67 x 1025 simulations!
§ Combinatorial Explosion § Care must be taken in choice of examined settings!
→ Analysis of Variance (ANOVA)
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
6
Some Terminology
§ Response Variables:• Outcome of experiment with regards to a performance index
§ Primary Factors:• Freely chosen parameter that may influence the response variable
§ Secondary Factors:• Parameters whose influence is not of interest (e.g. fixed ones)
§ Levels:• Possible values of factors (e.g. true/false, 100Mb/1Gb/10Gb, 5s, …)
§ Replications:• Number of runs performed
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
![Page 4: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/4.jpg)
7
More Terminology
§ Interaction:• In which experiment do factors A & B influence each other?
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
A1 A2
B1 3 5B2 6 8
A1 A2
B1 3 5B2 6 9
8
Rules for the Planning of Experiments
§ Identify & Control all important parameters
§ Isolate effects of different parameters• Do not changes too many parameters at once
§ Aggregate parameters changes reasonably• Do not check each parameter combination
§ Treat interactions• Check for parameters that influence each other (positive and negative!)
§ Regard variation of responses with fixed parameters• Perform multiple runs• Use confidence intervals
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
![Page 5: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/5.jpg)
9
Systematic Evaluation of Complex Systems
§ Motivation: Analysis of TCP Congestion Control§ 2k - Factorial Designs§ 2kr - Factorial Designs with Replications§ 2k-p – Fractional Factorial Designs§ One Factor Experiments§ Two Factor Experiments§ Two Factor Experiments with Replications§ General Full Factorial Designs
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
10
2k - Factorial Designs
§ Examine effects of • k factors F• with two levels• No replications• Single scenario
§ Which of the factors has the largest impact?• To get a feeling of the factors in the beginning of a study• To find a point to start further optimization• To reduce number of primary factors
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
![Page 6: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/6.jpg)
11
2k - Factorial Designs – Approximation of Responses
§ Observation: Impact of factors often follows strictly monotone functions
§ Idea: Approximate effects with a linear function
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
Level 1 Level 2
12
2k - Factorial Designs – Example for a 22 design (I)
§ 2 Factors at two levels
§ Response variable with• - Estimated average throughput (Buffer = 15Kb, MSS = 250 Bytes)• - Impact by changing MSS• - Impact of different buffer sizes• - Impact due to interaction of MSS and Buffers (0 iff no interaction)
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
TCP ThroughputMSS = 100 Bytes MSS = 400 Bytes
Buffer = 10Kb 15 45
Buffer = 20Kb 25 75
q0
y = q0 + qAxA + qBxB + qABxAxB
qAqBqAB
xA =
(�1 if MSS = 100
1 if MSS = 400xB =
(�1 if Bu↵er = 10
1 if Bu↵er = 20
![Page 7: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/7.jpg)
13
2k - Factorial Designs – Example for a 22 design (II)
§ Calculation of by linear equation system:
§ Influence of factor A twice as high as B§ Positive correlation between both factors
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
q0, qA, qB , qAB
y = 40 + 20xA + 10xB + 5xAxB
15 = y0 = q0 � qA � qB + qAB
45 = y1 = q0 + qA � qB � qAB
25 = y2 = q0 � qA + qB � qAB
75 = y3 = q0 + qA + qB + qAB
14
2k - Factorial Designs – Example for a 22 design (III)
§ Sign Table Method
§ Column AB is calculated by Column A * Column B§ Total is calculated sum of yj values with sign of row
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
I A B AB yj
+1 -1 -1 +1 15+1 +1 -1 -1 45+1 -1 +1 -1 25+1 +1 +1 +1 75160 80 40 20 Total40 20 10 5 Total / 4
![Page 8: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/8.jpg)
15
2k - Factorial Designs – Estimating the Variation
§ Variance may be better indicator to determine variables with high impact
§ Variance of response variable:
§ Simplification: compare changes of variation (without normalizing)
§ This is by definition ( , vectors x are orthogonal)
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
q0 = y
SST =X
f2P(F )
SSf =X
f2P(F )
2kq2f = 2kX
f2P(F )
q2f
s2 =
P2n
i=1(yi � y)2
2n � 1
SST =2nX
i=1
(yi � y)2
16
2k - Factorial Designs – Example for a 22 design (IV)
§ Total variation in the example:
§ Quadratic influence of the factors:
→ Conclusion for the study: evaluate A further!
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
SST = SSA + SSB + SSAB
= 22q2A + 22q2B + 22q2AB
= 22202 + 22102 + 2252
= 1600 + 400 + 100 = 2100
influence of A =SSASST
=1600
2100= 76.2%
influence of B =SSBSST
=400
2100= 19.0%
![Page 9: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/9.jpg)
17
Exercise: Estimating the impact in a 23 design
§ Which of the factors in the following measurements require further analysis?
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
A1 A2
C1 C2 C1 C2
B1 100 15 120 10
B2 40 30 20 50
18
Exercise: Solution
A B C AB AC BC ABC y!1 !1 !1 1 1 1 !1 100!1 !1 1 1 !1 !1 1 15!1 1 !1 !1 1 !1 1 40!1 1 1 !1 !1 1 !1 301 !1 !1 !1 !1 1 1 1201 !1 1 !1 1 !1 !1 101 1 !1 1 !1 !1 !1 201 1 1 1 1 1 1 5015 '105 '175 '15 15 215 65 Total
1.875 '13.125 '21.875 '1.875 1.875 26.875 8.125 Total4/48
3.515625 172.265625 478.515625 3.515625 3.515625 722.265625 66.01562528.125 1378.125 3828.125 28.125 28.125 5778.125 528.125 11596.875
0.2% 11.9% 33.0% 0.2% 0.2% 49.8% 4.6%
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
![Page 10: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/10.jpg)
19
Systematic Evaluation of Complex Systems
§ Motivation: Analysis of TCP Congestion Control§ 2k - Factorial Designs§ 2kr - Factorial Designs with Replications§ 2k-p – Fractional Factorial Designs§ One Factor Experiments§ Two Factor Experiments§ Two Factor Experiments with Replications§ General Full Factorial Designs
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
20
2kr - Factorial Designs with Replications
§ Examine effects of • k factors• with two levels• r replications• Single scenario
§ How sure can we be that the factors are correctly estimated?
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
![Page 11: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/11.jpg)
21
2kr - Factorial Designs with Replications
§ Response variables where i represents the factorial combination and j reflects the run
§ Measurements influenced by errors § Thus
§ The errors for each parameter set are expected to sum to zero, thus
§ Estimation of and the influence of the factors as in 2k design but by using
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
yi,j
ei,j
q0 qP(F )
yi
yi,j = q0 +X
f2P(F )
qfxf + ei,j
yi =1
r
rX
j=1
yi,j
22
Variation in 2kr Designs
§ If we additionally assume that errors are statistically independent:
§ The quadratic impact is again
§ Replications also allow for estimation of errors done during estimation!
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
Impact of f =SSfSST
SST =X
f2P(F )
SSf + SSE =X
f2P(F )
2krq2f +2kX
i=1
rX
j=1
e2i,j
![Page 12: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/12.jpg)
23
Confidence Intervals in 2kr Designs
§ Replications allow for assurance checks of effects§ Variance for confidence intervals:
§ Q: Why not ?§ As errors are assumed to be distributed independently:
§ Confidence intervals for :
§ Similar for responses
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
s2e =SSE
2k(r � 1)
2kr � 1
s2q0 = s2qA = s2qf =s2e2kr
qf
qf ± t[1�↵/2,2k(r�1)] · sf
yi
24
Assumptions on the errors
§ Assumptions about the errors• Statistically independent• Additive• Normally distributed• Constant standard deviation • Effects of factors are additive
§ Must be verified for every response variable!• Take system knowledge into account• Or use statistical Tests: Chi-Squared-Test, F-Test• Or use visual verification
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
![Page 13: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/13.jpg)
25
Example for visual error verification
§ Consider the following measurements
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
A1 A2
B1 (15, 18, 12) (25, 28,19)
B2 (45, 48, 51) (75, 81, 75)
26
Calculated Effects and Variations
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
I A B AB y yx1 yx2 yx3 ex1 ex2 ex31 "1 "1 1 15.00&&&&&& 15 18 12 0 3 "31 "1 1 "1 48.00&&&&&& 45 48 51 "3 0 31 1 "1 "1 24.00&&&&&& 25 28 19 1 4 "51 1 1 1 77.00&&&&&& 75 75 81 "2 "2 4
165.00 39.00 85.00 19.00 Total41.25 9.75 21.25 4.75 Total7/74
SSE= 102SE2= 12.75
Effect 380.25 1806.25 90.25 2378.75 SQ0= 3.092Percent7of7Variation 15.99% 75.93% 3.79% Conf= 7.131
Confidence 16.88 28.38 11.88Intervals 2.62 14.12 "2.38
"0.2940321"3.10624980.89337968"1.52987271.5298727"0.29403213.10624976"1.5298727"3.10624981.5298727"4.34256853.10624976
"6&
"5&
"4&
"3&
"2&
"1&
0&
1&
2&
3&
4&
5&
0& 20& 40& 60& 80& 100&
"4.00&
"2.00&
0.00&
2.00&
4.00&
6.00&
"6& "4& "2& 0& 2& 4&
![Page 14: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/14.jpg)
27
Plot Errors
§ Plot errors against projected y-value
§ Make sure: no direct relationship visible
→ Looks good!
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
-6
-5
-4
-3
-2
-1
0
1
2
3
4
5
0 50 100
28
Normal-Quantile-Quantile-Plot
§ Plot measured errors vs. where they are expected in a perfect normal distribution
§ Should follow the linear function y = x
§ Ensures errors follow a normal distribution
§ In Excel Speak: =NORMINV(RANG(P21,P$18:P$29,1)/(12+1),0,STABW(P$18:P$29))
→ Looks good!
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
-6,00
-4,00
-2,00
0,00
2,00
4,00
6,00
-5 0 5
![Page 15: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/15.jpg)
29
Exercise: Discussion of a scenario
§ Take the following (hypothetical) performance measurements to determine if one of the operating systems is particularly well suited one of the scenarios.
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
TCP ThroughputWindows Linux
Scenario 1 (9.55, 9.33, 11.22) (20.58, 19.51, 21.28)
Scenario 2 (96.48, 97.99, 102.67) (395.39, 407.22, 366.43)
30
Exercise: Solution (1st Try)
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
I A B AB y yx1 yx2 yx3 ex1 ex2 ex31 "1 "1 1 10.00 9.55 9.33 11.13 "0.45 "0.67 1.131 "1 1 "1 99.05 96.48 97.99 102.67 "2.57 "1.06 3.631 1 "1 "1 20.46 20.58 19.51 21.28 0.12 "0.95 0.821 1 1 1 389.68 395.40 407.22 366.43 5.71 17.54 "23.25
519.31 301.21 458.15 280.06 Total129.83 75.30 114.54 70.01 Total7/74
SSE= 905.348SE2= 113.168
Effect 22681.87 52474.68 19607.94 95669.83 SQ0= 9.21284Percent7of7Variation 23.71% 54.85% 20.50% Conf= 21.2448
Confidence 96.55 135.78 91.26Intervals 54.06 93.29 48.77
41.1653.5
65.17@@@@@@@@@@@50.08
"30.00@
"25.00@
"20.00@
"15.00@
"10.00@
"5.00@
0.00@
5.00@
10.00@
15.00@
20.00@@"@@@@ @50.00@@@100.00@@@150.00@@@200.00@@@250.00@@@300.00@@@350.00@@@400.00@@@450.00@@
Error7vs.7y7
"30.00@"25.00@"20.00@"15.00@"10.00@"[email protected]@[email protected]@[email protected]@
"15@ "10@ "5@ 0@ 5@ 10@ 15@
Normal7QuanHle7QuanHle7Plot7
![Page 16: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/16.jpg)
31
Exercise: Solution (1st Try)
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
-30,00
-25,00
-20,00
-15,00
-10,00
-5,00
0,00
5,00
10,00
15,00
20,00 - 100,00 200,00 300,00 400,00 500,00
Error vs. y
32
Exercise: Solution (1st Try)
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
-30,00
-25,00
-20,00
-15,00
-10,00
-5,00
0,00
5,00
10,00
15,00
20,00-15 -10 -5 0 5 10 15
Normal Quantile Quantile Plot
![Page 17: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/17.jpg)
33
Exercise: Solution (2nd Try – Using logarithms)
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
I A B AB y yx1 yx2 yx3 ex1 ex2 ex31 "1 "1 1 1.00 0.98 0.97 1.05 "0.02 "0.03 0.051 "1 1 "1 2.00 1.98 1.99 2.01 "0.01 0.00 0.021 1 "1 "1 1.31 1.31 1.29 1.33 0.00 "0.02 0.021 1 1 1 2.59 2.6 2.61 2.56 0.01 0.02 "0.03
6.90 0.95 2.29 0.28 Total1.72 0.24 0.57 0.07 Total7/74 SSE= 0.01
SE2= 0S0= 0.02
Effect 0.22 1.31 0.02 1.56 Conf= 0.05Percent7of7Variation 14.36% 84.02% 1.25%
Confidence 0.29 0.63 0.12Intervals 0.18 0.52 0.02
"0.02"0.010.000.01"0.030.00"0.020.020.050.020.02"0.03
"0.04?"0.03?"0.02?"0.01?
0?0.01?0.02?0.03?0.04?0.05?0.06?
0? 0.5? 1? 1.5? 2? 2.5? 3?
Error7vs.7y7
0?
0.01?
0.02?
0.03?
0.04?
0.05?
34
Exercise: Solution (2nd Try – Using logarithms)
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
-0,04
-0,03
-0,02
-0,01
0
0,01
0,02
0,03
0,04
0,05
0,06
0 0,5 1 1,5 2 2,5 3
Error vs. y
![Page 18: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/18.jpg)
35
Exercise: Solution (2nd Try – Using logarithms)
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
-0,04
-0,03
-0,02
-0,01
0
0,01
0,02
0,03
0,04
0,05
-0,04 -0,02 0,00 0,02 0,04 0,06
36
2kr Designs – Lessons learned
§ False assumptions lead to false correlations
§ But replications allow for verification of experiments§ Always perform F-Test or visual verification otherwise confidence
intervals are useless!§ Be sure to understand your system!
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
![Page 19: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/19.jpg)
37
Systematic Evaluation of Complex Systems
§ Motivation: Analysis of TCP Congestion Control§ 2k - Factorial Designs§ 2kr - Factorial Designs with Replications§ 2k-p – Fractional Factorial Designs§ One Factor Experiments§ Two Factor Experiments§ Two Factor Experiments with Replications§ General Full Factorial Designs
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
38
2k-p – Fractional Factorial Designs
§ Examine effects of • k factors• with two levels• without replications• Single scenario• But less experiments…
§ How can we save work?§ Idea: Neglect relationship between factors (e.g. )
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
qABCD
![Page 20: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/20.jpg)
39
Example for a 2k-p Design
§ Example for a 23-1 Design:
§ Factors AB, AC, BC, and ABC are neglected!
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
I A B C yj
+1 -1 -1 +1 15+1 +1 -1 -1 45+1 -1 +1 -1 25+1 +1 +1 +1 75160 80 40 20 Total40 20 10 5 Total / 4
40
Requirements on 2k-p Designs
§ Values of must form orthogonal vector space, s.t.
• Sum of any column j equals 0
• Sum of product of any two columns j and g equals 0
• Sum of the squares of any column j is 2k-p
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
xi,j
2k�pX
i=1
xi,j = 0
2k�pX
i=1
xi,jxi,g = 0
2k�pX
i=1
x2i,j = 2k�p
![Page 21: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/21.jpg)
41
Example for a 2k-p Design
§ Effect calculated like before:
§ However, what about ?
§ Both effects are confounded!§ Which effects confound depends on the choice of § Within the example:
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
qC =y1 � y2 � y3 + y4
4
qC
qAB
qAB =y1 � y2 � y3 + y4
4
xi,j
I = ABC ) A = A2BC = BC,B = AC,C = AB
42
Calculating the Variation in a 2k-p Design
§ Variation was defined as
§ Where SST is the variation of all factors and combinations, but not all might be calculated…
§ However due to all x adding to 0
§ Thus
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
Impact of f =SSfSST
SSY = SS0 + SST
SST = SSY� SS0
=2k�pX
i=1
y2i � 2k�pq20
![Page 22: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/22.jpg)
43
Exercise: A 24-1 Design
§ Calculate all effects and variations of effects. Which factors are negligible?
§ Which factors interact? How would you plan such an experiment?
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
A1 A2
C1 C2 C1 C2
B1D1 - 1.5 10 -D2 4 - - 3
B2D1 - 2 12 -D2 1 - - 5
44
Exercise: A 24-1 Design
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
I A B C D y1 "1 "1 "1 1 41 "1 "1 1 "1 1.51 "1 1 "1 1 11 "1 1 1 "1 21 1 "1 "1 "1 101 1 "1 1 1 31 1 1 "1 "1 121 1 1 1 1 5
38.5 21.5 1.5 -15.5 -12.5 Total4.8125 2.6875 0.1875 -1.9375 -1.5625 Total/8
57.78125 0.28125 30.03125 19.53125 115.9687549.8% 0.2% 25.9% 16.8%
SSY/= 301.25SS0/= 185.28125SST/=/ 115.96875
![Page 23: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/23.jpg)
45
Exercise: Confoundings
§ I = ACD§ A = CD§ B = ABCD§ C = AD§ D = AC§ BA = BCD§ BC = BAD§ BD = BAC
§ Different design would be better....
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
46
2k-p Designs – Lessons learned
§ Choose vector space wisely!§ Do it only if interaction between factors is expected be negligible!
§ May safe quite a lot of runs….
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
![Page 24: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/24.jpg)
47
Systematic Evaluation of Complex Systems
§ Motivation: Analysis of TCP Congestion Control§ 2k - Factorial Designs§ 2kr - Factorial Designs with Replications§ 2k-p – Fractional Factorial Designs§ One Factor Experiments§ Two Factor Experiments§ Two Factor Experiments with Replications§ General Full Factorial Designs
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
48
One Factor Experiments
§ Examine effects of • one factors F• with a levels• r replications• Single scenario
§ Standard setup: • We can measure each level separately• Calculate a mean for each level• Calculate confidence intervals for each level
§ But, we can do better! (if certain presumptions are fulfilled)
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
![Page 25: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/25.jpg)
49
One Factor Experiments
§ Model:
§ Average response without levels § Influence of a level j:§ Errors (independent of level)§ Estimating
§ Estimating
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
rX
i=1
aX
j=1
yi,j = arµ+ raX
j=1
↵j +rX
i=1
aX
j=1
ei,j
µ
↵j
ei,jµ
↵j
µ =
Pri=1
Paj=1 yi,j
ar
↵j =1
r
rX
i=1
yi,j � µ
50
One Factor Experiments – Estimating Variations
§ Total variation explained by the factor
§ Helps to compare factors vs. errors
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
SSA
SST=
SSA
SSY� SS0
=rPa
j=1 ↵2jPr
i=1
Paj=1 y
2i,j � arµ2
![Page 26: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/26.jpg)
51
One Factor Experiments – Confidence Intervals
§ Error variance like before:
§ Variances of average and effects
§ Confidence intervals for response variables:
§ Again: Distribution & Independence must be verified for every response variable!
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
s2e =SSE
a(r � 1)
sµ =separ
yj = µ+ ↵j ± t[1�↵/2,a(r�1)]sµ+↵
s↵ =
rs2e(a� 1)
ar= se
ra� 1
arsµ+↵ =
qs2µ + s2↵
52
Exercise: Comparing the approach to the naïve one
§ Compare the average throughput on the three machines• by estimating a common µ• by estimating the parameters separately
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
TCP ThroughputWindows (206.9 144.7 95.3 198.1 103.9)
Linux (172.4 122.2 157.7 163.5 149.8)MacOS (157.3 204.2 172.6 203.2 189)
![Page 27: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/27.jpg)
53
Exercise: Solution
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
Y Mean Alpha206.9 144.7 95.3 198.1 103.9 149.78 ,12.94 0.05172.4 122.2 157.7 163.5 149.8 153.12 ,9.60 0.03157.3 204.2 172.6 203.2 189 185.26 22.54 0.14
Total 162.72 Variation 22%Errors
57.12 ,5.08 ,54.48 48.32 ,45.8819.28 ,30.92 4.58 10.38 ,3.32,27.96 18.94 ,12.66 17.94 3.74
SSE 13800.47 SST= 17638.744
s t Var VarNaivee 33.91 1.78 76.32mu 8.76 1.78 15.61 28.27alpha 12.38 1.78 22.07 29.85mu_alpha 15.17 1.78 27.03
189
0@
50@
100@
150@
200@
250@
0@ 0.5@ 1@ 1.5@ 2@ 2.5@ 3@ 3.5@
0@
50@
100@
150@
200@
250@
0@ 1@ 2@ 3@ 4@
,40.00@
,30.00@
,20.00@
,10.00@
0.00@
10.00@
20.00@
30.00@
40.00@
50.00@
0@ 1@ 2@ 3@ 4@
54
Confidence Intervals: α-Values
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
-40,00
-30,00
-20,00
-10,00
0,00
10,00
20,00
30,00
40,00
50,00
0 0,5 1 1,5 2 2,5 3 3,5
![Page 28: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/28.jpg)
55
Confidence Intervals: µ + α-Values
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
0
50
100
150
200
250
0 0,5 1 1,5 2 2,5 3 3,5
56
Confidence Intervals: Separate Estimation
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
0
50
100
150
200
250
0 0,5 1 1,5 2 2,5 3 3,5
![Page 29: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/29.jpg)
57
One Factor Experiments - Lessons learned
§ Calculate effects based on common average estimation leads to cleaner results
§ Saves runs!
§ If and only if presumptions are met!
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
58
Systematic Evaluation of Complex Systems
§ Motivation: Analysis of TCP Congestion Control§ 2k - Factorial Designs§ 2kr - Factorial Designs with Replications§ 2k-p – Fractional Factorial Designs§ One Factor Experiments§ Two Factor Experiments§ Two Factor Experiments with Replications§ General Full Factorial Designs
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
![Page 30: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/30.jpg)
59
Two Factor Experiments without replications
§ Examine effects of • two factors F• with a and b levels• one replication• Single scenario
§ Full factorial design§ Better estimation of parameter importance (no longer limited to two
levels)
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
60
Two Factor Experiments - Model
§ Model: Additive effects & additive errors
§ Effects and errors sum up to 0, thus
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
yi,j = µ+ ↵i + �j + ei,j
µ =1
ab
aX
i=1
bX
j=1
yi,j
↵i =1
b
bX
j=1
yi,j � µ
�j =1
a
aX
i=1
yi,j � µ
![Page 31: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/31.jpg)
61
Example: a 32 Design
§ Calculation of Effects in a 3x3 Design
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
A1 A2 A3 Mean Effect
B1 15.9 33.9 45.4 31.73 -7.8
B2 25.5 38.6 50.0 38.03 -1.5B3 33.5 47.2 65.8 48.83 9.3
Mean 24.97 39.90 53.73 39.53Effect -14.57 0.37 14.20
62
Two Factor Experiments - Variations
§ By design (again):
§ Variation of factor A
§ Variation of factor B analogous
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
SSY = SS0 + SST
SSA
SST=
SSA
SSY� SS0
=bPa
i=1 ↵2iPa
i=1
Pbj=1 y
2i,j � abµ2
![Page 32: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/32.jpg)
63
Two Factor Experiments – Confidence Intervals
§ Error variance similar to designs before:
§ Variances of average and effects
§ Confidence intervals for response variables:
§ Yet again: Distribution & independence must be verified for every response variable!
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
s2e =SSE
(a� 1)(b� 1)
sµ =sepab
s↵ = se
ra� 1
ab
yi,j = µ+ ↵i + �j ± t[1�↵/2,(a�1)(b�1)]sµ+↵+�
sµ+↵+� =qs2µ + s2↵ + s2�
64
Exercise: A Two Factor Experiment
§ How large are the variations in the following experiment?
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
MSS
200 400 600 800 1000
Tim
eout 1s 153.5 139.8 184.7 231.1 266.5
2s 333.2 339.4 369.3 377.7 414.23s 194.7 239.3 221.1 281.9 306.5
![Page 33: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/33.jpg)
65
Solution: A Two Factor Experiment
Values Mean Effect Variation153.5 139.8 184.7 231.1 266.5 195.12 775.07 27.9%333.2 339.4 369.3 377.7 414.2 366.76 96.57 46.2%194.7 239.3 221.1 281.9 306.5 248.7 721.49 2.3%
Mean 227.13 239.50 258.37 296.90 329.07 270.19 76.5%Effect 743.06 730.69 711.83 26.71 58.87Variation 5.5% 2.8% 0.4% 2.1% 10.3% 21.2%
SSY= 1195933.51SS0= 1095066.56SST= 100866.949
Errors1.44 724.63 1.41 9.27 12.519.50 3.33 14.37 715.77 711.43
710.94 21.29 715.77 6.49 71.07SSE= 2405.21067Error5Variation 2.38%
y error normAinv153.5 1.44 0333.2 9.50 8.84072532194.7 710.94 76.4065288139.8 724.63 720.108146339.4 3.33 2.06191503239.3 21.29 20.1081459184.7 1.41 72.061915369.3 14.37 15.0779502221.1 715.77 715.07795231.1 9.27 6.40652877377.7 715.77 711.628077281.9 6.49 4.17649503266.5 12.51 11.6280774414.2 711.43 78.8407253306.5 71.07 74.176495
730.00A
720.00A
710.00A
0.00A
10.00A
20.00A
30.00A
0A 50A 100A 150A 200A 250A 300A
725A
720A
715A
710A
75A
0A
5A
10A
15A
20A
25A
730.00A 720.00A 710.00A 0.00A 10.00A
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
66
Solution: A Two Factor Experiment
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
-30,00
-20,00
-10,00
0,00
10,00
20,00
30,00
0 100 200 300 400 500
![Page 34: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/34.jpg)
67
Solution: A Two Factor Experiment
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
-25
-20
-15
-10
-5
0
5
10
15
20
25
-30,00 -20,00 -10,00 0,00 10,00 20,00 30,00
68
Solution: A Two Factor Experiment
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
0,00
100,00
200,00
300,00
400,00
500,00
600,00
0 200 400 600 800 1000 1200
![Page 35: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/35.jpg)
69
Two Factor Experiments – Lessons learned
§ Allows systematic evaluation of parameters§ May calculate confidence intervals without replication!
§ Experiment must (of course) qualify
§ More complex evaluations possible• Analysis of incomplete datasets (e.g. if your program crashes)• Testing relevance of parameter variations• Evaluating multiplicative effects, etc.• See [Jai91] for details
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
70
Systematic Evaluation of Complex Systems
§ Motivation: Analysis of TCP Congestion Control§ 2k - Factorial Designs§ 2kr - Factorial Designs with Replications§ 2k-p – Fractional Factorial Designs§ One Factor Experiments§ Two Factor Experiments§ Two Factor Experiments with Replications§ General Full Factorial Designs
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
![Page 36: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/36.jpg)
71
Two Factor Experiments with Replications
§ Examine effects of • two factors F• with a and b levels• r replications• Single scenario
§ Full factorial design§ Determination of influences between two parameters§ Even better confidence in correctness
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
72
Two Factor Experiments – Model (I)
§ Model: Additive effects & additive correlation between effects & additive errors
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
aX
i=1
bX
j=1
rX
k=1
yi,j,k = abrµ
+ braX
i=1
↵i + arbX
j=1
�j
+ raX
i=1
bX
j=1
�i,j
+aX
i=1
bX
j=1
rX
k=1
ei,j,k
![Page 37: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/37.jpg)
73
Two Factor Experiments – Model (II)
§ Effects and errors sum up to 0, thus
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
µ =1
abr
aX
i=1
bX
j=1
rX
k=1
yi,j,k
�j =1
ar
aX
i=1
rX
k=1
yi,j,k � µ
�i,j =1
r
rX
k=1
yi,j,k � ↵i � �j � µ
↵i =1
br
bX
j=1
rX
k=1
yi,j,k � µ
74
Two Factor Experiments – Variations
§ Variations are calculated similarly:
§ Degrees of freedom
§ Variances:
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
SSY = SS0 + SST = SS0 + SSA + SSB + SSAB + SSE
abr = 1 + (a� 1) + (b� 1) + (a� 1)(b� 1) + ab(r � 1)
se =
sSSE
ab(r � 1)sµ =
sepabr
s↵ = se
ra� 1
abr
sµ+↵+�+� = se
r1 + (a� 1) + (b� 1) + (a� 1)(b� 1)
abr
![Page 38: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/38.jpg)
75
Exercises: Two-Factor Experiments
§ Which formulas can be used to calculate confidence intervals?§ Which factor has more influence in the following design:
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
MSS
200 400 600 800 1000
Tim
eout
1s 320 315 325
470 474 466
320 322 318
510 520 500
680 676.5 683.5
2s 512 510 514
940 930 950
416 410 422
561 557.5 564.5
1224 1229 1219
3s 896 890 884
1974 1979 1969
736 730 742
2234 2244 2254
2856 2836 2876
76
Systematic Evaluation of Complex Systems
§ Motivation: Analysis of TCP Congestion Control§ 2k - Factorial Designs§ 2kr - Factorial Designs with Replications§ 2k-p – Fractional Factorial Designs§ One Factor Experiments§ Two Factor Experiments§ Two Factor Experiments with Replications§ General Full Factorial Designs
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
![Page 39: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/39.jpg)
77
General Full Factorial Designs
§ Examine effects of • k factors• with different levels• r replications
§ Full factorial design§ Determination of influences between k factors§ Even better confidence in correctness
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
78
Full Factorial Designs
§ Model:• Sum of response values is sum of µ, factors, two-factor-interactions, three-
factor-interactions, …, and errors• E.g. for three factors
• Calculations happen analogous to two factor design, but even more complex!
• E.g.
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems
yijkl = µ+ ↵i + �j + ⇠k + �ABij + �ACik + �BCjk + �ABCijk + eijkl
Pai=1
Pbj=1
Pck=1
Prl=1 yijkl
abcr= µ
Pbj=1
Pck=1
Prl=1 yijkl
bcr� µ = ↵i
![Page 40: Systematic Evaluation of Complex Systems · Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] ... 2kq2 f =2 k X f2P(F) q2 f s2 =](https://reader034.vdocument.in/reader034/viewer/2022042417/5f32793e5a3dd179e27ccaf8/html5/thumbnails/40.jpg)
79
Exercises: Full-Factor Experiments
§ Which formula can be used to calculate variations?§ What is the variance of
• se
• sα?
§ How can confidence intervals be calculated?
Tele2 / Perf Eval (WS 19/20): 10 – Systematic Evaluation of Complex Systems