analysis of simulation results andy wang cis 5930-03 computer systems performance analysis
TRANSCRIPT
![Page 1: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/1.jpg)
Analysis of Simulation Results
Andy WangCIS 5930-03
Computer SystemsPerformance Analysis
![Page 2: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/2.jpg)
Analysis of Simulation Results
• Check for correctness of implementation– Model verification
• Check for representativeness of assumptions– Model validation
• Handle initial observations• Decide how long to run the simulation
2
![Page 3: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/3.jpg)
3
Model Verification Techniques
• Verification is similar to debugging– Programmer’s responsibility
• Validation– Modeling person’s responsibility
![Page 4: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/4.jpg)
Top-Down Modular Design
• Simulation models are large computer programs– Software engineering techniques apply
• Modularity– Well-defined interfaces for pieces to
coordinate• Top-down design
– Hierarchical structure
4
![Page 5: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/5.jpg)
Antibugging
• Sanity checks– Probabilities of events should add up to 1– No simulated entities should disappear
• Packets sent = packets received + packets lost
5
![Page 6: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/6.jpg)
Structured Walk-Through
• Explaining the code to another person• Many bugs are discovered by reading
the code carefully
6
![Page 7: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/7.jpg)
Deterministic Models
• Hard to verify simulation against random inputs
• Should debug by specifying constant or deterministic distributions
7
![Page 8: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/8.jpg)
Run Simplified Cases
• Use only one packet, one source, one intermediary node
• Can compare analyzed and simulated results
8
![Page 9: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/9.jpg)
Trace
• Time-ordered list of events– With associated variables
• Should have levels of details– In terms of occurred events, procedure
called, or variable updates– Properly indented to show levels
• Should allow the traces to be turned on and off
9
![Page 10: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/10.jpg)
On-line Graphic Displays
• Important when viewing a large amount of data– Verifying a CPU scheduler preempt
processes according to priorities and time budgets
10
![Page 11: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/11.jpg)
Continuity Test
• Run the simulation with slightly different values of input parameters– Δ change in input should lead to Δ change
in output– If not, possibly bugs
11
![Page 12: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/12.jpg)
Degeneracy Test
• Test for extreme simulation input and configuration parameters– Routers with zero service time– Idle and peak load
• Also unusual combinations – Single CPU without disk
12
![Page 13: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/13.jpg)
Consistency Tests
• Check results for input parameters with similar effects– Two sources with arrival rate of 100
packets per second– Four sources with arrival rate of 50 packets
per second• If dissimilar, possibly bugs
13
![Page 14: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/14.jpg)
Seed Independence
• Different random seeds should yield statistically similar results
14
![Page 15: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/15.jpg)
• Should validate– Assumptions– Input values and distributions– Output values and conclusions
• Against– Expert intuition– Real system measurements– Theoretical results
15
Model Validation Techniques
![Page 16: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/16.jpg)
Model Validation Techniques
• May not be possible to check all nine possibilities– Real system not available
• May not be possible at all– The reason why the simulation was built
• As the last resort– E.g., economic model
16
![Page 17: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/17.jpg)
Expert Intuition
• Should validate assumptions, input, and output separately and as early as possible
17
% packet loss
Throughput
Why would increased packet loss lead to better throughput?
![Page 18: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/18.jpg)
Real-System Measurements
• Most reliable way for validation– Often not feasible
• System may not exist• Too expensive to measure
• Apply statistical techniques to compare model and measured data
• Use multiple traces under different environments
18
![Page 19: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/19.jpg)
Theoretical Results
• Can apply queueing models• If too complex
– Can validate a small subset of simulation parameters• E.g., compare analytical equations with CPU
simulation models with one and two cores– Use validated simulation to simulate many cores
– Can validate only the common scenarios
19
![Page 20: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/20.jpg)
Transient Removal
• In most cases, we care only about steady-state performance
• We need to perform transient removal to remove initial data from analysis
• Difficulty– Find out where transient state ends
20
Transient Removal
![Page 21: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/21.jpg)
Long Runs
• Just run the simulation for a long time– Waste resources– Not sure if it’s long enough
21
![Page 22: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/22.jpg)
Proper Initialization
• Start the simulation in a state close to steady state– Pre-populate requests in various queues– Pre-load memory cache content
• Reduce the length of transient periods
22
![Page 23: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/23.jpg)
Truncation
• Assume steady state variance < transient state variance
• Algorithm– Measure variability in terms of range– Remove the first L observations, one at a
time– Until the (L + 1)th observation is neither
min nor max or the remaining observations
23
![Page 24: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/24.jpg)
Truncation
24
L
0 5 10 15 20 250
5
10
15
observation number
value
![Page 25: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/25.jpg)
Initial Data Deletion
• m replications• n data points for each replication• Xij = jth data point in ith replication
25
0 2 4 6 8 10 120
5
10
15
j
xij
![Page 26: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/26.jpg)
Initial Data Deletion
• Step 1: average across replications
26
0 2 4 6 8 10 120
2
4
6
8
10
j
mean xj
![Page 27: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/27.jpg)
Initial Data Deletion
• Step 2: compute grand mean µ• Step 3: compute µL = average last n – L values, L
27
0 2 4 6 8 10 120
2
4
6
8
10
starting L
mean of j = L
to n
![Page 28: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/28.jpg)
Initial Data Deletion
• Step 4: offset µL by µ and normalize the result to µ by computing relative change ΔµL = (µL - µ)/µ
28
0 2 4 6 8 10 120
0.1
0.2
0.3
0.4
starting L
relative change
ΔµL
Transient interval
![Page 29: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/29.jpg)
Moving Average of Independent Replications• Similar to initial data deletion
• Requires computing the mean over a sliding time window
29
![Page 30: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/30.jpg)
Moving Average of Independent Replications• m replications
• n data points for each replication• Xij = jth data point in ith replication
30
0 2 4 6 8 10 120
5
10
15
j
xij
![Page 31: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/31.jpg)
Moving Average of Independent Replications• Step 1: average across replications
31
0 2 4 6 8 10 120
2
4
6
8
10
j
mean xj
![Page 32: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/32.jpg)
Moving Average of Independent Replications• Step 2: pick a k, say 1; average (j – k)th
data point to (j + k)th data point, j; increase k as necessary
32
0 2 4 6 8 10 120
2
4
6
8
10
j
Moving avrg for j -
1, j, j + 1
Transient interval
![Page 33: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/33.jpg)
Batch Means
• Used for very long simulations• Divide N data points into m batches of n
data points each• Step 1: pick n, say 1; compute the
mean for each batch• Step 2: compute the mean of means• Step 3: compute the variance of means• Step 4: n++, go to Step 1
33
![Page 34: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/34.jpg)
Batch Means
• Rationale: as n approaches the transient size, the variance peaks
• Does not work well with few data points
34
0 1 2 3 4 5 6 7 8 90
5
10
15
batch size n
vari-ance of batch means
Transient interval
![Page 35: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/35.jpg)
• Terminating simulations: for systems that never reach a steady state– Network traffic consists of the transfer of
small files• Transferring large files to reach steady state is
not useful– System behavior changes with time
• Cyclic behavior– Less need for transient removal
35
Terminating Simulations
![Page 36: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/36.jpg)
Final Conditions
• Handling the end of simulations• Might need to exclude some final data
points– E.g., Mean service time
= total service time/n completed jobs
36
![Page 37: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/37.jpg)
Stopping Criteria: Variance Estimation
• If the simulation run is too short– Results highly variable
• If too long– Wasting resources
• Only need to run until the confidence interval is narrow enough
• Since confidence interval is a function of variance, how do we estimate variance?
37
![Page 38: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/38.jpg)
Independent Replications
• m runs with different seed values• Each run has n + n0 data points
– First n0 data points discarded due to transient phase
• Step 1: compute mean for each replication based on n data points
• Step 2: compute µ, mean of means• Step 3: compute 2, variance of means
38
![Page 39: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/39.jpg)
Independent Replications
• Confidence interval: µ ± z1-α/22
– Use t[1-α/2; m – 1], for m < 30• This method needs to discard mn0 data
points– A good idea to keep m small– Increase n to get narrower confidence
39
![Page 40: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/40.jpg)
Batch Means
• Given a long run of N + n0 data points– First n0 data points discarded due to
transient phase• N data points are divided into m batches
of n data points
40
![Page 41: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/41.jpg)
Batch Means
• Start with n = 1• Step 1: compute the mean for each
batch• Step 2: compute µ, mean of means• Step 3: compute 2, variance of means• Confidence interval: µ ± z1-α/22
– Use t[1-α/2; m – 1], for m < 30
41
![Page 42: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/42.jpg)
Batch Means
• Compared to independent replications– Only need to discard n0 data points
• Problem with batch means – Autocorrelation if the batch size n is small
• Can use the mean of ith batch to guess the mean of (i + 1)th batch
– Need to find a batch size n
42
![Page 43: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/43.jpg)
Batch Means
• Plot batch size n vs. variance of means• Plot batch size n vs. autocovariance
– Cov(batch_meani, batch_meani+1), i
43
0 1 2 3 4 5 6 7 8 9 100
1
2
3
4
5
batch size n
var
0 1 2 3 4 5 6 7 8 9-100%0%
100%200%300%400%500%
batch size n
% of the
sample vari-ance
![Page 44: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/44.jpg)
Method of Regeneration
• Regeneration– Measured effects for a computational cycle
are independent of the previous cycle
44
0
2
4
6
8
time
queue length
Regeneration points
Regeneration cycle
![Page 45: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/45.jpg)
Method of Regeneration
• m regeneration cycles with ni data points each
• Step 1: compute yi, sum for each cycle• Step 2: compute grand mean, µ• Step 3: compute the difference
between expected and observed sums wi = yi - niµ
• Step 4: compute 2 based on wi
45
![Page 46: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/46.jpg)
Method of Regeneration
• Step 5: compute average cycle length, c
• Confidence interval: µ ± z1-α/22/(cm)– Use t[1-α/2; m – 1], for m < 30
46
![Page 47: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/47.jpg)
Method of Regeneration
• Advantages– Does not require removing transient data
points• Disadvantages
– Can be hard to find regeneration points
47
![Page 48: Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dde5503460f94ad68ba/html5/thumbnails/48.jpg)
48
White Slide