September 14, 2004 DCS Colloquium 1
Some Systems, Applicationsand Models I Have Known
Ken Sevcik
University of Toronto
… a retrospective look at many performance evaluation studies
June, 2004 Sigmetrics and Performance 2004 2
Overview
In the past 35 years, … Systems Have Changed Applications Have Grown Models Have Matured and Adapted
… and some interesting problems
have been encountered
June, 2004 Sigmetrics and Performance 2004 3
[One-slide Tutorial:]Approaches To Performance EvaluationHow to answer “What if …” questions
(about hardware, software, and workload) Three alternatives:
Analysis using queueing theory Abstract model, but fast and cheap
Stochastic Simulation Detailed model, and takes some time and work
Experimentation Actual system, but lots of time and work
June, 2004 Sigmetrics and Performance 2004 4
Problems with Voting Systems Defn: “Majority Winner”
A candidate who wins every pairwise election Problems:
Voting for a single candidate … Primaries and Drop Last can eliminate a majority winner
Expressing a full preference ordering … There may be no majority winner!
Question: How likely is a “cyclical majority” (or “voters’ paradox)
where there is no majority winner?
June, 2004 Sigmetrics and Performance 2004 5
Elections and the “will of the people” Assume voter preferences are:
30% : L > M > R 10% : M > L > R 20% : M > R > L 40% : R > M > L
Single Vote: R wins with 40% Yet pairwise M beats both R and L
Preference order and 40% : R > L > M
M
L
R
M
L
R
60%
70%
70%
60%
60%
60%
“Cyclical Majority”
30%
30%
40%
June, 2004 Sigmetrics and Performance 2004 6
First Research Application: Exact Probability of a Voters’ Paradox C candidates for election V voters with strict preference orderings Can one candidate beat each other pairwise?
Example: V = 3 & C = 3 V1 : X > Y > Z V2 : Y > Z > X V3 : Z > X > Y
Then, in pair-wise elections, X beats Y ; and Y beats Z ; yet Z beats X !
Paradox occurs in 12 of the (3!)3 = 216 possible configurations.
In general, there are (C!)V voting configurations.
June, 2004 Sigmetrics and Performance 2004 7
My first “personal” computer: IBM System 360 Model 30 with BOS
June, 2004 Sigmetrics and Performance 2004 8
Exact Probabilities of Voters’ Paradox V = 3 & C = 3 12 cycles in 216 configs. V = 7 & C = 7
26,295,386,028,643,902,475,468,800 cycles in
82,606,411,253,903,523,840,000,000 configs.(Computed in approximately 40 hours of CPU time.)
C = 3 5 7 ~ 40
V = 3 .0555… .1600… .238798185941 ~ .61V = 5 .06944… .19999525 .295755170299 ~ .71V = 7 .075017 .215334 .318321370333 ~ .74
V ~ 40 ~ .09 ~ .24 ~ .36 ~ .80
June, 2004 Sigmetrics and Performance 2004 9
Exact Probabilities of Voters’ Paradox
C = 3 5 7 9 ~ 40
V = 3 .0555… .1600… .238798 .298917 ~ .61V = 5 .06944… .199995 .295755 .367573 ~ .71V = 7 .075017 .21533 .318321 .?????? ~ .74V = 9 .070549 .223717 .330239 .?????? ~ .76
V ~ 40 ~ .09 ~ .24 ~ .36 ~ .45 ~ .80
Recent results:
V = 5 & C = 9 2,312,910,445,872,026,769,020,928,000 cycles 6,292,383,221,978,976,013,516,800,000 configs.
V = 9 & C = 7 692,953,571,964,418,337,059,197,419,520,000 cycles
2,098,335,016,107,155,751,174,144,000,000,000 configs.
June, 2004 Sigmetrics and Performance 2004 10
Job Sequencing on a Single Processor(using service time distribution knowledge)
Given N jobs and their service time distributions,Specify a schedule that minimizes average completion time.
Example with two jobs: job 1 t1 = k
job 2 t2 = s w. prob. 1 - p = t w. prob. p
j1 1st:
j2 1st:
[j2, j1, j2]: j1 j2j2
j1
j1
j2
j2
j2
j2[j2, j1, j2] BEST IFF:
< min [k, s +p (t – s)] s(1 – p)
June, 2004 Sigmetrics and Performance 2004 11
Job Sequencing on a Single Processor
Minimize Investment (quantum length)
Payoff (Pr [Completion])=
Service Time Knowledge exact average distribution
No SPT SEPT SEPTPreemption Allowed?
Yes SRPT SERPT SR
“Smallest Rank” (SR) Scheduling:
(using service time distribution knowledge)
June, 2004 Sigmetrics and Performance 2004 12
Job Sequencing with Two Processors & Two Customers
Extending “Shortest First” to Multiple Resources
SBT-RSBT -- Based on average service time per visit of each customer at each resource
SBT: A gets priority at k
RSBT: A gets priority at 1
2,2, , BA tt
1,1, , BA tt
kBkA tt ,,
2,1,
1,
2,1,
1,
BB
B
AA
A
tt
t
tt
t
June, 2004 Sigmetrics and Performance 2004 13
In the Beginning … Single Server Queue
Many variations arrival process, service process multiple servers, finite buffer size scheduling discipline
FCFS, RR, FBn, PS, SRPT, …
RR, FBn, and PS increased relevance of models
N , Z
S
June, 2004 Sigmetrics and Performance 2004 14
Queuing Network Models
N customers
Z avg. think time
K centersDj demand at j
“Central Server” Model
Variants: Open, Closed, Mixed scheduling disciplines
“Separable”(or “product form”) models
and efficientcomputational algorithms
June, 2004 Sigmetrics and Performance 2004 15
The “Great Debate”:Operational Analysis vs. Stochastic Modeling SM
Ergodic stationary Markov process in equilibrium Coxian distributions of service times independence in service times and routing
OA finite time interval measurable quantities testable assumptions
OA made analytic modelling accessible to capacity
planners in large computing environments
June, 2004 Sigmetrics and Performance 2004 16
Uses and Analysis of Queuing Network Models Applications
System Sizing; Capacity Planning; Tuning Analysis Techniques
Global Balance Solution Massive sets of Simultaneous Linear Equations
Bounds Analysis Asymptotic Bounds (ABA), Balanced System Bounds (BSB)
Solutions of “Separable” Models Exact (Convolution, eMVA) Approximate (aMVA)
Generalizations beyond “Separable” Models aMVA with extended equations
June, 2004 Sigmetrics and Performance 2004 17
Bounding Analysis Case Study: Insurance Company with 20 sites Upgrade alternatives:
Upgrade Dcpu Dio Dtot ImprovementCurrent 4.6 4.0 10.6 ----- # 1 5.1 1.9 7.0 1.5 to 2.0 # 2 3.1 1.9 5.0 2.0 to 3.5
ABA Inputs: N, Z, Dtot, Dmax
Throughput Bound:
Response Time Bound:
max
1,minDZD
NX
tot
ZDNDR tot max,max
June, 2004 Sigmetrics and Performance 2004 18
Bounding Analysis Case Study: Insurance Company with 20 sites Upgrade alternatives:
Upgrade Dcpu Dio Dtot ImprovementCurrent 4.6 4.0 10.6 # 1 5.1 1.9 7.0 1.5 to 2.0 # 2 3.1 1.9 5.0 2.0 to 3.5
X
N
.1
.2
.3
.4
2 6 8 104
Cur
#1
#2
June, 2004 Sigmetrics and Performance 2004 19
Bounding Analysis Case Study: Insurance Company with 20 sites Upgrade alternatives:
Upgrade Dcpu Dio Dtot ImprovementCurrent 4.6 4.0 10.6 # 1 5.1 1.9 7.0 1.5 to 2.0 # 2 3.1 1.9 5.0 2.0 to 3.5
R
N
# 2
Cur# 1
2 4 6 8 10
5
10
15
20
June, 2004 Sigmetrics and Performance 2004 20
Exact Mean Value Analysis Algorithm
00, kQk
for n = 1, … , N
)1(, nQnAk kk
)(1, nADnRk kkk
Understandable and Easy to Implement
Initialize (for zero customers):
Iterate up to N customers:
Set Arrival Instant Queue Lengths:
Set Residence Time:
June, 2004 Sigmetrics and Performance 2004 21
Approximate Mean Value Analysis
K
NNQk k ,
loop until Qk ( N ) are stable
)(1
, NQN
NNAk kk
)(1, NADNRk kkk
Substantial time savings; Little loss of accuracy
Initialize to Equal Queue Lengths:
Iterate until convergence:
Revise Arrival Instant Queue Lengths:
Revise Residence Times:
June, 2004 Sigmetrics and Performance 2004 22
“Details” of Real Systems Going beyond “Separable” models
Priority Scheduling Alter Residence Time equation
FCFS with high variance service times Reflect coefficient of variation in service times
Memory Constraints Alter MPL limit N , or Dpaging
I/O Subsystems (simultaneous resource possession) Reflect contention by inflating Ddisk
Enhanced Utility of QNM’s for Real Systems
)(1 NHDNR hepkk
June, 2004 Sigmetrics and Performance 2004 23
QNM’s for Capacity Planning & Tuning Existing system with measurable workload “What if …”
… the workload volume increases? … the workload mix changes? … the processor is upgraded? … memory is added? … the I/O configuration is enhanced? … class priorities are adjusted? … file placements are changed? … changing usage of memory?
Answer by changing model parameters
CAPACITYPLANNING
TUNING
June, 2004 Sigmetrics and Performance 2004 24
System Sizing Case Study:NASA Numerical Aerodynamic Simulator GOAL: to attain a sustainable Gigaflop
QNM’s proved more useful than a simulation model
Cray 1
Cray 2
Cray 3
Data Mgmt
Graphics
Work Stations
June, 2004 Sigmetrics and Performance 2004 25
Capacity Planning Case Study: FAA Air Traffic Control System ~ 40 distributed air traffic control centers Each with the SAME:
software hardware family 35 transaction types
But DIFFERENT: transaction volumes and mixes
Single QNM (one class per transaction type) supports capacity planning for all sites
June, 2004 Sigmetrics and Performance 2004 26
QNM’s for System and Architecture Analysis Architectures
caching structures
Communication networks Local Area Networks
Rings, buses Store and Forward
flow control end to end response time
Interconnection networks omega, shuffle-exchange, …
June, 2004 Sigmetrics and Performance 2004 27
Network for NASA’s Space Station (circa 1984) Distributed LAN for many components
Ground Station
Results: Some properties of the FDDI Protocol
Space Station
TetheredPlatform
OrbitalPlatform
Shuttle
Extra-Vehicular Activity
June, 2004 Sigmetrics and Performance 2004 28
Architectural Analysis Case Study: NUMAchine 4 x 4 x 4 Hierarchical Ring Architecture
Continuing vs. Upward
Exiting vs. Entering
Setting Routing Priorities:
Contiguous vs. InterleavedShortest First ?
Message Handling:
June, 2004 Sigmetrics and Performance 2004 29
SE&EU Interconnection NetworkSource
000
001
010
011
100
101
110
111
Destination
000
001
010
011
100
101
110
111
Shuffle Exchange
ExchangeUnshuffle
June, 2004 Sigmetrics and Performance 2004 30
SE&EU operation
Up to 40% increase in throughput
Sn Sn-1 Sn-2 S4 S3 S2 S1
Dn Dn-1 Dn-2 D4 D3 D2 D1
(Longest MatchingBit String)
Combination Lock Algorithm:
Bn-3 B4
B3
B2 B1
Bn-2
Bn-1Bn
EU: Right 2SE: Left 5EU: Right 1
June, 2004 Sigmetrics and Performance 2004 31
Job Scheduling for Parallel Processing
time
321
P
processors
Job j = ( tj , pj )Variants: Rigid Moldable Evolving Malleable
June, 2004 Sigmetrics and Performance 2004 32
Parallelism: Early or Late ? Problem
Schedule N jobs of two tasks each on two processors
to minimize average residence time Each pair of jobs can be executed as …
PARALLEL: SEQUENTIAL:
overhead of parallel execution
j2j2
j1j1
June, 2004 Sigmetrics and Performance 2004 33
Parallelism: Early or Late ? Results of two similar studies:
[RN et al.] Start parallel; Finish sequential
P PP P PPSS
SS
SS
June, 2004 Sigmetrics and Performance 2004 34
Parallelism: Early or Late ? Results of two similar studies:
[RN et al.] Start parallel; Finish sequential
[KCS] Start sequential; Finish parallel
S P
P
PP P PP
PP P PP
S
SS
SS
SS
SS
S S
June, 2004 Sigmetrics and Performance 2004 35
Parallelism: Early or Late ? Results of two similar studies:
[RN et al.] Start parallel; Finish sequential
[KCS] Start sequential; Finish parallel
S P
P
PP P PP
PP P PP
S
SS
SS
SS
SS
S S
Differences in assumptions: Some variability in task service times ( or ) [RN] Some overhead of parallelism ( ) [KCS]
June, 2004 Sigmetrics and Performance 2004 36
Parallelism: Early or Late ? Resolution
P P P P P P S S S SS S S S S
P P P P P P P P P PP P P P P
P P P P P S P P S SP P S S S
P P P S P P S S S PS S S S S
P P P P P S P S S SS S S S S
P P P S P P S S P PS S S P P
S S S S S S S S S SS S S S S
increasingincreasing
June, 2004 Sigmetrics and Performance 2004 37
The Case for Popt = 1 :
(Assume p > 1 Ej (p) < 1 ) Argument:
Demand is insatiable (unbounded backlog) Economies of scale (100’s of users) “Good” systems will be heavily used Parallelism overhead decreases throughput
and increases queuing times
pp
WppT jj
jjj
June, 2004 Sigmetrics and Performance 2004 38
Distributed Processing Models Processor selection strategies
local vs. global execution
Load Sharing sender-initiated vs. receiver-initiated
June, 2004 Sigmetrics and Performance 2004 39
Small example: Individual Versus Social Optimum Arriving customers must pick one of two
processors, one fast and one slow:
F
S
F
SIndividual Optimum: Pick server with lower response time ( response times are equalized)Social Optimum: Control pF to minimize average response time
pF
pS
June, 2004 Sigmetrics and Performance 2004 40
Satisfying Social and Individual Goals
SFINDFp
2
1
SOC
FS
SOCF
SOCFF
SOCF
p
p
p
p
1
1
Individual Goal: Equalize Response Times
Social Goal: Minimize Average Response Time
Social Optimum: SF
FSSFSOCFp
Individual Optimum:
IND
FSINDFF pp 1
11
min
June, 2004 Sigmetrics and Performance 2004 41
Resolution of Social and Individual Goals
)1(/1
11
1
SOCF
SOCF
SOCFF
SOCFS
pp
pp
Toll on users of F:
Rebate to users of S:
SOCF
SOCF
p
p
1
RESULT: Individual Choice Yields Social Optimum So Everybody Wins !!!
1. Charge a Toll on the Fast processor;2. Give a Rebate to users of the Slow processor;3. Set total of Rebates to equal the total of Tolls.
June, 2004 Sigmetrics and Performance 2004 42
Resolution of Social and Individual Goals
Example:
90.08.12.00.1 SF
pF RF RS R IND: .87 16.7 16.7 16.7SOC: .85 12.1 27.0 14.3
pF RF RS R CF CS CToll: .85 12.1 27.0 14.3 14.3 14.3 14.3
With Toll = 2.2 (and Rebate = 12.7):
June, 2004 Sigmetrics and Performance 2004 43
Anomaly of High Dimensional Spaces
-2 0 +2
+2
-2
01. Pointy-ness Property
2. Radius of Inner Sphere
3. Volume Ratio kas
V
V
cube
red
1 kRred
kD
D
side
corner
R2 = .414 R10 = 2.16 !!!
2k Spheres (radius = 1) inCube (vol. 4k & 2 k sides) and an Inner sphere
June, 2004 Sigmetrics and Performance 2004 44
Diagonal of a k-dimensional Cube (Example: k = 25 )
Blues =
Red =
Corners =
12 k
2
1k
June, 2004 Sigmetrics and Performance 2004 45
Diagonals of Cube
K = 2
K = 1
K = 3
K = 4
Blue width =
Red width =
Corner width =
12 k
2
1k
June, 2004 Sigmetrics and Performance 2004 46
Diagonals of CubeK = 9
K = 121
1k 2
(There are 2121
blue spheres)
June, 2004 Sigmetrics and Performance 2004 47
Multidimensional DatabasesRelational View:
Multidimensional View:
(Records of k Attributes)
(Points in k-dimensional space)
A1 A2 A3 A4 … Ak-1 Ak
Indexing Support for: -- point search -- range search -- similarity search -- clustering
A1
A2
A3
June, 2004 Sigmetrics and Performance 2004 48
Bounding Spheres and Rectangles
circumscribed inscribed ratio ofDim k sphere cube sphere volumes-------- ---------------- ---------- --------------- ------------- 2 1.57 1.00 .785 2 4 4.93 1.00 .308 16 8 64.94 1.00 .0159 4096 16 15422.64 1.00 .000004 4294967296
2
krsphere 2
1spherer
June, 2004 Sigmetrics and Performance 2004 49
Edge Density in High-Dimensions Proportion of points near some side:
Fraction near some edge:
k eps = .002 .020 .200---- ------ ------ ----- 1 .004 .040 .400 2 .007 .078 .640 4 .015 .150 .870 8 .031 .278 .983 16 .062 .479 .999
kedged 211Pr
21
1
June, 2004 Sigmetrics and Performance 2004 50
Lessons and Conclusions Exact answers are overrated
accurate approximate answers often suffice (e.g., Voters’ Paradox and Exact QNM solutions )
Analytic models have an important role quick, inexpensive answers in many situations
(e.g., Insurance Co., NAS System, and FAA System )
Assumptions matter subtle differences can have big effects
(e.g., in Early or Late Parallelism, NUMAchine analysis and PRI vs. FCFS or PS)
June, 2004 Sigmetrics and Performance 2004 51
What is the “best” way to attain largeimprovements in computer performance?
-- Analysis? -- Simulation? -- Experimentation?
June, 2004 Sigmetrics and Performance 2004 52
What is the “best” way to attain largeimprovements in computer performance?
-- Analysis? -- Simulation? -- Experimentation?
None of the above … Just wait 30 years!!!
September 14, 2004 DCS Colloquium 53
ACM Sigmetrics & IFIP W.G. 7.3 & Dept. of Computer ScienceThanks for the memories …
June, 2004 Sigmetrics and Performance 2004 54
Problems with Voting Systems Problems have occurred recently in ..
France (lowest eliminated) R > M > L 40% L > M > L 40% M > (R, L) 20%
Middle eliminated in first round though rank score (2.2) Beats rank score of others (1.9)
USA (primaries, and electoral college) E.g., McCain loses to Bush in primaries although he Might beat both candidates in a final election
June, 2004 Sigmetrics and Performance 2004 55
Exact Mean Value Analysis Algorithm 00, kQk
for n = 1, … , N
end for
))((/)(
)()(1
ZnRnnX
nRnRK
kk
)1(, nQnAk kk
)(1, nADnRk kkk
)()(, nRnXnQk kk -- Understandable-- Easy to implement-- Arrival Instant Theorem
June, 2004 Sigmetrics and Performance 2004 56
Approximate Mean Value Analysis KNNQk k /,
loop
exit when X(N) and R(N) converge
))((/)(
)()(1
ZNRNNX
NRNRK
kk
)(]/)1[(, NQNNNAk kk
)(1, NHDNRk hepkk
)()(, NRNXNQk kk
-- Substantial time savings -- Little loss of accuracy
end loop
June, 2004 Sigmetrics and Performance 2004 57
System Sizing Case Study:NASA Numerical Aerodynamic Simulator
June, 2004 Sigmetrics and Performance 2004 58
Quiz #1: Sequence Two Jobs on a Processor Service Times:
Rank Calculations:
t1 = 4
t2 = 1 w. prob. .5 10 w. prob. .5
Job Attained Investment Payoff Rank 1 0 4 1.0 4.0 2 0 1 .5 2.0 2 0 5.5 1.0 5.5 2 1 9 1.0 9.0
June, 2004 Sigmetrics and Performance 2004 59
Exact Probabilities of Voters’ Paradox
C = 3 5 7 9 ~ 40
V = 3 .0555… .1600… .238798 .298917 ~ .61V = 5 .06944… .199995 .295755 .367573 ~ .71V = 7 .075017 .21533 .318321 .?????? ~ .74V = 9 .070549 .223717 .?????? .?????? ~ .76
V ~ 40 ~ .09 ~ .24 ~ .36 ~ .45 ~ .80
Recent results:
V = 5 & C = 9 2,312,910,445,872,026,769,020,928,000 cycles
6,292,383,221,978,976,013,516,800,000 configs.
V = 9 & C = 5 1,154,330,758,425,600,000 cycles
5,159,780,352,000,000,000 configs.