september 14, 2004 dcs colloquium 1 some systems, applications and models i have known ken sevcik...

September 14, 2004 DCS Colloquium 1

Some Systems, Applicationsand Models I Have Known

Ken Sevcik

University of Toronto

… a retrospective look at many performance evaluation studies

June, 2004 Sigmetrics and Performance 2004 2

Overview

In the past 35 years, … Systems Have Changed Applications Have Grown Models Have Matured and Adapted

… and some interesting problems

have been encountered


[One-slide Tutorial:]Approaches To Performance EvaluationHow to answer “What if …” questions

(about hardware, software, and workload) Three alternatives:

Analysis using queueing theory Abstract model, but fast and cheap

Stochastic Simulation Detailed model, and takes some time and work

Experimentation Actual system, but lots of time and work


Problems with Voting Systems Defn: “Majority Winner”

A candidate who wins every pairwise election Problems:

Voting for a single candidate … Primaries and Drop Last can eliminate a majority winner

Expressing a full preference ordering … There may be no majority winner!

Question: How likely is a “cyclical majority” (or “voters’ paradox)

where there is no majority winner?


Elections and the “will of the people” Assume voter preferences are:

30% : L > M > R 10% : M > L > R 20% : M > R > L 40% : R > M > L

Single Vote: R wins with 40% Yet pairwise M beats both R and L

Preference order and 40% : R > L > M

M

L

R

M

L

R

60%

70%

70%

60%

60%

60%

“Cyclical Majority”

30%

30%

40%


First Research Application: Exact Probability of a Voters’ Paradox C candidates for election V voters with strict preference orderings Can one candidate beat each other pairwise?

Example: V = 3 & C = 3 V1 : X > Y > Z V2 : Y > Z > X V3 : Z > X > Y

Then, in pair-wise elections, X beats Y ; and Y beats Z ; yet Z beats X !

Paradox occurs in 12 of the (3!)3 = 216 possible configurations.

In general, there are (C!)V voting configurations.


My first “personal” computer: IBM System 360 Model 30 with BOS


Exact Probabilities of Voters’ Paradox V = 3 & C = 3 12 cycles in 216 configs. V = 7 & C = 7

26,295,386,028,643,902,475,468,800 cycles in

82,606,411,253,903,523,840,000,000 configs.(Computed in approximately 40 hours of CPU time.)

C = 3 5 7 ~ 40

V = 3 .0555… .1600… .238798185941 ~ .61V = 5 .06944… .19999525 .295755170299 ~ .71V = 7 .075017 .215334 .318321370333 ~ .74

V ~ 40 ~ .09 ~ .24 ~ .36 ~ .80


Exact Probabilities of Voters’ Paradox

C = 3 5 7 9 ~ 40

V = 3 .0555… .1600… .238798 .298917 ~ .61V = 5 .06944… .199995 .295755 .367573 ~ .71V = 7 .075017 .21533 .318321 .?????? ~ .74V = 9 .070549 .223717 .330239 .?????? ~ .76

V ~ 40 ~ .09 ~ .24 ~ .36 ~ .45 ~ .80

Recent results:

V = 5 & C = 9 2,312,910,445,872,026,769,020,928,000 cycles 6,292,383,221,978,976,013,516,800,000 configs.

V = 9 & C = 7 692,953,571,964,418,337,059,197,419,520,000 cycles

2,098,335,016,107,155,751,174,144,000,000,000 configs.


Job Sequencing on a Single Processor(using service time distribution knowledge)

Given N jobs and their service time distributions,Specify a schedule that minimizes average completion time.

Example with two jobs: job 1 t1 = k

job 2 t2 = s w. prob. 1 - p = t w. prob. p

j1 1st:

j2 1st:

[j2, j1, j2]: j1 j2j2

j1

j1

j2

j2

j2

j2[j2, j1, j2] BEST IFF:

< min [k, s +p (t – s)] s(1 – p)


Job Sequencing on a Single Processor

Minimize Investment (quantum length)

Payoff (Pr [Completion])=

Service Time Knowledge exact average distribution

No SPT SEPT SEPTPreemption Allowed?

Yes SRPT SERPT SR

“Smallest Rank” (SR) Scheduling:

(using service time distribution knowledge)


Job Sequencing with Two Processors & Two Customers

Extending “Shortest First” to Multiple Resources

SBT-RSBT -- Based on average service time per visit of each customer at each resource

SBT: A gets priority at k

RSBT: A gets priority at 1

2,2, , BA tt

1,1, , BA tt

kBkA tt ,,

2,1,

1,

2,1,

1,

BB

B

AA

A

tt

t

tt

t


In the Beginning … Single Server Queue

Many variations arrival process, service process multiple servers, finite buffer size scheduling discipline

FCFS, RR, FBn, PS, SRPT, …

RR, FBn, and PS increased relevance of models

N , Z

S


Queuing Network Models

N customers

Z avg. think time

K centersDj demand at j

“Central Server” Model

Variants: Open, Closed, Mixed scheduling disciplines

“Separable”(or “product form”) models

and efficientcomputational algorithms


The “Great Debate”:Operational Analysis vs. Stochastic Modeling SM

Ergodic stationary Markov process in equilibrium Coxian distributions of service times independence in service times and routing

OA finite time interval measurable quantities testable assumptions

OA made analytic modelling accessible to capacity

planners in large computing environments


Uses and Analysis of Queuing Network Models Applications

System Sizing; Capacity Planning; Tuning Analysis Techniques

Global Balance Solution Massive sets of Simultaneous Linear Equations

Bounds Analysis Asymptotic Bounds (ABA), Balanced System Bounds (BSB)

Solutions of “Separable” Models Exact (Convolution, eMVA) Approximate (aMVA)

Generalizations beyond “Separable” Models aMVA with extended equations


Bounding Analysis Case Study: Insurance Company with 20 sites Upgrade alternatives:

Upgrade Dcpu Dio Dtot ImprovementCurrent 4.6 4.0 10.6 ----- # 1 5.1 1.9 7.0 1.5 to 2.0 # 2 3.1 1.9 5.0 2.0 to 3.5

ABA Inputs: N, Z, Dtot, Dmax

Throughput Bound:

Response Time Bound:

max

1,minDZD

NX

tot

ZDNDR tot max,max



Upgrade Dcpu Dio Dtot ImprovementCurrent 4.6 4.0 10.6 # 1 5.1 1.9 7.0 1.5 to 2.0 # 2 3.1 1.9 5.0 2.0 to 3.5

X

N

.1

.2

.3

.4

2 6 8 104

Cur

#1

#2



Upgrade Dcpu Dio Dtot ImprovementCurrent 4.6 4.0 10.6 # 1 5.1 1.9 7.0 1.5 to 2.0 # 2 3.1 1.9 5.0 2.0 to 3.5

R

N

# 2

Cur# 1

2 4 6 8 10

5

10

15

20


Exact Mean Value Analysis Algorithm

00, kQk

for n = 1, … , N

)1(, nQnAk kk

)(1, nADnRk kkk

Understandable and Easy to Implement

Initialize (for zero customers):

Iterate up to N customers:

Set Arrival Instant Queue Lengths:

Set Residence Time:


Approximate Mean Value Analysis

K

NNQk k ,

loop until Qk ( N ) are stable

)(1

, NQN

NNAk kk

)(1, NADNRk kkk

Substantial time savings; Little loss of accuracy

Initialize to Equal Queue Lengths:

Iterate until convergence:

Revise Arrival Instant Queue Lengths:

Revise Residence Times:


“Details” of Real Systems Going beyond “Separable” models

Priority Scheduling Alter Residence Time equation

FCFS with high variance service times Reflect coefficient of variation in service times

Memory Constraints Alter MPL limit N , or Dpaging

I/O Subsystems (simultaneous resource possession) Reflect contention by inflating Ddisk

Enhanced Utility of QNM’s for Real Systems

)(1 NHDNR hepkk


QNM’s for Capacity Planning & Tuning Existing system with measurable workload “What if …”

… the workload volume increases? … the workload mix changes? … the processor is upgraded? … memory is added? … the I/O configuration is enhanced? … class priorities are adjusted? … file placements are changed? … changing usage of memory?

Answer by changing model parameters

CAPACITYPLANNING

TUNING


System Sizing Case Study:NASA Numerical Aerodynamic Simulator GOAL: to attain a sustainable Gigaflop

QNM’s proved more useful than a simulation model

Cray 1

Cray 2

Cray 3

Data Mgmt

Graphics

Work Stations


Capacity Planning Case Study: FAA Air Traffic Control System ~ 40 distributed air traffic control centers Each with the SAME:

software hardware family 35 transaction types

But DIFFERENT: transaction volumes and mixes

Single QNM (one class per transaction type) supports capacity planning for all sites


QNM’s for System and Architecture Analysis Architectures

caching structures

Communication networks Local Area Networks

Rings, buses Store and Forward

flow control end to end response time

Interconnection networks omega, shuffle-exchange, …


Network for NASA’s Space Station (circa 1984) Distributed LAN for many components

Ground Station

Results: Some properties of the FDDI Protocol

Space Station

TetheredPlatform

OrbitalPlatform

Shuttle

Extra-Vehicular Activity


Architectural Analysis Case Study: NUMAchine 4 x 4 x 4 Hierarchical Ring Architecture

Continuing vs. Upward

Exiting vs. Entering

Setting Routing Priorities:

Contiguous vs. InterleavedShortest First ?

Message Handling:


SE&EU Interconnection NetworkSource

000

001

010

011

100

101

110

111

Destination

000

001

010

011

100

101

110

111

Shuffle Exchange

ExchangeUnshuffle


SE&EU operation

Up to 40% increase in throughput

Sn Sn-1 Sn-2 S4 S3 S2 S1

Dn Dn-1 Dn-2 D4 D3 D2 D1

(Longest MatchingBit String)

Combination Lock Algorithm:

Bn-3 B4

B3

B2 B1

Bn-2

Bn-1Bn

EU: Right 2SE: Left 5EU: Right 1


Job Scheduling for Parallel Processing

time

321

P

processors

Job j = ( tj , pj )Variants: Rigid Moldable Evolving Malleable


Parallelism: Early or Late ? Problem

Schedule N jobs of two tasks each on two processors

to minimize average residence time Each pair of jobs can be executed as …

PARALLEL: SEQUENTIAL:

overhead of parallel execution

j2j2

j1j1


Parallelism: Early or Late ? Results of two similar studies:

[RN et al.] Start parallel; Finish sequential

P PP P PPSS

SS

SS




[KCS] Start sequential; Finish parallel

S P

P

PP P PP

PP P PP

S

SS

SS

SS

SS

S S




[KCS] Start sequential; Finish parallel

S P

P

PP P PP

PP P PP

S

SS

SS

SS

SS

S S

Differences in assumptions: Some variability in task service times ( or ) [RN] Some overhead of parallelism ( ) [KCS]


Parallelism: Early or Late ? Resolution

P P P P P P S S S SS S S S S

P P P P P P P P P PP P P P P

P P P P P S P P S SP P S S S

P P P S P P S S S PS S S S S

P P P P P S P S S SS S S S S

P P P S P P S S P PS S S P P

S S S S S S S S S SS S S S S

increasingincreasing


The Case for Popt = 1 :

(Assume p > 1 Ej (p) < 1 ) Argument:

Demand is insatiable (unbounded backlog) Economies of scale (100’s of users) “Good” systems will be heavily used Parallelism overhead decreases throughput

and increases queuing times

pp

WppT jj

jjj


Distributed Processing Models Processor selection strategies

local vs. global execution

Load Sharing sender-initiated vs. receiver-initiated


Small example: Individual Versus Social Optimum Arriving customers must pick one of two

processors, one fast and one slow:

F

S

F

SIndividual Optimum: Pick server with lower response time ( response times are equalized)Social Optimum: Control pF to minimize average response time

pF

pS


Satisfying Social and Individual Goals

SFINDFp

2

1

SOC

FS

SOCF

SOCFF

SOCF

p

p

p

p

1

1

Individual Goal: Equalize Response Times

Social Goal: Minimize Average Response Time

Social Optimum: SF

FSSFSOCFp

Individual Optimum:

IND

FSINDFF pp 1

11

min


Resolution of Social and Individual Goals

)1(/1

11

1

SOCF

SOCF

SOCFF

SOCFS

pp

pp

Toll on users of F:

Rebate to users of S:

SOCF

SOCF

p

p

1

RESULT: Individual Choice Yields Social Optimum So Everybody Wins !!!

1. Charge a Toll on the Fast processor;2. Give a Rebate to users of the Slow processor;3. Set total of Rebates to equal the total of Tolls.


Resolution of Social and Individual Goals

Example:

90.08.12.00.1 SF

pF RF RS R IND: .87 16.7 16.7 16.7SOC: .85 12.1 27.0 14.3

pF RF RS R CF CS CToll: .85 12.1 27.0 14.3 14.3 14.3 14.3

With Toll = 2.2 (and Rebate = 12.7):


Anomaly of High Dimensional Spaces

-2 0 +2

+2

-2

01. Pointy-ness Property

2. Radius of Inner Sphere

3. Volume Ratio kas

V

V

cube

red

1 kRred

kD

D

side

corner

R2 = .414 R10 = 2.16 !!!

2k Spheres (radius = 1) inCube (vol. 4k & 2 k sides) and an Inner sphere


Diagonal of a k-dimensional Cube (Example: k = 25 )

Blues =

Red =

Corners =

12 k

2

1k


Diagonals of Cube

K = 2

K = 1

K = 3

K = 4

Blue width =

Red width =

Corner width =

12 k

2

1k


Diagonals of CubeK = 9

K = 121

1k 2

(There are 2121

blue spheres)


Multidimensional DatabasesRelational View:

Multidimensional View:

(Records of k Attributes)

(Points in k-dimensional space)

A1 A2 A3 A4 … Ak-1 Ak

Indexing Support for: -- point search -- range search -- similarity search -- clustering

A1

A2

A3


Bounding Spheres and Rectangles

circumscribed inscribed ratio ofDim k sphere cube sphere volumes-------- ---------------- ---------- --------------- ------------- 2 1.57 1.00 .785 2 4 4.93 1.00 .308 16 8 64.94 1.00 .0159 4096 16 15422.64 1.00 .000004 4294967296

2

krsphere 2

1spherer


Edge Density in High-Dimensions Proportion of points near some side:

Fraction near some edge:

k eps = .002 .020 .200---- ------ ------ ----- 1 .004 .040 .400 2 .007 .078 .640 4 .015 .150 .870 8 .031 .278 .983 16 .062 .479 .999

kedged 211Pr

21

1


Lessons and Conclusions Exact answers are overrated

accurate approximate answers often suffice (e.g., Voters’ Paradox and Exact QNM solutions )

Analytic models have an important role quick, inexpensive answers in many situations

(e.g., Insurance Co., NAS System, and FAA System )

Assumptions matter subtle differences can have big effects

(e.g., in Early or Late Parallelism, NUMAchine analysis and PRI vs. FCFS or PS)


What is the “best” way to attain largeimprovements in computer performance?

-- Analysis? -- Simulation? -- Experimentation?


What is the “best” way to attain largeimprovements in computer performance?

-- Analysis? -- Simulation? -- Experimentation?

None of the above … Just wait 30 years!!!

September 14, 2004 DCS Colloquium 53

ACM Sigmetrics & IFIP W.G. 7.3 & Dept. of Computer ScienceThanks for the memories …


Problems with Voting Systems Problems have occurred recently in ..

France (lowest eliminated) R > M > L 40% L > M > L 40% M > (R, L) 20%

Middle eliminated in first round though rank score (2.2) Beats rank score of others (1.9)

USA (primaries, and electoral college) E.g., McCain loses to Bush in primaries although he Might beat both candidates in a final election


Exact Mean Value Analysis Algorithm 00, kQk

for n = 1, … , N

end for

))((/)(

)()(1

ZnRnnX

nRnRK

kk

)1(, nQnAk kk

)(1, nADnRk kkk

)()(, nRnXnQk kk -- Understandable-- Easy to implement-- Arrival Instant Theorem


Approximate Mean Value Analysis KNNQk k /,

loop

exit when X(N) and R(N) converge

))((/)(

)()(1

ZNRNNX

NRNRK

kk

)(]/)1[(, NQNNNAk kk

)(1, NHDNRk hepkk

)()(, NRNXNQk kk

-- Substantial time savings -- Little loss of accuracy

end loop


System Sizing Case Study:NASA Numerical Aerodynamic Simulator


Quiz #1: Sequence Two Jobs on a Processor Service Times:

Rank Calculations:

t1 = 4

t2 = 1 w. prob. .5 10 w. prob. .5

Job Attained Investment Payoff Rank 1 0 4 1.0 4.0 2 0 1 .5 2.0 2 0 5.5 1.0 5.5 2 1 9 1.0 9.0


Exact Probabilities of Voters’ Paradox

C = 3 5 7 9 ~ 40

V = 3 .0555… .1600… .238798 .298917 ~ .61V = 5 .06944… .199995 .295755 .367573 ~ .71V = 7 .075017 .21533 .318321 .?????? ~ .74V = 9 .070549 .223717 .?????? .?????? ~ .76

V ~ 40 ~ .09 ~ .24 ~ .36 ~ .45 ~ .80

Recent results:

V = 5 & C = 9 2,312,910,445,872,026,769,020,928,000 cycles

6,292,383,221,978,976,013,516,800,000 configs.

V = 9 & C = 5 1,154,330,758,425,600,000 cycles

5,159,780,352,000,000,000 configs.

september 14, 2004 dcs colloquium 1 some systems, applications and models i have known ken sevcik...

Documents