The AutoSimOA Project

Katy Hoad, Stewart Robinson, Ruth DaviesWarwick Business School

WSC 07

A 3 year, EPSRC funded project in collaboration with SIMUL8 Corporation.

http://www.wbs.ac.uk/go/autosimoa

Objective

To provide an easy to use method, that can be incorporated into existing simulation software, that enables practitioners to

obtain results of a specified accuracy from their discrete event simulation model.

(Only looking at analysis of a single scenario)

OUTLINE

IntroductionMethods in literatureOur AlgorithmTest Methodology & ResultsDiscussion & Summary

Underlying Assumptions

Any warm-up problems already dealt with.

Run length (m) decided upon.

Modeller decided to use multiple replications to obtain better estimate of mean

performance.

N

jjXN

X1

1Response measure

of interest

summary statistic from each replication

Perform N replications

QUESTION IS…

How many replications are needed?

Limiting factors: computing time and expense.

4 main methods found in the literature for choosing the number of replications N to perform.

1. Rule of Thumb (Law & McComas 1990)

Run at least 3 to 5 replications.

Advantage: Very simple.

Disadvantage: Does not use characteristics of model output.

No measured precision level.

2. Simple Graphical Method (Robinson 2004)

Cumulative mean graph

45

47

49

51

53

55

1 9 17 25 33 41 49 57 65 73 81 89 97 105

Number of replications (n)

Cum

ula

tive m

ean

Advantages: Simple Uses output of interest in decision.

Disadvantages: Subjective No measured precision level.

3. Confidence Interval Method (Robinson 2004, Law 2007, Banks et al. 2005).

Advantages: Uses statistical inference to determine N.

Uses output of interest in decision.

Provides specified precision.

Disadvantage: Many simulation users do not have the skills to apply approach.

Cumulative mean graph

46

48

50

52

54

56

1 8 15 22 29 36 43 50 57 64 71 78 85 92 99 106

Number of replications (n)

Cum

ula

tive m

ean

4. Prediction Formula (Banks et al. 2005)

• Decide size of error ε that can be can tolerated.• Run ≥ 2 replications - estimate variance s2.• Solve to predict N.

• Check desired precision achieved – if not recalculate N with new estimate of variance.

Advantages: Uses statistical inference to determine N. Uses output of interest in decision. Provides specified precision.

Disadvantage: Can be very inaccurate especially for small number of replications.

2

1,2

st

NN

Run

Model START:

Load Input

Produce Output Results

Run Replication Algorithm

Precision criteria met?

Recommend replication number

Run one more

replication

YES

NO

AUTOMATE Confidence Interval Method: Algorithm interacts with simulation model sequentially.

2,1 nt

n

nn

nX

nt

d

s2,1

100

is the student t value for n-1 df and a significance of 1-α,

nX

sn is the estimate of the standard deviation,

calculated using results Xi (i = 1 to n) of the n current replications.

Where

n is the current number of replications carried out,

We define the precision, dn, as the ½ width of the Confidence Interval expressed as a percentage of the cumulative mean:

is the cumulative mean,

ALGORITHM DEFINITIONS

Stopping Criteria

• Simplest method:

Stop when dn 1st found to be ≤ desired precision, drequired . Recommend that number of replications, Nsol, to user.

• Problem: Data series could prematurely converge, by chance, to incorrect estimate of the mean, with precision drequired , then diverge again.

• ‘Look-ahead’ procedure: When dn 1st found to be ≤ drequired, algorithm performs set number of extra replications, to check that precision remains ≤ drequired.

0

20

40

60

80

100

120

140

3 100

137

174

211

248

285

322

359

396

433

470

replication number (n )

f(kLim

it)

kLimit=0 kLimit=5

kLimit=10 kLimit=25

‘Look-ahead’ procedure

kLimit = ‘look ahead’ value. Actual number of replications checked ahead is

Relates ‘look ahead’ period length with current value of n.

100,100

100,)(

nkLimitn

nkLimitkLimitf

23

25

27

29

31

33

35

37

3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Replication number (n)

NsolNsol + f(kLimit)

f(kLimit)

Precision ≤ 5%X

X

95% confidence limits

Cumulative mean,

Replication Algorithm

0.8

0.85

0.9

0.95

1

1.05

1.1

1.15

1.2

3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

Replication number (n)

Precision

≤ 5%

Precision

> 5%

Precision ≤ 5%

f(kLimit)

Nsol2Nsol2 + f(kLimit)

Nsol1

• 24 artificial data sets: Left skewed, symmetric, right skewed; Varying values of relative st.dev (st.dev/mean).

• 100 sequences of 2000 data values.

• 8 real models selected.

• Different lengths of ‘look ahead’ period tested:

kLimit values = 0 (i.e. no ‘look ahead’), 5, 10, 25.

• drequired value kept constant at 5%.

TESTING METHODOLOGY

5 performance measures

1. Coverage of the true mean2. Bias3. Absolute Bias4. Average Nsol value5. Comparison of 4. with Theoretical Nsol

value

• For real models: ‘true’ mean & variance values - estimated from whole sets of output data (3000 to 11000 data points).

Microsoft Excel Worksheet

Results

• Nsol values for individual algorithm runs are very variable.

• Average Nsol values for 100 runs per model close to the theoretical values of Nsol.

• Normality assumption appears robust.

• Using a ‘look ahead’ period improves performance of the algorithm.

Mean bias significantly different to zero

Failed in coverage of true mean

Mean est. Nsol significantly different to theoretical Nsol (>3)

No ‘look-ahead’ period

Proportion of Artificial models

4/24 2/24 9/18

Proportion of Real models

1/8 1/8 3/5

kLimit = 5 Proportion of Artificial models

1/24 0 1/18

Proportion of Real models

0 0 0

% decrease in absolute mean bias

kLimit = 0 tokLimit = 5

kLimit = 5 tokLimit = 10

kLimit = 10 tokLimit = 25

ArtificialModels

8.76% 0.07% 0.26%

RealModels

10.45% 0.14% 0.33%

Impact of different look ahead periods on performance of algorithm

Number of times the Nsol value changes (out of 100 runs of the algorithm per model) because of the lengthening of the ‘look ahead’ period.

Model ID

kLimit = 0 to kLimit = 5

kLimit = 5 tokLimit = 10

kLimit = 10 to kLimit = 25

R1 0 0 0

R3 2 0 0

R5 24 0 1

R8 24 4 1

A5 30 1 3

A6 26 6 3

A15 1 0 0

A17 22 0 1

A21 25 2 1

A24 37 0 0

Model ID

kLimit Nsol Theoretical Nsol (approx)

Mean estimate significantly different to the true mean?

A9 0 4 112 Yes

  5 120 No

A24 0 3 755 Yes

  5 718 No

R7 0 3 10 Yes

  5 8 No

R4 0 3 6 Yes

5 7 No

R8 0 3 45 Yes

  5 46 No

Examples of changes in Nsol & improvement in estimate of true mean

DISCUSSION

• kLimit default value set to 5.

• Initial number of replications set to 3.

• Multiple response variables - Algorithm run with each response - use maximum estimated value for Nsol.

• Different scenarios - advisable to repeat algorithm every few scenarios to check that precision has not degraded significantly.

• Implementation into Simul8 simulation package.

SUMMARY

• Selection and automation of Confidence Interval Method for estimating the number of replications to be run in a simulation.

• Algorithm created with ‘look ahead’ period -efficient and performs well on wide selection of artificial and real model output.

• ‘Black box’ - fully automated and does not require user intervention.

ACKNOWLEDGMENTSThis work is part of the Automating Simulation Output

Analysis (AutoSimOA) project (http://www.wbs.ac.uk/go/autosimoa) that is funded by

the UK Engineering and Physical Sciences Research Council (EP/D033640/1). The work is being carried out in

collaboration with SIMUL8 Corporation, who are also providing sponsorship for the project.

Katy Hoad, Stewart Robinson, Ruth DaviesWarwick Business School

WSC 07