automating estimation of warm-up length katy hoad, stewart robinson, ruth davies warwick business...
Post on 19-Dec-2015
213 views
TRANSCRIPT
Automating estimation of warm-up length
Katy Hoad, Stewart Robinson, Ruth DaviesWarwick Business School
WSC08
The AutoSimOA ProjectA 3 year, EPSRC funded project in collaboration with SIMUL8 Corporation.
http://www.wbs.ac.uk/go/autosimoa
Research Aim
• To create an automated system for dealing with the problem of initial bias, for implementation into simulation software.
• Target audience: non- (statistically) expert simulation users.
The Initial Bias Problem
• Model may not start in a “typical” state.
• Can cause initial bias in the output.
• Method used: Deletion of the initial transient data by specifying a warm-up period (or truncation point).
• How do you estimate the length of the warm-up period required?
• Literature search – 44 methods
• Short-listing of methods• Accuracy & robustness
• Ease of automation
• Generality
• Computer running time
• Preliminary Testing – 6 methods
• MSER-5 most accurate and robust method.
MSER-5 warm-up method
0
0.002
0.004
0.006
0.008
0.01
0.012
0.014
0.016
0.018
0.02
0 50 100 150 200 250 300 350 400
Truncation Point
Tes
t S
tatis
tic
0
1
2
3
4
5
6
Bat
ch M
eans
MSER-5 test statistic
Rejection zone
Estimated warm-up period
Estimated truncation point, Lsol
Output data (batched means values)
Further Testing of MSER-5
1. Artificial data – controllable & comparable initial bias functions steady state functions
2. Full factorial design.
3. Set of performance criteria.
Parameters Levels
Data Type Single run
Data averaged over 5 reps
Error type N(1,1), Exp(1)
Auto-correlation
None, AR(1), AR(2), MA(2), AR(4), ARMA(5,5)
Bias Severity 1, 2, 4
Bias Length 0%, 10%, 40%, 100% (of n = 1000)
Bias direction Positive, Negative
Bias shape 7 shapes
1. Artificial Data Parameters
• Mean Shift:
• Linear:
• Quadratic:
• Exponential:
• Oscillating (decreasing):
Quadratic ExponentialLinear
Add Initial Bias to Steady state:
Superpostion: Bias Fn, a(t), added onto end of steady state function:
e.g.
2. Full factorial design
3048 types of artificial data set
MSER-5 run with each type 100 times
...
)(1
etc
taXY
XX
tt
ttt
i. Coverage of true mean.
ii. Closeness of estimated truncation point (Lsol) to true truncation point (L).
iii. Percentage bias removed by truncation.
iv. Analysis of the pattern & frequency of rejections of Lsol (i.e. Lsol > n/2).
3. Performance Criteria
MSER-5 Results
Does the true mean fall into the 95% CI for the estimated mean?
Non-truncated data sets
Truncated data sets
% of cases
yes yes 7.7%
no yes 72.5%
no no 19.8%
yes no 0%
i. Coverage of true mean.
-70
-50
-30
-10
10
30
50
0 20 40 60 80 100run
Lsol -
L
Quadratic bias Mean-shift bias
ii. Closeness of Lsol to L.
• Wide range of Lsol values.
e.g.
(Positive bias functions, single run data, N(1,1) errors, MA(2) auto-correlation, bias severity value of 2 and true L = 100.)
iii. Percentage bias removed by truncation.
0
5
10
15
20
25
300-4
0
40-5
0
50-6
0
60-7
0
70-8
0
80-9
0
90-9
5
95-9
9
99-1
00
100+
% bias removed
% o
f to
tal v
alid
runs
All valid runs
Effect of data parameters on bias removal
No significant effect: Error type Bias direction
Significant effect: Data type Auto-correlation
type Bias shape Bias severity Bias length
0
50
100
0-4
0
40
-50
50
-60
60
-70
70
-80
80
-90
90
-95
95
-99
99
-10
0
10
0+
% of bias removed
cum
ula
tive
% o
f va
lid c
ase
s Single run
Averaged replications
More bias removed by using averaged replications rather than a single run.
0
50
1000
-40
40
-50
50
-60
60
-70
70
-80
80
-90
90
-95
95
-99
99
-10
0
10
0+
% of bias removed
cu
mu
lative
% o
f va
lid
ca
se
s no a-c AR(1)
AR(2) AR(4)
MA(2) ARMA(5,5)
The stronger the auto-correlation, the less accurate the bias removal.
Effect greatly reduced by using averaged data.
0
50
100
0-4
0
40
-50
50
-60
60
-70
70
-80
80
-90
90
-95
95
-99
99
-10
0
10
0+
% of bias removed
cu
mu
lative
% o
f va
lid
ca
se
s
mean-shift Linear
Quad Exp
OscL OscQ
OscE
The more sharply the initial bias declines, the more likely MSER-5 is to underestimate the warm-up period and to remove increasingly less bias.
0
50
1000
-40
40
-50
50
-60
60
-70
70
-80
80
-90
90
-95
95
-99
99
-10
0
10
0+
% of bias removed
cum
ula
tive
% o
f va
lid c
ase
s 1
2
4
As the bias severity increases, MSER-5 removes an increasingly higher percentage of the bias.
0
50
100
0-4
0
40
-50
50
-60
60
-70
70
-80
80
-90
90
-95
95
-99
99
-10
0
10
0+
reje
ctio
ns
% of bias removed
cum
ula
tive
% o
f va
lid c
ase
s
10%
40%
Longer bias removed slightly more efficiently than shorter bias.
Shorter bias - more overestimations - partly due to longer bias overestimations being more likely to be rejected.
0
100
200
300
400
500
600
700
800
900
x=
0
0<
x≤1
1<
x≤5
5<
x≤1
0
10
<x≤2
0
20
<x≤4
0
40
<x≤6
0
60
<x≤8
0
80
<x≤1
00
x = no. of Lsol rejections
no
. o
f ca
se
s
ARMA(5,5)
MA(2)
AR(4)
AR(2)
AR(1)
No auto-correlation
Rejections caused by: high auto-correlation, bias close to n/2, smooth end to data = ‘end point’ rejection.
Averaged data slightly increases probability of getting ‘end point’ rejection but increases probability of more accurate L estimates.
iv. Lsol rejections
0
10
20
30
40
50
1000 1100 1200 1300 1400 1500 1600 1700 1800n
Lso
l re
ject
ion
co
un
t
+ meanshift
+ linear
+ quadratic
+ exp
+ osclinear
+ oscquad
+ oscexp
Giving more data to MSER-5 in an iterative fashion produces a valid Lsol value where previously the Lsol value had been rejected.
e.g. ARMA(5,5)
Lsol values Percentage of cases
Lsol = 0 71%
Lsol ≤ 50 93%
Testing MSER-5 with data that has no initial bias.
Want Lsol = 0
Lsol > 50 mainly due to highest auto-correlated data sets - AR(1) & ARMA(5,5).
Rejected Lsol values: 5.6% of the 2400 Lsol values produced. 93% from the highest auto-correlated data ARMA(5,5).
Testing MSER-5 with data that has 100% bias.
Want 100% rejection rate: Actual rate = 61%
0
1020
30
4050
60
70
8090
100
Line
ar
Qua
d
Exp
Osc
Line
ar
Osc
Qua
d
Osc
Exp
Bias shape
Per
cent
age
of L
sol
reje
ctio
ns
0
10
20
30
40
50
60
70
80
90
M1 M2 M4
Bias severity
Per
cent
age
of L
sol
rej
ectio
ns
Single data Averaged data
Summary
• MSER-5 most promising method for automation– Not model or data type specific. – No estimation of parameters needed. – Can function without user intervention. – Shown to perform robustly and effectively
for the majority of data sets tested. – Quick to run. – Fairly simple to understand.
Heuristic framework around MSER-5
Run k (= 5) replications of length, n ≥ 100
Create averaged
data
Batch data into b batches of length m, where number of
batches = bmn and n* =
b×m ≤ n
MSER-5 returns Lsol value
Produce more data to create
batches of no. orig of %10 or a user specified
number.
Dynamic graph of batched data; single reps, or
MSER-5 statistic
Graph of batched data; single reps,
or MSER-5 statistic with valid Lsol value shown.
Input data into MSER-5 algorithm.
Yes
Yes
No
No
Does User wish to keep running with more data? END
Lsol valid.
Lsol invalid.
Is Lsol ≤ (n* - (m × 5))/2
?
Yes
Have there been 10 invalid Lsol
values in a row?
No
Yes No
Does User wish to keep running with more data?
Produce more data to create
batches of no. orig of %10
Iterative procedure for procuring more data when required.
‘Failsafe’ mechanism - to deal with possibility of data not in steady state; insufficient data provided when highly auto-correlated.
Being implemented in SIMUL8.
ACKNOWLEDGMENTSThis work is part of the Automating Simulation Output
Analysis (AutoSimOA) project (http://www.wbs.ac.uk/go/autosimoa) that is funded by
the UK Engineering and Physical Sciences Research Council (EP/D033640/1). The work is being carried out in
collaboration with SIMUL8 Corporation, who are also providing sponsorship for the project.
Katy Hoad, Stewart Robinson, Ruth DaviesWarwick Business School
WSC08