Evaluation of Measurement Uncertainty
using Adaptive Monte Carlo Methods
Gerd Wübbeler1, Peter M. Harris2, Maurice G. Cox2, Clemens Elster1
1) Physikalisch-Technische Bundesanstalt (PTB)2) National Physical Laboratory (NPL)
Emerging Topics in Mathematics for Metrology – From Measurement Uncertainty to Metrology of Complex Systems
Physikalisch-Technische Bundesanstalt (PTB)
21-22 June 2010, Berlin, Germany
2
Content
� Evaluation of measurement uncertainty according to GUM S1
� GUM S1 adaptive Monte Carlo scheme
� Alternative approach: Stein’s Two-stage scheme
3
GUM Supplement 1 (GUM S1)
� PDF based method
� Numerical evaluation by a Monte Carlo Method (MCM)
MessgrößeSchätzwert(Messergebnis)
UnsicherheitPDF
Measured data
Further information
Probability density function (PDF)
probabilitydensity
Standarduncertainty
Estimate Measurand
4
Propagation of distributions
PDFs for input quantities PDF for measurandMeasurement model
Change-of-variables
5
GUM S1 Monte Carlo Method (MCM)
Model
),,( 1 NXXfY K=PDF of input quantities
( )NXX Ng ξξ ,,1,,1
K
K
random draw from
evaluation of measurement model ),,( 1 Nf ξξη K=
( )ηYgrandom sample from
( )Nξξ ,,1 K( )NXX N
g ξξ ,,1,,1K
K
η
many repetitions ���� PDF ( )ηYg
[ ] ξξξX d)()()( ∫ −= fggY ηδη
6
321 XXXY =
Illustration
trials
7
)1,0(~ NX i
Convergence
Law of large numbers
8
MCM results exhibit random variations
)1,0(~ NX i
Repetition of the MCM calculation
9
GUM S1 Adaptive Monte Carlo scheme (7.9)
Goal Estimation of the expectation with accuracy with a
coverage probability of about 95 %.
y δ
� Sequential batch-processing mode (e.g. 10 000 trials per batch)
iy� mean of the trials within batch i
iy� for sufficiently large batch size Gaussian distributed (central limit theorem)
),,( 1 hyy K
∑=
−−
=h
iiy hyy
hhs
1
22 ))((1
1)(
∑=
=h
iiy
hhy
1
1)(
10
Start: Batch 1 and 2
Stopping-rule yes
)(new batch hy
δ≤⋅
h
hsy )(2
no
21, yy
2=h
1+= hh
)(ˆ hyy =
GUM S1 Adaptive Monte Carlo scheme (7.9)
11
Assessment of the adaptive schemes
-1 0 1 2 3 4 5 6 70
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Y
prob
abili
ty d
ensi
ty
22
21 XXY +=
121 == xx
1)()( 21 == xuxu
Model
Estimates
UncertaintiesGaussian distributions (uncorrelated)
y = 1.812 9
u(y) = 0.844 6
Rice distribution
12
Goal Determination of the expectation value of the Rice
distribution with an accuracy of δ = 0.005 (95 %)
Assessment Adaptive scheme is repeatedly executed 100 000 times
Result per run
� Estimate of the expectation value
� Number of required batches
Assessment of the adaptive schemes
13
Assessment of the GUM S1 adaptive scheme
Distribution of 10 5 estimates
],[ δδ +− yy
Only 80 % of the results found in the specified accuracy interval
95 % Interval
y
y
(Batch size 10 4, δδδδ=0.005)
14
Distribution of the number of batches
Large number of early
terminations at h = 2
batches (about 32 %)
Assessment of the GUM S1 adaptive scheme
(Batch size 10 4, δδδδ=0.005)
15
Reasons for the behavior of GUM S1 adaptive scheme
Sequential estimation scheme
� Random sample size (random number of batches)
� Multiple (dependent) testing for termination of sampling
Resulting confidence level does not necessarily mee t
the confidence level applied in individual test
� Confidence level of GUM S1 stopping rule adequate for fixed sample size
� Multiple testing and random sample size not taken into account
16
Alternative Adaptive Monte-Carlo scheme
Goal Carry out the MCM until a prescribed accuracy is achieved
at a specified confidence level
(with lowest possible numerical effort)
17
Given i.i.d. from unknown
Goal
so that is a confidence interval
for µ at confidence level 1-α
Stein‘s Two-stage scheme
∑=
=h
iiy
hhyh
1
1)( and
])(,)([ δδ +− hyhy
Step 1
Make random draws
�Variance
11 >h
∑=
−−
=1
1
21
11
2 ))((1
1)(
h
iiy hyy
hhs
1,,1 hyy K
K,, 21 yy 22 ,),,( σµσµN
18
Two-stage scheme
Step 2
Number of additionally required drawings
� make further random draws
� no additional random draws are made
( )
+−
⋅= −− 0,1
)(max 12
22/1,11
2
21 h
thsh hy
δα
02 >h 2h
02 =h
αδµδ −≥++≤≤−+ 1))()(Pr( 2121 hhyhhy
Proof C Stein, 1945, Ann. Math. Statist. 16 243-58
)( 21 hhy +
2h
19
Application of the two-stage scheme within GUM S1
Sequential batch-processing mode
Mean of the trials in batch i
Variance of the trials in batch i
sufficiently large batch size (CLT) ���� ,
)(2 yu i
unbiasediy
iy )(2 yu i
Two-stage scheme applicable for
� Estimate
� Squared uncertainty
y
)(2 yu
approximately Gaussian distributed
20
Assessment of the adaptive schemes
-1 0 1 2 3 4 5 6 70
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Y
prob
abili
ty d
ensi
ty
22
21 XXY +=
121 == xx
1)()( 21 == xuxu
Model
Estimates
UncertaintiesGaussian distributions (uncorrelated)
y = 1.812 9
u(y) = 0.844 6
Rice distribution
21
Assessment of the two-stage scheme
y
(h1=10, Batch size 10 3, δδδδ=0.005, αααα=0.05)
y
],[ δδ +− yy
Interval covers95 % of the estimates �
Distribution of 10 5 estimates
22
Distribution of the number of batches
Assessment of the two-stage scheme
(h1=10, Batch size 10 3, δδδδ=0.005, αααα=0.05)
No earlyterminations
23
��
�
�� 157 271(196)0.969(0.001)10 000
146 383(218)0.951(0.001)1 000
146 423(218)0.952(0.001)100
146 176(218)0.951(0.001)10
145 802(211)0.952(0.001)1
ANT*Success rateBatch size
Determination of the estimate of the measurand: dependence on batch size
)05.0,005.0,10( 1 === αδh
*) ANT: Average Number of Trials
Result for a requested confidence level of 99.9 %
���� Success rate 0.999 08 (0.000 1)
ANT 654 760 (972)
1000) size Batch ,001.0( =α
�
Assessment of the two-stage scheme
24
Summary: Adaptive schemes for GUM S1
� GUM S1 adaptive scheme does not (intend to) meet a 95 % confidence level
� Alternative approach: Two-stage scheme
� Attains specified confidence level for a Gaussian d istribution (Proof by C. Stein, important for metrological appl ications)
� Applicable for the estimate and the squared uncerta inty
� Allows prediction of computation time (number of tr ials)
Wübbeler, Harris, Cox, Elster Metrologia 47 (2010) 317–324