evaluation of measurement uncertainty using adaptive monte ......evaluation of measurement...

Evaluation of Measurement Uncertainty

using Adaptive Monte Carlo Methods

Gerd Wübbeler1, Peter M. Harris2, Maurice G. Cox2, Clemens Elster1

1) Physikalisch-Technische Bundesanstalt (PTB)2) National Physical Laboratory (NPL)

Emerging Topics in Mathematics for Metrology – From Measurement Uncertainty to Metrology of Complex Systems

Physikalisch-Technische Bundesanstalt (PTB)

21-22 June 2010, Berlin, Germany

2

Content

� Evaluation of measurement uncertainty according to GUM S1

� GUM S1 adaptive Monte Carlo scheme

� Alternative approach: Stein’s Two-stage scheme

3

GUM Supplement 1 (GUM S1)

� PDF based method

� Numerical evaluation by a Monte Carlo Method (MCM)

MessgrößeSchätzwert(Messergebnis)

UnsicherheitPDF

Measured data

Further information

Probability density function (PDF)

probabilitydensity

Standarduncertainty

Estimate Measurand

4

Propagation of distributions

PDFs for input quantities PDF for measurandMeasurement model

Change-of-variables

5

GUM S1 Monte Carlo Method (MCM)

Model

),,( 1 NXXfY K=PDF of input quantities

( )NXX Ng ξξ ,,1,,1

K

K

random draw from

evaluation of measurement model ),,( 1 Nf ξξη K=

( )ηYgrandom sample from

( )Nξξ ,,1 K( )NXX N

g ξξ ,,1,,1K

K

η

many repetitions ���� PDF ( )ηYg

[ ] ξξξX d)()()( ∫ −= fggY ηδη

6

321 XXXY =

Illustration

trials

7

)1,0(~ NX i

Convergence

Law of large numbers

8

MCM results exhibit random variations

)1,0(~ NX i

Repetition of the MCM calculation

9

GUM S1 Adaptive Monte Carlo scheme (7.9)

Goal Estimation of the expectation with accuracy with a

coverage probability of about 95 %.

y δ

� Sequential batch-processing mode (e.g. 10 000 trials per batch)

iy� mean of the trials within batch i

iy� for sufficiently large batch size Gaussian distributed (central limit theorem)

),,( 1 hyy K

∑=

−−

=h

iiy hyy

hhs

1

22 ))((1

1)(

∑=

=h

iiy

hhy

1

1)(

10

Start: Batch 1 and 2

Stopping-rule yes

)(new batch hy

δ≤⋅

h

hsy )(2

no

21, yy

2=h

1+= hh

)(ˆ hyy =

GUM S1 Adaptive Monte Carlo scheme (7.9)

11

Assessment of the adaptive schemes

-1 0 1 2 3 4 5 6 70

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

Y

prob

abili

ty d

ensi

ty

22

21 XXY +=

121 == xx

1)()( 21 == xuxu

Model

Estimates

UncertaintiesGaussian distributions (uncorrelated)

y = 1.812 9

u(y) = 0.844 6

Rice distribution

12

Goal Determination of the expectation value of the Rice

distribution with an accuracy of δ = 0.005 (95 %)

Assessment Adaptive scheme is repeatedly executed 100 000 times

Result per run

� Estimate of the expectation value

� Number of required batches

Assessment of the adaptive schemes

13

Assessment of the GUM S1 adaptive scheme

Distribution of 10 5 estimates

],[ δδ +− yy

Only 80 % of the results found in the specified accuracy interval

95 % Interval

y

y

(Batch size 10 4, δδδδ=0.005)

14

Distribution of the number of batches

Large number of early

terminations at h = 2

batches (about 32 %)

Assessment of the GUM S1 adaptive scheme

(Batch size 10 4, δδδδ=0.005)

15

Reasons for the behavior of GUM S1 adaptive scheme

Sequential estimation scheme

� Random sample size (random number of batches)

� Multiple (dependent) testing for termination of sampling

Resulting confidence level does not necessarily mee t

the confidence level applied in individual test

� Confidence level of GUM S1 stopping rule adequate for fixed sample size

� Multiple testing and random sample size not taken into account

16

Alternative Adaptive Monte-Carlo scheme

Goal Carry out the MCM until a prescribed accuracy is achieved

at a specified confidence level

(with lowest possible numerical effort)

17

Given i.i.d. from unknown

Goal

so that is a confidence interval

for µ at confidence level 1-α

Stein‘s Two-stage scheme

∑=

=h

iiy

hhyh

1

1)( and

])(,)([ δδ +− hyhy

Step 1

Make random draws

�Variance

11 >h

∑=

−−

=1

1

21

11

2 ))((1

1)(

h

iiy hyy

hhs

1,,1 hyy K

K,, 21 yy 22 ,),,( σµσµN

18

Two-stage scheme

Step 2

Number of additionally required drawings

� make further random draws

� no additional random draws are made

( )

+−

⋅= −− 0,1

)(max 12

22/1,11

2

21 h

thsh hy

δα

02 >h 2h

02 =h

αδµδ −≥++≤≤−+ 1))()(Pr( 2121 hhyhhy

Proof C Stein, 1945, Ann. Math. Statist. 16 243-58

)( 21 hhy +

2h

19

Application of the two-stage scheme within GUM S1

Sequential batch-processing mode

Mean of the trials in batch i

Variance of the trials in batch i

sufficiently large batch size (CLT) ���� ,

)(2 yu i

unbiasediy

iy )(2 yu i

Two-stage scheme applicable for

� Estimate

� Squared uncertainty

y

)(2 yu

approximately Gaussian distributed

20

Assessment of the adaptive schemes

-1 0 1 2 3 4 5 6 70

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

Y

prob

abili

ty d

ensi

ty

22

21 XXY +=

121 == xx

1)()( 21 == xuxu

Model

Estimates

UncertaintiesGaussian distributions (uncorrelated)

y = 1.812 9

u(y) = 0.844 6

Rice distribution

21

Assessment of the two-stage scheme

y

(h1=10, Batch size 10 3, δδδδ=0.005, αααα=0.05)

y

],[ δδ +− yy

Interval covers95 % of the estimates �

Distribution of 10 5 estimates

22

Distribution of the number of batches

Assessment of the two-stage scheme

(h1=10, Batch size 10 3, δδδδ=0.005, αααα=0.05)

No earlyterminations

23

��

�

�� 157 271(196)0.969(0.001)10 000

146 383(218)0.951(0.001)1 000

146 423(218)0.952(0.001)100

146 176(218)0.951(0.001)10

145 802(211)0.952(0.001)1

ANT*Success rateBatch size

Determination of the estimate of the measurand: dependence on batch size

)05.0,005.0,10( 1 === αδh

*) ANT: Average Number of Trials

Result for a requested confidence level of 99.9 %

���� Success rate 0.999 08 (0.000 1)

ANT 654 760 (972)

1000) size Batch ,001.0( =α

�

Assessment of the two-stage scheme

24

Summary: Adaptive schemes for GUM S1

� GUM S1 adaptive scheme does not (intend to) meet a 95 % confidence level

� Alternative approach: Two-stage scheme

� Attains specified confidence level for a Gaussian d istribution (Proof by C. Stein, important for metrological appl ications)

� Applicable for the estimate and the squared uncerta inty

� Allows prediction of computation time (number of tr ials)

Wübbeler, Harris, Cox, Elster Metrologia 47 (2010) 317–324