reliable error estimation for monte carlo and quasi...

Simulation Problems Adaptive Monte Carlo Adaptive Quasi-Monte Carlo Discussion References

Reliable Error Estimation forMonte Carlo and Quasi-Monte Carlo SimulationWhen do you have enough samples to stop?

Fred J. Hickernell

Department of Applied Mathematics, Illinois Institute of [email protected] mypages.iit.edu/~hickernell

Joint work with Sou-Cheng Choi (NORC, U Chicago), Jiang Lan (IIT),Lluıs Antoni Jimenez Rugama (IIT), and Art Owen (Stanford)

Supported by NSF-DMS-1115392

Thank you for your kind invitation to visit!

October 7, 2014

[email protected] Reliable (Q)MC Simulation Northwestern U, 10/7/2014 1 / 18

mailto:[email protected]

[email protected]

http://mypages.iit.edu/~hickernell

mypages.iit.edu/~hickernell


Stopping the Simulation When the Error is Small Enoughoption price

probability of process failurepixel intensity

option payoff under random scenariorandom failure or not, i.e., 1 or 0intensity of random incident light ray

µ “ EpY q “ ?, where Y „ complicated

« µn :“1

n

nÿ

i“1

Yi, Y1, Y2, . . . IID

Want Pr|µ´ µn| ď εas ě 99% guaranteed for user-specified εa

What about Y “ fpXq, µ “

ż

r0,1qdfpxqdx “?

|µ´ µn| ď maxpεa, εr |µ|qY “ the limit of Y pdq as dÑ8, e.g.,

discretizing an SDE with d steps, orthe dth time step of a Markov Chain

p “ PpY ď µq, µ “? quantiles

µ “ ErY pxq |Y px1q, . . . , Y pxnqs “? kriging







« µn :“1

n

nÿ

i“1

Yi, Y1, Y2, . . . IID

Monte Carlo method “ statistical sampling with computersoriginated at LANL ca. 1947 (Eckhardt, 1987)



ż

r0,1qdfpxqdx “?











« µn :“1

n

nÿ

i“1

Yi, Y1, Y2, . . . IID



n “ ? based on Y1, Y2, . . .eyeball it? as much time as possible?Central Limit Theorem confidence interval?


ż

r0,1qdfpxqdx “?











« µn :“1

n

nÿ

i“1

Yi, Y1, Y2, . . . IID



n “ ? based on Y1, Y2, . . .under certain reasonable conditions on Y ?

(H. et al., 2014; Jiang and H., 2014)


ż

r0,1qdfpxqdx “?







Heuristic, Adaptive Simulation Based on the CLTCLT Adaptive Algorithm. Given an IID random generator for Y1, Y2, . . ., anabsolute error tolerance, εa, and parameters, nσ P N and C ą 1,

Step 1. Estimate the Variance. Compute the sample variance based on nσsamples of Y :

σ2 “1

nσ ´ 1

nσÿ

i“1

pYi ´ µσq2, µσ “

1

nσ

nσÿ

i“1

Yi,

and take C2σ2 as your estimate of varpY q.

Step 2. Estimate the Mean. Use the Central Limit Theorem to determine howmany samples are needed to estimate the mean, and estimate the meanindependently of the sample variance:

n “

S

ˆ

2.58Cσ

εa

˙2W

, µn “1

n

nσ`nÿ

i“nσ`1

Yi.

Hope that Pr|µ´ µn| ď εas ě 99%[email protected] Reliable (Q)MC Simulation Northwestern U, 10/7/2014 3 / 18


Heuristic, Adaptive Simulation Based on the CLTCLT Adaptive Algorithm. Given an IID random generator for Y1, Y2, . . ., anabsolute error tolerance, εa, and parameters, nσ P N and C ą 1,

Step 1. Estimate the Variance. Compute the sample variance based on nσsamples of Y :

σ2 “1

nσ ´ 1

nσÿ

i“1

pYi ´ µσq2, µσ “

1

nσ

nσÿ

i“1

Yi,

and take C2σ2 How good is this? as your estimate of varpY q.

Step 2. Estimate the Mean. Use the Central Limit Theorem (only good fornÑ8) to determine how many samples are needed to estimate the mean, andestimate the mean independently of the sample variance:

n “

S

ˆ

2.58Cσ

εa

˙2W

, µn “1

n

nσ`nÿ

i“nσ`1

Yi.

Hope that Pr|µ´ µn| ď εas ě 99%[email protected] Reliable (Q)MC Simulation Northwestern U, 10/7/2014 3 / 18


Guaranteed Adaptive Monte Carlo Simulation

Adaptive Algorithm meanMC g. Given an IID random generator for Y1, Y2, . . .,an absolute error tolerance, εa, and parameters, nσ P N and C ą 1,

Step 1. Bound the Variance. Compute the sample variance based on nσsamples of Y . We know that PrC2σ2 ě varpY qs ě 99.5% forkurtpY q :“ ErpY ´ µq4s{ varpY q2 ď κmaxpnσ,Cq by Cantelli’s inequality.

Step 2. Estimate the Mean. Use a Berry-Esseen Inequality to determine thesample size, n, needed for the sample mean by solving

Φ`

´?nεa{pCσq

˘

looooooooomooooooooon

CLT part

`∆np?nεa{pCσq, κmaxq

looooooooooooomooooooooooooon

Berry-Esseen extra part

ď 0.0025.

Then compute the sample mean, µn, of an independent sample of size n.

Theorem. (H. et al., 2014) If kurtpY q ď κmaxpnσ,Cq, then we must havePr|µ´ µn| ď εas ě 99%. The computational cost is nσ ` n “ OpvarpY q{ε2aq withhigh probability, even though varpY q is unknown a priori.



Numerical Examples for meanMC g

Asian Geometric Mean Call Optiond “ 1, 2, 4, . . . , 64 time steps

cd

ż

Rde´‖x‖2

cosp‖x‖qdx “ 1

d “ 1, 2, 3, 4 Keister (1996)

« 100% success « 100% success§ Sample size for bounding varpY q should be ě 103 to get a reasonable κmax

§ meanMC g is conservative, see Bayer et al. (2014) for a heuristic alternative§ Cannot yet handle relative error tolerances







« µn :“1

n

nÿ

i“1

Yi, Y1, Y2, . . . IID


n “ ? based on Y1, Y2, . . .under certain reasonable conditions on Y ?

(H. et al., 2014; Jiang and H., 2014)


ż

r0,1qdfpxqdx “?

using X1,X2, . . . more even than IID(H. and Jimenez Rugama, 2014)(Jimenez Rugama and H., 2014)







Quasi-Monte Carlo Cubature Using Digital Nets∣∣∣∣∣ż

r0,1qdfpxqdx

looooooomooooooon

µ“ErfpXqs

´1

2m

2m´1ÿ

i“0

fpziq

∣∣∣∣∣ ď εa

ď

8ÿ

λ“1

∣∣fλ2m∣∣ ď pωpmqqωp`qrSm´`,mpfq

1´ pωp`qqωp`qproof ď εa

digital net nodes

§ n “ 2m “? samples neededto satisfy error tolerance

§ Highly stratified sampling

§ Can be random (Owen, 2000)

§ Sobol’ sequences are apopular choice (Dick and

Pillichshammer, 2010)

0 0.25 0.5 0.75 10

0.25

0.5

0.75

1




r0,1qdfpxqdx

looooooomooooooon

µ“ErfpXqs

´1

2m

2m´1ÿ

i“0

fpziq

∣∣∣∣∣ ď D`

tziu2m´1i“0

˘

V pfqloooooooooomoooooooooon

(Niederreiter, 1992; H., 1998)

ď εa

ď

8ÿ

λ“1



digital net nodes

§ n “ 2m “? samples neededto satisfy error tolerance

§ D`

tziu2m´1i“0

˘

requires atleast Opn2q “ Op22mqoperations

§ V pfq is impractical tocalculate because it involvesLp-norms of mixed partialderivatives of f

0 0.25 0.5 0.75 10

0.25

0.5

0.75

1




r0,1qdfpxqdx

looooooomooooooon

µ“ErfpXqs

´1

2m

2m´1ÿ

i“0

fpziq

∣∣∣∣∣ ď 8ÿ

λ“1

∣∣fλ2m∣∣

ďpωpmqqωp`qrSm´`,mpfq


digital net nodes Walsh coefficients in dual net

Walsh functions & coefficients

fpxq “8ÿ

κ“0

p´1qxkpκq,xy fκ

fm,κ :“1

2m

2m´1ÿ

i“0

p´1qxkpκq,ziyfpziq

“

8ÿ

λ“0

fκ`λ2m aliasing




r0,1qdfpxqdx

looooooomooooooon

µ“ErfpXqs

´1

2m

2m´1ÿ

i“0

fpziq

∣∣∣∣∣ ď 8ÿ

λ“1



digital net nodes Walsh coefficients in dual net

pS`,mpfq :“2`´1ÿ

κ“t2`´1u

8ÿ

λ“1

∣∣fκ`λ2m∣∣,qSmpfq :“

8ÿ

κ“2m

∣∣fκ∣∣S`pfq :“

2`´1ÿ

κ“t2`´1u

∣∣fκ∣∣rS`,mpfq :“

2`´1ÿ

κ“t2`´1u

∣∣fm,κ∣∣Ð can get 100

101

102

103

104

10−15

10−10

10−5

100

κ

|fκ|

err ≤ S(0, 12)S(12)S(8)

conditions: pS`,mpfq ď pωpm´ `qqSmpfq, qSmpfq ď qωp`qSm´`pfq.



Guaranteed Adaptive Quasi-Monte Carlo Simulation

Have

∣∣∣∣∣ż

r0,1qdfpxqdx

looooooomooooooon

µ“ErfpXqs

´1

2m

2m´1ÿ

i“0

fpziq

looooooomooooooon

µm

∣∣∣∣∣ ď pωpmqqωp`qrSm´`,mpfq

1´ pωp`qqωp`qloooooooooooomoooooooooooon

errpmq

.

We want to find m and µm that guarantees |µ´ µm| ď εa.

Adaptive Algorithm cubSobol g. Given a black-box function evaluator, f , anda tolerance, εa, fix ` and initalize m ą `.

Step 1. Compute the data-based error bound, errpmq.Step 2. If errpmq is small enough such that errpmq ď εa, then return µm.Step 3. Otherwise, increase m by one, and return to Step 1.

Theorem. (H. and Jimenez Rugama, 2014) For integrands satifying the

conditions cubSobol g succeeds proof , and the computational cost isOprm` $pfqs2mq “ Oprlogpnq ` $pfqsnq, for some

m ď min

"

m1 :r1` pωp`qqωp`qspωpm1qqωp`qSm1´`pfq

1´ pωp`qqωp`qď εa

*

proof more proof



Numerical Examples for meanMC g and cubSobol g

Asian Geometric Mean Call Option, d “ 1, 2, 4, . . . , 64

meanMC g cubSobol g

« 100% success « 100% success




cd

ż

Rde´‖x‖2

cosp‖x‖qdx “ 1 Keister (1996)

d “ 1, . . . , 4 d “ 1, . . . , 4

meanMC g cubSobol g





cd

ż

Rde´‖x‖2

cosp‖x‖qdx “ 1 Keister (1996)

d “ 1, . . . , 4 d “ 1, . . . , 19

meanMC g cubSobol g




Observations on meanMC g and cubSobol g

§ These algorithms require Y or f to lie in a , which describes how

nasty they are allowed to be. Bigger ùñ more robust algorithms.

§ We focus on of random variables or functions (instead of other

shapes) because our problems are homogeneous, and our error bounds arepositively homogeneous.

§ Lyness (1983) warned against adaptive algorithms that use C |µn ´ µn1 | as anerror estimate (stop when the change is small). We avoid this type error ofestimate, but it is prevalent in popular adaptive algorithms.

§ There are rather general sufficient conditions under which adaption providesno advantage (Bahadur and Savage, 1956 ; Traub et al., 1988, Chapter 4,

Theorem 5.2.1 ; Novak, 1996). To violate those conditions we consider

nonconvex of random variables or integrands.§ cubSobol g accommodates hybrid error requirements,|µ´ µn| ď maxpεa, εr |µ|q, and meanMC g will soon.

§ These algorithms are featured in our Guaranteed Automatic IntegrationLibrary (GAIL) code.google.com/p/gail/ (Choi et al., 2013–2014).


http://code.google.com/p/gail/






« µn :“1

n

nÿ

i“1

Yi, Y1, Y2, . . . IID



ż

r0,1qdfpxqdx “?







References I

Bahadur, R. R. and L. J. Savage. 1956. The nonexistence of certain statistical procedures innonparametric problems, Ann. Math. Stat. 27, 1115–1122.

Bayer, C., H. Hoel, E. von Schwerin, and R. Tempone. 2014. On nonasymptotic optimalstopping criteria in monte carlo simulations on nonasymptotic optimal stopping criteria inMonte Carlo Simulations, SIAM J. Sci. Comput. 36, A869–A885.

Choi, S.-C. T., Y. Ding, F. J. H., L. Jiang, and Y. Zhang. 2013–2014. GAIL: GuaranteedAutomatic Integration Library (versions 1, 1.3).

Dick, J. and F. Pillichshammer. 2010. Digital nets and sequences: Discrepancy theory andquasi-Monte Carlo integration, Cambridge University Press, Cambridge.

Eckhardt, R. 1987. Stan Ulam, John von Neumann, and the Monte Carlo method, Los AlamosScience, 131–136.

H., F. J. 1998. A generalized discrepancy and quadrature error bound, Math. Comp. 67,299–322.

H., F. J., L. Jiang, Y. Liu, and A. B. Owen. 2014. Guaranteed conservative fixed widthconfidence intervals via Monte Carlo sampling, Monte Carlo and quasi-Monte Carlo methods2012, pp. 105–128.

H., F. J. and Ll. A. Jimenez Rugama. 2014. Reliable adaptive cubature using digital sequences.submitted for publication.



References II

Jiang, L. and F. J. H. 2014. Guaranteed conservative confidence intervals for means of Bernoullirandom variables. in preparation.

Jimenez Rugama, Ll. A. and F. J. H. 2014. Adaptive multidimensional integration based onrank-1 lattices. in preparation.

Keister, B. D. 1996. Multidimensional quadrature algorithms, Computers in Physics 10,119–122.

Lyness, J. N. 1983. When not to use an automatic quadrature routine, SIAM Rev. 25, 63–87.

Niederreiter, H. 1992. Random number generation and quasi-Monte Carlo methods,CBMS-NSF Regional Conference Series in Applied Mathematics, SIAM, Philadelphia.

Novak, E. 1996. On the power of adaption, J. Complexity 12, 199–237.

Owen, A. B. 2000. Monte Carlo, quasi-Monte Carlo, and randomized quasi-Monte Carlo, MonteCarlo, quasi-Monte Carlo, and randomized quasi-Monte Carlo, pp. 86–97.

Traub, J. F., G. W. Wasilkowski, and H. Wozniakowski. 1988. Information-based complexity,Academic Press, Boston.



Proof of Cubature Error bound

S`pfq “2`´1ÿ

κ“t2`´1u

∣∣fκ∣∣ “ 2`´1ÿ

κ“t2`´1u

∣∣∣∣∣fm,κ ´ 8ÿ

λ“1

fκ`λ2m

∣∣∣∣∣ď

2`´1ÿ

κ“t2`´1u

∣∣fm,κ∣∣looooooomooooooon

rS`,mpfq

`

2`´1ÿ

κ“t2`´1u

8ÿ

λ“1

∣∣fκ`λ2m ∣∣loooooooooooomoooooooooooon

pS`,mpfq

ď rS`,mpfq ` pωpm´ `qqωpm´ `qS`pfq

S`pfq ďrS`,mpfq

1´ pωpm´ `qqωpm´ `qprovided that pωpm´ `qqωpm´ `q ă 1

∣∣∣∣∣ż

r0,1qdfpxqdx´

1

2m

2m´1ÿ

i“0

fpziq

∣∣∣∣∣ ď 8ÿ

λ“1

fλ2m “ pS0,mpfq

ď pωpmqqωp`qS`pfq ďpωpmqqωp`qrS`,mpfq

1´ pωpm´ `qqωpm´ `qback



Bounding errpmq in Terms of S`pfq

rS`,mpfq “2`´1ÿ

κ“t2`´1u

∣∣fm,κ∣∣ “ 2`´1ÿ

κ“t2`´1u

∣∣∣∣∣fκ ` 8ÿ

λ“1

fκ`λ2m

∣∣∣∣∣ď

2`´1ÿ

κ“t2`´1u

∣∣fκ∣∣looooomooooon

S`pfq

`

2`´1ÿ

κ“t2`´1u

8ÿ

λ“1

∣∣fκ`λ2m ∣∣loooooooooooomoooooooooooon

pS`,mpfq

ď r1` pωpm´ `qqωpm´ `qsS`pfq

which implies that

errpmq “pωpmqqωp`qrSm´`,mpfq

1´ pωp`qqωp`qď

pωpmqqωp`qr1` pωp`qqωp`qsSm´`pfq

1´ pωp`qqωp`qalways.

Therefore, we know that errpmq ď ∆`,m must be satisfied when

pωpmqqωp`qr1` pωp`qqωp`qsSm´`pfq

1´ pωp`qqωp`qď

maxpεa, εr |µ|q1´ εr

. back


reliable error estimation for monte carlo and quasi...

Documents