algorithms for uncertainty quantification...repetition from previous lecture sampling methods!a...
TRANSCRIPT
Algorithms for Uncertainty QuantificationTobias Neckel, Ionut,-Gabriel Farcas,
Lehrstuhl Informatik V
Summer Semester 2017
Lecture 4: More advanced samplingtechniques
Repetition from previous lecture
Repetition from previous lecture• Sampling methods→ a popular technique for uncertainty propagation
• Most widely used sampling approach→ Monte Carlo sampling
• Monte Carlo sampling→ simple, robust, independent of probability distribution, number of randomparameters, ...
• ... but slow convergence rate
• Model problem→ damped linear oscillator
• Uncertainty in some input parameters→ higher impact in the output uncertainty
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 4
Monte Carlo sampling error analysis
Short error analysis for standard Monte CarloRemember
• for N samples, the MCS error is O( 1√N)
How did we get that?
• Monte Carlo sampling↔ averaging
• let f : [0,1]→ R and I :=∫ 1
0 f (x)dx
• I ≈ If = 1N ∑
Ni=1 f (Ui), Ui ∼U (0,1)
• if σ2f = Var[f (x)], E[(I− If )2] = σ2
f /N
• σ2f ≈ σ2
f = 1N−1 ∑
Ni=1(f (Ui)− If )2
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 6
Short error analysis for standard Monte CarloRemember
• for N samples, the MCS error is O( 1√N)
How did we get that?
• Monte Carlo sampling↔ averaging
• let f : [0,1]→ R and I :=∫ 1
0 f (x)dx
• I ≈ If = 1N ∑
Ni=1 f (Ui), Ui ∼U (0,1)
• if σ2f = Var[f (x)], E[(I− If )2] = σ2
f /N
• σ2f ≈ σ2
f = 1N−1 ∑
Ni=1(f (Ui)− If )2
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 6
Short error analysis for standard Monte CarloRemember
• for N samples, the MCS error is O( 1√N)
How did we get that?
• Monte Carlo sampling↔ averaging
• let f : [0,1]→ R and I :=∫ 1
0 f (x)dx
• I ≈ If = 1N ∑
Ni=1 f (Ui), Ui ∼U (0,1)
• if σ2f = Var[f (x)], E[(I− If )2] = σ2
f /N
• σ2f ≈ σ2
f = 1N−1 ∑
Ni=1(f (Ui)− If )2
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 6
Short error analysis for standard Monte CarloRemember
• for N samples, the MCS error is O( 1√N)
How did we get that?
• Monte Carlo sampling↔ averaging
• let f : [0,1]→ R and I :=∫ 1
0 f (x)dx
• I ≈ If = 1N ∑
Ni=1 f (Ui), Ui ∼U (0,1)
• if σ2f = Var[f (x)], E[(I− If )2] = σ2
f /N
• σ2f ≈ σ2
f = 1N−1 ∑
Ni=1(f (Ui)− If )2
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 6
Short error analysis for standard Monte CarloRemember
• for N samples, the MCS error is O( 1√N)
How did we get that?
• Monte Carlo sampling↔ averaging
• let f : [0,1]→ R and I :=∫ 1
0 f (x)dx
• I ≈ If = 1N ∑
Ni=1 f (Ui), Ui ∼U (0,1)
• if σ2f = Var[f (x)], E[(I− If )2] = σ2
f /N
• σ2f ≈ σ2
f = 1N−1 ∑
Ni=1(f (Ui)− If )2
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 6
Short error analysis for standard Monte CarloRemember
• for N samples, the MCS error is O( 1√N)
How did we get that?
• Monte Carlo sampling↔ averaging
• let f : [0,1]→ R and I :=∫ 1
0 f (x)dx
• I ≈ If = 1N ∑
Ni=1 f (Ui), Ui ∼U (0,1)
• if σ2f = Var[f (x)], E[(I− If )2] = σ2
f /N
• σ2f ≈ σ2
f = 1N−1 ∑
Ni=1(f (Ui)− If )2
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 6
Improving standard Monte Carlosampling
Towards more advanced sampling techniques• How to increase the accuracy of standard Monte Carlo? (E[(I− If )2] = σ2
f /N)
− improve your code• parallelize• vectorize• remove if statements• use memory efficiently• etc.
− increase N• not desirable
− decrease σ2f
• desirable
− improve the random number generation
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 8
Towards more advanced sampling techniques• How to increase the accuracy of standard Monte Carlo? (E[(I− If )2] = σ2
f /N)− improve your code• parallelize• vectorize• remove if statements• use memory efficiently• etc.
− increase N• not desirable
− decrease σ2f
• desirable
− improve the random number generation
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 8
Towards more advanced sampling techniques• How to increase the accuracy of standard Monte Carlo? (E[(I− If )2] = σ2
f /N)− improve your code• parallelize• vectorize• remove if statements• use memory efficiently• etc.
− increase N• not desirable
− decrease σ2f
• desirable
− improve the random number generation
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 8
Towards more advanced sampling techniques• How to increase the accuracy of standard Monte Carlo? (E[(I− If )2] = σ2
f /N)− improve your code• parallelize• vectorize• remove if statements• use memory efficiently• etc.
− increase N• not desirable
− decrease σ2f
• desirable
− improve the random number generation
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 8
Towards more advanced sampling techniques• How to increase the accuracy of standard Monte Carlo? (E[(I− If )2] = σ2
f /N)− improve your code• parallelize• vectorize• remove if statements• use memory efficiently• etc.
− increase N• not desirable
− decrease σ2f
• desirable
− improve the random number generation
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 8
Improving standard Monte Carlo sampling
Variance-minimization techniques
• antithetic sampling
• importance sampling
• stratified sampling
• control variates
Alternative random number generation techniques
• Fibonacci generators
• latin hypercube sampling
• Sobol’ sequences
• Halton sequences
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 9
Improving standard Monte Carlo samplingVariance-minimization techniques
• antithetic sampling
• importance sampling
• stratified sampling
• control variates
Alternative random number generation techniques
• Fibonacci generators
• latin hypercube sampling
• Sobol’ sequences
• Halton sequences
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 9
Improving standard Monte Carlo samplingVariance-minimization techniques
• antithetic sampling
• importance sampling
• stratified sampling
• control variates
Alternative random number generation techniques
• Fibonacci generators
• latin hypercube sampling
• Sobol’ sequences
• Halton sequences
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 9
Improving standard Monte Carlo samplingVariance-minimization techniques
• antithetic sampling
• importance sampling
• stratified sampling
• control variates
Alternative random number generation techniques
• Fibonacci generators
• latin hypercube sampling
• Sobol’ sequences
• Halton sequences
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 9
Improving standard Monte Carlo samplingVariance-minimization techniques
• antithetic sampling
• importance sampling
• stratified sampling
• control variates
Alternative random number generation techniques
• Fibonacci generators
• latin hypercube sampling
• Sobol’ sequences
• Halton sequences
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 9
Improving standard Monte Carlo samplingVariance-minimization techniques
• antithetic sampling
• importance sampling
• stratified sampling
• control variates
Alternative random number generation techniques
• Fibonacci generators
• latin hypercube sampling
• Sobol’ sequences
• Halton sequences
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 9
Improving standard Monte Carlo samplingVariance-minimization techniques
• antithetic sampling
• importance sampling
• stratified sampling
• control variates
Alternative random number generation techniques
• Fibonacci generators
• latin hypercube sampling
• Sobol’ sequences
• Halton sequences
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 9
Improving standard Monte Carlo samplingVariance-minimization techniques
• antithetic sampling
• importance sampling
• stratified sampling
• control variates
Alternative random number generation techniques
• Fibonacci generators
• latin hypercube sampling
• Sobol’ sequences
• Halton sequences
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 9
Improving standard Monte Carlo samplingVariance-minimization techniques
• antithetic sampling
• importance sampling
• stratified sampling
• control variates
Alternative random number generation techniques
• Fibonacci generators
• latin hypercube sampling
• Sobol’ sequences
• Halton sequences
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 9
Improving standard Monte Carlo samplingVariance-minimization techniques
• antithetic sampling
• importance sampling
• stratified sampling
• control variates
Alternative random number generation techniques
• Fibonacci generators
• latin hypercube sampling
• Sobol’ sequences
• Halton sequences
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 9
Variance reduction techniques
Antithetic sampling
• let X denote a (continuous) random variable with pdf ∼ fX (x), supp(X )⊂ R• assume fX (x) is symmetric, c being the center of symmetry
• let t(X1, . . . ,Xn) denote an estimator
• let x ∈ supp(X ); the symmetric of x w.r.t. c is x = 2c−x
• symmetry implies fX (x) = fX (x)
• sample n/2 samples X1, . . . ,Xn/2 from ∼ fX (x)
• the remaining n/2 samples X1, . . . , Xn/2 are obtained via reflection
• then t(X1, . . . ,Xn) = t(X1, . . . ,Xn/2)+ t(X1, . . . , Xn/2)
• example: if U ∼U (0,1), take U ∼U (0,1); the antithetic samples are, U = 1−U
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 11
Antithetic sampling• let X denote a (continuous) random variable with pdf ∼ fX (x), supp(X )⊂ R
• assume fX (x) is symmetric, c being the center of symmetry
• let t(X1, . . . ,Xn) denote an estimator
• let x ∈ supp(X ); the symmetric of x w.r.t. c is x = 2c−x
• symmetry implies fX (x) = fX (x)
• sample n/2 samples X1, . . . ,Xn/2 from ∼ fX (x)
• the remaining n/2 samples X1, . . . , Xn/2 are obtained via reflection
• then t(X1, . . . ,Xn) = t(X1, . . . ,Xn/2)+ t(X1, . . . , Xn/2)
• example: if U ∼U (0,1), take U ∼U (0,1); the antithetic samples are, U = 1−U
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 11
Antithetic sampling• let X denote a (continuous) random variable with pdf ∼ fX (x), supp(X )⊂ R• assume fX (x) is symmetric, c being the center of symmetry
• let t(X1, . . . ,Xn) denote an estimator
• let x ∈ supp(X ); the symmetric of x w.r.t. c is x = 2c−x
• symmetry implies fX (x) = fX (x)
• sample n/2 samples X1, . . . ,Xn/2 from ∼ fX (x)
• the remaining n/2 samples X1, . . . , Xn/2 are obtained via reflection
• then t(X1, . . . ,Xn) = t(X1, . . . ,Xn/2)+ t(X1, . . . , Xn/2)
• example: if U ∼U (0,1), take U ∼U (0,1); the antithetic samples are, U = 1−U
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 11
Antithetic sampling• let X denote a (continuous) random variable with pdf ∼ fX (x), supp(X )⊂ R• assume fX (x) is symmetric, c being the center of symmetry
• let t(X1, . . . ,Xn) denote an estimator
• let x ∈ supp(X ); the symmetric of x w.r.t. c is x = 2c−x
• symmetry implies fX (x) = fX (x)
• sample n/2 samples X1, . . . ,Xn/2 from ∼ fX (x)
• the remaining n/2 samples X1, . . . , Xn/2 are obtained via reflection
• then t(X1, . . . ,Xn) = t(X1, . . . ,Xn/2)+ t(X1, . . . , Xn/2)
• example: if U ∼U (0,1), take U ∼U (0,1); the antithetic samples are, U = 1−U
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 11
Antithetic sampling• let X denote a (continuous) random variable with pdf ∼ fX (x), supp(X )⊂ R• assume fX (x) is symmetric, c being the center of symmetry
• let t(X1, . . . ,Xn) denote an estimator
• let x ∈ supp(X ); the symmetric of x w.r.t. c is x = 2c−x
• symmetry implies fX (x) = fX (x)
• sample n/2 samples X1, . . . ,Xn/2 from ∼ fX (x)
• the remaining n/2 samples X1, . . . , Xn/2 are obtained via reflection
• then t(X1, . . . ,Xn) = t(X1, . . . ,Xn/2)+ t(X1, . . . , Xn/2)
• example: if U ∼U (0,1), take U ∼U (0,1); the antithetic samples are, U = 1−U
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 11
Antithetic sampling• let X denote a (continuous) random variable with pdf ∼ fX (x), supp(X )⊂ R• assume fX (x) is symmetric, c being the center of symmetry
• let t(X1, . . . ,Xn) denote an estimator
• let x ∈ supp(X ); the symmetric of x w.r.t. c is x = 2c−x
• symmetry implies fX (x) = fX (x)
• sample n/2 samples X1, . . . ,Xn/2 from ∼ fX (x)
• the remaining n/2 samples X1, . . . , Xn/2 are obtained via reflection
• then t(X1, . . . ,Xn) = t(X1, . . . ,Xn/2)+ t(X1, . . . , Xn/2)
• example: if U ∼U (0,1), take U ∼U (0,1); the antithetic samples are, U = 1−U
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 11
Antithetic sampling• let X denote a (continuous) random variable with pdf ∼ fX (x), supp(X )⊂ R• assume fX (x) is symmetric, c being the center of symmetry
• let t(X1, . . . ,Xn) denote an estimator
• let x ∈ supp(X ); the symmetric of x w.r.t. c is x = 2c−x
• symmetry implies fX (x) = fX (x)
• sample n/2 samples X1, . . . ,Xn/2 from ∼ fX (x)
• the remaining n/2 samples X1, . . . , Xn/2 are obtained via reflection
• then t(X1, . . . ,Xn) = t(X1, . . . ,Xn/2)+ t(X1, . . . , Xn/2)
• example: if U ∼U (0,1), take U ∼U (0,1); the antithetic samples are, U = 1−U
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 11
Antithetic sampling• let X denote a (continuous) random variable with pdf ∼ fX (x), supp(X )⊂ R• assume fX (x) is symmetric, c being the center of symmetry
• let t(X1, . . . ,Xn) denote an estimator
• let x ∈ supp(X ); the symmetric of x w.r.t. c is x = 2c−x
• symmetry implies fX (x) = fX (x)
• sample n/2 samples X1, . . . ,Xn/2 from ∼ fX (x)
• the remaining n/2 samples X1, . . . , Xn/2 are obtained via reflection
• then t(X1, . . . ,Xn) = t(X1, . . . ,Xn/2)+ t(X1, . . . , Xn/2)
• example: if U ∼U (0,1), take U ∼U (0,1); the antithetic samples are, U = 1−U
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 11
Antithetic sampling• let X denote a (continuous) random variable with pdf ∼ fX (x), supp(X )⊂ R• assume fX (x) is symmetric, c being the center of symmetry
• let t(X1, . . . ,Xn) denote an estimator
• let x ∈ supp(X ); the symmetric of x w.r.t. c is x = 2c−x
• symmetry implies fX (x) = fX (x)
• sample n/2 samples X1, . . . ,Xn/2 from ∼ fX (x)
• the remaining n/2 samples X1, . . . , Xn/2 are obtained via reflection
• then t(X1, . . . ,Xn) = t(X1, . . . ,Xn/2)+ t(X1, . . . , Xn/2)
• example: if U ∼U (0,1), take U ∼U (0,1); the antithetic samples are, U = 1−U
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 11
Antithetic sampling• let X denote a (continuous) random variable with pdf ∼ fX (x), supp(X )⊂ R• assume fX (x) is symmetric, c being the center of symmetry
• let t(X1, . . . ,Xn) denote an estimator
• let x ∈ supp(X ); the symmetric of x w.r.t. c is x = 2c−x
• symmetry implies fX (x) = fX (x)
• sample n/2 samples X1, . . . ,Xn/2 from ∼ fX (x)
• the remaining n/2 samples X1, . . . , Xn/2 are obtained via reflection
• then t(X1, . . . ,Xn) = t(X1, . . . ,Xn/2)+ t(X1, . . . , Xn/2)
• example: if U ∼U (0,1), take U ∼U (0,1); the antithetic samples are, U = 1−U
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 11
Stratified sampling
• let X denote a (continuous) random variable with pdf ∼ fX (x), supp(X )⊂ R• assume, without loss of generality, that supp(X ) = [0,1]
• let t(X1, . . . ,Xn) denote an estimator
• idea: prevent samples from clustering in a particular region of the interval
• select λ ∈ (0,1)
• then draw n1 = λn samples in [0,λ ] and n2 = n−n1 = (1−λ )n samples in [λ ,1]
• t(X1, . . . ,Xn) = t(X1, . . . ,Xn1) + t(X1, . . . ,Xn2)
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 12
Stratified sampling• let X denote a (continuous) random variable with pdf ∼ fX (x), supp(X )⊂ R• assume, without loss of generality, that supp(X ) = [0,1]
• let t(X1, . . . ,Xn) denote an estimator
• idea: prevent samples from clustering in a particular region of the interval
• select λ ∈ (0,1)
• then draw n1 = λn samples in [0,λ ] and n2 = n−n1 = (1−λ )n samples in [λ ,1]
• t(X1, . . . ,Xn) = t(X1, . . . ,Xn1) + t(X1, . . . ,Xn2)
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 12
Stratified sampling• let X denote a (continuous) random variable with pdf ∼ fX (x), supp(X )⊂ R• assume, without loss of generality, that supp(X ) = [0,1]
• let t(X1, . . . ,Xn) denote an estimator
• idea: prevent samples from clustering in a particular region of the interval
• select λ ∈ (0,1)
• then draw n1 = λn samples in [0,λ ] and n2 = n−n1 = (1−λ )n samples in [λ ,1]
• t(X1, . . . ,Xn) = t(X1, . . . ,Xn1) + t(X1, . . . ,Xn2)
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 12
Stratified sampling• let X denote a (continuous) random variable with pdf ∼ fX (x), supp(X )⊂ R• assume, without loss of generality, that supp(X ) = [0,1]
• let t(X1, . . . ,Xn) denote an estimator
• idea: prevent samples from clustering in a particular region of the interval
• select λ ∈ (0,1)
• then draw n1 = λn samples in [0,λ ] and n2 = n−n1 = (1−λ )n samples in [λ ,1]
• t(X1, . . . ,Xn) = t(X1, . . . ,Xn1) + t(X1, . . . ,Xn2)
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 12
Stratified sampling• let X denote a (continuous) random variable with pdf ∼ fX (x), supp(X )⊂ R• assume, without loss of generality, that supp(X ) = [0,1]
• let t(X1, . . . ,Xn) denote an estimator
• idea: prevent samples from clustering in a particular region of the interval
• select λ ∈ (0,1)
• then draw n1 = λn samples in [0,λ ] and n2 = n−n1 = (1−λ )n samples in [λ ,1]
• t(X1, . . . ,Xn) = t(X1, . . . ,Xn1) + t(X1, . . . ,Xn2)
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 12
Stratified sampling• let X denote a (continuous) random variable with pdf ∼ fX (x), supp(X )⊂ R• assume, without loss of generality, that supp(X ) = [0,1]
• let t(X1, . . . ,Xn) denote an estimator
• idea: prevent samples from clustering in a particular region of the interval
• select λ ∈ (0,1)
• then draw n1 = λn samples in [0,λ ] and n2 = n−n1 = (1−λ )n samples in [λ ,1]
• t(X1, . . . ,Xn) = t(X1, . . . ,Xn1) + t(X1, . . . ,Xn2)
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 12
Stratified sampling• let X denote a (continuous) random variable with pdf ∼ fX (x), supp(X )⊂ R• assume, without loss of generality, that supp(X ) = [0,1]
• let t(X1, . . . ,Xn) denote an estimator
• idea: prevent samples from clustering in a particular region of the interval
• select λ ∈ (0,1)
• then draw n1 = λn samples in [0,λ ] and n2 = n−n1 = (1−λ )n samples in [λ ,1]
• t(X1, . . . ,Xn) = t(X1, . . . ,Xn1) + t(X1, . . . ,Xn2)
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 12
Control variates
• remember Monte Carlo integration: estimate∫ 1
0 f (x)dx via sampling
• assume there exists φ : [0,1]→ R that can be easily integrated
• therefore∫ 1
0 f (x)dx =∫ 1
0 (f (x)+φ(x)−φ(x))dx =∫ 1
0 φ(x)dx +∫ 1
0 (f (x)−φ(x))dx
• Var(f - φ ) = Var(f) + Var(φ ) - 2cov(f, φ )
• if cov(f, φ ) is high, i.e. f and φ are “similar”, Var(f - φ ) < Var(f)
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 13
Control variates• remember Monte Carlo integration: estimate
∫ 10 f (x)dx via sampling
• assume there exists φ : [0,1]→ R that can be easily integrated
• therefore∫ 1
0 f (x)dx =∫ 1
0 (f (x)+φ(x)−φ(x))dx =∫ 1
0 φ(x)dx +∫ 1
0 (f (x)−φ(x))dx
• Var(f - φ ) = Var(f) + Var(φ ) - 2cov(f, φ )
• if cov(f, φ ) is high, i.e. f and φ are “similar”, Var(f - φ ) < Var(f)
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 13
Control variates• remember Monte Carlo integration: estimate
∫ 10 f (x)dx via sampling
• assume there exists φ : [0,1]→ R that can be easily integrated
• therefore∫ 1
0 f (x)dx =∫ 1
0 (f (x)+φ(x)−φ(x))dx =∫ 1
0 φ(x)dx +∫ 1
0 (f (x)−φ(x))dx
• Var(f - φ ) = Var(f) + Var(φ ) - 2cov(f, φ )
• if cov(f, φ ) is high, i.e. f and φ are “similar”, Var(f - φ ) < Var(f)
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 13
Control variates• remember Monte Carlo integration: estimate
∫ 10 f (x)dx via sampling
• assume there exists φ : [0,1]→ R that can be easily integrated
• therefore∫ 1
0 f (x)dx =∫ 1
0 (f (x)+φ(x)−φ(x))dx =∫ 1
0 φ(x)dx +∫ 1
0 (f (x)−φ(x))dx
• Var(f - φ ) = Var(f) + Var(φ ) - 2cov(f, φ )
• if cov(f, φ ) is high, i.e. f and φ are “similar”, Var(f - φ ) < Var(f)
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 13
Control variates• remember Monte Carlo integration: estimate
∫ 10 f (x)dx via sampling
• assume there exists φ : [0,1]→ R that can be easily integrated
• therefore∫ 1
0 f (x)dx =∫ 1
0 (f (x)+φ(x)−φ(x))dx =∫ 1
0 φ(x)dx +∫ 1
0 (f (x)−φ(x))dx
• Var(f - φ ) = Var(f) + Var(φ ) - 2cov(f, φ )
• if cov(f, φ ) is high, i.e. f and φ are “similar”, Var(f - φ ) < Var(f)
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 13
Control variates• remember Monte Carlo integration: estimate
∫ 10 f (x)dx via sampling
• assume there exists φ : [0,1]→ R that can be easily integrated
• therefore∫ 1
0 f (x)dx =∫ 1
0 (f (x)+φ(x)−φ(x))dx =∫ 1
0 φ(x)dx +∫ 1
0 (f (x)−φ(x))dx
• Var(f - φ ) = Var(f) + Var(φ ) - 2cov(f, φ )
• if cov(f, φ ) is high, i.e. f and φ are “similar”, Var(f - φ ) < Var(f)
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 13
Importance sampling
• remember Monte Carlo integration: estimate∫ 1
0 f (x)dx via sampling
• standard Monte Carlo solution∫[0,1] f (x)dx ≈ I = 1
N ∑Ni=1 f (Ui), Ui ∼U (0,1)
• however, Ui are spread all over the domain
• idea: sample from another distribution gX that better captures the structure of f
•∫ 1
0 f (x)dx =∫ 1
0f (x)
gX (x)gX (x)dx =
∫ 10 h(x)gX (x)dx
• therefore, instead of sampling from the uniform distribution, sample according to gX
• variance reduced if f and gX have similar shapes
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 14
Importance sampling• remember Monte Carlo integration: estimate
∫ 10 f (x)dx via sampling
• standard Monte Carlo solution∫[0,1] f (x)dx ≈ I = 1
N ∑Ni=1 f (Ui), Ui ∼U (0,1)
• however, Ui are spread all over the domain
• idea: sample from another distribution gX that better captures the structure of f
•∫ 1
0 f (x)dx =∫ 1
0f (x)
gX (x)gX (x)dx =
∫ 10 h(x)gX (x)dx
• therefore, instead of sampling from the uniform distribution, sample according to gX
• variance reduced if f and gX have similar shapes
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 14
Importance sampling• remember Monte Carlo integration: estimate
∫ 10 f (x)dx via sampling
• standard Monte Carlo solution∫[0,1] f (x)dx ≈ I = 1
N ∑Ni=1 f (Ui), Ui ∼U (0,1)
• however, Ui are spread all over the domain
• idea: sample from another distribution gX that better captures the structure of f
•∫ 1
0 f (x)dx =∫ 1
0f (x)
gX (x)gX (x)dx =
∫ 10 h(x)gX (x)dx
• therefore, instead of sampling from the uniform distribution, sample according to gX
• variance reduced if f and gX have similar shapes
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 14
Importance sampling• remember Monte Carlo integration: estimate
∫ 10 f (x)dx via sampling
• standard Monte Carlo solution∫[0,1] f (x)dx ≈ I = 1
N ∑Ni=1 f (Ui), Ui ∼U (0,1)
• however, Ui are spread all over the domain
• idea: sample from another distribution gX that better captures the structure of f
•∫ 1
0 f (x)dx =∫ 1
0f (x)
gX (x)gX (x)dx =
∫ 10 h(x)gX (x)dx
• therefore, instead of sampling from the uniform distribution, sample according to gX
• variance reduced if f and gX have similar shapes
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 14
Importance sampling• remember Monte Carlo integration: estimate
∫ 10 f (x)dx via sampling
• standard Monte Carlo solution∫[0,1] f (x)dx ≈ I = 1
N ∑Ni=1 f (Ui), Ui ∼U (0,1)
• however, Ui are spread all over the domain
• idea: sample from another distribution gX that better captures the structure of f
•∫ 1
0 f (x)dx =∫ 1
0f (x)
gX (x)gX (x)dx =
∫ 10 h(x)gX (x)dx
• therefore, instead of sampling from the uniform distribution, sample according to gX
• variance reduced if f and gX have similar shapes
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 14
Importance sampling• remember Monte Carlo integration: estimate
∫ 10 f (x)dx via sampling
• standard Monte Carlo solution∫[0,1] f (x)dx ≈ I = 1
N ∑Ni=1 f (Ui), Ui ∼U (0,1)
• however, Ui are spread all over the domain
• idea: sample from another distribution gX that better captures the structure of f
•∫ 1
0 f (x)dx =∫ 1
0f (x)
gX (x)gX (x)dx =
∫ 10 h(x)gX (x)dx
• therefore, instead of sampling from the uniform distribution, sample according to gX
• variance reduced if f and gX have similar shapes
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 14
Importance sampling• remember Monte Carlo integration: estimate
∫ 10 f (x)dx via sampling
• standard Monte Carlo solution∫[0,1] f (x)dx ≈ I = 1
N ∑Ni=1 f (Ui), Ui ∼U (0,1)
• however, Ui are spread all over the domain
• idea: sample from another distribution gX that better captures the structure of f
•∫ 1
0 f (x)dx =∫ 1
0f (x)
gX (x)gX (x)dx =
∫ 10 h(x)gX (x)dx
• therefore, instead of sampling from the uniform distribution, sample according to gX
• variance reduced if f and gX have similar shapes
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 14
Importance sampling• remember Monte Carlo integration: estimate
∫ 10 f (x)dx via sampling
• standard Monte Carlo solution∫[0,1] f (x)dx ≈ I = 1
N ∑Ni=1 f (Ui), Ui ∼U (0,1)
• however, Ui are spread all over the domain
• idea: sample from another distribution gX that better captures the structure of f
•∫ 1
0 f (x)dx =∫ 1
0f (x)
gX (x)gX (x)dx =
∫ 10 h(x)gX (x)dx
• therefore, instead of sampling from the uniform distribution, sample according to gX
• variance reduced if f and gX have similar shapes
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 14
Quasi-Monte Carlo: alternative samplingtechniques
Quasi-Monte Carlo sampling• standard Monte Carlo: pseudo-random
samples
⇒
• quasi-Monte Carlo (QMC):deterministic samples
• in this lecture: QMC based on low discrepancy sequences• note: QMC methods defined for [0,1]d ; for any other domain, we need transformations
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 16
Quasi-Monte Carlo sampling• standard Monte Carlo: pseudo-random
samples
⇒
• quasi-Monte Carlo (QMC):deterministic samples
• in this lecture: QMC based on low discrepancy sequences• note: QMC methods defined for [0,1]d ; for any other domain, we need transformations
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 16
Quasi-Monte Carlo sampling• standard Monte Carlo: pseudo-random
samples
⇒
• quasi-Monte Carlo (QMC):deterministic samples
• in this lecture: QMC based on low discrepancy sequences
• note: QMC methods defined for [0,1]d ; for any other domain, we need transformations
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 16
Quasi-Monte Carlo sampling• standard Monte Carlo: pseudo-random
samples
⇒
• quasi-Monte Carlo (QMC):deterministic samples
• in this lecture: QMC based on low discrepancy sequences• note: QMC methods defined for [0,1]d ; for any other domain, we need transformations
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 16
Upper bound of integration error
Koksma-Hlawa inequalityRemember
• I :=∫ 1
0 f (x)dx
• If = 1N ∑
Ni=1 f (xi)
Theorem
Koksma-Hlawka inequality: |I− If | ≤ V (f )DN
• V (f )→ variation of f
• DN = supA⊂[0,1]
∣∣∣card(A)N −vol(A)
∣∣∣→ discrepancy of {xi}Ni=1
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 18
Koksma-Hlawa inequalityRemember
• I :=∫ 1
0 f (x)dx
• If = 1N ∑
Ni=1 f (xi)
Theorem
Koksma-Hlawka inequality: |I− If | ≤ V (f )DN
• V (f )→ variation of f
• DN = supA⊂[0,1]
∣∣∣card(A)N −vol(A)
∣∣∣→ discrepancy of {xi}Ni=1
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 18
Koksma-Hlawa inequalityRemember
• I :=∫ 1
0 f (x)dx
• If = 1N ∑
Ni=1 f (xi)
Theorem
Koksma-Hlawka inequality: |I− If | ≤ V (f )DN
• V (f )→ variation of f
• DN = supA⊂[0,1]
∣∣∣card(A)N −vol(A)
∣∣∣→ discrepancy of {xi}Ni=1
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 18
Low discrepancy sequences
Low discrepancy sequencesBasic idea
• in |I− If | ≤ V (f )DN , assume that V (f ) = constant
• idea: minimize error by reducing DN , i.e.
• produce {xi}Ni=1 that are “well” spaced
• in this way: O( 1√N)→ O( log(N)d
N ), where d is the dimension
⇐ ⇒
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 20
Low discrepancy sequencesBasic idea
• in |I− If | ≤ V (f )DN , assume that V (f ) = constant
• idea: minimize error by reducing DN , i.e.
• produce {xi}Ni=1 that are “well” spaced
• in this way: O( 1√N)→ O( log(N)d
N ), where d is the dimension
⇐ ⇒
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 20
Low discrepancy sequencesBasic idea
• in |I− If | ≤ V (f )DN , assume that V (f ) = constant
• idea: minimize error by reducing DN , i.e.
• produce {xi}Ni=1 that are “well” spaced
• in this way: O( 1√N)→ O( log(N)d
N ), where d is the dimension
⇐ ⇒
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 20
Low discrepancy sequences example:Halton sequences
Halton sequences• start with a prime number p
• construct a sequence based on finer and finer p-based divisions of sub-intervals of [0,1]• e.g. let p = 3− break [0,1] into 3 equal subintervals
− break each sub-interval into 3 equal subintervals
− now, the sequence is 1/3, 2/3, 1/9, 4/9, 7/9, 2/9, 5/9, 8/9− repeat until desired length
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 22
Halton sequences• start with a prime number p
• construct a sequence based on finer and finer p-based divisions of sub-intervals of [0,1]
• e.g. let p = 3− break [0,1] into 3 equal subintervals
− break each sub-interval into 3 equal subintervals
− now, the sequence is 1/3, 2/3, 1/9, 4/9, 7/9, 2/9, 5/9, 8/9− repeat until desired length
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 22
Halton sequences• start with a prime number p
• construct a sequence based on finer and finer p-based divisions of sub-intervals of [0,1]• e.g. let p = 3
− break [0,1] into 3 equal subintervals
− break each sub-interval into 3 equal subintervals
− now, the sequence is 1/3, 2/3, 1/9, 4/9, 7/9, 2/9, 5/9, 8/9− repeat until desired length
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 22
Halton sequences• start with a prime number p
• construct a sequence based on finer and finer p-based divisions of sub-intervals of [0,1]• e.g. let p = 3− break [0,1] into 3 equal subintervals
− break each sub-interval into 3 equal subintervals
− now, the sequence is 1/3, 2/3, 1/9, 4/9, 7/9, 2/9, 5/9, 8/9− repeat until desired length
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 22
Halton sequences• start with a prime number p
• construct a sequence based on finer and finer p-based divisions of sub-intervals of [0,1]• e.g. let p = 3− break [0,1] into 3 equal subintervals
− break each sub-interval into 3 equal subintervals
− now, the sequence is 1/3, 2/3, 1/9, 4/9, 7/9, 2/9, 5/9, 8/9− repeat until desired length
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 22
Halton sequences• start with a prime number p
• construct a sequence based on finer and finer p-based divisions of sub-intervals of [0,1]• e.g. let p = 3− break [0,1] into 3 equal subintervals
− break each sub-interval into 3 equal subintervals
− now, the sequence is 1/3, 2/3, 1/9, 4/9, 7/9, 2/9, 5/9, 8/9
− repeat until desired length
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 22
Halton sequences• start with a prime number p
• construct a sequence based on finer and finer p-based divisions of sub-intervals of [0,1]• e.g. let p = 3− break [0,1] into 3 equal subintervals
− break each sub-interval into 3 equal subintervals
− now, the sequence is 1/3, 2/3, 1/9, 4/9, 7/9, 2/9, 5/9, 8/9− repeat until desired length
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 22
Halton sequences example• 2D Halton grid with 100 elements
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 23
Quasi-Monte Carlo sampling: example
Model problem – damped linear oscillator
d2ydt2 (t)+c dy
dt (t)+ky(t) = f cos(ω t)y(0) = y0dydt (0) = y1
• t ∈ [0,30]
• k = 0.035
• f = 0.100
• ω = 1.000
• y0 = 0.500
• y1 = 0.000
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 25
Model problem – damped linear oscillator
d2ydt2 (t)+c dy
dt (t)+ky(t) = f cos(ω t)y(0) = y0dydt (0) = y1
• t ∈ [0,30]
• k = 0.035
• f = 0.100
• ω = 1.000
• y0 = 0.500
• y1 = 0.000
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 25
Quasi-Monte Carlo – example• t0 = 15
Deterministic result
• y(t0) =−1.51e−01
Stochastic results
• assume c ∼U (0.08,0.12)
• 100 samples, standard Monte Carlo→ E [y(t0)] =−1.61e−01, Var[y(t0)] = 6.51e−04
• 100 samples, QMC, Halton sequences→ E [y(t0)] =−1.53e−01, Var[y(t0)] = 7.78e−04
• 1000 samples, standard Monte Carlo→ E [y(t0)] =−1.52e−01, Var[y(t0)] = 7.30e−04
• 1000 samples, QMC, Halton sequences→ E [y(t0)] =−1.52e−01, Var[y(t0)] = 7.81e−04
• 10000 samples, standard Monte Carlo→ E [y(t0)] =−1.52e−01, Var[y(t0)] = 7.84e−04
• 10000 samples, QMC, Halton sequences→ E [y(t0)] =−1.52e−01, Var[y(t0)] = 7.80e−04
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 26
Quasi-Monte Carlo – example• t0 = 15
Deterministic result
• y(t0) =−1.51e−01
Stochastic results
• assume c ∼U (0.08,0.12)
• 100 samples, standard Monte Carlo→ E [y(t0)] =−1.61e−01, Var[y(t0)] = 6.51e−04
• 100 samples, QMC, Halton sequences→ E [y(t0)] =−1.53e−01, Var[y(t0)] = 7.78e−04
• 1000 samples, standard Monte Carlo→ E [y(t0)] =−1.52e−01, Var[y(t0)] = 7.30e−04
• 1000 samples, QMC, Halton sequences→ E [y(t0)] =−1.52e−01, Var[y(t0)] = 7.81e−04
• 10000 samples, standard Monte Carlo→ E [y(t0)] =−1.52e−01, Var[y(t0)] = 7.84e−04
• 10000 samples, QMC, Halton sequences→ E [y(t0)] =−1.52e−01, Var[y(t0)] = 7.80e−04
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 26
Quasi-Monte Carlo – example• t0 = 15
Deterministic result
• y(t0) =−1.51e−01
Stochastic results
• assume c ∼U (0.08,0.12)
• 100 samples, standard Monte Carlo→ E [y(t0)] =−1.61e−01, Var[y(t0)] = 6.51e−04
• 100 samples, QMC, Halton sequences→ E [y(t0)] =−1.53e−01, Var[y(t0)] = 7.78e−04
• 1000 samples, standard Monte Carlo→ E [y(t0)] =−1.52e−01, Var[y(t0)] = 7.30e−04
• 1000 samples, QMC, Halton sequences→ E [y(t0)] =−1.52e−01, Var[y(t0)] = 7.81e−04
• 10000 samples, standard Monte Carlo→ E [y(t0)] =−1.52e−01, Var[y(t0)] = 7.84e−04
• 10000 samples, QMC, Halton sequences→ E [y(t0)] =−1.52e−01, Var[y(t0)] = 7.80e−04
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 26
Quasi-Monte Carlo – example• t0 = 15
Deterministic result
• y(t0) =−1.51e−01
Stochastic results
• assume c ∼U (0.08,0.12)
• 100 samples, standard Monte Carlo→ E [y(t0)] =−1.61e−01, Var[y(t0)] = 6.51e−04
• 100 samples, QMC, Halton sequences→ E [y(t0)] =−1.53e−01, Var[y(t0)] = 7.78e−04
• 1000 samples, standard Monte Carlo→ E [y(t0)] =−1.52e−01, Var[y(t0)] = 7.30e−04
• 1000 samples, QMC, Halton sequences→ E [y(t0)] =−1.52e−01, Var[y(t0)] = 7.81e−04
• 10000 samples, standard Monte Carlo→ E [y(t0)] =−1.52e−01, Var[y(t0)] = 7.84e−04
• 10000 samples, QMC, Halton sequences→ E [y(t0)] =−1.52e−01, Var[y(t0)] = 7.80e−04
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 26
Quasi-Monte Carlo – example• t0 = 15
Deterministic result
• y(t0) =−1.51e−01
Stochastic results
• assume c ∼U (0.08,0.12)
• 100 samples, standard Monte Carlo→ E [y(t0)] =−1.61e−01, Var[y(t0)] = 6.51e−04
• 100 samples, QMC, Halton sequences→ E [y(t0)] =−1.53e−01, Var[y(t0)] = 7.78e−04
• 1000 samples, standard Monte Carlo→ E [y(t0)] =−1.52e−01, Var[y(t0)] = 7.30e−04
• 1000 samples, QMC, Halton sequences→ E [y(t0)] =−1.52e−01, Var[y(t0)] = 7.81e−04
• 10000 samples, standard Monte Carlo→ E [y(t0)] =−1.52e−01, Var[y(t0)] = 7.84e−04
• 10000 samples, QMC, Halton sequences→ E [y(t0)] =−1.52e−01, Var[y(t0)] = 7.80e−04
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 26
Summary
Summary• the accuracy of standard Monte Carlo can be improved via− optimizing your code− increasing the number of samples− decreasing the variance of the estimators− changing the sampling technique
• variance reduction techniques− antithetic sampling− importance sampling− stratified sampling− control variates• alternative random number generation techniques− Fibonacci generators− latin hypercube sampling− Sobol’ sequences− Halton sequences
• example low-discrepancy sequences: Halton sequences
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 28
Summary• the accuracy of standard Monte Carlo can be improved via− optimizing your code− increasing the number of samples− decreasing the variance of the estimators− changing the sampling technique• variance reduction techniques− antithetic sampling− importance sampling− stratified sampling− control variates
• alternative random number generation techniques− Fibonacci generators− latin hypercube sampling− Sobol’ sequences− Halton sequences
• example low-discrepancy sequences: Halton sequences
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 28
Summary• the accuracy of standard Monte Carlo can be improved via− optimizing your code− increasing the number of samples− decreasing the variance of the estimators− changing the sampling technique• variance reduction techniques− antithetic sampling− importance sampling− stratified sampling− control variates• alternative random number generation techniques− Fibonacci generators− latin hypercube sampling− Sobol’ sequences− Halton sequences
• example low-discrepancy sequences: Halton sequences
Dr. rer. nat. Tobias Neckel | Algorithms for Uncertainty Quantification | Summer Semester 2017 28