probabilistic methods in open earth tools ferdinand diermanse kees den heijer bas hoonhout
TRANSCRIPT
Probabilistic methods in Open Earth Tools
Ferdinand Diermanse
Kees den Heijer
Bas Hoonhout
2
Open Earth Tools
• Deltares software• Open source• Sharing code for users of matlab, python, R, …• https://publicwiki.deltares.nl/display/OET/OpenEarth
3
Application: probabilities of unwanted events (failure)
Floods (too much)
Droughts (too little)
Contamination (too dirty)
4
Example application: flood risk analysis
Rainfall
Upstream river
Discharge
Sea water
level
Sobek
5
General problem definition
X1
System/model
X2
Xn
.
.
.
Z
“Boundary
conditions”
“system
variable”
6
Notation
X1
X2
Xn
.
.
.
Z
X = (X1, X2, …, Xn)
Z = Z(X)
System/model
7
General problem definition
X1
model
X2
Xn
.
.
.
Z
?
Statistical
analysis
Probabilistic
analysis
complex
Time consuming
8
failure domain: unwanted events
x1
x2
“failure”
Z(x)=0no “failure”
Z(x)>0
Z(x)<0
Wanted: probability of failure, i.e. probability that Z<0
9
Example Z-function
Failure: if water level (h) exceeds crest height (k): Z = k - h
10
Probability functions of x-variables
-4 -3 -2 -1 0 1 2 3 40
0.1
0.2
0.3
0.4
0.5
0.6
0.7
x2
f(x 2)
-4 -2 0 2 40
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
x1
f(x 1)
11
Correlations need to be included
x2
f(x)
x1
x1
x2
f(x)
Multivariate distribution function
12
Combination of f(x) and Z(x)
x2
x1
f(x)
Z(x)=C*
“failure”
no “failure”
13
Probability of failure
x2
x1
f(x)
fail
0
P
Z
f d
x
x x
Z(x)=0
14
Problem definition
Problem cannot be solved analytically
Probabilistic estimation techniques are required
Evaluation of Z(x) can be very time consuming
fail
0
P
Z
f d
x
x x
15
Probabilistic methods in Open Earth Tools
Crude Monte Carlo
Monte Carlo with importance sampling
First Order Reliability Method (FORM)
Directional sampling
1616
Crude Monte Carlo sampling
X1
X2
Z=0
failureno failure
1. Take N random samples of the x-variables 2. Count the number of samples (M) that lead to “failure” 3. Estimate Pf = M/N
-4 -2 0 2 40
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
x1
f(x 1)
-4 -3 -2 -1 0 1 2 3 40
0.1
0.2
0.3
0.4
0.5
0.6
0.7
x2
f(x 2)
17
Simple example Crude Monte Carlo: ¼ circle
Uniform 0,1f x
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
x1
x 2
4
14
18
Samples crude Monte Carlo
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
x1
x 2
no failure
failure
19
MC estimate
100
101
102
103
104
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
number of samples
MC
est
imat
e
1-/4
20
New example: smaller probability of failure
-5 -4 -3 -2 -1 0 1 2 3 4 5-5
-4
-3
-2
-1
0
1
2
3
4
5
z=0
u1
u 2
example limit state: Z = 5 - (u1+u
2)
failure probability: 0.00020
-3 -2 -1 0 1 2 30
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
U1;U2
1000 samples
-5 -4 -3 -2 -1 0 1 2 3 4 5-5
-4
-3
-2
-1
0
1
2
3
4
5
z=0
u1
u 2
1000 samples crude MC
21
How many samples required?
100
101
102
103
104
105
106
0
1
2
x 10-4
number of samples
estim
ated
fai
lure
pro
babi
lity
exact
crude MC
22
23
Crude Monte Carlo
• Can handle a large number of random variables
• Number of samples required for a sufficiently accurate estimate is inversely proportional to the probability of failure
• For small failure probabilities, crude MC is not a good choice, especially if each sample brings with it a time consuming computation/simulation
2424
“Smart” MC method 1: importance sampling
x
f(x)h(x)
f xh x
Manipulation of probability denstity function
Allowed with the use of a correction:
Potentially much faster than Crude Monte Carlo
25
-8 -6 -4 -2 0 2 4 6 80
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
u
prob
abili
ty d
ensi
tyexample strategy: increase standard deviation by a factor 2
f(u)
h(u)
Example strategy: increase variance
26
-5 -4 -3 -2 -1 0 1 2 3 4 5-5
-4
-3
-2
-1
0
1
2
3
4
5
u1
u 21000 samples MC-IS
Samples
27
Convergence of MC estimate
100
101
102
103
104
105
106
0
1
2
3
4
5
6x 10
-4
number of samples
estim
ated
fai
lure
pro
babi
lity
importance sampling; scaling factor 2
exact
crude MCimportance sampling
28
Example strategy 2
-8 -6 -4 -2 0 2 4 6 8 100
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
u
prob
abili
ty d
ensi
tyexample strategy: mu=2 sigma=2
f(u)
h(u)
29
Samples
-3 -2 -1 0 1 2 3 4 5 6 7-3
-2
-1
0
1
2
3
4
5
6
7
u1
u 2
1000 samples MC-IS
30
Convergence of MC estimate
100
101
102
103
104
105
106
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5x 10
-4
number of samples
estim
ated
fai
lure
pro
babi
lity
exact
crude MC
importance sampling
31
Monte Carlo with importance sampling
• Potentially much faster than Crude Monte Carlo
• Proper choice of h(x) is crucial
• Therefore: Proper system knowledge is crucial
32
FORM
Design point: most likely combination leading to failure
33
x
u
F(x)
real world variable X
transformed normallydistributed variable u
(u) = F(x)
f(x)
(u )
(u)
Method is executed with standard normally distributed variables
34
Probability density independent normal values
Probability density decreases
away from origin
35
example
0 0.5 1 1.5 2 2.5 3 3.5 40
0.5
1
1.5
2
2.5
3
3.5
4 -10-9-8
-7
-6
-6
-5
-5
-4
-4
-3
-3
-2
-2
-2
-1
-1
-1
0
0
0
1
1
1
1
2
2
2
2
3
3
3
3
4
4
4
44
55
5
5 5 5
u
v
0.8 1.25Z u v
u en v standard
normally distributed
36
Design point
0 0.5 1 1.5 2 2.5 3 3.5 40
0.5
1
1.5
2
2.5
3
3.5
4 -10-9-8
-7
-6
-6
-5
-5
-4
-4
-3
-3
-2
-2
-2
-1
-1
-1
0
0
0
1
1
1
1
2
2
2
2
3
3
3
3
4
4
4
44
55
5
5 5 5
u
v
Z=0 & shortest distance to origin
37
Start iterative procedure
0 0.5 1 1.5 2 2.5 3 3.5 40
0.5
1
1.5
2
2.5
3
3.5
4 -10-9
-8-7
-6-5
-5
-4
-4
-3
-3
-2
-2
-1
-1
-1
0
0
0
1
1
1
2
2
2
2
3
3
3
3
44
4
4
55
5 5 5
u1
u 2
38
Estimation of derivatives
0.4 0.6 0.8 1 1.2 1.4 1.60.4
0.6
0.8
1
1.2
1.4
1.6
u
v
3.5
4
4.5
u
v
39
Resulting tangent
0 0.5 1 1.5 2 2.5 3 3.5 40
0.5
1
1.5
2
2.5
3
3.5
4
55
5
5 5 5
u
v
40
Linearisation of Z-function based on tangent
0 0.5 1 1.5 2 2.5 3 3.5 40
0.5
1
1.5
2
2.5
3
3.5
4
-10
0
1
1
1
2
2
2
3
3
4
u1
u 2
41
First estimate of design point
0 0.5 1 1.5 2 2.5 3 3.5 40
0.5
1
1.5
2
2.5
3
3.5
4 -1
0
0
1
1
1
2
2
2
3
3
4
u1
u 2
42
3D view: Z-function
43
3D view: linearisation of Z-function
44
Smaller steps to prevent “accidents” (relaxation)
0 0.5 1 1.5 2 2.5 3 3.5 40
0.5
1
1.5
2
2.5
3
3.5
4
-1
0
0
1
1
1
2
2
2
3
3
4
u1
u 2
45
2nd iteration step
0 0.5 1 1.5 2 2.5 3 3.5 40
0.5
1
1.5
2
2.5
3
3.5
4
55
5
5 5 5
u
v
46
Linearisation in 2nd iteration step
0 0.5 1 1.5 2 2.5 3 3.5 40
0.5
1
1.5
2
2.5
3
3.5
4
u
v
47
3D view
48
All iteration steps
0 0.5 1 1.5 2 2.5 3 3.5 40
0.5
1
1.5
2
2.5
3
3.5
4 -10-9-8-7-6
-6
-5
-5
-4
-4
-3
-3
-2
-2
-2
-1
-1
-1
0
0
0
1
1
1
1
2
2
2
2
3
3
3
3
4
4
4
44
55
55 5 5
u
v
49
-value of design point in standard normal space
0 0.5 1 1.5 2 2.5 3 3.5 40
0.5
1
1.5
2
2.5
3
3.5
4 -10-9-8
-7
-6
-6
-5
-5
-4
-4
-3
-3
-2
-2
-2
-1
-1
-1
0
0
0
1
1
1
1
2
2
2
2
3
3
3
3
44
4
44
55
5
5 5 5
u
v
||||
Pfail
50
-values in design point
Limit state(Z = 0)
Design point
u1
u2 Z < 0
β
u2,d = -α2β
u1,d = -α1β
51
FORM
• Very fast method
• Risk: iteration method does not converge, or converges to the wrong design point
52
Directional sampling
Z=0Selected directions
Z-function evaluations
u1
u2
1
2
34
Z<0
Z>0
0
53
Search along 1 direction
z
0
1
2
4
3
54
Resume
Crude Monte Carlo (MC)
Monte Carlo with importance sampling (MC-IS)
First Order Reliability Method (FORM)
Directional Sampling (DS)
Towards the exercises
56
Generic problem statement
x2
x1
f(x)
fail
0
P
Z
f d
x
x x
Z(x)=0
57
Generic problem statement
fail
0
P
Z
f d
x
x x
1. Probability functions, f(x): P -> X
2. Z-function: X -> Z