monte carlo filtering and regional sensitivity analysis (rsa)
TRANSCRIPT
http://eemc.jrc.ec.europa.eu1
Monte Carlo filtering and
regional sensitivity analysis (RSA)
M. Ratto, IPSC-JRC, European Commission
DYNARE Workshop, CEF2011, San Francisco
http://eemc.jrc.ec.europa.eu2
Background
• environmental sciences, early 80’s;
• complex numerical and analytical models, based on first principles, conservation laws, … ;
• ill-defined parameters, competing model structures(different constitutive equations, different types of process considered, spatial/temporal resolution, … )
• need to establish magnitude and sources of prediction uncertainty;
• Monte Carlo simulation analyses.
http://eemc.jrc.ec.europa.eu3
MC filtering and RSA
One of the earliest landmark application of MC simulation: eutrophication in the Peel Inlet, SW Australia.
Regional Sensitivity Analysis developed and employed.
Hornberger G.M., Spear R.C., An approach to the preliminary analysis of environmental systems, J. Environm. Management, 12, 7-18, 1981.
Young, P.C., Parkinson, S. D. and Lees, M. Simplicity out of complexity: Occam’s razor revisited. Journal of Applied Statistics 23, 165-210, 1996.
Young, P.C., Data-based mechanistic modelling, generalised sensitivity and dominant mode analysis. Computer Physics Communication 117, 113-129, 1999.
http://eemc.jrc.ec.europa.eu4
MC filtering and RSA
Two tasks for RSA:
• qualitative definition of the system behaviour
[a set of constraints: thresholds, ceilings, time bounds based on available information on the system];
• binary classification of model outputs based on the specified behaviour definition.
[qualifies a simulation as behaviour (B) if the model output lies within constraints, non-behaviour ( ) otherwise]B
http://eemc.jrc.ec.europa.eu5
MC filtering and RSA
Define a range for k input factors xi [1 ≤ i ≤ k], reflecting uncertainty in parameters and model constituent hypotheses.
Each Monte Carlo simulation is associated to a vector of values of the input factors.
Classifying simulations as either B or , a set of binary elements is defined allowing to distinguish two sub-sets for each xi:
B
)|( Bxi
)|( Bxi
http://eemc.jrc.ec.europa.eu6
)( BXi
iX
Y
)( BXi
B
B
MC filtering and RSA
http://eemc.jrc.ec.europa.eu7
MC filtering and RSA
The Kolmogorov-Smirnov two-sample test (two-sided version) is performed for each factor independently
)|()|(:)|()|(:
1
0
BxfBxfH
BxfBxfH
inim
inim
≠=
Test statistic:
)|()|(sup)(,
BxFBxFxd inimyinm −=
whereF are marginal cumulative probability functions, f are probability density functions
http://eemc.jrc.ec.europa.eu8
MC filtering and RSA
At what significance level α does the computed value of dm,n determine the rejection of Ho?
α is the probability to reject Ho when it is true (i.e. to recognise a factor as important when it is not).
The importance of each parameter is inversely related to this significance level.
http://eemc.jrc.ec.europa.eu9
MC filtering and RSA
Derive the critical level Dα at which the computed value of dm,n determines the rejection of Ho (the smaller α, the higher Dα).
If dm,n > Dα, then Ho is rejected at significance level α.
http://eemc.jrc.ec.europa.eu10
MC filtering and RSA
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1
Xi
F(X
i)
PriorF
m(X
i|B)
Fn(X
i|B)
dm,n
http://eemc.jrc.ec.europa.eu11
RSA: comment
RSA has many ‘global’ properties:
The whole range of value of input factors is considered, all factors are varied at the same time.
CAVEAT:
RSA classification analyses univariate marginal distributions.
Lacks for interaction structure.
http://eemc.jrc.ec.europa.eu12
Mapping stability: example (1)
Phillips curve:
GDP y, inflation i, output gap c
titytttt
tcttt
tptt
ttt
actiEii
acAcAc
app
cpy
,11
,22
1
,1
)1(
)/2cos(2
++−+⋅=+−=
+=+=
+−
−−
−
ωωτπ
http://eemc.jrc.ec.europa.eu13
Mapping stability: example (1)
Stability is assured if
5.0>ω
http://eemc.jrc.ec.europa.eu14
Mapping stability: example (1)
0 0.5 10
5
10
15
20A
0 0.5 10
5
10
15
20ω
0 50 100 1500
5
10
15
20τ
0 0.5 10
0.2
0.4
0.6
0.8
1
x
F(x
)
A. K-S prob 1
0 0.5 10
0.2
0.4
0.6
0.8
1
x
F(x
)
ω. K-S prob 4.5031e-221
0 50 100 1500
0.2
0.4
0.6
0.8
1
x
F(x
)
τ. K-S prob 1
http://eemc.jrc.ec.europa.eu15
Mapping stability: example (2)
Phillips curve, extended version:
GDP y, inflation i, output gap c
tityttftbt
tcttt
tptt
ttt
actiEii
acAcAc
app
cpy
,11
,22
1
,1
)/2cos(2
+++⋅=+−=
+=+=
+−
−−
−
ωωτπ
http://eemc.jrc.ec.europa.eu16
Mapping stability: example (2)
Stability is assured if
1<+ fb ωω
http://eemc.jrc.ec.europa.eu17
Mapping stability: example (2)
0 0.5 10
5
10
15
20A
0 0.5 10
10
20
30
40
ωb
0 0.5 10
10
20
30
40
ωf
0 50 100 1500
5
10
15
20τ
0 0.5 10
0.2
0.4
0.6
0.8
1
x
F(x
)
A. K-S prob 1
0 0.5 10
0.2
0.4
0.6
0.8
1
x
F(x
)
ωb. K-S prob 1.0423e-054
0 0.5 10
0.2
0.4
0.6
0.8
1
x
F(x
)
ωf. K-S prob 2.2889e-057
0 50 100 1500
0.2
0.4
0.6
0.8
1
x
F(x
)
τ. K-S prob 0.99541
http://eemc.jrc.ec.europa.eu18
Mapping stability: example (2)
0 0.5 10
0.5
1
ωb
ω fcc = -0.50539
Bivariate analysis
http://eemc.jrc.ec.europa.eu19
Mapping stability: example (3)
• As the complexity of the model structure and its parameterization increases, it is not trivial to know a priory the set of model coefficients assuring stability;
http://eemc.jrc.ec.europa.eu20
Mapping stability : example (3)
• The Lubik Schorfheide (2005) model
• 12 parameters;
http://eemc.jrc.ec.europa.eu21
Lubik Schorfheide model (2005)
http://eemc.jrc.ec.europa.eu22
Lubik Schorfheide model (2005)
http://eemc.jrc.ec.europa.eu23
Lubik Schorfheide model (2005)
http://eemc.jrc.ec.europa.eu24
Lubik Schorfheide model (2005)
http://eemc.jrc.ec.europa.eu25
Toolbox documentation
http://eemc.jrc.ec.europa.eu26
Toolbox documentation
http://eemc.jrc.ec.europa.eu27
Mapping the fit
How do parameters adjust to fit the multivariate set of observed covariates?
Are there trade-off’s?
http://eemc.jrc.ec.europa.eu28
Mapping the fit
- Sample structural coefficients prior distributions (or ranges);
OR
- Use Metropolis posterior sample;
Compute RMSE’s of the 1-step ahead model predictions for each of the N observed series.
http://eemc.jrc.ec.europa.eu29
Mapping the fit
N filtering rules: defining as B the
samples corresponding to the best 10% RMSE’s, -B otherwise.
For each parameter we get N ‘behavioural’ sets:
fj(Xi|B); j = 1,…, N
http://eemc.jrc.ec.europa.eu30
Mapping the fit
Trade off occurs when:
1) one structural parameter drives significantly the fit of more than one observed series;
AND
2) fj(Xi|B)≠ fk(Xi|B)
http://eemc.jrc.ec.europa.eu31
Mapping the fit: prior
0 100 200 300 4000
0.2
0.4
0.6
0.8
1e_R
0 200 400 600 8000
0.2
0.4
0.6
0.8
1e_q
0 100 200 300 4000
0.2
0.4
0.6
0.8
1e_A
0 100 200 300 4000
0.2
0.4
0.6
0.8
1e_ys
0 200 400 6000
0.2
0.4
0.6
0.8
1e_pies
0 2 4 6 80
0.2
0.4
0.6
0.8
1psi1
basey
obs
Robs
pieobs
dqde
http://eemc.jrc.ec.europa.eu32
Mapping the fit: prior
0 0.5 1 1.5 20
0.2
0.4
0.6
0.8
1psi2
0 0.5 1 1.5 20
0.2
0.4
0.6
0.8
1psi3
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1rho_R
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1alpha
0 5 10 150
0.2
0.4
0.6
0.8
1rr
0 1 2 3 40
0.2
0.4
0.6
0.8
1k
basey
obs
Robs
pieobs
dqde
http://eemc.jrc.ec.europa.eu33
Mapping the fit: prior
0 1 2 30
0.2
0.4
0.6
0.8
1tau
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1rho_q
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1rho_A
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1rho_ys
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1rho_pies
basey
obs
Robs
pieobs
dqde
http://eemc.jrc.ec.europa.eu34
Mapping the fit: posterior
0.2 0.3 0.4 0.5 0.6 0.70
0.2
0.4
0.6
0.8
1
σR
1 1.2 1.4 1.6 1.8 20
0.2
0.4
0.6
0.8
1
σq
0.4 0.6 0.8 1 1.2 1.40
0.2
0.4
0.6
0.8
1
σz
0 1 2 3 40
0.2
0.4
0.6
0.8
1
σy*
1.4 1.6 1.8 2 2.20
0.2
0.4
0.6
0.8
1
σπ*
1 2 3 40
0.2
0.4
0.6
0.8
1
ψ1
basey
R
π∆q
∆e
basey
R
π∆q
∆e
basey
R
π∆q
∆e
base
y
R
π∆q
∆e
base
y
R
π∆q
∆e
base
y
R
π∆q
∆e
http://eemc.jrc.ec.europa.eu35
0 0.2 0.4 0.6 0.80
0.2
0.4
0.6
0.8
1
ψ2
0 0.2 0.4 0.6 0.80
0.2
0.4
0.6
0.8
1
ψ3
0.5 0.6 0.7 0.8 0.90
0.2
0.4
0.6
0.8
1
ρR
0 0.1 0.2 0.3 0.40
0.2
0.4
0.6
0.8
1α
0 2 4 6 80
0.2
0.4
0.6
0.8
1r
0 0.5 1 1.50
0.2
0.4
0.6
0.8
1κ
basey
R
π∆q
∆e
basey
R
π∆q
∆e
basey
R
π∆q
∆e
base
y
R
π∆q
∆e
base
y
R
π∆q
∆e
base
y
R
π∆q
∆e
Mapping the fit: posterior
http://eemc.jrc.ec.europa.eu36
0 0.2 0.4 0.6 0.80
0.2
0.4
0.6
0.8
1τ
0 0.2 0.4 0.6 0.80
0.2
0.4
0.6
0.8
1
ρq
0.4 0.5 0.6 0.70
0.2
0.4
0.6
0.8
1
ρz
0.8 0.85 0.9 0.95 10
0.2
0.4
0.6
0.8
1
ρy*
0 0.2 0.4 0.6 0.80
0.2
0.4
0.6
0.8
1
ρπ*
basey
R
π∆q
∆e
basey
R
π∆q
∆e
base
y
R
π∆q
∆e
base
y
R
π∆q
∆e
base
y
R
π∆q
∆e
Mapping the fit: posterior
http://eemc.jrc.ec.europa.eu37
Mapping the fit: posterior
psi1-psi3 conflict: an interpretation?
http://eemc.jrc.ec.europa.eu38
Toolbox documentation
http://eemc.jrc.ec.europa.eu39
RSA: problems
Correlation and interaction structures of the B subset [⇒ also Beck’s review, 1987]:
• Smirnov test is a sufficient test only if Hois rejected
(i.e. the factor is IMPORTANT);
• any covariance structure induced by the classification is not detected by the univariate dm,n statistic.
e.g. factors combined as products or quotients may compensate
• bivariate correlation analysis is not revealing, either.
http://eemc.jrc.ec.europa.eu40
RSA problems: example (1)
Example 1. Y = X1*X2, X1,X2 ∼U[-0.5,0.5]
Behavioural runs: Y>0.
-0.5 0 0.5-0.5
0
0.5
X1
X2 Smirnov test fails
Correlation analysis would help (e.g. PCA)
Scatter plot of the B subset in the X1,X2 plane
http://eemc.jrc.ec.europa.eu41
RSA problems: example (1)
Example 1. Cumulative distributions of X1
(Behavioural runs: Y>0.)
-0.5 0 0.50
0.2
0.4
0.6
0.8
1
X1
F(X
1)
originalB sampleB- sample
http://eemc.jrc.ec.europa.eu42
RSA problems: example (1)
In this case, a correlation analysis would be helpful.
the correlation coefficient of (X1, X2) for the B set is ρ=0.75.
This suggests performing a Principal Component Analysis on the B set, obtaining the two components (the eigenvectors of the correlation matrix):
http://eemc.jrc.ec.europa.eu43
RSA problems: example (1)
PC1 = (0.7079, 0.7063); accounting for the 87.5% of the variation of the B set.
PC2 = (-0.7063, 0.7079); accounting for the 12.5% of the variation of the B set.
The direction of the principal component PC1 (associated with the highest eigenvalue) indicates the privileged orientation for acceptable runs.
http://eemc.jrc.ec.europa.eu44
RSA problems: example (1)
-0.5 0 0.5-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
PC1
PC
2
B subsetB subset
http://eemc.jrc.ec.europa.eu45
RSA problems: example (1)
-0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.80
0.5
1
PC1
cdf(P
C1)
-0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.80
0.5
1
PC2
cdf(P
C2)PriorB subsetB subset
PriorB subsetB subset
http://eemc.jrc.ec.europa.eu46
RSA problems: example (1)
Now the level of significance for rejecting the
null hypothesis when it is true is very small
(α<0.1%), implying a very strong relevance of the linear combinations of the two input factors,
defined by the principal component analysis
http://eemc.jrc.ec.europa.eu47
RSA problems: example (2)
Example 2. Y = X12 + X2
2, X1,X2 ∼U[-0.5,0.5]
Behavioural runs: [0.2 < Y < 0.25]
-0.5 0 0.5-0.5
0
0.5
X1
X2
Smirnov test fails
Correlation fails as well
(sample ρ≈-0.04)
Scatter plot of the B subset in the X1,X2 plane
http://eemc.jrc.ec.europa.eu48
RSA problems (ctd.)
To address the RSA limitations and to better understand the impact of uncertainty and interaction in the high-dimensional parameter spaces of models, Spear et al. (1994) developed the computer intensive tree-structured density estimation technique (TSDE),
Interesting applications of TSDE in environmental sciences can be found in Spear (1997), Grieb at al. (1999) and Osidele and Beck (2001).