methods for: modeling uncertainty / model uncertainty · quantification of modeling uncertainty of...
TRANSCRIPT
Methods for:
Modeling uncertainty /
Model uncertainty Peter Hessling,
Peter Hedberg
SP Mätteknik,
SP Sveriges Tekniska Forskningsinstitut, Borås
Contact info
E-mail: [email protected]
Tel. +46 10 516 54 79, +46 702 92 54 79,
Adress: SP Measurement Technology,
Brinellgatan 4, Box 857, SE-501 15 Borås, Sweden 1
Peter Hessling, tekn dr
Peter Hedberg, tekn dr
• SP offers quality-checked traceable calibration services
for measurements against national references, for a number of units of the SI-system.
• SP services also include consulting (”science partner”) and are available to the industry.
• The research at the department of measurement science (MT~100 pers.) includes but is not limited to fundamental metrology.
• Our group (2 pers) at MT develops novel methods for evaluating and propagating the uncertainty of complex calculation models.
• Fundamental to metrology (analysis of measurements) is the standardized concept measurement uncertainty.
4
Measurement
uncertainty
Modeling uncertainty
Object Measurements Models
Standardization Yes No (complete concept missing)
Model Simple/rudimentary (often) Complex! FEM, CFD, Dyn, …
Uncertainty contributions
Systematic errors A few.
Unsatisfactory treatment.
Main problem: mesh-size, validity step
length/discretization, extrapolation
Stocastic errors
Scatter Lack of repeatability None! (Fixed model)
Lack of knowledge
discrete/cont. par.
Standard contribution Omitted? Competing methods
Methods
Linearisation According to standard ’GUM’ Standard / sensitivity analysis
Sampling / Ensemble Large acceptable [Mkt] Small absolute requirement!
Slump sampling
(Monte Carlo)
(Stratified, LHS)
Almost always possible
Notorious overmodeling
Often unacceptable calculation time
Overmodeling,
not robust (when few samples)
Deterministic
sampling
(Unknown method)
Simple, robust Well adapted
Statistical analysis separated from
model evaluation (’non-invasive’)
EXAMPLES OF UNCERTAINTIES
• Model Constants in turbulence models, determined from experiments
• Boundary conditions • Material properties • Geometry • Mesh size
Standard Deviation of Cp
Standard Deviation of Cp
Model(-ing) uncertainty
– The field of Uncertainty Quantification (UQ)
• Dedicated research groups at:
– Res. Lab. Sandia, Los Alamos, NASA
– US univ. high rank: Stanford, MIT, Cornell,…
• Two new journals
– International Journal for Uncertainty quantification (IJ4UQ) (Begell house, 2011)
– the Journal on Uncertainty Quantification (JUQ) [8] SIAM (Society for Industrial and Applied Mathematics) and ASA (American
Statistical Association) (2013)
• SIAM Conference on UQ (2-5 April, 2012)
• Target applications:
– Computational Fluid Dynamics…
– Advanced signal processing (included by myself, not formal UQ)
– Meteorology (SMHI ensemble prognoses),…
– Nuclear physics???
UQ
• Two directions of uncertainty propagation
– (Direct) Uncertainty Quantification (of result of modeling)
– Inverse Uncertainty Quantification (calibration, (system) identification)
• Common denominators:
– Heavy numerical models, large-scale simulations
– Complex publications
– Multidisciplinary – many fields of math involved!
– Non-invasive methods dominates
– Random sampling (RS)
• Painfully/unacceptable ineffective – Conventional ways to speed up:
– Improve sample distribution (more uniform) to reduce sampling
• Stratified Sampling, Latin Hypercube Sampling
– Coarse (’surrogate’) model approximation for full sampling
Response Surface Methodology
Deterministic sampling of model parameters
Our proposal: Sample with a rule, not random generator!
• Origin: Signal Processing, Kalman filter,
Simon Julier, Jeffrey Uhlmann Unscented propagation of covariance (~1994)
• Compromise statistical fidelity but not model evaluation accuracy
– Alt. 1: Sampling on confidence boundaries – Hessling JP., Svensson T., Propagation of uncertainty by sampling on confidence
boundaries, Int’l Journal of Uncertainty Quantification, 3 (5): 421-444 (2013))
– Alt. 2: Propagation of covariance
• Quantification of modeling uncertainty (result of models) – Hessling J P, in Digital Filters, chapter “Integration of digital filters and measurements”
(www.intechopen.com) ISBN 978-953-307-190-9 (INTECH, 2011)
– Hessling J P, in Digital Filters and Signal Processing, chapter “Deterministic sampling for
Quantification of Modeling Uncertainty of Signals” (www.intechopen.com) ISBN 978-953-
51-0871-9 (INTECH, 2013)
– Hessling JP, Deterministic sampling for propagating model covariance, Journal of
Uncertainty Quantification, minor rev.
• Inverse quantification of model uncertainty (calibration) – Hessling JP, Identification of complex models, in preparation.
Conventional statistical modeling
• De facto standard of modeling: Probability density function
– Random generators
– Not observable
– Mathematical construct (probability density)
– Never known (non-trivial common cases)
– One possible representation of statistical information
• Preference in mathematical statistics: Probability distribution function
– F(x)=P(y<=x), cumulative probability of not exceeding a value (x)
– Observable
– ’Physical’ meaning (probability)
– Never known (non-trivial common cases)
– Awkward representation…
– Terrible habit of using percentiles (deficiency of Wilk’s rule?)
Novel(?) statistical modeling
• Practice: Expectation values
– Mean, covariance, skewness, kurtosis, higher order
dependencies…
– Best(?) representation of stat. info in statistical moments
• Hierarchy, contained in Taylor series
• Complete representation
• Can be estimated
• Variable estimation accuracy (bias, variance of estimators)
• Robust
– Can be evaluated for every set of samples,
discrete or continuous
– Statistics of finite (RS,DS) ensembles can be compared to
continuous probability distributions (pdf)
• Generalized concept of sampling of pdf, loss of information
comparable to sampling of signals
(Lack of) Consistency of statistical modeling
• Fidelity of modeling, expressed in statistical moments
– Location: Mean
– Variation: Variance
– Dependency: Covariance
– Asymmetry: Skewness
– Shape: Kurtosis
• Consistency of distinguishing different marginal distributions (shape)
requires representation of dependencies beyond covariance!
• Models are deviced to match reality (calibration data/benchmarks)
=>Strong dependencies (also beyond covariance)!
• Higher order dependencies can not be modeled in MC (not normal)
• Hence, virtually every MC model simulation is inconsistent!
• Any mixed moment can be controlled and consistently represented
with a finite calculated set of samples
Multi- / univariate analyses fundamentally different
• The majority of us thinks/understands ’univariate’
• ’The pdfs’ of MC are nothing but marginal pdfs
• How control marginal pdfs in presence of covariance (non-normal
pdf)
• Marginal information is a minute fraction of all information
Ex. 20 parameters, marginal 4th moments < 0.3% of all 4th moments
• Why represent a negligible fraction of all information much more
accurately than the rest?
(by distinguishing different marginal pdfs)
• Full multivariate pdfs are required to account for their ’shape’
– Exceptionally difficult to determine!
Illustration of deterministic sampling…
• Model
• Random sampling (Monte Carlo)
• Deterministic sampling (one alt.)
16
4.2,6.1
,4,3,2,,~
4.0,2,
2,1
)(
4
k
MC
nkNq
qqh
100
101
102
103
0
0.5
1
1.5
2
2.5
Antal sampel
Parameter
DS
DS
Mean MC
Std MC
Mean
Std
100
101
102
103
6
8
10
12
14
16
18
20
22
24
26
Antal sampel
Model
DS
DS
Mean MC
Std MC
Mean LIN
Std LIN
Alternative sampling…
Sampling with brute force MC (*), stratified (solid), fixed grid (dashed),
q~N(0,1)
M(2): M(4):
Deterministic sampling
Deterministic sampling, general
• Deterministic ensemble
– Scaling
– Correlation
– Excitation matrix of canonical ensembles
• Generic (normalized, de-correlated), i.e. re-used indefinitely
• Discrete equivalent to continuous probability distributions
• ’Tiny’ deterministic variant of large Monte Carlo ensemble
• Types so far:
– Standard ensemble (STD)
– Simplex ensemble (SPX)
– Binary ensemble (BIN)
– Combined (CMB)
USU
VIVVVmVVSUU
T
m
T
mn
T
mmn
2
11
cov
01,,ˆ,ˆ1
V
U
S
ˆ
Key features of excitation matrix
– For constructing novel ensembles
• Orthogonality
• Lack of constraints
=>Transformation,
with change of ens size
• Combination of ensembles
• Correlated samples
(reduce ens size)
• Normalization, scaling and correlation of parameters eliminated
• Ensemble size aspect of deterministic sampling rule
4/5ceil2:BIN
2:STD
1:SPX
nm
nm
nm
IVV
VVV
nmIWWVVWV
VV
IVV
T
nxn
T
nxmnxm
T
T
)(Blockdiag
,,~
21
Several parameters…
• The standard (STD) ensemble (largest maximum excitation)
• The simplex (SPX) ensemble (minimum ensemble)
Several parameters…
• The binary (BIN) ensemble (minimum maximal excitation)
Several parameters…
• The binary (BIN) ensemble (minimum maximal excitation)
Invalid/inaccurate model?
0 0.5 1 1.5-0.2
0
0.2
0.4
0.6
0.8
1
1.2
'Field' x
'Me
asu
red
'/Pri
or
pre
dic
tio
n h
(x, )
Cal data
+2
-2
Prior Model Ens
?cov?,
sin,, 2121
xxh
Calibration – Identification!
0 0.5 1 1.50
0.2
0.4
0.6
0.8
1
1.2
1.4
'Field' x
'Me
asu
red
'/Pri
or
pre
dic
tio
n h
(x, )
Cal data
+2
-2
Post Model Ens
05.0,96.013.0
022.0014.0
014.0012.0cov,970.0139.0
22
22
NTRUE
Calibration of deterministic ensemble,
with adjustments of samples
-0.1 -0.05 0 0.05 0.1 0.15 0.2 0.250.7
0.75
0.8
0.85
0.9
0.95
1
1.05
(1)
1=
PRIOR(:,1)
(1)
2=
PRIOR(:,2)
(1)
3=
PRIOR(:,3)
(1)
4=
PRIOR(:,4)
POST
=(N)
1
2
True
Surrogate model quality and spread of regr. points
1 2 3 4 5 6 7 8 9 100.98
0.99
1
1.01
1 2 3 4 5 6 7 8 9 10-2
-1.5
-1
-0.5
log
10
(||
||)
Iteration
A complete concept for modeling uncertainty…
”What’s in it for us (UU/SP)?”
• Discussion partner: methods for evaluation of uncertainty
• ’Export’ of SP’s prototype methods
No one will be happier than SP if you are interested to try our
methods…
• Apply for joint projects (VR, ÅF, companies…)
Reactor physics, UU
Novel UQ methods, SP
• Scientific collaboration
Articles
Conferences
Workshops etc.
Thanks for your attention!