challenges in uncertainty, calibration, validation and ... · epistemic & aleatory uncertainty....
Post on 05-Jul-2020
9 Views
Preview:
TRANSCRIPT
1GE – Aviation & Global Research© 2011 General Electric Company - All Rights Reserved
Challenges In Uncertainty, Calibration, Validation and Predictability of Engineering Analysis Models
Dr. Liping WangGE Global ResearchManager, Probabilistics LabNiskayuna, NY
2011 UQ WorkshopUniversity of Minnesota, MNJune 02, 2011
Team MembersGE Global Research: Arun Subramaniyan, Nataraj Chennimalai,
Xingjie Fang, Giridhar Jothiprasad, Martha Gardner, Amit Kale
GE Aviation: Don Beeson, Gene Wiggs, and John Nelson
2GE – Aviation & Global Research© 2011 General Electric Company - All Rights Reserved
Outline
•Motivation
• History of Development
– How far are we along the path?
• GE capabilities
• Technical Challenges & Solutions
• Future Direction
• Summary
3GE – Aviation & Global Research© 2011 General Electric Company - All Rights Reserved
Input
Factors (Xs)
Motivation
• Why Model Calibration, Validation, Prediction & Uncertainty Quantification?
• What has been accomplished? �Literature Review �GE Experience
• Possible technical solutions and future direction
Output
Factors (Ys)Uncertainty:
• Aleatory – Random and usually modeled by
probability distributions.
Methods include probability theory
and classical statistics
• Epistemic – Lack of knowledge. Methods include
fuzzy logic or evidence/possibility
theory
X1X2
Etc.
.
.
Etc.
Deterministic
Simulation
X1
X2
Etc.
Input
Factors (Xs)
Y2
Etc.
Output
Factors (Ys)
Y1
Model parametersModel discrepancy
.
θ
δδδδ
Uncertainty Quantification (UQ)
4GE – Aviation & Global Research© 2011 General Electric Company - All Rights Reserved
•One task at a time (calibration, validation, prediction & uncertainty quantification) - since the 1980s
�Calibration - data matching, or inverse problems, or parameter estimation
- applied to heat transfer, fluid mechanics, solid mechanics, etc.
�Verification & Validation (V&V) - introduced by DoD, AIAA, ASME, National Labs …
�Prediction -well-established physics models, calibrated empirical models, and
meta-models (Response Surface, Kriging, Gaussian Process, Radial Basis Function, etc)
�Uncertainty quantification - Monte Carlo, First Order Second Moment, moments
based, polynomial chaos, etc
•All tasks simultaneously - first introduced by Kennedy and O’Hagan in 2001 (Bayesian framework)
History of Development
– How far are we along the path?
5GE – Aviation & Global Research© 2011 General Electric Company - All Rights Reserved
Kennedy & O’Hagan (2001)
y(xi)= η(xi, θ) + δ(xi) + ε(xi), i=1,2,…,n
Observations from Observations from the physical systemthe physical system Output of a simulator, Output of a simulator,
with design inputs (with design inputs (xx) and ) and calibration parameters (calibration parameters (θθ))
Discrepancy between Discrepancy between the simulator and the the simulator and the physical systemphysical system
Observation Observation (measurement system) (measurement system) errorerror
• Build & calibrate Gaussian Process (GP) models for both ηηηη and δ...δ...δ...δ...
� Specify beliefs about θ , δ through prior probability distributions � Use Markov Chain Monte Carlo (MCMC) to obtain parameter estimates
• Similar approaches by Higdon et al. and Liu et al.
•Kennedy & O’Hagan Hybrid Model Formulation:
Most implementations are for single output
Given data
( | ) ( )( | )
( | ) ( )
f y ff y
f y f
θ θθ
θ θ=
∫
( | )f yθ
Posterior
θ
( )f θ
pdfPriorθ
Likelihood function
( ) ( | )L f yθ θ=
θ
•What is Bayesian Statistics?
6GE – Aviation & Global Research© 2011 General Electric Company - All Rights Reserved
Multiple Outputs
•Implementation by Los Alamos National Lab (LANL) - Higdon, William et al.
� Principal Components Analysis (PCA) for dimension reduction & efficiency improvements
� Correlated outputs
),(...),(),( 11 θθθηηη
xwkxwkx pp++=
)(...)()( 11 xvdxvdx pp δδδ ++=
ηpkkk ,...,, 21 and are the principal componentsδpddd ,...,, 21
w & v are the GP models for simulator and model correction
More Applicable to Real Problems with LANL Implementation
7GE – Aviation & Global Research© 2011 General Electric Company - All Rights Reserved
Maximum Likelihood Estimation (MLE)
• Alternative approach to Bayesian (Xiong, Chen, Tsui and Apley)
• Investigated three possible formulations
• Implementation only for single output
• Sensitivity analysis prior to MLE optimization to avoid numerical instability
εθη +=Θ ),(),( xxy
εδθη ++=Θ )(),(),( xxxy
εδθη ++=Θ )())(,(),( xxxxy
best
8GE – Aviation & Global Research© 2011 General Electric Company - All Rights Reserved
Model Inadequacy Correction & Prediction
(No Calibration)
• Capture model inadequacy with no model calibration
(Wang et al. and Chen et al.)
• Closed form Bayesian posterior
• Solve GP hyper-parameters using MLE
• Improved efficiency for high dimensional design space
• Useful for well established physical models where calibration is not necessary or performed previously
εδη ++= )()()( xxxy
9GE – Aviation & Global Research© 2011 General Electric Company - All Rights Reserved
V&V and Model Validation Metrics
Most Common
Desired State
• Current and desired state
of validation metrics(Oberkampf et al. 2004)
• Quantitative Metrics using
classical hypothesis testing,
Bayes factor, frequentist’s
metrics, and area metrics
• Quantitative Metrics using
Kennedy & O’Hagan Bayesian
Framework (Chen et al.)
• Preliminary elements of model validation
(Paez, Swiler, Mayes, Miller, et al.
– 2009 International Modal Analysis
Conference, Orlando, FL)
• Epistemic uncertainty (Paez & Swiler, Paez)
Analysis
Modelers
Customers Stakeholders
Experimentalists
Validation Analysts
10GE – Aviation & Global Research© 2011 General Electric Company - All Rights Reserved
GE Capabilities
• Deterministic Inverse modeling since 2003�Methods development and implementation
�Efficient transient data matching using PCA based hybrid
metamodels & zoning techniques
�Partial probabilistic data matching to update standard deviations
•Widely used across GE businesses
Materials Design Acceleration – Material Modeling
GEnx – 29 XsGP7000 – 100XsGE90 – 78Xs
Analysis time savings >50%Data mismatch reduced by half
Transient Analysis and Performance
23 Xs, 35 TCs simultaneously
Others:Transient cycle models 3D transient clearances Undercowl heat transfer Empirical model tuning
Heat Transfer and Fluid Systems
11GE – Aviation & Global Research© 2011 General Electric Company - All Rights Reserved
Probabilistic Inverse modeling (Bayesian Hybrid Modeling) since 2006
� Built on Kennedy & O’Hagan Bayesian Method and LANL Implementation
� Efficiency improvement (~2X), flexibility, robustness …
� Investigated possible formulations
� Key drivers of model inadequacy & insight to possible model improvements
� Validated with multiple benchmark problems
GE Capabilities
εθη
εδη
δη
η
+=
++=
+=
=
),()(
)()()()(
)()()(
)()(
xxy
xxxxy
xxxy
xxy
εδθη
εδθη
εδθη
εδθη
+=
++=
++=
++=
))(,,()(
)())(,()(
))(,()(
)()(),()(
xxxy
xxxxy
xxxy
xxxxy Kennedy & O’Hagan
12GE – Aviation & Global Research© 2011 General Electric Company - All Rights Reserved
Demonstration with challenging engineering problems
GE Capabilities
Corrected flow
PT_ra
tio
1201151101051009590858075706560555045
1.9
1.8
1.7
1.6
1.5
1.4
1.3
1.2
1.1
10.9
1201151101051009590858075706560555045
1.9
1.8
1.7
1.6
1.5
1.4
1.3
1.2
1.1
10.9
Missing vertical parts of high speed lines
Test data : y(x)
Test data uncertainty: ε(x)
BRM model: η(x,θ)
Design parameters: x
Model parameters: θ
104 104.5 105 105.5 106 106.5 107 107.5 108 108.5 1091.54
1.56
1.58
1.6
1.62
1.64
1.66
1.68
1.7
1.72
1.74
Test
Hybrid Modeling
Calibrated Simulator only
105% speed-line
104 104.5 105 105.5 106 106.5 107 107.5 108 108.5 1091.54
1.56
1.58
1.6
1.62
1.64
1.66
1.68
1.7
1.72
1.74
Test
Hybrid Modeling
Calibrated Simulator only
105% speed-line
Performance Maps at Speed = 105%
δ(x)
Model Discrepancy & Updatingδ(x)
HybridModeling
-1 -0.5 0 0.5 1
Mean=0.1797Std=0.39367
θ (calibrated)
Test
HM Mean
90% CI
ConfidenceBounds
Corrected flow
PT_ra
tio
Corrected flow
PT_ra
tio
1201151101051009590858075706560555045
1.9
1.8
1.7
1.6
1.5
1.4
1.3
1.2
1.1
10.9
1201151101051009590858075706560555045
1.9
1.8
1.7
1.6
1.5
1.4
1.3
1.2
1.1
10.9
Missing vertical parts of high speed lines
Test data : y(x)
Test data uncertainty: ε(x)
BRM model: η(x,θ)
Design parameters: x
Model parameters: θ
104 104.5 105 105.5 106 106.5 107 107.5 108 108.5 1091.54
1.56
1.58
1.6
1.62
1.64
1.66
1.68
1.7
1.72
1.74
Test
Hybrid Modeling
Calibrated Simulator only
105% speed-line
104 104.5 105 105.5 106 106.5 107 107.5 108 108.5 1091.54
1.56
1.58
1.6
1.62
1.64
1.66
1.68
1.7
1.72
1.74
Test
Hybrid Modeling
Calibrated Simulator only
105% speed-line
Performance Maps at Speed = 105%
δ(x)
Model Discrepancy & Updatingδ(x)
HybridModeling
-1 -0.5 0 0.5 1
Mean=0.1797Std=0.39367
θ (calibrated)
Test
HM Mean
90% CI
ConfidenceBounds
Corrected flow
PT_ra
tio
95 100 105 1101.35
1.4
1.45
1.5
1.55
1.6
1.65
1.7
1.75
Test
Calibrated Simulator only
Hybrid Modeling
105%
100%
95%
95 100 105 1101.35
1.4
1.45
1.5
1.55
1.6
1.65
1.7
1.75
Test
Calibrated Simulator only
Hybrid Modeling
105%
100%
95%
98 100 102 104 106 108 1101.4
1.45
1.5
1.55
1.6
1.65
1.7
1.75
1.8
Test
Calibrated Simulator only
Hybrid Modeling
100%
105%
98 100 102 104 106 108 1101.4
1.45
1.5
1.55
1.6
1.65
1.7
1.75
1.8
Test
Calibrated Simulator only
Hybrid Modeling
100%
105%
2 Speed Lines 3 Speed Lines
95 100 105 1101.35
1.4
1.45
1.5
1.55
1.6
1.65
1.7
1.75
Test
Calibrated Simulator only
Hybrid Modeling
105%
100%
95%
95 100 105 1101.35
1.4
1.45
1.5
1.55
1.6
1.65
1.7
1.75
Test
Calibrated Simulator only
Hybrid Modeling
105%
100%
95%
98 100 102 104 106 108 1101.4
1.45
1.5
1.55
1.6
1.65
1.7
1.75
1.8
Test
Calibrated Simulator only
Hybrid Modeling
100%
105%
98 100 102 104 106 108 1101.4
1.45
1.5
1.55
1.6
1.65
1.7
1.75
1.8
Test
Calibrated Simulator only
Hybrid Modeling
100%
105%
2 Speed Lines 3 Speed Lines
-0.4 -0.2 0 0.2 0.4 0.6 0.8 10
50
100
150
200
250
300
350
400
450
Mean=0.036642
Std=0.1968
-0.4 -0.2 0 0.2 0.4 0.6 0.8 10
50
100
150
200
250
300
350
400
450
Mean=0.036642
Std=0.1968
Matches Test Data Well for Single & Multiple Speed Lines
13GE – Aviation & Global Research© 2011 General Electric Company - All Rights Reserved
GE Capabilities GE Capabilities
13
Transient AIR Model (Benchmark problem)
• 6 calibration parameters
• 9 Outputs (Ys)
• 52 time points in each transient
(> 3000 DoE points)
• Only 52 DoE (simulation) points used
for Hybrid modeling
BHM calibrates transient model accurately with very little data
ηηηη(t, θθθθ)
y(t)= ηηηη(t, θθθθ) + δδδδ(t)
δδδδ(t)Calibrated Model
Calibrated &
Discrepancy Adjusted
Model
Discrepancy
14GE – Aviation & Global Research© 2011 General Electric Company - All Rights Reserved
Demonstration with challenging engineering problems
GE Capabilities
Demonstration with IPE Status Match, TOW Uncertainty Severity Modeling, Cycle Deck Performance …
IPE Status Match TOW Uncertainty
Cycle Deck
0 0.2 0.4 0.6 0.8 1500
550
600
650
700
750
800
ZT
49
calibrated simulator
0 0.2 0.4 0.6 0.8 1
ZPCN12
discrepancy-adjusted
0 0.2 0.4 0.6 0.8 1-8
-6
-4
-2
0
2
4
6
8
10discrepancy
-0.04 -0.03 -0.02 -0.01 0 0.01 0.02 0.03 0.040
50
100
150
200
250
300
350
DE42DD
His
togra
m C
oun
ts
θ
Posterior Distribution
of HPT Efficiency
Fan Speed
Exh
aust Gas Temperature
Calibrated Simulator Discrepancy-Adjusted Discrepancy
Fan Speed
Exh
aust Gas Temperature
Calibrated Simulator Discrepancy-Adjusted Discrepancy
• Improved calibration results by capturing model discrepancy• More confidence in solution with probabilistic estimation
• Characterizing model discrepancy and uncertainty in severity models • Main effects able to point high thrust severity for improvement of
current models
15GE – Aviation & Global Research© 2011 General Electric Company - All Rights Reserved
Extension to Model Validation
GE Capabilities
Acoustic fluctuations (p’)
Flame heat release fluctuations (q’)
Acoustic fluctuations (p’)
Flame heat release fluctuations (q’)
Acoustic fluctuations (p’)
Flame heat release fluctuations (q’)
Xs
θθθθs
δδδδ(x)
εεεε(x)
Combustion Dynamics
GE90 Fan Blade Row Model
-40 -30 -20 -10 0 10 20 30 40 50 600
5
10
15
20
25
30
δδδδ(x) at untested points
-50 -40 -30 -20 -10 0 10 20 30 40 500
5
10
15
20
25
θ (calibrated)
Co
mp
uta
tio
n v
. E
xp
eri
men
t
-2.5
-2
-1.5
-1
-0.5
0
0.5
δδδδ(x)
Freq1
Calibrated Simulator Model Discrepancy δδδδ(x)
δδδδ(x) at tested points
-40 -30 -20 -10 0 10 20 30 40 50 600
5
10
15
20
25
30
δδδδ(x) at untested points
-50 -40 -30 -20 -10 0 10 20 30 40 500
5
10
15
20
25
θ (calibrated)
Co
mp
uta
tio
n v
. E
xp
eri
men
t
-2.5
-2
-1.5
-1
-0.5
0
0.5
δδδδ(x)
Freq1
Calibrated Simulator Model Discrepancy δδδδ(x)
δδδδ(x) at tested points
Enabler for Probabilistic Validation Metrics at Tested & Untested Points
16GE – Aviation & Global Research© 2011 General Electric Company - All Rights Reserved
Going-forward�Continue improving MCMC speed issues for high dimensional (>100Xs)
and solving challenging engineering data matching applications
� Model Validation for All Engineering Models (3-year program)
(Flexible to all models based on data availability, affordable & accurate,
account for all types of uncertainty, probabilistic quantitative metrics)
GE Capabilities
TTPTVrVthKω
combustor exit
purge flow
geom etrytip clearance,
core shift, film
hole d rilling
Xs
TTPTVrVthKω
combustor exit
purge flow
geom etrytip clearance,
core shift, film
hole d rilling
Xs
θθθθsεεεε(x)
Acoustic fluctuations (p’)
Flame heat release fluctuations (q’)
Acoustic fluctuations (p’)
Flame heat release fluctuations (q’)
Acoustic fluctuations (p’)
Flame heat release fluctuations (q’)
Xs
θθθθsδδδδ(x)
εεεε(x)
δδδδ(x) = y(x) – (ηηηη(x, θθθθ) + εεεε(x))δδδδ(x) = y(x) – (ηηηη(x) + εεεε(x)) δδδδ(x) = y(x) – (ηηηη(x, θθθθ) + εεεε(x))
Aero
Mechanical
2011 Hot Gas Path Heat Transfer 2012 Combustion Dynamics 2013 All Engineering Models
Complicated Physics, Unknown Uncertainty, High Dimension … Challenges Remain!
17GE – Aviation & Global Research© 2011 General Electric Company - All Rights Reserved
Technical Challenges and Solutions
• Curse of dimensionality�Large number of calibration parameters. MCMC speed issues.
�Transient data matching
�PCA/Sensitivity, sparse matrix inversion, adaptive convergence criteria for MCMC, sequential MC or other optimization techniques …
• Source of Uncertainty�Epistemic & Aleatory uncertainty.
�Probability, statistics, fuzzy logic or evidence/possibility theory
• Model inadequacy, uncertainty and characterization�Identifiability of parameter calibration and model inadequacy
Use as much knowledge as you can on the prior. Carefully choose
the range. Uncertainty quantification of experiments.
�Better understand model inadequacy and key drivers
�Post-processing to model discrepancy terms. Bayesian model comparison for possible suggestion to model improvement
18GE – Aviation & Global Research© 2011 General Electric Company - All Rights Reserved
Technical Challenges and Solutions
• Lack of data and extrapolation�Limited test & simulation data
�No overlap between simulation & test data (extrapolation)
�Scientific knowledge, designer’s belief (prior)
• Confounding Effects�High-dimensionality coupled with lack of data
�Challenging mathematical issues
�Scientific knowledge, designer’s belief (prior)
• Model validation and quantitative metrics�Account for all source of uncertainty (epistemic & aleatory)
�Flexible for all analysis models - empirical, physics (no calibration) & metamodels … based on data availability (complete, partial and extrapolation)
�Affordable & accurate
�Confidence and probabilistic metrics
19GE – Aviation & Global Research© 2011 General Electric Company - All Rights Reserved
Technical Challenges and Solutions
• Multiple sets of experimental data�Mixed datasets – multiple speed lines, multiple engine data …
�Multiple modes of posterior distributions
�Numerical and speed issues
• Measurement error and uncertainty�Uncertainty quantification for both epistemic and aleatory uncertainty
�Statistical analysis
�Outlier detection (sensor failure)
• Multi-physics & multi-fidelity models�Direct simulations are prohibitively expensive
�Vast scale difference among the lowest atomic to the highest macroscopic scale
�Difficult to establish criteria & strategies when switching design space from one scale to another
20GE – Aviation & Global Research© 2011 General Electric Company - All Rights Reserved
Summary
• Advancements in model calibration, validation, prediction and uncertainty quantification in the past three decades
• Much research and many publications from industry, government, academia
• GE has been very active
• Technical challenges remain and are being worked
top related