representing and analyzing uncertainty in large-scale complex system models
DESCRIPTION
Representing and Analyzing Uncertainty in Large-Scale Complex System Models. Doug Allaire. Outline. Modeling Uncertainty Research Objectives The Aviation Environmental Portfolio Management Tool Global Sensitivity Analysis Future Work. Modeling Uncertainty. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/1.jpg)
Partnership for AiR Transportation Noise and Emission Reduction Partnership for AiR Transportation Noise and Emission Reduction
An FAA/NASA/TC-sponsored Center of Excellence
Representing and Analyzing Representing and Analyzing Uncertainty in Large-Scale Uncertainty in Large-Scale Complex System ModelsComplex System Models
Doug Allaire
![Page 2: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/2.jpg)
2
Outline
1. Modeling Uncertainty
2. Research Objectives
3. The Aviation Environmental Portfolio Management Tool
4. Global Sensitivity Analysis
5. Future Work
![Page 3: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/3.jpg)
3
Modeling Uncertainty
• Uncertainties in modeling are unavoidable
• Uncertainty should be properly represented– Estimates and predictions– Risk analysis– Cost-benefit analysis– Furthering model development
• Uncertainty in models can be represented with three different methodologies– Probabilistic (aleatory)– Fuzzy sets (epistemic)– Fuzzy Randomness (both)
![Page 4: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/4.jpg)
4
Large-Scale Complex System Models
• Multiple disciplines– Economics, aerodynamics, atmospheric science, …
• Many inputs, many outputs
• Systems of models of different character– Physics-based, empirical
• Many assumptions– Subjective, objective
• Computationally intensive
• Characterizing, representing, quantifying, and accounting for uncertainty is key– To both development and application of models
![Page 5: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/5.jpg)
5
Uncertainty in Large-Scale Complex System Models
Goals of a formal uncertainty analysis:
• Further the development of a model– Identify gaps in functionality that significantly impact the
achievement of model requirements, leading to the identification of high-priority areas for further development.
– Rank inputs based on contributions to output variability to inform future research
• Inform decision-making– Provide quantitative evaluation of the performance of the model
relative to fidelity requirements for various analysis scenarios.– Properly represent different types of uncertainty in the model
Especially important for complex systems that comprise multiple models of different disciplines, different character, different assumptions
![Page 6: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/6.jpg)
6
What exists today
• Probabilistic and fuzzy random uncertainty models
• Rigorous model independent sensitivity analysis method – Expensive
• Several model dependent sensitivity analysis methods– Inexpensive but difficult to ensure proper use
• Limited research into the use of surrogate models in uncertainty analysis– Critical for models with long runtimes
• Numerous sampling techniques for improving runtimes of probabilistic analyses
![Page 7: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/7.jpg)
7
Research Objectives
• For specific large-scale, complex system models, identify how to properly represent and analyze uncertainty.
• Systematically develop surrogate models for situations where proper representation and analysis of uncertainty is computationally prohibitive. Limitations and uncertainty associated with the use of a surrogate should be identified.
• Use appropriate sampling techniques to improve computational costs in uncertainty analysis. Limitations and uncertainty associated with the use of chosen sampling methods should be identified.
![Page 8: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/8.jpg)
8
Aviation Environmental Portfolio Management Tool (APMT)
APMT
Policy scenarios•Certification stringency•Market-based measures•Land-use controls•Sound insulation
Market scenarios•Demand•Fuel prices•Fleet
Environmental scenarios•CO2 growth
Technology and operational advances•CNS/ATM, NGATS•Long term technology forecasts
Cost-effectiveness•$/kg NOx reduced•$/# people removed from 65dB DNL •$/kg PM reduced•$/kg CO2 reduced
Benefit-cost•Health and welfare impacts•Change in societal welfare ($)
Distributional analyses•Who benefits, who pays•Consumers•Airports•Airlines•Manufacturers•People impacted by noise and pollution•Special groups•Geographical regions
Global, Regional, Airport-local
inp
uts
ou
tpu
ts
![Page 9: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/9.jpg)
9
APMT PARTIAL EQUILIBRIUM BLOCK
NOISE IMPACTS
LOCAL AIR QUALITY IMPACTS
CLIMATE IMPACTS
APMT COSTS & BENEFITS
New Aircraft
Emissions
Noise
APMT BENEFITS VALUATION BLOCK
Monetized Benefits
CollectedCosts
Emissions
Emissions & Noise
Policy and Scenarios
AEDT
Fares
DEMAND (Consumers)
SUPPLY (Carriers)
Operations
Schedule &
Fleet
Airport-levelEmissions
Tools
GlobalNoise
Assessment
Airport-levelNoiseTools
GlobalEmissionsInventories
EDS VehicleNoise
Design Tools
VehicleEmissions
Design Tools
Technology
ImpactForecastin
gVehicle
CostAssessme
nt
DesignTools
Interface
APMT approach
![Page 10: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/10.jpg)
10
APMT Uncertainty Analysis Goals
• AQ1. What are the key assumptions employed within the module? How do these assumptions translate into quantifiable uncertainty in module outputs?
• AQ2. What are the key assumptions employed within the module databases? How do these assumptions translate into quantifiable uncertainty in module outputs?
• AQ3. How do assumptions/limitations in modeling and databases impact the applicability of the module for certain classes of problems? What are the implications for future development efforts?
• AQ4. How do uncertainties in module inputs propagate to uncertainties in module outputs? Further, what are the key inputs that contribute to variability in module outputs?
• AQ5. For assumptions, limitations and inputs where effects cannot be quantified, what are the expected influences (qualitatively) on module outputs
• AQ6. How do assessment results translate into guidelines for use?
![Page 11: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/11.jpg)
11
Answering AQ4 Probabilistically
• AQ4. How do uncertainties in module inputs propagate to uncertainties in module outputs? Further, what are the key inputs that contribute to variability in module outputs?
• Assign factor distributions
• Propagate uncertainty
• Apportion output variance to model factors
![Page 12: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/12.jpg)
12
Assigning Factor Distributions
• Example: APMT Aircraft Emissions Module
EINOx: ( ) ( )( )
17.36 .24 , .24 0
17.36 .24 , 0 .24EI
x for xP x
x for x
+ − ≤ ≤⎧⎪=⎨
− ≤ ≤⎪⎩
( ) ( )( )
82.64 .11 , .11 0
82.64 .11 , 0 .11T
x for xP x
x for x
+ − ≤ ≤⎧⎪=⎨
− ≤ ≤⎪⎩Temperature:
( ) ( )( )
1111.11 .03 , .03 0
1111.11 .03 , 0 .03P
x for xP x
x for x
+ − ≤ ≤⎧⎪=⎨
− ≤ ≤⎪⎩Pressure:
( ) ( )( )
34.60 .17 , .17 0
34.60 .17 , 0 .17H
x for xP x
x for x
+ − ≤ ≤⎧⎪=⎨
− ≤ ≤⎪⎩Rel. Humidity:
( ) 10, .05 .05FFP x for x= − ≤ ≤Fuel Flow:
.24-.24 0
4.17
-.11 0 .11
9.09
-.03 0 .03
33.33
-.17 0 .17
5.88
-.05 0 .05
10
![Page 13: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/13.jpg)
13
Propagate Uncertainty
gT g(T)
Assume a model:
o.w.
• Would like to compute the mean value of g(T)
( )2m−Ο• Trapezoidal Rule Error goes as and 1/ 1sm N= −
[2][1]
![Page 14: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/14.jpg)
14
Propagate Uncertainty: Monte Carlo Simulation
• Assume is a random variable with a probability density function fX(x).
• Monte Carlo mean estimate (for any number of factors)
• Error goes probabilistically as– Strong law of large numbers– Central limit theorem
( ) ( ) ( ) ( )1
1
s
N
nXnI
g X g x f x d x g xN =
Ε = ≈⎡ ⎤⎣ ⎦ ∑∫
X
( )1/ 2N −Ο
• Compute mean value of g(X) using random samples
• X is any factor (temperature, pressure, etc.)gX g(X)
Assume same model:
![Page 15: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/15.jpg)
15
Partitioning Output Variance
Analysis of Variance (ANOVA)– Assumes a linear statistical model– Requires specification of factors and factor interactions total to
be comprehensive, where S is the number of factors
Vary-all-but-one methods– Calculate factor variance contributions by fixing a factor and performing a
Monte Carlo simulation. – Requires specification of where to fix a factor– N(S+1) model evaluations
Global Sensitivity Analysis– All factors varying– Computes total sensitivity indices for each factor in N(S+1) model
evaluations using Monte Carlo simulation– Variance contributions take into account underlying factor distributions
( )2 1S −
![Page 16: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/16.jpg)
16
Global Sensitivity Analysis
• What is the goal?
• ANOVA Decomposition
• Partitioning output variance
• Monte Carlo Estimates
• Total Sensitivity Indices
![Page 17: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/17.jpg)
17
The goal of Global Sensitivity Analysis
• Ranking model factors on the basis of contribution to output variability– Useful for model development– Useful for understanding model outputs
• The goal is to partition output variance amongst the factors of the model– Main effects– Interaction effects
Factor 1
Factor 2
Factor 1, Factor 2 Interaction
Output Variance
![Page 18: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/18.jpg)
18
ANOVA Decomposition
• High-Dimensional Model Representation of f(x) [4]
• Example HDMR for a function of three parameters
![Page 19: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/19.jpg)
19
ANOVA Decomposition
• High-Dimensional Model Representation of f(x) [4]
• Example HDMR for a function of three parameters
Mean value
![Page 20: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/20.jpg)
20
ANOVA Decomposition
• High-Dimensional Model Representation of f(x) [4]
• Example HDMR for a function of three parameters
Mean value Main effects
![Page 21: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/21.jpg)
21
ANOVA Decomposition
• High-Dimensional Model Representation of f(x) [4]
• Example HDMR for a function of three parameters
Mean value Main effects
First-order interactions
![Page 22: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/22.jpg)
22
ANOVA Decomposition
• High-Dimensional Model Representation of f(x) [4]
• Example HDMR for a function of three parameters
Mean value Main effects
First-order interactions
Second-order interaction
![Page 23: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/23.jpg)
23
Example of ANOVA Decomposition
• Consider a function
• Then the ANOVA decomposition is:
![Page 24: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/24.jpg)
24
Calculating Variances
• Given a function f(x) that is square integrable,
• Similarly,
• Example for a function of two parameters:
![Page 25: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/25.jpg)
25
The goal of global sensitivity analysis
• The goal is to partition output variance amongst the factors of the model– Main effects– Interaction effects
Factor 1
Factor 2
Factor 1, Factor 2 Interaction
Output Variance
D D1
D12
D2
![Page 26: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/26.jpg)
26
Monte Carlo Estimates
• The expected value of f(x) may be estimated from:
![Page 27: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/27.jpg)
27
Monte Carlo Estimates
• The expected value of f(x) may be estimated from:
• The variance of f(x) may then be estimated from:
![Page 28: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/28.jpg)
28
Monte Carlo Estimates
• The expected value of f(x) may be estimated from:
• The variance of f(x) may then be estimated from:
• For the one indexed terms the variance may be estimated from:
![Page 29: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/29.jpg)
29
Total Sensitivity Indices
• Sensitivity indices:
• Total Sensitivity indices:
![Page 30: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/30.jpg)
30
APMT ResultsGlobal Sensitivity Analysis
0
0.2
0.4
0.6
0.8
HousingGrowth Rate
DiscountRate
Quiet Level ContourUncertainty
NDI
Total Global Sensitivity Index
Total Sensitivity Indices
0.69 0.670.67
0.360.300.28
0.110.05 0.010.02 0.02
0
0.2
0.4
0.6
0.8
1.0
1.2
IntegratedTemperature Change
Damage NPV
OthersRF* short-livedclimate sensitivityDamage coefficient
BVB-Noise Module Analysis
BVB-Climate Module Analysis
![Page 31: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/31.jpg)
31
Future WorkUncertainty Model
Selection
Problem Statement
Define model Define methods
Compare methods
Test functions
Surrogate Reqs. Sampling Reqs.
Application
Large-scale model (APMT Module)
Probabilistic
Problem Statement
Define model Define methods
Compare methods
Test functions
Application
Large-scale model (APMT Module)
Fuzzy Random
Surrogate Reqs. Sampling Reqs.
Compare
![Page 32: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/32.jpg)
32
Questions?
![Page 33: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/33.jpg)
33
References
[1]http://www.krellinst.org/UCES/archive/modules/potential/trap/index.html
[2]http://www.statisticalengineering.com/curse_of_dimensionality.htm
[3] T. Homma, A. Saltelli, “Importance Measures in Global Sensitivity Analysis of Nonlinear Models.” Reliability Engineering and System Safety, 52(1996) pp1-17.
[4] I.M. Sobol’, “Global Sensitivity Indices for Nonlinear Mathematical Models and their Monte Carlo Estimates,” Mathematics and Computation in Simulation 55(2001) 271-280.
![Page 34: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/34.jpg)
34
Aviation Environmental Portfolio Management Tool (APMT): Motivation
• Aviation benefits and environmental effects result from a complex system of interdependent technologies, operations, policies and market conditions
• Community responses, policy and R&D options typically considered in a limited context
– only noise, only local air quality, only climate change
– only partial economic effects
• Actions in one domain may produce unintended negative consequences in another
• Tools and processes do not support recommended practice
– NPV of benefits-costs is recommended basis for informing policy decisions in U.S., Canada and Europe
![Page 35: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/35.jpg)
35
ANOVA Decomposition (Cont.)
• A high-dimensional model representation is unique if
where and
• The individual functions are orthogonal
• The expected value of each individual function is zero
e.g.
• The expected value of f(x) is then
1 1
1
,...,
0
( ,..., ) 0s si i i i kf x x dx =∫ 1,..., sk i i= 1,2,...,s n=
1 1 1 1( , , ) ( , , ) 0
s s t ti i i i j j j jf x x f x x dx =∫ K KK K
1 1 1( ) 0f x dx =∫
0( )f x dx f=∫
![Page 36: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/36.jpg)
36
Improving Runtime: Surrogate Modeling
• A surrogate is a less expensive, (often) lower-fidelity model that represents the system input/output behavior
• Surrogates are often classified into three categories:1. Data fit, e.g. response surface models, Kriging models EDS2. Reduced-order models, derived using
system mathematical structure3. Hierarchical models,
e.g. coarser grid, neglected physics, aggregation AEDT
• Why surrogates?– If one analysis takes 1 minute, then a Monte Carlo simulation
with 10,000 samples takes 10,000 minutes = 7 days– A global sensitivity analysis requiring a Monte Carlo simulation
for each factor would take 7(S+1) days, where S is the number of factors
• Relating uncertainty analysis results done with a surrogate model to results from the full model is critical
![Page 37: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/37.jpg)
37
Surrogate Modeling Approach for APMT
• Adaptive approach for Aircraft Emissions Module– Monte Carlo Simulation of a single day of flights– Build up from a small number of flights to representative set
• Statistical approach for Aircraft Performance Module– Determine critical factors using expert opinion
• Fuel burn per meter• Emissions per meter• Segment level thrust distribution• …
– Group aircraft/engine pairs based on statistical similarities– Select representative from each group
• Important questions to ask– How do these surrogates impact Monte Carlo results?– How do these surrogates impact global sensitivity results?
![Page 38: Representing and Analyzing Uncertainty in Large-Scale Complex System Models](https://reader035.vdocument.in/reader035/viewer/2022062500/56815563550346895dc32dbc/html5/thumbnails/38.jpg)
38
Improving Runtime: Sampling Methods
• Some well-known methods:– Brute-force Monte Carlo Methods
• Use pseudorandom numbers on [0,1] and the inversion method• The probabilistic error bound is .
– Quasi-Monte Carlo Methods• Use well-chosen deterministic points rather than random samples• Low discrepancy sequences (e.g. Halton, Hammersley)• Deterministic error bound
– Stratification -- Latin Hypercube Sampling• Stratify on all factor dimensions simultaneously• Leads to lower variance in integral estimates than independent,
identically distributed samples• Probabilistic error bound .
• Things to keep in mind– How bias in sampling impacts analysis metrics– Obtaining more samples if a metric has not converged sufficiently
( )1/ 2N −Ο
((log ) / )sN NΟ
( )1/ 2N −Ο