error and uncertainty in modeling
DESCRIPTION
Error and Uncertainty in Modeling. George H. Leavesley, Research Hydrologist, USGS, Denver, CO. Sources of Error and Uncertainty. Model Structure Parameters Data Forecasts of future conditions. Sacramento. Conceptualization of Reality. measurement. Effective Parameters and States. - PowerPoint PPT PresentationTRANSCRIPT
Error and Uncertainty in Modeling
George H. Leavesley, Research Hydrologist, USGS, Denver, CO
Sources of Error and Uncertainty
• Model Structure
• Parameters
• Data
• Forecasts of future conditions
INTERFLOWSURFACERUNOFF
INFILTRATIONTENSION
TENSION TENSION
PERCOLATION
LOWERZONE
UPPERZONE
PRIMARYFREE
SUPPLE-MENTAL
FREE
RESERVED RESERVED
FREE
EVAPOTRANSPIRATION
BASEFLOW
SUBSURFACEOUTFLOW
DIRECTRUNOFF
Precipitation Sacramento
Conceptualization of Reality
homog.
(xeff,eff)
input
input
output
output
identical Identical ?
heterog.
measurement
After Grayson and Blöschl, 2000, Cambridge Univ. Press
real world
model
Effective Parameters and States
1000M
Mountain blockage of radar
2000M3000M
Precipitation Measurement
Source: Maddox, et. al. Weather and Forecasting, 2002.
Sparse precip gauge distribution
SAHRA – NSF STC
Streamflow Measurement Accuracy (USGS)
• Excellent– 95% of daily discharges are within 5% of true value
• Good– 95% of daily discharges are within 10% of true value
• Fair– 95% of daily discharges are within 15% of true value
• Poor– Do not meet Fair criteria
Different accuracies may be attributed to different parts of a given record
Dimensions of Model Evaluation
From Wagener 2003, Hydrological Processes
Dimensions of Model Evaluation
From Wagener 2003, Hydrological Processes
Performance Measures
Mean (observed vs simulated)
Standard deviation (observed vs simulated)
Root Mean Square Error (RMSE)
Mean Absolute Error (MAE)
Performance Measures
Coefficient of Determination R2
Not a good measure -- High correlations can be achieved for mediocre or poor models
Performance Measures
Coefficient of Efficiency E
Nash and Sutcliffe, 1970, J. of Hydrology
Widely used in hydrology Range – infinity to +1.0 Overly sensitive to extreme values
Seasonal Variability of Nash-Sutcliffe Efficiency
Seasonal Analysis of Daily Nash-Sucliffe Efficiency01608500
-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 2 4 6 8 10 12 14
Month
Eff
icie
ncy SAC
GR4J
PRMS
Performance Measures
Index of Agreement d
Willmott, 1981, Physical Geography
Range 0.0 – 1.0 Overly sensitive to extreme values
Analysis of Residuals
Performance Measure Issues
• Different measures have different sensitivities for different parts of the data.
• Are assumptions correct regarding the nature of the error structures (i.e. zero mean, constant variance, normality, independence, …)?
• Difficulty in defining what constitutes an acceptable level of performance for different types of data.
Dimensions of Model Evaluation
From Wagener 2003, Hydrological Processes
Definitions
• Uncertainty analysis: investigation of the effects of lack of knowledge or potential errors on model components and output.
• Sensitivity analysis: the computation of the effect of changes in input values or assumptions on model output.
EPA, CREM, 2003
Parameter Sensitivity
The single, “best-fit model” assumption
Magnitude of Parameter Error
5% 10% 20% 50%
%> VAR 0.23963 0.95852 3.83408 23.96303 %> SE 0.11974 0.47812 1.89901 11.33868 soil_moist_max 0.15243 0.60973 2.43891 15.24316
%> VAR 0.16210 0.64840 2.59359 16.20993 %> SE 0.08102 0.32367 1.28849 7.80071 hamon_coef 0.10311 0.41245 1.64981 10.31133
%> VAR 0.07889 0.31556 1.26224 7.88900 %> SE 0.03944 0.15766 0.62914 3.86963 ssrcoef_sq 0.05018 0.20073 0.80293 5.01829
joint 0.32477 1.29908 5.19632 32.47698
Error Propagation
routing
gw rech
soil moisture
et
et
soil moisture
gw rech
routing
baseflow
Objective Function Selection
Relative Sensitivity
SR = (QPRED / PI) * (PI / QPRED)
Relative Sensitivity Analysis
Soil available water holding capacity
Evapotranspiration coefficient
Relative Sensitivity Analysis
Snow/rain threshold temperature
Snowfall adjustment
Parameter Sensitivity
The “parameter equifinality” assumption
• Consider a population of models• Define the likelihood that they are
consistent with the available data
Regional Sensitivity Analysis
• Apply a random sampling procedure to the parameter space to create parameter sets
•Classify the resulting model realizations as “behavioural” (acceptable) or “non-behavioural”
•Significant difference between the set of “behavioural” and “non-behavioural” parameters identifies the parameter as sensitive
Spear and Hornberger, 1980, WRR
ji
0
11
0 cum
ulat
ive
dist
ribu
tion
cum
ulat
ive
dist
ribu
tion
)B|(F i
)B|(F i
)(F i
)B|(F j
)B|(F j
)(F j
Sensitive Not sensitive
B = behavioural B = non-behavioural
Regional Sensitivity Analysis
• Monte Carlo generated simulations are classified as behavioural or non-behavioural, and the latter are rejected.
• The likelihood measures of the behavioural set are scaled and used to weight the predictions associated with individual behavioural parameter sets.
• The modeling uncertainty is then propagated into the simulation results as confidence limits of any required percentile.
Generalized Likelihood Uncertainty Analysis (GLUE)
Dotty Plots and Identifiability Analysis
behavioural
GLUE computed 95%confidence limits
Animas Carson Cle Elum
rad_trncf
soil_moist_max
tmax_allsnow
Obj
ecti v
e Fu
ncti
on
o
bsQ
pre
dQ–
0.0 0.2 0.4 0.6 0.850000.0
100000.0
150000.0
200000.0
250000.0
300000.0
0.0 5.0 10.0 15.050000.0
100000.0
150000.0
200000.0
250000.0
300000.0
25.0 30.0 35.0 40.050000.0
100000.0
150000.0
200000.0
250000.0
300000.0
0.0 0.2 0.4 0.6 0.850000.0
100000.0
150000.0
200000.0
250000.0
300000.0
0.0 5.0 10.0 15.050000.0
100000.0
150000.0
200000.0
250000.0
300000.0
25.0 30.0 35.0 40.050000.0
100000.0
150000.0
200000.0
250000.0
300000.0
0.0 0.2 0.4 0.6 0.850000.0
100000.0
150000.0
200000.0
250000.0
300000.0
0.0 5.0 10.0 15.050000.0
100000.0
150000.0
200000.0
250000.0
300000.0
25.0 30.0 35.0 40.050000.0
100000.0
150000.0
200000.0
250000.0
300000.0
Figure 3
Uncalibrated Estimate
Parameter Equifinality
(deg F)
(inches)
Rockies Sierras Cascades
Regional Variability
Increasing the information content of the data
Multi-criteria Analysis
A single objective function:• cannot capture the many performance attributes
that an experienced hydrologist might look for• uses only a limited part of the total information
content of a hydrograph• when used in calibration it will tend to bias model
performance to match a particular aspect of the hydrograph
A multi-criteria approach overcomes these problems (Wheater et al., 1986, Gupta et al., 1998, Boyle et al., 2001, Wagener et al., 2001).
Identifying Characteristic Behavior
Developing Objective Measures
1 2
1 2
1 2
Qobs QcomnDnD
FD
FQ Qobs
Qobs
Qcom
Qcomnsns
nQnQ
FS
peaks/timing
baseflow
quick recession
Pareto Optimality
Paret
o Solu
tions
500 Pareto Solutions
Normalized Parameter Space
FS
FQ
FD FD FQ
FS
Function Space
SAC-SMA Hydrograph Range
Overall Performance Measures
RMSE min
BIAS min
Parameter Sensitivity by Objective Function
Dimensions of Model Evaluation
From Wagener 2003, Hydrological Processes
Evaluation of Model Evaluation of Model Component ProcessesComponent Processes
PET
Day
Annual Runoff
Percent Groundwater
Nash-Sutcliffe
Daily Q
0.7
0
.8 0
.9
Observed
SCE
Final SCE
1991 1993 1995 1997 Year
1991 1993 1995 1997 Year
1991 1993 1995 1997 Year
J F M A M J J A S O N DMonth
J F M A M J J A S O N DMonth
Solar Radiatio
n
Coupling SCA remote sensing products with point measures and modeled SWE to evaluate snow component process
East River
0
0.2
0.4
0.6
0.8
1
1/0 1/20 2/9 2/29 3/20 4/9 4/29 5/19 6/8 6/28 7/18
1996
Pe
rce
nt
Ba
sin
Sn
ow
co
ve
r
MODELSATELLITE
Integrating Remotely Sensed Data
Identifiability Analysis
Identification of the model structure and a corresponding parameter set that are most representative of the catchment under investigation, while considering aspects such as modeling objectives and available data.
Wagener et al., 2001, Hydrology Earth System Sciences
Dynamic Identifiability Analysis - DYNIA Information content by parameter
Dynamic Identifiability Analysis - DYNIA
Identifiability measure and 90% confidence limits
A Wavelet Analysis Strategy
• Daily time series
• Seasonally varying daily variance (row sums)
• Seasonally varying variance frequency decomposition (column sums)
• Annual average variance frequency decomposition
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 81
2
3
4
5
6
7
8
9
10
11
12Precipitation
John Schaake, NWS
Variance Decomposition
1
2
3
4
5
6
7
8
9
10
1 2 3 4 5 6 7 81
2
3
4
5
6
7
8
9
10
11
12
Component
Mon
th
OBSQ
1
2
3
4
5
6
7
8
9
10
1 2 3 4 5 6 7 81
2
3
4
5
6
7
8
9
10
11
12
Component
Mon
th
OBSQ2
Precipitation Streamflow Variance Transfer Functions
8 da
y w
i ndo
w64
day
win
dow
1-D
ay8-
Day
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 81
2
3
4
5
6
7
8
9
10
11
12Precipitation
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
1 2 3 4 5 6 7 81
2
3
4
5
6
7
8
9
10
11
12Variance Transfer Function - Observed
0
2
4
6
8
10
12
14
16
18
20
1 2 3 4 5 6 7 81
2
3
4
5
6
7
8
9
10
11
128-day Average Precipitation
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
1 2 3 4 5 6 7 81
2
3
4
5
6
7
8
9
10
11
128-day Avg Variance Transfer Function - Observed
Streamflow
0
1
2
3
4
5
6
7
8
9
10
1 2 3 4 5 6 7 81
2
3
4
5
6
7
8
9
10
11
12Estimated Streamflow - Linear
ObsLinear
PRMS SAC
0
1
2
3
4
5
6
7
8
9
10
1 2 3 4 5 6 7 81
2
3
4
5
6
7
8
9
10
11
12
Component
Mon
th
ESTQ H
0
1
2
3
4
5
6
7
8
9
10
1 2 3 4 5 6 7 81
2
3
4
5
6
7
8
9
10
11
12
Component
Mon
th
OBSQ
0
1
2
3
4
5
6
7
8
9
10
1 2 3 4 5 6 7 81
2
3
4
5
6
7
8
9
10
11
12
Component
Mon
th
ESTQ B
Variance Transfer Functions
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
1 2 3 4 5 6 7 81
2
3
4
5
6
7
8
9
10
11
12Variance Transfer Function - Linear
Obs Linear
PRMS SAC
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
1 2 3 4 5 6 7 81
2
3
4
5
6
7
8
9
10
11
12
Component
Mon
thOBSH
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
1 2 3 4 5 6 7 81
2
3
4
5
6
7
8
9
10
11
12
Component
Mon
th
ESTH H
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
1 2 3 4 5 6 7 81
2
3
4
5
6
7
8
9
10
11
12
Component
Mon
th
ESTH B
Forecast Uncertainty
Ensemble Streamflow Prediction
Using history as an analog for the future
Simulate to today
Predict future using historic data
Probability of exceedence
NOAA
USGS
BOR
ESP - Animas River @ Durango
010002000
30004000500060007000
80009000
10000
4/3
/20
05
4/1
7/2
00
5
5/1
/20
05
5/1
5/2
00
5
5/2
9/2
00
5
6/1
2/2
00
5
6/2
6/2
00
5
7/1
0/2
00
5
7/2
4/2
00
5
8/7
/20
05
8/2
1/2
00
5
9/4
/20
05
9/1
8/2
00
5
Str
ea
mfl
ow
(c
fsd
)1982
1983
1987
1991
1992
1993
1994
1995
1997
1998
2002
2003
2004
2005
ESP Animas River @ Durango
0
2000
4000
6000
8000
10000
120004/
3/20
05
4/17
/200
5
5/1/
2005
5/15
/200
5
5/29
/200
5
6/12
/200
5
6/26
/200
5
7/10
/200
5
7/24
/200
5
8/7/
2005
8/21
/200
5
9/4/
2005
9/18
/200
5
Str
eam
flo
w (
cfsd
)
1981
1982
1983
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2005 ESP Forecast
Forecast Period 4/3 – 9/30
Made 4/2/2005
All historic years
Only el nino years
Observed 2005
Ranked Probability Skill Score (RPSS) for each forecast day and month using measured runoff and
simulated runoff (Animas River, CO) produced using: (1) SDS output and (2) ESP technique
For
ecas
t D
ay
Month MonthJ F M A M J J A S O N D J F M A M J J A S O N D
8
6
4
2
0
8
6
4
2
0
0.1 0.3 0.5 0.7 0.9
RPSSRPSS
ESPSDS
Perfect Forecast: RPSS=1
Given current uncertainty in long-term atmospheric-model forecasts, seasonal to annual forecasts may be better with ESP
• This presentation has been a selected review of uncertainty and error analysis techniques.
• No single approach provides all the information needed to assess a model. The appropriate mix is a function of model structure, problem objectives, data constraints, and spatial and temporal scales of application.
• Still searching for the unified theory of uncertainty analysis.
Summary
• Input-output behaviour of the model is consistent with measured behaviour -performance
• Model predictions are accurate (negligible bias) and precise (prediction uncertainty relatively small)
• Model structure and behaviour are consistent with the understanding of reality
Necessary Conditions for a Model to be Considered Properly
Calibrated
Gupta, H.V., et al, in review
http://www.es.lancs.ac.uk/hfdg/uncertainty_workshop/uncert_intro.htm
National and international groups are collaborating to assess existing methods and tools for uncertainty analysis and to explore potential avenues for improvement in this area.
A Federal Interagency
Working Group is developing a Calibration,
Optimization, and Sensitivity and
Uncertainty Analysis Toolbox
International Workshop Proceedings describes this effort: available at http://www.iscmem.org
Aquatic, Riparian& Terrestrial
GISLanduse
GeochemicalFlowpaths
Coupled Hydrological Modelling Systems
HydrologicalModelling
INPUTS
ModelComplexity
ScaleUncertainty
Analysis
PREDICTIONS
Increased Model Complexity More Parameters
More Spatial Interactions
More Complex Responses
but still data limited ….
MORE MODELLING UNCERTAINTY
Future Model Development and Application
ImprovedRepresentations of
Hydrological Processorsand Predictions
ModelStructures
Visualisations ofModels &
Uncertainty
FieldMeasurements
Visualisations of Measurements
0
1
2
3
4
5
BedrockDepth (m)
C onceptualised Bedrock
D eeper ValleyZoneModular Modelling
System - USGS
Visual Uncertainty Analysis Framework
Freer, et al., Lancaster Univ., UK