Uncertainty analysis

and

Model Validation

Final Project

Summary of Results &

Conclusions

Calibration Targets

calibration value

associated error

20.24 m

0.80 m

Target with relativelylarge associated error.

Target with smaller associated error.

In a real-world problem we need to establish model specific calibration criteria and define targets including associated error.

Smith Creek Valley (Thomas et al., 1989)

Calibration Objectives

1. Heads within 10 ft of measured heads. Allows forMeasurement error and interpolation error.

2. Absolute mean residual between measured and simulated heads close to zero (0.22 ft) and standard deviation minimal (4.5 ft).

3. Head difference between layers 1&2 within 2 ft of field values.

4. Distribution of ET and ET rates match field estimates.

Also need to identify calibration parameters and theirreasonable ranges.

Calibration   Prediction

Group ARM h ARM ET (x10e7)

  ARM h (at targets)

ARM h(at pumping wells)

1 0.92 1.38   1.60 4.16

2 0.73 1.11   1.99 3.03

3 0.69 0.51   0.95 1.76

4 1.34 1.27   1.46 2.57

5  1.56

 0.89

   2.79

 1.43

6  1.29

 0.16

   2.58

 2.92

Calibration to Fluxes

When recharge rate (R) is a calibration parameter, calibrating to fluxes can help in estimating K and/or R.

H1H2

q = KI

In this example, flux information helps calibrate K.

In this example, discharge information helps calibrate R.

All water discharges to the playa.Calibration to ET merely fine tunesthe discharge rates within the playaarea.

1 2 3 4 5 6 7 8 9 10 11

16

15

14

13

12

11

10

9

8

7

6

5

4

3

2

1

In our example, total recharge is known/assumed to be 7.14E08 ft3/year and discharge = recharge.

Calibration   Prediction

Group ARM h ARM ET (x10e7)

  ARM h (at targets)

ARM h(at pumping wells)

1 0.92 1.38   1.60 4.16

2 0.73 1.11   1.99 3.03

3 0.69 0.51   0.95 1.76

4 1.34 1.27   1.46 2.57

5  1.56

 0.89

   2.79

 1.43

6  1.29

 0.16

   2.58

 2.92

00.5

1

1.52

2.53

3.54

4.5

0 0.5 1 1.5 2

Calibrated ARM

Pre

dic

ted

AR

M-t

arg

ets

Includes results from 2000, 2001, 2003

0

2

4

6

8

10

12

0 0.5 1 1.5 2

Calibrated ARM

Pre

dic

ted

AR

M-p

um

pin

g w

ells

Includes results from 2000, 2001, 2003

Group P1 P2 P3 P4 P5 P6 P7

1 2320 PW1 3970 PW2 2310 playa 1920 PW4

1500 PW4 4810 PW2 684 PW2

2 74,000 PW2

39,000 PW2 393,000playa 3.93E6 PW4

252 PW4 1084 playa

1576 playa

3 1.21E6 PW1

2.15E6 PW2

3.90E6 playa 1110 PW4

1.58E6 playa

1860 playa

893 playa

4 1200 PW1 1900 PW2 6.7E6 PW3 290 PW4

2800 PW5

760 PW1 1200 PW2

5 1295 PW1 3160 PW2 503 playa 986 PW4

605 PW4 316 PW1 3100 PW2

6  3100 PW1

 982 PW1

 4.9E5 playa  603 PW4

 1450 PW4

 2000 PW1

 1380 PW2

Truth 802 PW1 1913 playa

620 playa 310 PW4

1933 PW5

690 playa

2009 PW2

Particle Tracking

Predicted ARM > Calibrated ARM

Predicted ARM at pumping wells > Predicted ARM at nodes with targets

Flow predictions are more robust (consistent among different calibrated models) than transport (particle tracking) predictions.

Observations

Conclusions

• Calibrations are non-unique.

• A good calibration (even if ARM = 0) does not ensure that the model will make good predictions.

• Need for an uncertainty analysis to accompany calibration results and predictions.

• You can never have enough field data.

• Modelers need to maintain a healthy skepticism about their results.

Uncertainty in the Calibration

Involves uncertainty in:

Parameter values

Conceptual model including boundary conditions,zonation, geometry, etc.

Targets

Ways to analyze uncertaintyin the calibration

Sensitivity analysis

Use an inverse model (automated calibration) to quantify uncertainties and optimize the calibration.

Uncertainty in the Prediction

Involves uncertainty in how parameter values(e.g., recharge) will vary in the future.

Reflects uncertainty in the calibration.

Stochastic simulation

Ways to quantify uncertaintyin the prediction

Sensitivity analysis

MADE site – Feehley and Zheng, 2000, WRR 36(9).

A Monte Carlo analysis considers 100 or more realizations.

0

20

40

60

80

100

120

140

1 2 3 4 5 6 7

Drawdown at pumping well

nu

mb

er o

f re

aliz

atio

ns

Stochastic modeling option in GW Vistas

Stochastic simulation

Ways to quantify uncertaintyin the prediction

Sensitivity analysis

Scenario analysis

How do we “validate” a model so thatwe have confidence that it will makeaccurate predictions?

Modeling Chronology

1960’s Flow models are great!

1970’s Contaminant transport models are great!

1975 What about uncertainty of flow models?

1980s Contaminant transport models don’t work. (because of failure to account for heterogeneity)

1990s Are models reliable? Concerns overreliability in predictions arose over efforts to modela geologic repository for high level radioactive waste.

“The objective of model validation is to determine how well the mathematical representation of the processes describes the actual system behavior in terms of the degree of correlation between model calculations and actual measured data”(NRC, 1990)

Oreskes et al. (1994): paper in Science

Calibration = forced empirical adequacy

Verification = assertion of truth (possible in a closed system, e.g., testing of codes)

Validation = establishment of legitimacy (does not contain obvious errors), confirmation, confidence building

What constitutes validation? (code vs. model)

NRC study (1990): Model validation is not possible.

How to build confidence in a model

Calibration (history matching) steady-state calibration(s) transient calibration

“Verification” requires an independent set of field data

Post-Audit: requires waiting for prediction to occur

Models as interactive management tools

HAPPY MODELING!