VIII: Methods for Evaluating Model Predictions

1. Define predictive quantity and calculate sensitivities and standard deviations (Ex8.1a)

2. Assess data needs for the predictions Which parameters are important to predictions?

Use composite and prediction scaled sensitivities (css and pss) and parameter correlation coefficients (pcc) (Ex8.1b)Use parameter-prediction (ppr) statistic (Ex8.1c)

What existing observations are important to predictions?

Use observation-prediction (opr) statistic (Ex8.1d)

How important are the proposed new observations?Use dss, css, pcc (Ex8.1e)Use opr (Ex8.1f)

3. Quantify prediction uncertaintyLinear confidence and prediction intervalsNonlinear confidence and prediction intervalsMonte Carlo – no exercise

Exercise filesMFI2005 does not support predictionsAll the files have been constructed for you, and are in initial\ex8.i

Copy the files to exer\ex8Execute batch files as described in the input instructions.Look at output files

For other problems, you can use the class files as a beginning point.

Define predictive quantityWe typically make predictions to assess the state of the simulated system at a future time.

Often, predictions occur under different conditions than those under which the model was calibrated – addition of pumping, change in boundary conditions, etc.

The predictive quantity might differ from the types of quantities used as observations.

You as the modeler need to help define what model-calculated values are best to use as predictions in the problem of concern

The predictions available can depend on the software used

MODFLOW-2005: any quantity that can be an observation can also be a prediction. Heads, temporal changes in head, some flows, advective transport.

UCODE_2005 or PEST: any quantity that can be calculated using values in the application model output files.

Prediction sensitivities (Book, p. 159)

Used to calculate measures for assessing:Parameters important to predictionsObservations important to predictionsPrediction uncertaintyData needs

Prediction sensitivities:

Usually need to scale these sensitivities to produce useful measures.

jb

z

Prediction scaled sensitivities (pss) (Book, p. 160-161)

Scaling depends on prediction and purpose. Four useful scalings are:1. pss is percent change in prediction caused by a 1-percent change in the parameter value bj (in _sppp file of UCODE_2005):

pssj = (z/ bj) (bj/100) (100/z)

2. pss is percent change in prediction caused by a change in bj equal to 1 percent of its standard deviation sbj (in _spsp file):

pssj = (z/ bj) (sbj/100) (100/z)

3,4. For both these scalings, (100/z) can be replaced by

(100/a), where a’ is some other meaningful quantity. On p.

161, using a reference value is suggested. (in _sppr and _spsr file of UCODE_2005)

In _sp** , third letter identifies pparameter value or sstandard deviation on the parameter; fourth letter identifies pprediction or rreference value)

Parameter Correlation Coefficients (pcc) in the Context of Predictions (Book, p. 162-166)

Calculate pcc using different combinations of observations, prior, and predictions to evaluate the existence and need for unique parameter estimates

For example, two parameters are extremely correlated given a set of observations. This is only a problem if the predictions need unique values.

Next: Use pss, css, and pcc together to assess the importance of parameters to the predictions.

Use pss and css to identify insensitivity problems

Are imprecise parameter estimates important to the predictions?

AcceptableAcceptableNot important: Small pss

Importance of the parameter to predictions of interest

Precise: Large cssImprecise: Small css

Acceptable – consider representing system feature(s)

with more parameters

Improve estimation of this parameter and representation

of system feature(s)

Important: Large pss

Precision of parameter estimate

AcceptableAcceptableNot important: Small pss

Importance of the parameter to predictions of interest

Precise: Large cssImprecise: Small css

Acceptable – consider representing system feature(s)

with more parameters

Improve estimation of this parameter and representation

of system feature(s)

Important: Large pss

Precision of parameter estimate

(from Hill & Tiedeman, Figure 8.2a, p. 166)

Use pcc to identify uniqueness problems

Are non-unique parameter estimates important to the predictions?

AcceptableAcceptableNot important: |pcc| with preds ~ 1

Importance of unique parameter estimates to predictions of interest

Unique: |pcc| < ~0.95Nonunique: |pcc| ~ 1

Acceptable – consider representing system feature(s)

with more parameters

Improve estimation of one or both parameters and

improve representation of system feature(s)

Important: |pcc| with preds < ~0.95

Uniqueness of the estimates for a parameter pair

AcceptableAcceptableNot important: |pcc| with preds ~ 1

Importance of unique parameter estimates to predictions of interest

Unique: |pcc| < ~0.95Nonunique: |pcc| ~ 1

Acceptable – consider representing system feature(s)

with more parameters

Improve estimation of one or both parameters and

improve representation of system feature(s)

Important: |pcc| with preds < ~0.95

Uniqueness of the estimates for a parameter pair

(from Hill & Tiedeman, Figure 8.2b, p. 166)

• Developer claims landfill effluent will go to river, not to wells.

• Water supply wells are being completed and a pump test could be used to collect more data on the system.

• Developer claims the model is inadequate for two reasons: 1. Model was calibrated using heads and flows and no

pumping, but is being used to predict advective transport under pumping conditions.

2. Need for prior suggests the observations are inadequate

• County government officials want to know:• Is existing model adequate? • Wait for new data?

• We, as the modelers, suggest using sensitivity analysis to:• Evaluate the developer’s claims. • Plan new data collection

Prediction Exercises – Book, p. 193-212

Prediction Exercises

Will effluent go to well or river?

Thorough analysis requires full transport model (advection, dispersion, etc.).

We will do preliminary analysis using Advective Transport. Use MODPATH.

?

?

Predicted transport from landfill

Predictions simulated using MODPATH (Pollock 1996)

Calculates motion in the x, y and z directions.When used for observations:

Observed advective transport inferred from concentrations.Three entries are added to the objective function for each advective-transport observation.

When used for predictions:Three predictions for each advective transport prediction – transport along rows (y), columns (x), and vertically (z).

Calculates sensitivities; these are used here to calculate pcc, parameter standard deviations, and linear confidence intervals.

Start of pathParticle path simulated using MODPATH.

The horizontal and vertical bars show one standard deviation at selected travel times.

End of path

Predictive transport: Questions to address (p. 194)

1. Transport destination and time? Ex. 8.1a. Forward model with MODPATH

2. Consider the parameters important to predictions. Are they precisely and uniquely estimated?

Prediction and composite scaled sensitivities and parameter correlation coefficients (pss, css, pcc). Ex. 8.1b. PPR statistic. Ex. 8.1c

3. What existing observations are important? OPR statistic. Ex. 8.1d

4. Are potential new observations worth waiting for?Dimensionless and prediction scaled sensitivities, parameter correlation coefficients (dss, pss, pcc). Ex. 8.1eOPR statistic. Ex. 8.1f

5. What is the prediction uncertainty? Linear confidence intervals. Ex. 8.2a. Nonlinear confidence intervals. Ex 8.2b.

Exercise 8.1a: Predict Advective Transport

Read general Exercises description on p. 193-195.Do Exercise 8.1a on p. 195-196

In the simulation, where does the landfill effluent go, and how long does it take to get there?

MODPATH Output (Equivalent to ADV output on Fig. 8.7a, p. 196

@ [ MODPATH Version 4.00 (V4, Release 3, 7-2003) (TREF= 0.000000E+00 ) ] 1 1.55000E+04 1.65000E+04 9.99900E-01 9.99950E+01 0.00000E+00 16 2 1 1 1 1.51565E+04 1.63903E+04 7.87957E-01 8.93978E+01 -3.15000E+08 16 2 1 1 1 1.51565E+04 1.63903E+04 7.87957E-01 8.93978E+01 3.15000E+08 16 2 1 1 1 1.50000E+04 1.63422E+04 7.10433E-01 8.55216E+01 4.46763E+08 15 2 1 1 1 1.40917E+04 1.60000E+04 4.02401E-01 7.01200E+01 1.11285E+09 15 3 1 1 1 1.40000E+04 1.59693E+04 3.82440E-01 6.91220E+01 1.16709E+09 14 3 1 1 1 1.32700E+04 1.56553E+04 2.56172E-01 6.28086E+01 -1.57000E+09 14 3 1 1 1 1.32700E+04 1.56553E+04 2.56172E-01 6.28086E+01 1.57000E+09 14 3 1 1 1 1.30000E+04 1.55379E+04 2.19734E-01 6.09867E+01 1.70786E+09 13 3 1 1 1 1.20897E+04 1.50000E+04 1.26071E-01 5.63035E+01 2.14586E+09 13 4 1 1 1 1.20000E+04 1.49508E+04 1.19582E-01 5.59791E+01 2.18187E+09 12 4 1 1 1 1.10000E+04 1.41688E+04 5.97149E-02 5.29857E+01 2.58234E+09 11 4 1 1 1 1.08476E+04 1.40000E+04 5.20098E-02 5.26005E+01 2.64385E+09 11 5 1 1 1 1.00382E+04 1.30000E+04 1.70220E-02 5.08511E+01 2.94240E+09 11 6 1 1 1 1.00000E+04 1.29501E+04 1.52036E-02 5.07602E+01 2.95493E+09 10 6 1 1 1 9.76378E+03 1.25238E+04 0.00000E+00 5.00000E+01 3.03896E+09 10 6 1 1 1 9.76378E+03 1.25238E+04 -3.23767E-01 4.67623E+01 -3.15000E+09 10 6 1 1 1 9.76378E+03 1.25238E+04 -3.23767E-01 4.67623E+01 3.15000E+09 10 6 1 1 1 9.76378E+03 1.25238E+04 1.00000E+00 4.00000E+01 3.38193E+09 10 6 2 1 1 9.56571E+03 1.20000E+04 9.22715E-01 3.61357E+01 3.83712E+09 10 7 2 1 1 9.38455E+03 1.10000E+04 6.69144E-01 2.34572E+01 4.32990E+09 10 8 2 1 1 9.38417E+03 1.00000E+04 3.71525E-01 8.57624E+00 4.49415E+09 10 9 2 1

Distance along rows from left

Distance along columns from bottom

Level above

bottom of model

Row, column, layer

Level above

bottom of model layer

Time, in

seconds

Exercise 8.1b: Determine problematic parameters using css and pss

1. Compare prediction and composite scaled sensitivities: Are pss large for any parameters with small css?

• Use pss scaling so that pss represent the percent change in distance traveled caused by a 1-percent change in a parameter value:

For UCODE_2005, these pss are in ex8.1b \ ex8.1b_sppp.

pssj = (z/ bj) (bj/100) |100/z| One-percent scaled sens

Exercise 8.1b: Determine problematic parameters using pcc

2. Compare two different sets of parameter correlation coefficients:

• pcc calculated using only calibration observations• pcc calculated with calibration observations AND

predictions.

• Use parameter correlation coeffeicients in ex8.1b \ ex8.1b._mc or _pcc

ex8.1b-obs-pred\

Weights for the advective travel predictions are calculated from specified standard deviations, in meters (Table 8.3, p. 198). Weights in this analysis can reflect desired prediction precision.

Time of advective travel

Direction 10 years 50 years 100 years

X 200 m 600 m 1000 m

Y 200 m 600 m 1000 m

Z 10 m 15 m 25 m

Exercise 8.1b: Prediction weights needed for

pcc with predictions

Do Exercise 8.1b (p. 196-199) and the Problems.

Question 2: Are parameters important to predictions

precisely and uniquely estimated?

Results of Exercise 8.1b

css-pss analysis for Question 2: Are parameters important to predictions precisely

estimated?

0

1

2

HK_1 K_RB VK_CB HK_2 RCH_1 RCH_2 POR_1&2 POR_CB

Parameter Name

0

10

20

30

40

Com

posi

te s

cale

d se

nsit

ivit

y ( css

)AD10x AD10y AD10z

AD50x AD50y AD50zA100x A100y A100z

css

Abs

olut

e va

lue

of p

redi

ctio

nsc

aled

sen

siti

vity

( pss

)

Figure 8.8, p. 198

Results of Exercise 8.1b

css-pss analysis for Question 2: Are parameters important to predictions precisely

estimated?

0

1

2

HK_1 K_RB VK_CB HK_2 RCH_1 RCH_2 POR_1&2 POR_CB

Parameter Name

0

10

20

30

40

Com

posi

te s

cale

d se

nsit

ivit

y ( css

)AD10x AD10y AD10z

AD50x AD50y AD50zA100x A100y A100z

css

Abs

olut

e va

lue

of p

redi

ctio

nsc

aled

sen

siti

vity

( pss

)

Figure 8.8, p. 198

Not all of them

HK_1 K_RB VK_CB HK_2 RCH_1 RCH_2

HK_1 1.00 -0.40 -0.90 -0.93 0.96 -0.90

K_RB 1.00 0.20 0.34 -0.32 0.32

VK_CB 1.00 0.97 -0.97 0.97

HK_2 symmetric 1.00 -0.99 0.996

RCH_1 1.00 -0.98

RCH_2 1.00

HK_1 K_RB VK_CB HK_2 RCH_1 RCH_2

HK_1 1.00 -0.16 0.078 -0.22 0.71 0.26

K_RB 1.00 -0.61 0.013 0.25 -0.070

VK_CB 1.00 0.33 -0.19 0.28

HK_2 symmetric 1.00 -0.52 0.83

RCH_1 1.00 -0.30

RCH_2 1.00

Observations only

With predictions

pcc analysis

for Question

2: Are parameter

s important

to predictions uniquely

estimated

?

Results of Exercise 8.1b

Tables 8.4 & 8.5,p. 199

No correlations larger than 0.85

Not all of them

Exercise 8.1c: PPR Individual Parameters

Which parameters rank as most important to the predictions by the ppr statistic?0

2

4

6

8

10

HK_1 K_RB VK_CB HK_2 RCH_1 RCH_2 POR_1&2

Parameter Name

Ave

rage

ppr

sta

tist

ic

(a) Average ppr statistic for all predictionsFigure 8.9a, p. 201

0

1

2

HK_1 K_RB VK_CB HK_2 RCH_1 RCH_2 POR_1&2 POR_CB

Parameter Name

0

10

20

30

40

Com

posi

te s

cale

d se

nsit

ivit

y ( css

)AD10x AD10y AD10z

AD50x AD50y AD50zA100x A100y A100z

css

Abs

olut

e va

lue

of p

redi

ctio

nsc

aled

sen

siti

vity

( pss

)

How does this differ from the rankings by the pss?

What causes these differences?

Response to developer concerns

Developer claims the model is inadequate for two reasons:

Model was calibrated using heads and flows and no pumping, but is being used to predict advective transport under pumping conditions.Need for prior suggests the observations are inadequate

Our response based on sensitivity analysisIndeed, the model is inadequate. The calibration data does not provide information on some parameters important to predictions

Also, we stress that we need to consider the results of an uncertainty analysis