Hydrologic Modeling: Verification, Validation, Calibration, and

Sensitivity Analysis

Fritz R. Fiedler, P.E., Ph.D.

Definitions (review)

• Verification: check if code solves equations correctly

• Validation: check if model reasonably represents physical process

• Calibration: adjust model parameters to match observations

• Sensitivity Analysis: relative effect of parameter changes on output

Verification

• Compare numerical results to analytical results

Level 1 Validation

• Compare model results to simple experiments (can estimate parameters a priori)

Calibration

• Adjust parameters to match observations

Level 2 Validation

• Compare model results to observations for a different input data set post-calibration– Reserve some data (do not use in calibration)– After finding parameters that result in “best fit,”

run model with reserved input and compare to output

• Problems with this?• What happens in practice?

Sensitivity Analysis

• Explore how parameter changes affect output

• Sensitivity index:

measureeperformancofvalueZ

iparameterofvaluex

x

dxdZ

S

i

i

ii

Calibration TargetsModel Parameters Observations

Channel Models n, S0, h-B relationship

Q, h

Watershed Models

watershed and channel

Q

Richards Equation

K(), h() head, moisture

Green-Ampt K, , flux, moisture (?)

Can physically based model parameters be measured? Why or why not?

Goodness of Fit

• Visual comparison between simulated and observed – look for trends in errors– A learned art– Use appropriate graph scales

• Statistical performance measures– Consider mean daily discharge as calibration

target– Q = observed– S = simulated

Means and Bias

N

SS

N

ii

1

N

QQ

N

ii

1

)100(

1

1

N

ii

N

iii

Q

QSbiaspercent

Common calibration strategy: fix bias first, revisit periodically, goal of no bias

• Maximum Error:

• Percent Average Absolute Error

NtoiforQSME ii 1)max(

)100(

1

1

Q

QSN

PAAE

N

iii

Sum of Squares of Errors

• Most common basis for statistical goodness of fit– e.g., least squares regression, seek to minimize

N

iii QS

1

2)(

Root Mean Squared Error

• Size of error usually related to size of events or values, thus RMSE typically smaller for dry periods, small watersheds (for example)

• How would you modify RMSE to facilitate comparison?

2/1

1

2)(

N

QSRMSE

N

iii

Percent RMSE

• Normalize RMSE by mean observed

• Because the magnitude of RMSE varies with magnitude of values, by minimizing RMSE only, which part of hydrographs are primarily best fit in calibration?

• How can this tendency be addressed?

)100(Q

RMSEPRMSE

Nash-Sutcliffe

• Very popular method of evaluating calibration

• Reading: McCuen, R. H., Evaluation of the Nash—Sutcliffe efficiency index, Journal of Hydrologic Engineering, 11(6), 597-602, 2006 (note: author uses different variables)

N

ii

N

iii

NS

QQ

QSR

1

2

1

2

2

)(

)(1

Line of Best Fit

ABSQ

Analyze as in regression: hypothesis testing on A and B, residual analysis, correlation coefficient…

Line of Best Fit – Correlation Coefficient

2/12222

iiii

iiii

QQNSSN

QSQSNR

How to Use Statistical Measures

• For a given time period, e.g., 1 year, and/or averages over multiple years

• Look for seasonal trends

Month Q S PB Bias ME PAAE RMSE PRMSE Jan Feb March … Annual

How to Use Statistical Measures

• By flow interval (value interval)

• Errors as f(Q) – aim for no systematic variation• How would you pick the intervals?

Interval N Q S Bias ME RMSE PRMSE 0-20 20-40 40-60 …

Exceedance Plots

Q, S

0 100percent days exceeded

x

x

x

x

x

x

xxx

x

Generalized Calibration Strategies

• Set realistic parameter bounds before starting• Fix insensitive parameters first; focus on most

sensitive• Eliminate most bias early in process, revisit• Use regionalized variables as appropriate• Combine manual and automatic techniques

Equifinality

• Multiple combinations of parameters can lead to similar results

• Issue with both multi-parameter lumped models (e.g., SAC-SMA) and spatially distributed models (e.g., CASC-2D)

• Reading: Ebel, B. A. and K. Loague, Physics-based hydrologic-response simulation: Seeing through the fog of equifinality, Hydrological Processes, 20(13), 2887–2900, 2006