bayesian approach for uncertainty analysis of an

Bayesian Approach for Uncertainty Analysis of an Urban Storm Water Model and Its Application to

a Heavily Urbanized Watershed

Misgana K Muleta MASCE1 Jonathan McMillan 2 Geremew G Amenu3 and Steven J Burian MASCE 4

Abstract The significance of uncertainty analysis (UA) to quantify reliability of model simulations is being recognized Consequently literature on parameter and predictive uncertainty assessment of water resources models has been rising Applications dealing with urban drainage systems are however very limited This study applies formal Bayesian approach for uncertainty analysis of a widely used storm water management model and illustrates the methodology using a highly urbanized watershed in the Los Angeles Basin California A flexible likelihood function that accommodates heteroscedasticity non-normality and temporal correlation of model residuals has been used for the study along with a Markov-chain Monte Carlo-based sampling scheme The solution of the UA model has been compared with the solution of the conventional calibration methodology widely practiced in water resources modeling Results indicate that the maximum likelihood solshyution determined using the UA model produced runoff simulations that are of comparable accuracy with the solution of the traditional calibration method while also accurately characterizing structure of the model residuals The UA model also successfully determined both parameter uncertainty and total predictive uncertainty for the watershed Contribution of parameter uncertainty to total predictive uncertainty was found insignificant for the study watershed underlying the importance of other sources of uncertainty including data and model strucshyture Overall the UA methodology proved promising for sensitivity analysis calibration parameter uncertainty and total predictive uncershytainty analysis of urban storm water management models DOI 101061(ASCE)HE1943-55840000705 copy 2013 American Society of Civil Engineers

CE Database subject headings Uncertainty principles Calibration Bayesian analysis Stormwater Management Parameters Markov process Monte Carlo method Urban areas

Author keywords Uncertainty analysis Calibration Bayesian approach Storm water management Parameter uncertainty Predictive uncertainty Markov-chain Monte Carlo

Introduction

Primarily by increasing imperviousness urbanization alters natural hydrology of a watershed and negatively impacts ecology geomorshyphology water quality and socioeconomic functions of the receivshying waters (National Research Council (NRC) 2008) Structural and nonstructural methods generally referred to as storm water control measures (SCMs) are often used to mitigate these impacts To improve effectiveness of the SCMs watershed-scale design soshylutions are advocated as opposed to the conventional approach of selecting and designing SCMs site-by-site (EPA 2007 NRC 2008) Watershed-scale design requires understanding hydrologic and water quality characteristics of individual SCMs as well as the

1Dept of Civil and Environmental Engineering California Polytechnic State Univ San Luis Obispo CA 93407 (corresponding author) E-mail mmuletacalpolyedu

2Dept of Civil and Environmental Engineering California Polytechnic State Univ San Luis Obispo CA 93407

3Dept of Civil Engineering and Construction Engineering Manageshyment California State Univ Long Beach CA 90840

4Dept of Civil and Environmental Engineering Univ of Utah Salt Lake City UT 84112

interaction between SCMs of various types sizes and relative locations in a watershed consequently making the design process more challenging

Computer models could be used for effective design of SCMs at watershed-scale Models can simulate responses of the watershed and the SCMs considering the factors relevant to the generation and routing of runoff and contaminants Models however must be properly calibrated before their use for planning and management of water resources The traditional calibration method seeks to identify an optimal set of parameters that forces model simulations closer to the observed counterparts The basis of this calibration approach is the assumption that the inputs used to build the model the observations used to evaluate goodness of model simulations and structure of the model that describes physics of the watershed are all error free Recent contributions to the water resources literature have seriously questioned the continued usefulness of this classic calibration method (Beven and Freer 2001 Muleta and Nicklow 2005 Kavetski et al 2006 Vrugt et al 2008 Montanari et al 2009) It is acknowledged that hydrologic predictions are plagued with uncertainties arising from errors associated with forcshyings (inputs) observations parameters and model structural inadshyequacies Consequently a prudent evaluation technique is to recognize these uncertainties and to quantify predictive uncertainty and parameter posteriors (Vrugt et al 2005 Moradkhani and Sorooshian 2008 Gotzinger and Bardossy 2008 Montanari et al 2009)

During the past two decades the Generalized Likelihood Uncershytainty Estimation (GLUE) technique of Beven and Binley (1992)

and Beven (2006) has found widespread application for unshycertainty analysis in water resources This informal Bayesian apshyproach is simple to implement but has been criticized for being statistically incoherent (Mantovan and Todini 2006 Stedinger et al 2008 Vrugt et al 2009b) In response to this various authors have proposed formal uncertainty analysis (UA) methods that use proper statistics and employ valid likelihood measures (Kuczera and Parent 1998 Thiemann et al 2001 Vrugt et al 2003 Schoups and Vrugt 2010) These techniques attempt to provide estimates of the probability density function (PDF) of the model parameters as well as total predictive uncertainty eg through Monte Carlo simulations For computational efficiency reason Markov-chain Monte Carlo (MCMC) schemes are preferred to classic Monte Carlo simulations that rely on random sampling

In addition to efficient and robust sampling schemes successful UA entails appropriate formulation of the likelihood function The formal UA applications often reported in the literature make unrealistic assumptions regarding the structure of the residuals (errors) between model simulations and the observed watershed response Common assumptions include that residuals are (1) temshyporally independent (ie no correlation between errors of successhysive time steps) (2) normally distributed and (3) homoscedastic (ie variance of the residuals does not depend on magnitude) Addressing these unrealistic assumptions a flexible and general formal likelihood (GL) function has been recently described by Schoups and Vrugt (2010)

The objective of this study is to examine effectiveness of a formal Bayesian approach for uncertainty analysis and calibration of the US EPA Storm Water Management Model (SWMM5) (Rossman 2010) The GL function and a recently developed effishycient MCMC sampling scheme known as DREAMethZSTHORN (Schoups and Vrugt 2010) has been used to identify parameter posteriors and to estimate runoff prediction uncertainty The methodology is illustrated using the Ballona Creek watershed a heavily urbanshyized watershed located in the Los Angeles Basin California Effecshytiveness of the UA method in removing heteroscedasticity and temporal correlation and in identifying representative PDF for the residuals has been scrutinized To examine robustness of the UA method for identifying the optimal solutions typically sought by classic calibration approaches the UA solution [ie the maxishymum likelihood (ML) parameter set and the associated runoff preshydictions] has been compared with the solution determined by an automated calibration algorithm known as dynamically dimenshysioned search (DDS) (Tolson and Shoemaker 2007)

Most UA studies in water resources that applied MCMC techshynique within the Bayesian framework used lumped conceptual models for rainfall-runoff analysis of rural watersheds (Kuczera et al 2006 Vrugt et al 2009a Schoups and Vrugt 2010) Few studshyies have been reported in spatially distributed modeling (Feyen et al 2008) Applications to urban watersheds are very limited Ball (2009) underlined the need for UA-based approaches for evaluation of urban drainage models in the discussion of the conventional calibration effort reported on Ballona Creek watershed by Barco et al (2008) Freni et al (2008 2009a) applied GLUE to an urban drainage model and tested sensitivity of the solutions to likelihood measures (Freni et al 2009a) and acceptability thresholds (Freni et al 2008) In a separate study Freni et al (2009b) compared performance of Bayesian Monte Carlo method to that of GLUE Mannina and Viviani (2010) applied GLUE for UA of storm water quality using a conceptual urban drainage model developed inshyhouse All these applications of UA to urban drainage models used GLUE an informal approach whose statistical validity has been questioned (Mantovan and Todini 2006 Stedinger et al 2008 Vrugt et al 2008)

Working under the Joint Committee on Urban Drainage estabshylished by the International Water Association and the International Association for Hydro-Environment Engineering and Research (IWAIAHR) the International Working Group on Data and Models has recently published findings of its effort to develop a framework for defining and assessing uncertainties in urban drainshyage models (Dotto et al 2012 Deletic et al 2012) The article by Deletic et al (2012) underscored the need for consistent use of terminologies and methods for UA of urban drainage models The authors defined various sources of uncertainties presented the linkages between the different uncertainty sources and proshyposed a framework for UA of urban storm water models The article by Dotto et al (2012) compared four different UA methods (three non-Bayesian and one Bayesian) in terms of the posterior PDFs and prediction intervals determined by the methods and their relative computational efficiencies They showed that the non-Bayesian methods required subjective decisions that affected the UA results whereas the Bayesian method used erroneous assumption regardshying structure of the residuals

This study is the first to the best knowledge of the authors to apply an MCMC scheme that works within formal Bayesian frameshywork for UA of urban watersheds and to apply Bayesian approach to UA of SWMM Besides demonstrating application of stateshyof-the-art in UA to urban storm water modeling the ensuing model will be used for an ongoing study that attempts to develop a simulation-optimization model for watershed-scale design of SCMs for urban watersheds

Methods and Materials

Uncertainty Analysis and the MCMC Algorithm

The watershed response (eg runoff) simulated by a storm water management model f such as SWMM5 can be described as

Y frac14 fethI θTHORN eth1THORN ^where Y frac14 n times 1 vector representing the runoff time series

(y1 yn) I = matrix of model forcings (eg precipitation) and θ signifies a d-dimensional vector of model parameters To test how well f describes runoff from a watershed the common practice

^is to compare the model predictions Yn frac14 fy1 yng with the corresponding observations Yn frac14 fy1 yng The difference between the two time series can be represented by a residual vector EnethθTHORN

^EnethθTHORN frac14 Y minus Y frac14 fy1 minus y1 yn minus yng frac14 fe1ethθTHORN enethθTHORNg

eth2THORN The traditional model calibration technique searches for a

single optimal combination of parameter values that minimizes this residual time series With the recognition that model simulations are plagued by many sources of uncertainty validity of this convenshytional parameter estimation method has been questioned A plaushysible alternative is to account for the various sources of uncertainty and to determine posterior PDF of the parameters for example using the Bayesian approach

From Bayes theorem the posterior PDF of the parameters pethθjIYnTHORN) can be given as

pethθjIYnTHORN prop LethθjYn ITHORNpethθTHORN eth3THORN

where LethθjYn ITHORN denotes the likelihood function that measures how well the model fits the data and pethθTHORN = prior distribution

of the model parameters Different likelihood functions have been proposed depending on the assumptions made about the statistical properties of the residual vector EnethθTHORN If the residuals are assumed to be temporally uncorrelated and normally distributed with zero mean and a homoscedastic error standard deviation σe the likeshylihood function takes on the well-known simple least-square (SLS) form (Box and Tiao 1992) as

P n 21 ifrac141 eiethθTHORNLethθjYn ITHORN prop exp minus eth4THORN 2 σe

2

Limitations of the SLS assumptions for hydrologic models have been documented by several authors (Sorooshian and Dracup 1980 Kuczera 1983 Thyer et al 2009 Schoups and Vrugt 2010) and different proposals have been suggested to relax the assumptions One of the latest recommendations is the formal likelihood function proposed by Schoups and Vrugt (2010) who described a general error model that embraces temporal correlation heteroscedasticity and non-Gaussian nature of the model residuals

The generalized log-likelihood function of Schoups and Vrugt (2010) can be written as

n n X X2σξωβLethθ φjY ITHORN frac14 n log minus log σt minus cβ jaξtj2=eth1thornβTHORN ξ thorn ξminus1

tfrac141 tfrac141

eth5THORN

where φ signifies parameters of the error model Temporal correshylation between the residuals is accounted for using a pth order autoregressive polynomial [ϕpethBTHORN] as

p X ϕpethBTHORNet frac14 σtat where ϕpethBTHORN frac14 1 minus ϕiBi

ifrac141

and Biet frac14 etminusi eth6THORN

where σt = standard deviation at time t and to account for heteroshyscedasticity it is assumed to increase linearly with the streamflow yt as

σt frac14 σ0 thorn σ1yt eth7THORN

where σ0 and σ1 are inferred from the data along with model parameters (θ) Finally at represents an independent and identishycally distributed random error with zero mean and a unit standard deviation whose probability is described by a skew exponential power (SEP) density with skewness (ξ) and kurtosis (β) parameters

pethatjξ βTHORN frac14 ξ thorn2σ

ξξ minus1 ωβ expfminuscβjaξtj2=eth1thornβTHORNg eth8THORN

aξt frac14 ξminussignethμξ thornσξ at THORNethμξ thorn σξatTHORN eth9THORN

where μξ σξ cβ and ωβ are computed as a function of ξ and β as described by Schoups and Vrugt (2010)

Besides the likelihood function a sampling scheme that effishyciently identifies posterior PDFs is crucial for effective application of Bayesian-based UAs Markov-chain Monte Carlo (MCMC) schemes are often used for this application and improving efficiency of MCMC schemes has been one of the focuses of UA research the past few years In this regard Laloy and Vrugt (2012) developed DREAMethZSTHORN an MCMC algorithm that capitalized on the strength of the DiffeRential Evolution Adaptive Metropolis

(DREAM) (Vrugt et al 2008) Effectiveness and efficiency of DREAMethZSTHORN for posterior sampling has been reported in several studies Schoups and Vrugt (2010) applied the GL function and DREAMethZSTHORN for rainfall-runoff analysis of two watersheds by using a lumped conceptual model This study examines DREAMethZSTHORN and GL for UA of SWMM5 using the Ballona Creek watershed which is one of the most urbanized watersheds in the world with approxshyimately 83 developed (Bay et al 2003) Schoups and Vrugt (2010) provides further description of DREAMethZSTHORN

Single-Objective Calibration

Single-objective automated calibration was performed primarily to compare solutions of the conventional model calibration techshynique to those identified by GL and DREAMethZSTHORN The dynamically dimensioned search (DDS) (Tolson and Shoemaker 2007) was used to identify optimal values of SWMM5 runoff parameters DDS has been developed to improve efficiency of calibrating computation-ally demanding models DDS is a simple single-objective heurisshytic search method that starts by globally searching the feasible region and incrementally localizes the search space as the number of simulations approaches the maximum allowable number of simshyulations (the only stopping criteria used by the algorithm) Progress from global to local search is achieved by probabilistically reducing the number of model parameters modified from their best value obtained thus far New potential solutions are created by perturbing the current parameter values of the randomly selected model parameters only The best solution identified thus far is maintained and is updated only when a solution with superior value of the objective function is found

DDS requires minimal algorithmic parameter tweaking because the only parameters to set are the maximum number of model evalshyuations and the scalar neighborhood size perturbation parameter (r) that defines the random perturbation size standard deviation as a fraction of the decision variable range The recommended value of 02 (Tolson and Shoemaker 2007) has been used for r in this study Efficiency and effectiveness of DDS has been reported by Tolson and Shoemaker (2007) and Muleta (2010) who comshypared its performance to that of widely used optimization methods including the Shuffled Complex Evolution-University of Arizona (SCE-UA) (Duan et al 1992) and the Genetic Algorithms (Holland 1975) For this study DDS has been integrated with SWMM5 to calibrate runoff for the study watershed

EPA Storm Water Management Model

SWMM was first developed in 1971 and it continues to be widely used throughout the world for planning analysis and design of storm water runoff combined sewers sanitary sewers and other drainage systems (Rossman 2010) SWMM5 the latest version of SWMM simulates hydrology hydraulics and water quality of urbanized and nonurbanized watersheds The hydrologic processes modeled include precipitation (rainfall or snow fall) evaporation surface runoff infiltration groundwater flow and snowpacks and snowmelt Both single event and continuous simulations can be performed accounting for spatial and temporal variability in the climate soil land use and topography in the watershed Surface runoff is estimated using the nonlinear reservoir method in which surface runoff occurs only when the depth of the overland flow exceeds the maximum surface storage provided by initial abstracshytions including depression storage and interception in which case the runoff rate is estimated using Manningrsquos equation Horton (1937) Green and Ampt (1911) and the Curve Number methods

(Soil Conservation Service 1964) are available to model infiltration losses

Runoff quality including buildup and washoff of pollutants can be simulated by using various approaches from both developed and nondeveloped land uses The runoff quantity and quality simulated from a subwatershed and the wastewater loads (if any) assigned to the receiving nodes are added and then transported by using either steady kinematic wave or dynamic wave routing through a conshyveyance system of pipes channels storagetreatment devices pumps and hydraulic regulators such as weirs orifices and other outlet types Hydraulic conditions of any level of complexity inshycluding those experiencing backwater effect flow reversal and pressurized flow can be accommodated In addition the capability to model the commonly used low impact developments (LIDs) including porous pavements bioretention cells infiltration trenches vegetative swales and rain barrels has been recently added to SWMM5 For this study source code of SWMM5 has been integrated with the UA and the single-objection calibration methods previously described

Application Watershed and Data

The Ballona Creek watershed (Fig 1) is used to illustrate the methshyods described in this study Total drainage area of the Ballona Creek watershed is approximately 337 km2 For this study the upper 230 km2 of the Ballona Creek watershed (ie the portion that drains to the streamflow gauging station used in this study) has been modeled Approximately 90 of the modeled watershed is

developed and its land-use distribution consists of 60 residential 10 commercial 35 industrial and 11 open space (Amenu 2011) The open spaces are in the Santa Monica Mountains located in the northern part of the watershed The drainage system is charshyacterized by extensive networks of storm drains that collect storm water from the watershed and convey it to the Ballona Creek a 145 km (9-mi)-long flood protection channel that discharges to the Santa Monica Bay [Los Angeles County Department of Public Works (LACDPW) 2011] The watershed has been identified as the major source of non-point-source pollution to the Santa Monica Bay (Stenstrom and Strecker 1993 Stein and Tiefenthaler 2005)

The data needed to build SWMM5 have been collected from various sources A digital elevation model land-use map and an imperviousness map were obtained from the USGS seamless data warehouse (USGS 2013) and soil survey geographic (SSURGO) soil map has been obtained from the Natural Resources Conservashytion Service (NRCS) soil data mart (NRCS 2013) Because of the difficulty to accurately delineate urban subwatersheds from digital elevation models alone (Gironaacutes et al 2010) the subwatershed delineation obtained from the LACDPW were used for this study The LACDPW delineation (which was created through a compreshyhensive hydrologic study based on USGS topo quads as built drawings and field surveys) divided the watershed into 134 sub-watersheds For this study the number of subwatersheds was furshyther reduced to 92 by merging smaller subwatersheds (area less than 041 km2) to the adjoining subwatershed Subcatchment inforshymation such as area slope and flow length were extracted from the digital elevation model The soil land-use and imperviousness

Fig 1 Location map of the Ballona Creek Watershed

maps were superimposed onto the subwatersheds to extract SWMM5 runoff parameters including percent imperviousness infiltration parameters and Manningrsquos roughness coefficient

Rainfall data at three gauges (Fig 1) and streamflow data for a monitoring station that drains approximately 70 of the Ballona Creek watershed were obtained from the LACDPW for 15 years (ie 1996ndash2010) at 15-min intervals Both rainfall and streamflow data were collected using automatic gauges that are equipped with real-time data telemetry and electronic data loggers (Amenu 2011) Proximity and altitude criteria were used to define the rain gauge that represents each subcatchment The climate of the watershed can be characterized as semiarid with average annual rainfall of approximately 380 mm and temperature of approximately 18degC Rainfall season for the region spans from October to April The elevation of the watershed varies from 750 m above mean seal level (AMSL) at the Santa Monica Mountains to 0 m AMSL at its disshycharge to the Santa Monica Bay

The watershed consists of an extensive network of storm drains (underground pipes and open channels) designed for flood protecshytion purposes With the assumption that overland flow from each of the 92 subwatersheds directly flows to a storm drain inlet located at the outlet of the subwatershed only 72 larger storm drains were considered in this model In reality each subwatershed may contain numerous streets swales and minor storm drains that can play significant roles in routing of runoff and contaminants within the subcatchment Neglecting the minor drainage systems and modshyeling only the major drainage systems as done in this study can have an effect on the hydraulics of the drainage system For examshyple the travel time could get longer as the faster conduit flows are replaced with the slower overland flows under this assumption The approach used in this study is however commonly used to simplify modeling complexity and to reduce the cost associated with data collection (Gironaacutes et al 2010) The study by Burian et al (2000) was used for storm drain information including shape size slope and length

Methodology

Simulation Durations and Analysis Methods

For both the DDS and GL-DREAMethZSTHORN methods rainfall and streamflow data from January 17 2010 to January 23 2010 were used for calibration Verification of the solutions was performed using data from January 23 2008 to January 28 2008 The event used for calibration had rainfall depth of approximately 124 cm in 7 days and the verification event produced 126 cm of rainfall in 6 days According to National Oceanic and Atmospheric Administration (NOAA) Atlas 14 (NOAA 2013) for a station in Ballona Creek mean rainfall depth for a 2-year 7-day event is approximately 121 cm which is comparable to the calibration verification storm events used for this study Both streamflow and rainfall data are available at 15-min intervals The curve number method was selected for infiltration modeling as the CN values (primary parameter for the curve number method) can be detershymined more readily compared to Horton or Green-Ampt parameshyters from the land cover and soil maps available for the watershed Because the GL-DREAMethZSTHORN algorithm used for the UA requires running SWMM5 repetitively (up to hundreds of thousands of runs) to converge computational time is a significant concern As such kinematic wave routing was selected to reduce computashytional burden of the dynamic wave routing option

Likewise continuous (long-term) simulation was not considshyered because of computational concern Short-term simulation

(ie duration of 7 days for calibration and 6 days for verification) were used for both single-objection calibration and uncertainty analysis Most studies that reported on calibration of urban drainshyage models used single-event simulations with a typical duration of a day or less (Barco et al 2008 Fang and Ball 2007) The simushylation durations considered in this study for both calibration and verification cases are therefore significant improvements comshypared to single-event simulations Although single-event and the short-duration simulation pursued in this study may suffice for cershytain applications such as flood control continuous (eg multiple year duration) simulation models are more appropriate for applicashytions that are sensitive to long-term watershed characteristics (eg contaminant buildup and washoff processes)

Parameters

A total of 11 SWMM5 runoff parameters were considered for calibration and uncertainty analysis Values of the parameters vary from subwatershed to subwatershed depending on soil land use imperviousness topography andor other characteristics of the subwatershed Initial values of the parameters have been extracted for each subwatershed from the soil land-use imperviousness and topography maps using geographic information system (GIS) During both calibration and uncertainty analysis these initial (baseshyline) values were altered by multiplying the parameters by the respective adjustments proposed by the calibration and the UA algorithm This way the initial values would be scaled up or down while preserving the heterogeneity determined from watershed characteristics The parameters were assumed to follow uniform distribution as done in Muleta and Nicklow (2005) and lower and upper percentage adjustment bounds were assigned based on litershyature and engineering judgment (Rossman 2010 Barco et al 2008) A list of the parameters and the assumed adjustment ranges are given in Table 1

Objective Function and Efficiency Criteria

The streamflow measured at the Swatelle station shown in Fig 1 was used for calibration and uncertainty analysis Mean absolute error (MAE) Eq (10) was used as objective function for the single-objection calibration MAE was selected as the objective function based on the findings of Muleta (2012) which compared relative effectiveness of the efficiency criteria commonly used in hydrologic modeling to describe goodness of model performances According to the study efficiency criteria such as MAE that describe the absolute deviation between observations and model simulations were found to be robust Goodness of the calibration result was further assessed using additional efficiency criteria including the Nash-Sutcliffe efficiency (NSE) criterion (Nash and Sutcliffe 1970) described in Eq (11) percent bias (PBIAS) [Eq (12)] and total volume of runoff

N X1MAE frac14 jYi minus Oij eth10THORN

N ifrac141

PN ethYi minus OiTHORN2 ifrac141NSE frac14 1 minus PN THORN2 eth11THORN

ifrac141 ethOi minus Omean

PN ifrac141ethOi minus YiTHORNPBIAS frac14 100 P eth12THORNN Oiifrac141

where Y = model simulated output O = observed runoff Omean = mean of the observations which the NSE uses as a benchmark

Table 1 Model Parameters and Ranges Used for Calibration and Uncertainty Analysis

Name Description Minimum Maximum

SWMM parameters Percent adjustment Width Subcatchment width (m) minus90 150 Slope Subcatchment slope () minus20 20 Imperv Percentage of impervious minus15 15

area () N-Imperv Manning n for impervious minus50 100

area N-Perv Manning n for pervious area minus50 100 Dstore-Imperv Depression storage for minus95 100

impervious area (mm) Dstore-Perv Depression storage for minus95 300

pervious area (mm) Zero-Imperv Percent of the impervious minus100 100

area with no depression storage ()

Curve number NRCS runoff curve number minus20 20 Drying Time Time for a fully saturated soil minus95 100

to completely dry (days) Conduit n Manningrsquos roughness minus50 100

coefficient for conduit

Error parameters Parameter values σ0 Heteroscedasticity intercept 0 100

(m3=s) σ1 Heteroscedasticity slope 0 1 ϕ1 Lag one autocorrelation minus1 1

coefficient ϕ2 Lag two autocorrelation minus1 1

coefficient ϕ3 Lag three autocorrelation minus1 1

coefficient β Kurtosis parameter minus1 1

against which performance of the model simulations is evaluated and N = number of data points (observations)

Parameter Posteriors and Predictive Uncertainty

Posterior distributions of the 11 runoff parameters were estimated with DREAMethZSTHORN and the GL function Six additional error model parameters were considered for the GL These include σ0 and σ1 in Eq (7) to explicitly account for heteroscedasticity of model residshyuals the kurtosis parameter β in Eq (5) to account for non-normality of the residuals and ϕ1 ϕ2 and ϕ3 in Eq (8) to allow for autocorrelation of the residuals It was initially assumed that the residual distribution is not skewed setting skewness parameter ξ equal to 1 and then this assumption was checked a posteriori A total of 100000 SWMM5 simulations were used to sample the posterior distribution of the parameters Convergence of GL-DREAMethZSTHORN to a stable posterior PDF was monitored using the R statistic of Gelman and Rubin (1992) Convergence is declared when Rj le 12 for all j frac14 1 d where d is the number of model parameters being analyzed (ie 17 in this study) The last 5000 GL-DREAMethZSTHORN runs that meet the convergence criteria were exshytracted and parameter posteriors were determined and reported for each individual parameter

Once the posterior distribution of the model parameters is known runoff predictive uncertainty can be estimated by propashygating the different samples of the posterior distribution through the SWMM5 model and reporting the respective prediction uncershytainty ranges (eg 95 confidence interval) This prediction interval however represents parameter uncertainty only it doesnrsquot consider other sources of error including model structure forcing

data and calibration data uncertainty Total predictive uncertainty was calculated using the methodology described in Schoups and Vrugt (2010) based on the error model parameters determined by GL-DREAMethZSTHORN The ML parameter values determined by GL-DREAMethZSTHORN are benchmarked against the calibration results obtained using DDS and are also compared in terms of their ability to fit different parts of the hydrograph

Results and Discussion


Results of the single-objection calibration are summarized in Fig 2 and Tables 2 and 3 Fig 2 compares the streamflow simulated using the optimal parameter values identified by DDS with the stream-flow observed at Swatelle station for both calibration [Figs 2(andashd)] and verification [Figs 2(endashh)] periods Performance of the calishybrated model was also tested using the efficiency criteria given in Table 2 The graphical comparison and the efficiency criteria indicate that the single-objection calibration performed very well for both calibration and verification periods However as shown in Fig 2 a closer look into the characteristics of the residuals (ie the difference between the observed streamflow and the streamflow simulated by the calibrated model) depicts that the asshysumptions (ie homoscedasticity Gaussian distribution and temshyporal independence) made by the objective functions almost always used in model calibrations including the MAE used for this study are unjustified The residuals exhibited heteroscedasticity (ie they increase with the magnitude of streamflow) as shown in Fig 2(b) they do not follow Gaussian distribution [Fig 2(c)] and they are temporally correlated [Fig 2(d)] for both calibration and verificashytion periods Similar findings have been reported by other studies including those by Schoups and Vrugt (2010) and regarding charshyacteristics of the residuals generated from solutions of the tradishytional calibration methods

In addition to relying on unrealistic assumptions on residuals the conventional calibration models attempt to identify a single best solution based on the assumption that data model structure and model parameters are all error-free It is now a common knowledge that input and output data are subjected to substantial uncertainty no model structure is a true representation of the watershed being studied and optimal parameter sets are not unique for a given watershed (ie multiple set of parameters can be equally good for the watershed) As such these calibration methods provide no inshyformation on reliability of the optimal solution The uncertainty analysis model described in this study has been developed to address these vital limitations

Parameter Uncertainty

As previously described 100000 SWMM5 simulations were run for the GL-DREAMethZSTHORN UA model Fig 3 shows progress of the R statistic of Gelman and Rubin (1992) that has been used to test convergence of the UA runs The plot indicates that the 100000 model runs used for the analysis were sufficient to meet the convergence criteria of R le 12 for all the SWMM5 and error model parameters considered for the study Fig 4 shows the posshyterior histograms obtained for the parameters using the last 3000 simulations As shown in Fig 3 the convergence criteria was met after approximately 62000 model simulations implying that the last 3000 simulations used to generate the posterior histograms have met the convergence criteria and each of these parameter sets represents a reasonable SWMM5 model for the watershed

500

400 (a)

300

200

100

0 0

100 200 300 400 500 600 Time Count [15 minute interval]

Res

idua

ls (

a)

Str

eam

flow

[m3 s

]R

esid

uals

(a)

S

trea

mflo

w [m

3 s

]

40 1 1

05

Par

tial a

utoc

orre

latio

nP

artia

l aut

ocor

rela

tion

(b) (c) (d)0820

Den

sity

D

ensi

ty 06

04 0

0

-05

-20 02

-40 0 0 200 400 -2 0 2 0 10 20 30 40 50

Simulated flow [m3s] Residuals (a) Lag [15 minute interval]

400 (e)

300

200

100

0 0 50 100 150 200 250 300 350 400 450 500 550

Time Count [15 minute interval]

50 1 1 (h)(f) (g)

0825 05

0

-05

06

04 0

-25 02

-50 0 0 100 200 300 -2 0 2 0 10 20 30 40 50


Fig 2 Comparison of observed and simulated streamflow and diagnostic plots of the residuals obtained using DDS for both calibration (andashd) and verification (endashh) periods (a and e) comparison between observed (dots) and simulated (solid line) streamflow (b and f) illustration of heterosceshydasticity of the residuals (c and g) comparison of observed PDF of the residuals to normal distribution (d and h) illustration of autocorrelation of the residuals dashed lines in (d) and (h) show lower and upper bounds of the 95 confidence interval

Fig 4(a) offers important information regarding the relative importance of the SWMM5 parameters considered for the analysis Except for percentage imperviousness (Imperv) depression storage for impervious subareas (Dstore-Imperv) and percentage of the imshypervious subarea with no depression storage ( Zero Imperv) the uncertainty bound of the other SWMM5 parameters is very wide Nonetheless the streamflow simulated by the last 3000 parameter

Table 2 Efficiency Criteria Values Obtained Using DDS and GL-DREAMethZSTHORN

Method Period MAE (m3=s) NSE

Bias ()

Volume (mm)

Observed Simulated

DDS

GL-DREAMethZSTHORN

C V C V

689 979 774 1057

094 1115 090 minus1882 093 1094 088 minus2040

596 836 596 836

662 689 661 676

sets did not show significant variability as shown in Fig 5 This suggests that the SWMM5 parameters that exhibited wide uncertainty range do not have substantial effect on rainfall-runoff characteristics of the Ballona Creek watershed This has practical implication for example in terms of prioritizing resources on data collection Availability of more accurate data that characterize the insensitive parameters may not help in improving accuracy of runshyoff simulation for the watershed On the contrary uncertainty of the model predictions can be further reduced from having more reliable imperviousness data Unlike the SWMM5 parameters the error model parameters given in Fig 4(b) produced narrow uncertainty range indicating their identifiablity

Tables 2 and 3 compare the solution of the GL-DREAMethZSTHORN model with the single-objection calibration results Performance of the maximum likelihood solution has been summarized using several efficiency criteria in Table 2 The results show that the ML solution performed very well but not as well as the DDS sol-

Note C= calibration V= verification ution This is not surprising because the objective function used by

Table 3 Percentage Adjustments Obtained by DDS the ML and the 95 Confidence Interval Obtained by GL-DREAMethZSTHORN and Optimal Parameter Values Determined for One of the Subcatchments in the Ballona Creek Watershed

Percentage adjustments Parameter valuesa

GL-DREAMethZSTHORN Lower Upper Initial Optimal

Parameter DDS ML bound bound value value

Width 1371 1170 minus126 1263 460 998 Slope 27 minus74 minus401 995 179 166 Imperv minus149 minus150 minus150 minus127 70 59 N-Imperv 321 793 58 995 001 002 N-Perv minus428 386 minus465 990 002 003 Dstore-Imperv 998 982 849 999 025 050 Dstore-Perv 2347 18 722 2936 127 129 Zero-Imperv minus999 minus1000 minus1000 minus945 100 0 Curve Number minus32 84 minus175 07 69 75 Drying Time minus69 minus931 minus680 897 4 028 Conduit n 310 280 340 960 0013 0016 aInitial values refer to the actual initial parameter values assigned to one of the subcatchments and the optimal values are determined from the ML percent adjustments and the initial parameter values See Table 1 for units

22

18

R S

tati

stic

This shows that the insensitive parameters have minimal impact on runoff from the watershed and thus on the likelihood function The ML adjustments recommended for these insensitive parameters are meaningless as any adjustment within the wide range of the posshyterior PDFs will produce almost identical runoff and likelihood function value

One interesting observation is the recommendation by both ML and DDS solutions to decrease percent imperviousness of the watershed by approximately 15 so that model simulations closely match observed runoff Given that only the largest 72 storm drains were considered in the study by ignoring hundreds of smaller storm drains and channels in the watershed one would intuitively expect a solution that speeds up travel time (eg increase in percent imperviousness and steeper slope) to compensate for impact of the ignored storm drains on travel time The suggested decrease in percentage imperviousness is believed to be related to the assumption made regarding connectivity of the impervious and pervious subareas in each subwatershed Both subarea types were assumed to directly flow to the outlet of each subwatershed whereas in reality only a fraction of the impervious subareas may be directly connected to the engineered drainage system This assumption might have led to overestimation of the effective pershycentage imperviousness of the watershed in the model

Predictive Uncertainty

Understanding the total predictive uncertainty associated with model simulations is very essential for decision makers Modeling uncertainties are believed to arise from imprecise knowledge of the temporal and spatial variability of input and observed system response from the assumptions and simplifications made in the simulation model to represent physical processes in the watershed and owing to the parameter uncertainty described in the previous section In the past parameter uncertainty has been assumed to represent all uncertainty sources (Beven and Freer 2001 Muleta and Nicklow 2005) However as shown in Fig 5 parameter uncertainty for the study watershed is very minimal indicating that at least for the Ballona Creek watershed parameter uncertainty alone cannot represent the total predictive uncertainty associated

0 50000 100000

16

14

12

1

Simulation

Fig 3 Convergence plot of the Gelman and Rubin R statistic for the parameters analyzed using GL-DREAMethZSTHORN dashed line shows convershygence threshold of R frac14 12

the two methods is different Although DDS attempts to minimize the MAE the GL-DREAMethZSTHORN model uses the GL function as obshyjective function to remove heteroscedasticity and autocorrelation from the residuals while also attempting to minimize the residuals The ML parameter percentage adjustments the upper and lower bounds of the 95 confidence interval extracted from the posterishyors and the optimal adjustments identified by the DDS are given in Table 3 The initial parameter values assigned to one of the subwatersheds in the study area and the optimal parameter values calculated from the initial values and the ML percent adjustments are also given in Table 3

The optimal parameter adjustments identified by the ML and the DDS for corresponding SWMM 5 parameters are substantially different except for the results of Imperv Dstore-Imperv Zero Imperv and Conduit n (ie Manningrsquos n for conduits) This further confirms insensitivity of runoff from the watershed to the majority of SWMM5 parameters considered in the analysis The 95 confidence interval is very wide for the insensitive model parameters For some of the insensitive parameters (ie Dstore-Perv Curve Number Drying Time and Conduit N) the ML solushytions are outside the 95 confidence interval as shown in Table 3

with simulation models Similar findings have been reported by previous studies including Kuczera et al (2006) using different hydrologic models and application watersheds

Fig 6(a) shows a 95 confidence interval for the total predicshytive uncertainty generated for the calibration [Figs 6(andashd)] and verification [Figs 6(endashh)] periods The predictive uncertainty has been determined using the last 3000 parameter sets identified by the GL-DREAMethZSTHORN model as previously described Fig 6 inshydicates substantial uncertainty for the watershed especially for low flow simulations Wider uncertainty bound for low flows might have been obtained because the likelihood function used in the UA (ie the GL function) is biased towards peak flows that seem to have been simulated with a higher degree of reliability (ie narshyrow bounds) Overall the total predictive uncertainty bounds seem reasonably accurate because they bracketed more than 75 of the observed data albeit lower than the theoretically expected value of 95 for both calibration and verification periods This indicates that the MCM-based formal Bayesian methodology pursued in this study is promising for UA of urban drainage models

Fig 6 also shows diagnostic plots of the residuals derived from the ML solution Fig 6(b) shows that the residuals are not sensitive to the magnitude of streamflow indicating that heteroscedasticity has been removed by the GL function Fig 6(c) clearly shows the Laplace (ie double-exponential) distribution used by the error model is consistent with the PDF of the residuals of the ML

1200 2500 1000 1200

0 0 0 0 -20 Width 150 -100 Slope 125 -15 Imperv -12 -20 120

N-Imperv

-75 100 80 100 0 320 -100 -94Dstore-Imperv Dstore-Perv Zero Imperv

N-Perv

-20 CN 10 -100 Drying Time 100 30 Mannings n 100 (a)

0 1 0117 012 096 1 s0 x 10-6 s1 β

0845 085 -0105 -01 0185 0205φ2φ φ3(b) 1

0

2000

0

1500

0

1000

0

1500

0

800

0

1200

0

1500

0

600

1200

0

900

1800

0

750

1500

0

1000

2000

0

750

1500

0

750

Fig 4 Posterior PDFs determined using GL-DREAMethZSTHORN for (a) SWMM5 runoff parameters (b) GL error model parameters ordinate of the plots show frequency of occurrence

solution Temporal dependence of the residuals is shown in Fig 6(d) which indicates that the residuals still exhibit substantial dependence at lag-one and lag-two autocorrelations However the temporal correlation has been significantly reduced compared to the DDS solution shown in Fig 2(d) Given the short simulation time interval (ie 15-min) used for the study which is typical in urban drainage modeling the difficulty of removing temporal correlation in its entirety is understandable Generally the diagshynostic plots demonstrate that the assumptions made by the GL-DREAMethZSTHORN model regarding the characteristics of the residshyuals are consistent with properties of the residuals derived from the ML solution

Conclusions

This paper describes an MCMC-based formal Bayesian methodshyology for parameter uncertainty and total predictive uncertainty analysis of a widely used urban storm water management model The methodology has been illustrated using the Ballona Creek watershed a heavily urbanized watershed located in the Los Angeles Basin California Solution of the UA model has been compared with the optimal solution typically derived by using the traditional calibration methods widely used in water reshysources modeling Furthermore validity of the assumptions comshymonly made with regard to characteristics of model residuals in the

objective functions often used for model calibration has been examined Flexibility of the likelihood function used in the UA model to accommodate the characteristics of the residuals has been

Calibration Period Verification Period demonstrated The subsequent paragraphs summarize major con-

Stre

amfl

ow [

m3

s]

Rai

nfal

l [m

m] clusions of the study

The runoff simulated using the optimal solution identified by the single-objection calibration attempt was in good agreement with the observed counterparts when evaluated graphically and by using several goodness-of-fit measures However diagnostic analysis of the residuals indicates that the assumptions of homoscedasticity temporal independence and Gaussian distribution commonly made in such traditional calibration models are unjustified On the other hand the maximum likelihood solutions determined using the UA model produced runoff simulations that are of comparable accuracy

Time Count [15 Minute Interval] with that of the single-objection calibration solutions while accushyrately characterizing the structure of the model residuals The

Fig 5 Comparison of observed runoff (dotted) to the 95 confidence assumptions made by the error model used in the UA methodology interval bounds (lines) determined using GL-DREAMethZSTHORN considering were found consistent with the characteristics of the residuals only parameter uncertainty for calibration and verification periods

generated from the ML solution

600 (a)

400

200

0

0 -200

100 200 300 400 500 600


Str

eam

flow

[m3 s

]R

esid

uals

(a)

S

trea

mflo

w [m

3 s

]

40 1 1

05

Par

tial a

utoc

orre

latio

n

(b) (c) (d)0820

Den

sity 06

04 0

0

-05

-20 02

-40 0 0 100 200 300 400 -2 0 2 0 10 20 30 40 50

Residuals (a) Lag [15 minute interval]Simulated flow [m3s]

600 (e)

400

200

0

-200 0 50 100 150 200 250 300 350 400 450 500 550


20 1 1

05

Par

tial a

utoc

orre

latio

n

(f) (g) (h) 10

Res

idua

ls (

a)

Den

sity

050

0

-05

-10

-20 0 0 100 200 300 -2 0 2 0 10 20 30 40 50


Fig 6 Total predictive uncertainty (95 confidence interval) and diagnostic plots of the residuals obtained using GL-DREAMethZSTHORN for calibration (andashd) and verification (endashh) periods (a and e) comparison between observed streamflow (dots) to the 95 confidence bounds (solid line) (b and f) show that the residuals are homoscedastic (c and g) comparison of observed PDF (dotted) of the residuals to the assumed distribution (solid line) in GL (d and h) show autocorrelation of the residuals dashed lines in (d) and (h) show lower and upper bounds of the 95 confidence interval

In addition to accurately simulating runoff and properly characshyterizing the residuals the UA model has successfully determined parameter uncertainty and total predictive uncertainty The paramshyeter posteriors showed that eight of the 11 SWMM5 parameters considered for the analysis exhibited wide uncertainty bound whereas the runoff simulated for the watershed considering paramshyeter uncertainty alone showed no appreciable variability This sugshygests that runoff is sensitive only to three (ie Imperv Dstore Imperv Zero Imperv) of the 11 parameters Additionally the ML solution identified by the UA model and the optimal solution determined by DDS showed good agreement only for four (Imperv Dstore Imperv Zero Imperv Conduit n) of the 11 SWMM5 parameters confirming nonidentifiablity of the insensitive parameshyters Results also suggest that contribution of parameter uncertainty to total predictive uncertainty is insignificant for the study watershyshed underlying the importance of the other sources of predictive uncertainty for Ballona Creek watershed

The 95 confidence interval determined for total predictive uncertainty using the UA model bracketed the majority of the observed data demonstrating reasonable accuracy of the UA result Satisfactory total predictive uncertainty bounds were genshyerated for both calibration and verification periods although the verification period results seem less adequate Overall the UA methodology proved promising for sensitivity analysis calibrashytion parameter uncertainty and total predictive uncertainty analysis of urban storm water drainage models at least for the short-simulation durations considered in this study Applications to additional watersheds in other hydroclimatic regions can help further examine this potential The subsequent study will investishygate application to continuous simulations and on ways to use the predictive uncertainty in decision making such as for optimal SCM design applications

References

Amenu G G (2011) ldquoA comparative study of water quality conditions between heavily urbanized and less urbanized watersheds of Los Angeles Basinrdquo Proc World Environmental and Water Resources Congress 2011 Bearing Knowledge for Sustainability ASCE Reston VA 680ndash690

Ball J E (2009) ldquoDiscussion of lsquoAutomatic calibration of the US EPA SWMM model for a large urban catchmentrsquo by J Barco K M Wong and M K Stenstromrdquo J Hydraul Eng 135(12) 1108ndash1110

Barco J Wong K M and Stenstrom M K (2008) ldquoAutomatic calibrashytion of the US EPA SWMM model for a large urban catchmentrdquo J Hydraul Eng 134(4) 466ndash474

Bay S Jones B H Schiff K and Washburn L (2003) ldquoWater quality impacts of stormwater discharges to Santa Monica Bayrdquo Marine Envishyron Res 56(1ndash2) 205ndash223

Beven K (2006) ldquoA manifesto for the equifinality thesisrdquo J Hydrol 320(1ndash2) 28ndash36

Beven K and Freer J (2001) ldquoEquifinality data assimilation and uncershytainty estimation in mechanistic modeling of complex environmental systems using the GLUE methodologyrdquo J Hydrol 249(1ndash4) 11ndash29

Beven K J and Binley A M (1992) ldquoThe future of distributed models Model calibration and uncertainty predictionrdquo Hydrol Processes 6(3) 279ndash298

Beven K J and Freer J (2001) ldquoEquifinality data assimilation and unshycertainty estimation in mechanistic modeling of complex environmental systems using the GLUE methodologyrdquo J Hydrol 249(1ndash4) 11ndash29

Box G E P and Tiao G C (1992) Bayesian inference in statistical analysis Wiley New York

Burian S J McPherson T N Brown M J and Turin H J (2000) ldquoDevelopment of a stormwater model for the Ballona Creek watershedrdquo Los Alamos Unclassified Rep (LA-UR-00-1849) Los Alamos National

Laboratory Presented at 1st Ballona Wetlands Symp Los Angeles CA

Deletic A et al (2012) ldquoAssessing uncertainties in urban drainage modelsrdquo Phys Chem Earth Parts ABC 42ndash44(2012) 3ndash10

Dotto C B S et al (2012) ldquoComparison of different uncertainty techniques in urban stormwater quantity and quality modellingrdquo Water Res 46(8) 2545ndash2558

Duan Q Sorooshian S and Gupta V K (1992) ldquoEffective and efficient global optimization for conceptual rainfall-runoff modelsrdquo Water Resour Res 28(4) 1015ndash1031

EPA (2007) ldquoNational water quality inventory Report to congress 2002 reporting cyclerdquo Document No EPA-841-R-07-001 Washington DC

Fang T and Ball J E (2007) ldquoEvaluation of spatially variable parameters in a complex system An application of a genetic algorithmrdquo J Hydro-inform 9(3) 163ndash173

Feyen L Khalas M and Vrugt J A (2008) ldquoSemi-distributed parameter optimization and uncertainty assessment for large-scale streamflow simshyulation using global optimizationrdquo Hydrol Sci J 53(2) 293ndash308

Freni G Mannina G and Viviani G (2008) ldquoUncertainty in urban stormwater quality modelling The effect of acceptability threshold in the GLUE methodologyrdquo Water Res 42(8ndash9) 2061ndash2072

Freni G Mannina G and Viviani G (2009a) ldquoUncertainty in urban stormwater quality modelling The influence of likelihood measure formulation in the GLUE methodologyrdquo Sci Total Environ 408(1) 138ndash145

Freni G Mannina G and Viviani G (2009b) ldquoUrban runoff modelling uncertainty Comparison among Bayesian and pseudo-Bayesian methshyodsrdquo Environ Modell Software 24(9) 1100ndash1111

Gelman A and Rubin D B (1992) ldquoInference from iterative simulation using multiple sequencesrdquo Stat Sci 7(4) 457ndash472

Gironaacutes J Niemann J D Roesner L A Rodriguez F and Andrieu H (2010) ldquoEvaluation of methods for representing urban terrain in storm-water modelingrdquo J Hydrol Eng 15(1) 1ndash14

Gotzinger J and Bardossy A (2008) ldquoGeneric error model for calibration and uncertainty estimation of hydrological modelsrdquo Water Resour Res 44(12) W00B07

Green W H and Ampt G (1911) ldquoStudies of soil physics Part 1 The flow of air and water through soilsrdquo J Agr Soc 4 1ndash24

Holland J H (1975) Adaptation in natural and artificial systems Unishyversity of Michigan Press Ann Arbor MI

Horton R E (1937) ldquoDetermination of infiltration-capacity for large drainage basinsrdquo Trans Am Geophys Union 18 371ndash385

Kavetski D Kuczera G and Franks S W (2006) ldquoBayesian analysis of input uncertainty in hydrological modeling 1 Theoryrdquo Water Resour Res 42(3) W03407

Kuczera G (1983) ldquoImproved parameter inference in catchment models 1 Evaluating parameter uncertaintyrdquo Water Resour Res 19(5) 1151ndash1162

Kuczera G Kavetski D Franks S and Thyer M (2006) ldquoTowards a Bayesian total error analysis of conceptual rainfall runoff models Charshyacterizing model error using storm-dependent parametersrdquo J Hydrol 331(1ndash2) 161ndash177

Kuczera G and Parent E (1998) ldquoMonte Carlo assessment of parameter uncertainty in conceptual catchment models The Metropolis algoshyrithmrdquo J Hydrol 211(1ndash4) 69ndash85

Laloy E and Vrugt J A (2012) ldquoHigh-dimensional posterior exploration of hydrologic models using multiple-try DREAM(ZS) and high-perforshymance computingrdquo Water Resour Res 48 W01526

Los Angeles County Department of Public Works (LACDPW) (2011) ldquoBallona Creek Watershedrdquo langhttpladpworgwmdwatershedbcrang (Jul 26 2011)

Mannina G and Viviani G (2010) ldquoAn urban drainage stormwater quality model Model development and uncertainty quantificationrdquo J Hydrol 381(3ndash4) 248ndash265

Mantovan P and Todini E (2006) ldquoHydrological forecasting uncertainty assessment Incoherence of the GLUE methodologyrdquo J Hydrol 330(1ndash2) 368ndash381

Montanari A Shoemaker C A and van de Giesen N (2009) ldquoIntroduction to special section on uncertainty assessment in surface

and subsurface hydrology An overview of issues and challengesrdquo Water Resour Res 45(12) W00B00

Moradkhani H and Sorooshian S (2008) ldquoGeneral review of rainfall-runoff modeling Model calibration data assimilation and uncertainty analysisrdquo Hydrological modeling and water cycle Coupling of the atmospheric and hydrological models Vol 63 Springer Water Science and Technology Library Berlin Heidelberg 1ndash23

Muleta M K (2010) ldquoComparison of model evaluation methods to develop a comprehensive watershed simulation modelrdquo World Environshymental and Water Resources Congress 2010 ASCE Reston VA 2492ndash2501

Muleta M K (2012) ldquoSensitivity of model performance to the efficiency criteria used as objective function during automatic calibrationsrdquo J Hydrol Eng 17(6) 756ndash767

Muleta M K and Nicklow J W (2005) ldquoSensitivity and uncertainty analysis coupled with automatic calibration for a distributed watershed modelrdquo J Hydrol 306(1ndash4) 127ndash145

Nash J E and Sutcliffe J V (1970) ldquoRiver flow forecasting through conceptual models Part I- A discussion of principlesrdquo J Hydrol 125 277ndash291

National Oceanic and Atmospheric Administration (NOAA) (2013) ldquoAtlas 14 point precipitation frequency estimatesrdquo Silver Spring MD langhttphdscnwsnoaagovhdscpfdspfds_map_prhtmlrang (Jun 25 2013)

Natural Resources Conservation Service (NRCS) (2013) ldquoSoil data martrdquo US Dept of Agriculture Washington DC langhttpsoildatamartnrcs usdagovrang (Jun 25 2013)

National Research Council (NRC) (2008) Urban stormwater management in the United States The National Academies Press Washington DC

Rossman L A (2010) ldquoStormwater management model userrsquos manual version 5rdquo EPA600R-05040 US EPA Cincinnati

Schoups G and Vrugt J A (2010) ldquoA formal likelihood function for parameter and predictive inference of hydrologic models with corshyrelated heteroscedastic and non-Gaussian errorsrdquo Water Resour Res 46(10) W10531

Soil Conservation Service (1964) ldquoNational engineering handbookrdquo Secshytion 4 Hydrology Department of Agriculture Washington DC 450

Sorooshian S and Dracup J A (1980) ldquoStochastic parameter estimation procedures for hydrologic rainfall-runoff models Correlated and hetershyoscedastic error casesrdquo Water Resour Res 16(2) 430ndash442

Stedinger J R Vogel R M Lee S U and Batchelder R (2008) ldquoAppraisal of the generalized likelihood uncertainty estimation (GLUE) methodrdquo Water Resour Res 44(12) W00B06

Stein E D and Tiefenthaler L L (2005) ldquoDry-weather metals and bacshyteria loading in an arid urban watershed Ballona Creek Californiardquo Water Air Soil Pollut 164(1ndash4) 367ndash382

Stenstrom M K and Strecker E (1993) ldquoAssessment of storm drain sources of contaminants to Santa Monica Bay Annual pollutants loadshyings to Santa Monica Bay from stormwater runoffrdquo Rep No UCLAshyENG-93-62 Vol 1 Univ of California Los Angeles 1ndash248

Thiemann M Trosset M Gupta H V and Sorooshian S (2001) ldquoBayesian recursive parameter estimation for hydrologic modelsrdquo Water Resour Res 37(10) 2521ndash2535

Thyer M Renard B Kavetski D Kuczera G Franks S W and Srikanthan S (2009) ldquoCritical evaluation of parameter consistency and predictive uncertainty in hydrological modeling A case study using Bayesian total error analysisrdquo Water Resour Res 45(12) W00B14

Tolson B A and Shoemaker C A (2007) ldquoDynamically dimensioned search algorithm for computationally efficient watershed model calibrashytionrdquo Water Resour Res 43(1) W01413

USGS (2013) ldquoThe national maprdquo USGS Washington DC langhttp seamlessusgsgovrang (Jun 25 2013)

Vrugt J A Diks C G H Gupta H V Bouten W and Verstraten J M (2005) ldquoImproved treatment of uncertainty in hydrologic modeling Combining the strengths of global optimization and data assimilationrdquo Water Resour Res 41(1) W01017

Vrugt J A Gupta H V Bouten W and Sorooshian S (2003) ldquoA Shuffled Complex Evolution Metropolis algorithm for optimization and uncertainty assessment of hydrologic model parametersrdquo Water Resour Res 39(8) SWC 1-1ndashSWC 1-14

Vrugt J A Ter Braak C J F Clark M P Hyman J M and Robinson B A (2008) ldquoTreatment of input uncertainty in hydrologic modeling Doing hydrology backward with Markov chain Monte Carlo simulashytionrdquo Water Resour Res 44(12) W00B09

Vrugt J A Ter Braak C J F Diks C G H Higdon D Robinson B A and Hyman J M (2009a) ldquoAccelerating Markov chain Monte Carlo simulation by differential evolution with self-adaptive randomshyized subspace samplingrdquo Int J Nonlinear Sci Numer Simul 10(3) 271ndash288

Vrugt J A Ter Braak C J F Gupta H V and Robinson B A (2009b) ldquoEquifinality of formal (DREAM) and informal (GLUE) Bayesian apshyproaches to hydrologic modelingrdquo Stoch Environ Res Risk Assess 23(7) 1011ndash1026

and Beven (2006) has found widespread application for unshycertainty analysis in water resources This informal Bayesian apshyproach is simple to implement but has been criticized for being statistically incoherent (Mantovan and Todini 2006 Stedinger et al 2008 Vrugt et al 2009b) In response to this various authors have proposed formal uncertainty analysis (UA) methods that use proper statistics and employ valid likelihood measures (Kuczera and Parent 1998 Thiemann et al 2001 Vrugt et al 2003 Schoups and Vrugt 2010) These techniques attempt to provide estimates of the probability density function (PDF) of the model parameters as well as total predictive uncertainty eg through Monte Carlo simulations For computational efficiency reason Markov-chain Monte Carlo (MCMC) schemes are preferred to classic Monte Carlo simulations that rely on random sampling

In addition to efficient and robust sampling schemes successful UA entails appropriate formulation of the likelihood function The formal UA applications often reported in the literature make unrealistic assumptions regarding the structure of the residuals (errors) between model simulations and the observed watershed response Common assumptions include that residuals are (1) temshyporally independent (ie no correlation between errors of successhysive time steps) (2) normally distributed and (3) homoscedastic (ie variance of the residuals does not depend on magnitude) Addressing these unrealistic assumptions a flexible and general formal likelihood (GL) function has been recently described by Schoups and Vrugt (2010)

The objective of this study is to examine effectiveness of a formal Bayesian approach for uncertainty analysis and calibration of the US EPA Storm Water Management Model (SWMM5) (Rossman 2010) The GL function and a recently developed effishycient MCMC sampling scheme known as DREAMethZSTHORN (Schoups and Vrugt 2010) has been used to identify parameter posteriors and to estimate runoff prediction uncertainty The methodology is illustrated using the Ballona Creek watershed a heavily urbanshyized watershed located in the Los Angeles Basin California Effecshytiveness of the UA method in removing heteroscedasticity and temporal correlation and in identifying representative PDF for the residuals has been scrutinized To examine robustness of the UA method for identifying the optimal solutions typically sought by classic calibration approaches the UA solution [ie the maxishymum likelihood (ML) parameter set and the associated runoff preshydictions] has been compared with the solution determined by an automated calibration algorithm known as dynamically dimenshysioned search (DDS) (Tolson and Shoemaker 2007)

Most UA studies in water resources that applied MCMC techshynique within the Bayesian framework used lumped conceptual models for rainfall-runoff analysis of rural watersheds (Kuczera et al 2006 Vrugt et al 2009a Schoups and Vrugt 2010) Few studshyies have been reported in spatially distributed modeling (Feyen et al 2008) Applications to urban watersheds are very limited Ball (2009) underlined the need for UA-based approaches for evaluation of urban drainage models in the discussion of the conventional calibration effort reported on Ballona Creek watershed by Barco et al (2008) Freni et al (2008 2009a) applied GLUE to an urban drainage model and tested sensitivity of the solutions to likelihood measures (Freni et al 2009a) and acceptability thresholds (Freni et al 2008) In a separate study Freni et al (2009b) compared performance of Bayesian Monte Carlo method to that of GLUE Mannina and Viviani (2010) applied GLUE for UA of storm water quality using a conceptual urban drainage model developed inshyhouse All these applications of UA to urban drainage models used GLUE an informal approach whose statistical validity has been questioned (Mantovan and Todini 2006 Stedinger et al 2008 Vrugt et al 2008)

Working under the Joint Committee on Urban Drainage estabshylished by the International Water Association and the International Association for Hydro-Environment Engineering and Research (IWAIAHR) the International Working Group on Data and Models has recently published findings of its effort to develop a framework for defining and assessing uncertainties in urban drainshyage models (Dotto et al 2012 Deletic et al 2012) The article by Deletic et al (2012) underscored the need for consistent use of terminologies and methods for UA of urban drainage models The authors defined various sources of uncertainties presented the linkages between the different uncertainty sources and proshyposed a framework for UA of urban storm water models The article by Dotto et al (2012) compared four different UA methods (three non-Bayesian and one Bayesian) in terms of the posterior PDFs and prediction intervals determined by the methods and their relative computational efficiencies They showed that the non-Bayesian methods required subjective decisions that affected the UA results whereas the Bayesian method used erroneous assumption regardshying structure of the residuals

This study is the first to the best knowledge of the authors to apply an MCMC scheme that works within formal Bayesian frameshywork for UA of urban watersheds and to apply Bayesian approach to UA of SWMM Besides demonstrating application of stateshyof-the-art in UA to urban storm water modeling the ensuing model will be used for an ongoing study that attempts to develop a simulation-optimization model for watershed-scale design of SCMs for urban watersheds

Methods and Materials

Uncertainty Analysis and the MCMC Algorithm

The watershed response (eg runoff) simulated by a storm water management model f such as SWMM5 can be described as

Y frac14 fethI θTHORN eth1THORN ^where Y frac14 n times 1 vector representing the runoff time series

(y1 yn) I = matrix of model forcings (eg precipitation) and θ signifies a d-dimensional vector of model parameters To test how well f describes runoff from a watershed the common practice

^is to compare the model predictions Yn frac14 fy1 yng with the corresponding observations Yn frac14 fy1 yng The difference between the two time series can be represented by a residual vector EnethθTHORN

^EnethθTHORN frac14 Y minus Y frac14 fy1 minus y1 yn minus yng frac14 fe1ethθTHORN enethθTHORNg

eth2THORN The traditional model calibration technique searches for a

single optimal combination of parameter values that minimizes this residual time series With the recognition that model simulations are plagued by many sources of uncertainty validity of this convenshytional parameter estimation method has been questioned A plaushysible alternative is to account for the various sources of uncertainty and to determine posterior PDF of the parameters for example using the Bayesian approach

From Bayes theorem the posterior PDF of the parameters pethθjIYnTHORN) can be given as

pethθjIYnTHORN prop LethθjYn ITHORNpethθTHORN eth3THORN

where LethθjYn ITHORN denotes the likelihood function that measures how well the model fits the data and pethθTHORN = prior distribution



2




tfrac141 tfrac141

eth5THORN



ifrac141


























Methodology





Parameters





N ifrac141
































500

400 (a)

300

200

100

0 0


Res

idua

ls (

a)

Str

eam

flow

[m3 s

]R

esid

uals

(a)

S

trea

mflo

w [m

3 s

]

40 1 1

05

Par

tial a

utoc

orre

latio

nP

artia

l aut

ocor

rela

tion

(b) (c) (d)0820

Den

sity

D

ensi

ty 06

04 0

0

-05

-20 02

-40 0 0 200 400 -2 0 2 0 10 20 30 40 50


400 (e)

300

200

100

0 0 50 100 150 200 250 300 350 400 450 500 550


50 1 1 (h)(f) (g)

0825 05

0

-05

06

04 0

-25 02

-50 0 0 100 200 300 -2 0 2 0 10 20 30 40 50






Bias ()

Volume (mm)

Observed Simulated

DDS

GL-DREAMethZSTHORN

C V C V

689 979 774 1057

094 1115 090 minus1882 093 1094 088 minus2040

596 836 596 836

662 689 661 676









22

18

R S

tati

stic





0 50000 100000

16

14

12

1

Simulation







1200 2500 1000 1200


N-Imperv


N-Perv


0 1 0117 012 096 1 s0 x 10-6 s1 β

0845 085 -0105 -01 0185 0205φ2φ φ3(b) 1

0

2000

0

1500

0

1000

0

1500

0

800

0

1200

0

1500

0

600

1200

0

900

1800

0

750

1500

0

1000

2000

0

750

1500

0

750



Conclusions




Stre

amfl

ow [

m3

s]

Rai

nfal

l [m






600 (a)

400

200

0

0 -200

100 200 300 400 500 600


Str

eam

flow

[m3 s

]R

esid

uals

(a)

S

trea

mflo

w [m

3 s

]

40 1 1

05

Par

tial a

utoc

orre

latio

n

(b) (c) (d)0820

Den

sity 06

04 0

0

-05

-20 02

-40 0 0 100 200 300 400 -2 0 2 0 10 20 30 40 50


600 (e)

400

200

0

-200 0 50 100 150 200 250 300 350 400 450 500 550


20 1 1

05

Par

tial a

utoc

orre

latio

n

(f) (g) (h) 10

Res

idua

ls (

a)

Den

sity

050

0

-05

-10

-20 0 0 100 200 300 -2 0 2 0 10 20 30 40 50





References































































2




tfrac141 tfrac141

eth5THORN



ifrac141


























Methodology





Parameters





N ifrac141
































500

400 (a)

300

200

100

0 0


Res

idua

ls (

a)

Str

eam

flow

[m3 s

]R

esid

uals

(a)

S

trea

mflo

w [m

3 s

]

40 1 1

05

Par

tial a

utoc

orre

latio

nP

artia

l aut

ocor

rela

tion

(b) (c) (d)0820

Den

sity

D

ensi

ty 06

04 0

0

-05

-20 02

-40 0 0 200 400 -2 0 2 0 10 20 30 40 50


400 (e)

300

200

100

0 0 50 100 150 200 250 300 350 400 450 500 550


50 1 1 (h)(f) (g)

0825 05

0

-05

06

04 0

-25 02

-50 0 0 100 200 300 -2 0 2 0 10 20 30 40 50






Bias ()

Volume (mm)

Observed Simulated

DDS

GL-DREAMethZSTHORN

C V C V

689 979 774 1057

094 1115 090 minus1882 093 1094 088 minus2040

596 836 596 836

662 689 661 676









22

18

R S

tati

stic





0 50000 100000

16

14

12

1

Simulation







1200 2500 1000 1200


N-Imperv


N-Perv


0 1 0117 012 096 1 s0 x 10-6 s1 β

0845 085 -0105 -01 0185 0205φ2φ φ3(b) 1

0

2000

0

1500

0

1000

0

1500

0

800

0

1200

0

1500

0

600

1200

0

900

1800

0

750

1500

0

1000

2000

0

750

1500

0

750



Conclusions




Stre

amfl

ow [

m3

s]

Rai

nfal

l [m






600 (a)

400

200

0

0 -200

100 200 300 400 500 600


Str

eam

flow

[m3 s

]R

esid

uals

(a)

S

trea

mflo

w [m

3 s

]

40 1 1

05

Par

tial a

utoc

orre

latio

n

(b) (c) (d)0820

Den

sity 06

04 0

0

-05

-20 02

-40 0 0 100 200 300 400 -2 0 2 0 10 20 30 40 50


600 (e)

400

200

0

-200 0 50 100 150 200 250 300 350 400 450 500 550


20 1 1

05

Par

tial a

utoc

orre

latio

n

(f) (g) (h) 10

Res

idua

ls (

a)

Den

sity

050

0

-05

-10

-20 0 0 100 200 300 -2 0 2 0 10 20 30 40 50





References







































































Methodology





Parameters





N ifrac141
































500

400 (a)

300

200

100

0 0


Res

idua

ls (

a)

Str

eam

flow

[m3 s

]R

esid

uals

(a)

S

trea

mflo

w [m

3 s

]

40 1 1

05

Par

tial a

utoc

orre

latio

nP

artia

l aut

ocor

rela

tion

(b) (c) (d)0820

Den

sity

D

ensi

ty 06

04 0

0

-05

-20 02

-40 0 0 200 400 -2 0 2 0 10 20 30 40 50


400 (e)

300

200

100

0 0 50 100 150 200 250 300 350 400 450 500 550


50 1 1 (h)(f) (g)

0825 05

0

-05

06

04 0

-25 02

-50 0 0 100 200 300 -2 0 2 0 10 20 30 40 50






Bias ()

Volume (mm)

Observed Simulated

DDS

GL-DREAMethZSTHORN

C V C V

689 979 774 1057

094 1115 090 minus1882 093 1094 088 minus2040

596 836 596 836

662 689 661 676









22

18

R S

tati

stic





0 50000 100000

16

14

12

1

Simulation







1200 2500 1000 1200


N-Imperv


N-Perv


0 1 0117 012 096 1 s0 x 10-6 s1 β

0845 085 -0105 -01 0185 0205φ2φ φ3(b) 1

0

2000

0

1500

0

1000

0

1500

0

800

0

1200

0

1500

0

600

1200

0

900

1800

0

750

1500

0

1000

2000

0

750

1500

0

750



Conclusions




Stre

amfl

ow [

m3

s]

Rai

nfal

l [m






600 (a)

400

200

0

0 -200

100 200 300 400 500 600


Str

eam

flow

[m3 s

]R

esid

uals

(a)

S

trea

mflo

w [m

3 s

]

40 1 1

05

Par

tial a

utoc

orre

latio

n

(b) (c) (d)0820

Den

sity 06

04 0

0

-05

-20 02

-40 0 0 100 200 300 400 -2 0 2 0 10 20 30 40 50


600 (e)

400

200

0

-200 0 50 100 150 200 250 300 350 400 450 500 550


20 1 1

05

Par

tial a

utoc

orre

latio

n

(f) (g) (h) 10

Res

idua

ls (

a)

Den

sity

050

0

-05

-10

-20 0 0 100 200 300 -2 0 2 0 10 20 30 40 50





References
























































































500

400 (a)

300

200

100

0 0


Res

idua

ls (

a)

Str

eam

flow

[m3 s

]R

esid

uals

(a)

S

trea

mflo

w [m

3 s

]

40 1 1

05

Par

tial a

utoc

orre

latio

nP

artia

l aut

ocor

rela

tion

(b) (c) (d)0820

Den

sity

D

ensi

ty 06

04 0

0

-05

-20 02

-40 0 0 200 400 -2 0 2 0 10 20 30 40 50


400 (e)

300

200

100

0 0 50 100 150 200 250 300 350 400 450 500 550


50 1 1 (h)(f) (g)

0825 05

0

-05

06

04 0

-25 02

-50 0 0 100 200 300 -2 0 2 0 10 20 30 40 50






Bias ()

Volume (mm)

Observed Simulated

DDS

GL-DREAMethZSTHORN

C V C V

689 979 774 1057

094 1115 090 minus1882 093 1094 088 minus2040

596 836 596 836

662 689 661 676









22

18

R S

tati

stic





0 50000 100000

16

14

12

1

Simulation







1200 2500 1000 1200


N-Imperv


N-Perv


0 1 0117 012 096 1 s0 x 10-6 s1 β

0845 085 -0105 -01 0185 0205φ2φ φ3(b) 1

0

2000

0

1500

0

1000

0

1500

0

800

0

1200

0

1500

0

600

1200

0

900

1800

0

750

1500

0

1000

2000

0

750

1500

0

750



Conclusions




Stre

amfl

ow [

m3

s]

Rai

nfal

l [m






600 (a)

400

200

0

0 -200

100 200 300 400 500 600


Str

eam

flow

[m3 s

]R

esid

uals

(a)

S

trea

mflo

w [m

3 s

]

40 1 1

05

Par

tial a

utoc

orre

latio

n

(b) (c) (d)0820

Den

sity 06

04 0

0

-05

-20 02

-40 0 0 100 200 300 400 -2 0 2 0 10 20 30 40 50


600 (e)

400

200

0

-200 0 50 100 150 200 250 300 350 400 450 500 550


20 1 1

05

Par

tial a

utoc

orre

latio

n

(f) (g) (h) 10

Res

idua

ls (

a)

Den

sity

050

0

-05

-10

-20 0 0 100 200 300 -2 0 2 0 10 20 30 40 50





References


































































22

18

R S

tati

stic





0 50000 100000

16

14

12

1

Simulation







1200 2500 1000 1200


N-Imperv


N-Perv


0 1 0117 012 096 1 s0 x 10-6 s1 β

0845 085 -0105 -01 0185 0205φ2φ φ3(b) 1

0

2000

0

1500

0

1000

0

1500

0

800

0

1200

0

1500

0

600

1200

0

900

1800

0

750

1500

0

1000

2000

0

750

1500

0

750



Conclusions




Stre

amfl

ow [

m3

s]

Rai

nfal

l [m






600 (a)

400

200

0

0 -200

100 200 300 400 500 600


Str

eam

flow

[m3 s

]R

esid

uals

(a)

S

trea

mflo

w [m

3 s

]

40 1 1

05

Par

tial a

utoc

orre

latio

n

(b) (c) (d)0820

Den

sity 06

04 0

0

-05

-20 02

-40 0 0 100 200 300 400 -2 0 2 0 10 20 30 40 50


600 (e)

400

200

0

-200 0 50 100 150 200 250 300 350 400 450 500 550


20 1 1

05

Par

tial a

utoc

orre

latio

n

(f) (g) (h) 10

Res

idua

ls (

a)

Den

sity

050

0

-05

-10

-20 0 0 100 200 300 -2 0 2 0 10 20 30 40 50





References





























































1200 2500 1000 1200


N-Imperv


N-Perv


0 1 0117 012 096 1 s0 x 10-6 s1 β

0845 085 -0105 -01 0185 0205φ2φ φ3(b) 1

0

2000

0

1500

0

1000

0

1500

0

800

0

1200

0

1500

0

600

1200

0

900

1800

0

750

1500

0

1000

2000

0

750

1500

0

750



Conclusions




Stre

amfl

ow [

m3

s]

Rai

nfal

l [m






600 (a)

400

200

0

0 -200

100 200 300 400 500 600


Str

eam

flow

[m3 s

]R

esid

uals

(a)

S

trea

mflo

w [m

3 s

]

40 1 1

05

Par

tial a

utoc

orre

latio

n

(b) (c) (d)0820

Den

sity 06

04 0

0

-05

-20 02

-40 0 0 100 200 300 400 -2 0 2 0 10 20 30 40 50


600 (e)

400

200

0

-200 0 50 100 150 200 250 300 350 400 450 500 550


20 1 1

05

Par

tial a

utoc

orre

latio

n

(f) (g) (h) 10

Res

idua

ls (

a)

Den

sity

050

0

-05

-10

-20 0 0 100 200 300 -2 0 2 0 10 20 30 40 50





References































































Stre

amfl

ow [

m3

s]

Rai

nfal

l [m






600 (a)

400

200

0

0 -200

100 200 300 400 500 600


Str

eam

flow

[m3 s

]R

esid

uals

(a)

S

trea

mflo

w [m

3 s

]

40 1 1

05

Par

tial a

utoc

orre

latio

n

(b) (c) (d)0820

Den

sity 06

04 0

0

-05

-20 02

-40 0 0 100 200 300 400 -2 0 2 0 10 20 30 40 50


600 (e)

400

200

0

-200 0 50 100 150 200 250 300 350 400 450 500 550


20 1 1

05

Par

tial a

utoc

orre

latio

n

(f) (g) (h) 10

Res

idua

ls (

a)

Den

sity

050

0

-05

-10

-20 0 0 100 200 300 -2 0 2 0 10 20 30 40 50





References































































References





























































bayesian approach for uncertainty analysis of an

Documents