a multiple criteria decision support system for testing integrated

Fuzzy Sets and Systems 113 (2000) 53–67www.elsevier.com/locate/fss

A multiple criteria decision support system for testing integratedenvironmental models

D. Scott Mackaya ;∗, Vincent B. RobinsonbaDepartment of Forest Ecology and Management and Institute for Environmental Studies, University of Wisconsin-Madison,

Madison, WI 53706, USAbDepartment of Geography, University of Toronto, Erindale Campus, Mississauga, Ont., Canada L5L 1C6

Received November 1998

Abstract

Spatial models of ecological and hydrological processes are widely used tools for studying natural systems over largeareas. However, these models lack speci�c mechanisms for reporting output uncertainty contributed by model structure, andso testing their suitability for studying a large range of problems is di�cult. This paper describes a method of evaluating theuncertainty contributed by underlying assumptions used in constructing integrated environmental models from two or moresub-models that were developed for di�erent purposes. Integrated environmental models are typically constructed from manyindividual process-based models. Con icting assumptions between these sub-models, e.g., spatial scale di�erences, are easilyoverlooked during model development and application. This “semantic error” cannot be predicted prior to simulation, as itmay only emerge through the interaction of sub-models applied to a particular set of data used to drive a simulation. Modelagreement is proposed and demonstrated as a way to detect problems of model integration at the state variable level withinan integrated ecosystem model. This model agreement is then propagated to model response variables using multiple criteriato examine their sensitivity, predictability, and synchronicity to the measured uncertainty in state variables. These threeproperties are combined under fuzzy logic in order to provide decision support on where, for a given time during simulationthe sub-models agree on a particular response variable. This paper describes the details of the approach and its applicationusing an existing integrated environmental model. The results show that, for a given set of model inputs and application,integrated environmental models may have spatially variable levels of agreement at the sub-model level. The results usingRHESSysD, a spatially integrated ecosystem hydrology model, indicate that semantic error in estimates of plant availablesoil moisture are consistent with observations of the need for resetting events, such as ooding, to initialize the model to apoint where further simulation results can be trusted. These results suggest that a dynamic selection of sub-models may bewarranted given a reasonable method of determining sub-model disagreement during simulation. Fuzzy set theory may be auseful tool in arriving at such a model selection process as it allows for a relatively straightforward synthesis of numerousmodel evaluation criteria with a large quantity of output from the model. c© 2000 Elsevier Science B.V. All rights reserved.

Keywords: Arti�cial intelligence; Control theory; Decision making; Environmental modeling; Operators; Multicriteriaanalysis; Spatial scale

∗ Corresponding author. Tel.: +1-608-262-1669; fax: +1-608-262-9922.E-mail addresses: dsmackay@facsta�.wisc.edu (D.S. Mackay);[email protected] (V.B. Robinson)

0165-0114/00/$ - see front matter c© 2000 Elsevier Science B.V. All rights reserved.PII: S 0165 -0114(99)00012 -3

54 D.S. Mackay, V.B. Robinson / Fuzzy Sets and Systems 113 (2000) 53–67

1. Introduction

Concern over the role of human activity on ourenvironment has increased the demand for integrated,spatially-distributed, environmental models that ad-dress the interactions of human activity, the terrestrialbiosphere, climate, and hydrology. Furthermore, thewidespread availability of geographical informationsystems (GIS) to support spatial data processing andanalysis is making spatial models accessible to alarger audience that includes policy makers. As a re-sult, there has never been a greater need for decisionsupport tools to help in evaluating the applicability ofcomplex environmental models to a given problem.One of many possible requirements for such modelsis that their underlying assumptions be consistent witha potential application of the model. Models abstractfrom reality to varying degrees by assuming thatcertain properties of the system represented can be ig-nored. Some of these simplifying assumptions will beknown to a model end-user, while others will not. Themore deeply rooted assumptions, such as those basedon spatial aggregation, are often poorly documentedand thus less well understood to model end-users. Ifexplicit knowledge of the assumptions underlying aprocess-based model is unknown, then con icting as-sumptions resulting from a combination of such mod-els will undoubtedly produce unexpected, and oftendi�cult to evaluate, output. Con icting assumptionsin geographical information systems have previouslybeen described as semantic errors [27,35], since theyare usually attributable to di�erences in interpretationof a modeled reality and usually contribute to the over-all error. Semantic error in the context of integratedmodels also contributes to error in the form of over-prediction or under-prediction of model output. Whilethere are many sources of error in environmentalmodels it is this semantic error contributed by modelstructure that we address here.For the purposes of this paper integrated models

are composed of many individual process-based mod-els (or sub-models) that describe di�erent processes.Each sub-model represents an interpretation of systembehavior, intended for the purpose of (1) elucidatingcomplex system behavior, or (2) predicting system be-havior [15]. A model developed as a null hypothesismay be used for making predictions if it demonstratesa convincing level of validity. Such adaptation is a po-

tential source of uncertainty for environmental mod-els, which are not subject to formal proof and so areneither unconditionally validated nor invalidated [28].Since the conditional requirements for using a modelare either not reported or are subject to interpretation,there is a danger that the perceived “: : : theoreticalrigor that underlies the models will engender uncriti-cal belief in their prediction” [20].This paper presents a methodology and example

prototype software system for testing combinations ofprocess models intended to be integrated into an envi-ronmental model and applied to a particular problem.A new method of managing semantic errors causedby the structure of integrated environmental models isdeveloped to evaluate model suitability for answeringspeci�c questions. Here it is suggested that a decisionabout sub-model suitability as part of an integratedmodel can be made given a knowledge of how modelresponse variables are a�ected by semantic error de-termined by a divergence from some expected sys-tem behavior. The decision-making framework uponwhich this decision is made is referred to as modelagreement. This framework is described in the nextsection. A method of measuring semantic error in thecontext of a particular integrated model is then de-scribed, and �nally we will step through an exampleto illustrate the application of the system.

2. Decision making framework

2.1. Model agreement

Hayes-Roth and Jacobstein [12] suggest that knowl-edge bases give application programs increased ex-ibility by lending them reasoning ability. Previouswork in knowledge-based simulation modeling hasdemonstrated improved control over model develop-ment, analysis of model results, and model use [4,14].Knowledge-based modeling has improved the rigorand provability of some earth science models. Forinstance, decision-support systems have aided in theapplication of both hydrological and ecosystem mod-els [7,10,30,34,36]. Decision-support systems help inchoosing, from a repository of models, a collectionof models that is suitable for a given problem. Whilethe model selection process remains largely human-guided a number of model description languages have

D.S. Mackay, V.B. Robinson / Fuzzy Sets and Systems 113 (2000) 53–67 55

Fig. 1. Model-based management can be approached from theperspective of the model as (a) black-box, or (b) an integral partof the software system in which all components, e.g., sub-models,must be viewed as acting together towards achieving a commongoal.

emerged to formalize some of the human decision pro-cesses and information management associated withmodel development [3,5,6,19,33,38].Model information management can be approached

at di�erent levels (Fig. 1). In the model informa-tion management strategy that is most commonlyseen within the GIS community (Fig. 1a), all deci-sion rules are held separate from the process modelsand decisions are made centrally [10,21,36]. This isconsidered a coarse level of integration in which se-mantic errors within the process models cannot bedetected, as they depend on the interactions betweensub-models during simulation. A �ner level suggestedhere (Fig. 1b) allows sub-models to measure seman-tic error that emerges through model interaction. Thedegree to which semantic errors in the linkages be-tween models a�ect model output is here referred toas model agreement.Model agreement follows from model theoretic

ideas, which are applied to databases and knowl-edge bases [18]. A model theory clearly states thesemantics of a knowledge base, KB, and providesthe software system an unambiguous understandingabout which statements added to KB agree with KB(Fig. 2a). The statements based on model theoryare either true or false in the context of the existingdatabase or knowledge base. Here we must deal withthe fact that process models do not lend themselvesto mathematical proof [28], and so, by de�nition, se-

Fig. 2. Model theory requires perfect agreement between knowl-edge bases KB 1 and KB 2. Model agreement measures a shiftfrom a central concept representing perfect agreement betweenmodels.

mantic equivalence of process models must follow adi�erent paradigm that is not based on a Boolean set.Instead, a continuous measure of agreement betweenmodels is suggested (Fig. 2b). Two models are in per-fect agreement if they give the same system responseunder the same conditions. Integrated models thatare designed top-down, in which all sub-models aredesigned together for a speci�c goal, will have sub-models that are compatible in their underlying space,time, and attribute assumptions. However, large in-tegrated models are more likely to be constructed ina bottom-up fashion, in which sub-models that wereoriginally designed for di�erent goals are combined tosolve a new set of goals. Thus, integrated models aremost likely to have con icting assumptions betweensub-models, and so the sub-models may diverge fromperfect agreement. Instead of being true or false, thecombined models do not totally agree on all variablesof interest to the end-user of the model.While it is obvious that a detected semantic error

can be corrected given a better model, we suggestthat a given integrated model structure may perform


admirably over a wide range of applications and theneasily trusted for all applications. However, there isas yet little theory on how to determine when an inte-grated model is misbehaving at a semantic level. Hereit is suggested that semantic errors can be detectedfor a larger number of variables than more tradi-tionally tested variables that are corroborated againstmeasurements. Many modeled variables have no�eld-measured values for which a rigorous modelcorroboration can be made. For instance, the spatialvariability of soil moisture is di�cult to measureover large areas, and yet it has a profound in uenceon surface runo�, which is routinely measured. Forthe model described here we found that semantic er-rors could be detected in storage, or state, variablessuch as water table depth. The propagation of thisdetectable error to other model variables, such assoil moisture, give the modeler insight on how andwhen to change model structure. Of course, even the“best” integrated models may have undetectable se-mantic errors, for which the methodology presentedhere cannot help. However, the methodology relieson the modeling being able to specify all simplifyingassumptions used to translate what is currently knownabout the physical system into a model structure.The discussion that follows assumes that semanticerror is directly related to violation of one or moreunderlying assumptions of the models and that thiserror is propagated to variables of interest to themodel developer. A multiple criteria method of trans-ferring model agreement from measured semanticerror to output variables is described in the nextsection.

2.2. Transfer of model agreement

Fig. 3 shows the overall ow of processes withinthe decision support system. If we assume that theresponse variable is de�ned in some query to the in-tegrated modeling system, then such a query couldhave the form, 〈�; �; �; �(��)〉, where � is the goal ofthe query (or response variable of interest), � and �respectively de�ne the spatial and temporal domainswithin which � is de�ned, and �(��) is a membershipfunction that describes the extent to which � is a mem-ber of the fuzzy set [23] describing model agreementfor the goal. We assume that a membership of 1.0 willbe used to denote that � was computed with perfect

agreement between sub-models, and a membership of0.0 will denote perfect disagreement.The simulation model provides agreement amounts

in both the response and state variables. Agreementin response variables and state variables are distin-guished in that agreement in a response variable, ��,is said to be functionally related to semantic error ina state variable, ��:

�� = f〈��;M 〉; (1)

where the function, f, combines one or more crite-ria, and M represents the integrated model itself. Theactual computation of �� is made by parallel simula-tion in which an output variable � is calculated twice:(1) �1 with �� uncorrected and (2) �2 with �� cor-rected. The di�erence between �1 and �2 is ��, whichcan then be compared to �� to assess how semanticerror is transferred to variables of interest. A one-to-one relationship between �� and �� is not necessaryto determine the actual transfer of agreement, as the�nal agreement is reported as a membership regard-less of the absolute sign of the relationship between�� and ��. For instance, there could be a form of hys-teresis in which �� may take on completely di�erentvalues at di�erent times while �� remains constant.Hysteresis is partially accounted for by incorporatinga synchronicity criterion as described below.We identi�ed three criteria for measuring the

transferral of semantic error to goal variables. Thecriteria are: (1) sensitivity, (2) predictability, and (3)synchronicity. These criteria were selected to be con-sistent with indicators used in sensitivity analysis toassess model performance. Sensitivity describes thetotal amount of variability in a response variable thatis attributable to a given amount of model disagree-ment measured for a state variable. The membershipfunction for sensitivity is computed as

�〈��〉 = min{1−

[��(i; t)−min(��)max(��)−min(��)

]}(2)

for all i patches and time intervals, t, within a de-�ned spatial domain. A spatial domain is representedas a collection of one or more patches. Patches maybe considered as a group in order to reduce the num-ber of end-member response functions and facilitateeasier presentation to a model designer. The member-ship function that results from the transformation in


Fig. 3. Flow of processes within the decision support system. RHESSysD is contained within the box, while all components to the rightof the box are considered part of the decision support system.

Eq. (2) describes the worst case sensitivity of the re-sponse variable to measured disagreement. The shapeof this function indicates how sensitive the responsevariable is to model disagreement.Predictability describes the rate of change of ��

with respect to ��. Slowly or smoothly varying re-sponses are predictable over a small range of variationin semantic error. Sharp transitions or discontinuitiesmake it di�cult to predict response variable behaviorfor even small amounts of model disagreement. A re-sponse variable whose behavior is linearly relatedto disagreement in a state variable would yield apredictability membership function with a value of1.0. The membership function for predictability iscomputed as

�(��) = min{1−

[��(i; t)−min(��)max(��)−min(��)

]}; (3)

where ��= @��=@�� is the rate of change of theresponse variable with respect to measured modeldisagreement.Synchronicity describes the degree to which agree-

ment in a response variable follows state variableagreement. In many cases a response variable dis-agreement may peak at a later time during a simula-tion than the peak disagreement in a state variable.This lag is attributable to a storage e�ect in manyphysical systems. For instance, some water stored inthe soil by a rising water table during a spring snowmelt often remains in the soil well into the summermonths at which time it may be depleted by transpir-ing vegetation. An error in calculating the positionof the water table can result in an error in soil waterstorage. Although the error in the water table positionmay occur over only a short time period, e.g., the time

it takes to melt a snow pack, the error propagated tosoil moisture storage remains for a longer period oftime. This storage e�ect results in an asynchronousresponse, possibly in the form of some hysteresis, asillustrated in Fig. 2b. The actual width of the greyband is variable and is conceptually de�ned as thedi�erence in relative residual semantic error betweenthe response and state variables, for a given spatiallocation, calculated over the full time domain of thesimulation. It is computed as follows:

� (!�(i)) = 1− | (i)− �(i)|; (4)

where

�(i) =��(i)−min(��)max(��)−min(��) ; (5)

(i) = 1− |��(i)| −min(|��|)max(|��|)−min(|��|) ; (6)

respectively, describe relative magnitude of the rangeof residual error in a response variable for a givenamount of state variable disagreement, and relativemagnitude of the state variable disagreement. Therange of residual error in the response variable isgiven by

��(i) = max(��(i))−min(��(i)): (7)

Finally, an overall membership function for modelagreement with respect to a particular goal variable isgiven by

�(��(i)) = min[�(��(i)); �(��(i)); �(!�(i))];

(8)

where �(��) gives the membership of response vari-able, �, in the model agreement set for that variable.


We choose to use the minimum function on thegrounds that lowmembership in any one of sensitivity,predictability, and synchronicity should produce un-desirable integrated model response. How these indi-vidual criteria interact with each other is otherwiseunknown. Furthermore, it is assumed that disagree-ment is given by the complement of agreement,@�(�), such that a group of sub-models are in agree-ment with respect to � if �(��)¿ 0:5.The preceding discussion is easily extended to

multi-variate goals by recognizing that each goal vari-able in a query can have its own membership functionand decision-making process. However, speci�c rulesfor combining the individual membership functionsto assess overall agreement of the multi-variate queryare beyond the scope of the present work. In the nextsection we describe a particular environment modeland a method of measuring semantic error within it.

3. Measuring model semantic error

To demonstrate our approach we used the Re-gional HydroEcological Simulation System-Dynamic(RHESSysD). RHESSysD is a spatial informationprocessing and modeling environment for simulat-ing water, carbon, and nutrient uxes (Fig. 4). Themodel incorporates a hydrology model, TOPMODEL[22,26], to link a collection of landscape patches viaground water. Vertical water balance is maintainedwithin each patch using a collection of models torepresent evaporation, transpiration, soil drainage,capillary rise, snow melt, and storage of water. De-tails of the mechanics of the integrated model aregiven in [8,25]. Some details of the hydrology modelthat determines the position of the water table at eachtime step during simulation are described here. Theposition of the water table in turn a�ects soil waterstorage, which is used by vegetation and determinesthe spatial extent of saturated areas that can generatesurface runo�. The position of the water table is alsoused to control the rate of ground water ow, whichsustains stream ow. In TOPMODEL the depth to awater table in thin, well-drained soils on moderate tosteep slopes is represented by

zi = 〈z〉 − 1f(�− TSI); (9)

where zi is the water table depth at a particular landsurface area (or patch), 〈z〉 is a mean water tabledepth, f is a soil hydraulic parameter, � is the meantopography-soils index (TSI) [29], where TSI ac-counts for horizontal water ow convergence in con-cave areas, water retention in areas with relatively lowsoil transmissivity, ow divergence in convex areas,and rapid drainage in areas of high soil transmissivity[26]. Patches with high TSI will tend to have smallz values while patches with low TSI will have largez values. Eq. (9) allows for a redistribution of waterbased on a probability distribution of TSI with noregard for spatial position other than its connected toa single simulation unit. This allows TOPMODELto e�ectively move water laterally without knowingexplicit connections between patches. This lack ofa spatially explicit hydrological model poses a chal-lenge for constructing spatially distributed modelssuch as RHESSysD, yet these simpler models remainin the mainstream because of their applicability to ad-dressing watershed management problems for whichthere may be a paucity of detailed spatial information[16,20].Patches are represented as groups of cells within a

raster GIS. Patches are in turn organized into subcatch-ments over which 〈z〉 and � are computed. Patcheswithin a given subcatchment that have the same TSIare considered hydrologically equivalent. Each patchis assigned vegetation, soil, and other attributes usingGIS analysis. TSI assumes that subsurface rechargeis spatially uniform in order to allow the water tableto rise and fall uniformly with the subcatchment av-erage water recharge. A net positive recharge raisesthe water table the same amount everywhere, while anet water loss lowers the water table the same every-where. The model generates runo� from any patch inwhich zi¡0. The uniform rise and fall of the watertable is only considered a reasonable representationunder conditions of spatially uniform water rechargeto the water table. Otherwise, transient localized wa-ter table mounding occurs for short periods of time inareas of greater than average recharge. Spatial varia-tions in snow melt rates, evaporation and transpirationrates, and drainage determine the actual spatial distri-bution of water recharge in RHESSysD, which is notrequired to be uniform. The result is a spatial scalemismatch between TOPMODEL and the other modelswithin RHESSysD. Mackay [9] tested the validity of


Fig. 4. Physical processes represented by RHESSysD. Shown are three patches (e.g., forest stands) with vertical water uxes and horizontalwater uxes.

Eq. (9) given spatial variation in surface-subsurfacewater ux processes, by incorporating a model self-evaluation tool that di�erentiates between expectedsubsurface ow and actual subsurface ow as repre-sented by TOPMODEL.Fig. 5 describes the process of di�erentiating be-

tween expected subsurface ow and semantic errordue to variable recharge. Patches are spatially arrangedinto groups by elevation, allowing for water redistri-bution using Eq. (9) within each elevation group, butrestricting between-group water redistribution to oc-

cur only from higher to lower elevation. This restric-tion assumes that the partitioning of the landscape intoa series of hillslope facets attached to streams capturesthe dominant landscape dissection and that major path-ways of surface runo� are not contained within hill-slopes. It is recognized that this assumption may notalways hold and it may be possible that runo� gener-ating areas higher on the slope will occur. However,it is the redistribution of excess recharge that occursin runo� generating areas to source areas that con-stitutes a semantic error here. By restricting water


Fig. 5. Conceptual representation of semantic error in the redistribution of water to compute water table depths.

redistribution we hope to approximate the behaviorof explicit routing models that move water in only adown slope direction without creating an explicit rout-ing model that would be di�cult to integrate into theexisting model structure. The unrestricted water redis-tribution of TOPMODEL allows water to e�ectivelybe moved in an up slope direction under certain con-ditions in which higher than average recharge occursin the lower elevation areas within the subcatchment.For example, snow melt rates are usually higher atlower elevation sites than at high elevation sites due tothe higher temperatures, and so snow melt processescan violate the uniform recharge assumption. The dif-ference between unrestricted water redistribution andrestricted water redistribution gives the patch level se-mantic error.Following Moore and Thompson [32] Eq. (9) is

solved for patch i using the current water table positionat patch j. Solving for zj gives an estimate for zj as

zji = zi +1f(TSIi − TSIj): (10)

A residual between stored water table depth, whichwas computed in a previous iteration of the model us-ing the correction described below in Eq. (13), derivedfrom Rules 1 and 2 (below), and the estimated depthis computed as

�Rji = zj − zji ; (11)

where zj is explicitly stored from the previous timestep. Eq. (11) describes the e�ect of redistributing wa-ter from (to) i to (from) j, but we still need to de-termine how this potentially contributes to semanticerror. Since we would like to preserve the down-slopemovement of water that is produced by explicit rout-

ing models, semantic error can be quanti�ed to de-termine if lower elevation patches contribute waterto higher elevation patches. The following rules de-�ne up-slope and down-slope components of the waterredistribution:Rule 1 (Water incorrectly redistributed to an up-

slope patch from patch i):

∀j; ∈ sub-area | elevation(j)¿elevation(i);i∈ sub-area

⇒ �iupslope = MAX

0;

k∑j=1

�Rjiwj

:

Rule 2 (Water incorrectly distributed from a down-slope patch to patch i):

∀j; ∈ sub-area | elevation(j)¡elevation(i);i∈ sub-area

⇒ �idownslope = MIN

0;

k∑j=1

�Rjiwj

;

wherewj is weighting of sub-area j by its relative area.Total semantic error, which is error in the position ofthe water table due to violation of the uniform rechargeassumption of TOPMODEL, at patch i is

��(i) = �iupslope + �idownslope (12)

where ��(i) is positive when the predicted water tableis too deep and negative when the modeled water tableis too near the surface. Given a knowledge of semantic


error a “true” patch water table is computed as

zi = (〈z〉 − ��(i))− 1f(TSIi − �); (13)

where ��(i) is used as a correction to the mean wa-ter table depth used in redistribution. This correctedmodel (Eq. (13)) is then applied iteratively withEq. (11) in the next simulation time step. By us-ing Eqs. (9) and (13) in parallel two sets of outputvariables are generated representing, respectively, un-corrected and corrected sets. The di�erences betweenrespective corrected and uncorrected variables, ��(i),that arise using the di�erent sets of state variable cal-culations are then applied in Eq. (8) to obtain modelagreement (or disagreement) about � at patch i.

4. Results

We tested the decision support system using datasets obtained for Onion Creek, a 13 km2 watershed inthe Central Sierra Nevada in California. Relief in thiswatershed ranges from about 1600m a.s.l to 2600ma.s.l. Annual precipitation averages about 1300mmand occurs primarily as snowfall. In this region hydro-logical processes are dominated by snow pack storageand snow melt. We partitioned the watershed [24] intosubcatchments, elevation bands, and TSI [29] patches(Fig. 6). We then ran the decision support system withthe following query:

Where do the sub-models agree on soil moisture

predictions at the start of the growing season?

In this query the goal is soil moisture, the spatialdomain is de�ned by the spatial layers shown inFig. 6, and the temporal domain is the start of thegrowing season, which corresponds with the end ofthe snow melt period. Fig. 7 shows six zones de�nedby spatially aggregating TSI patches. This aggrega-tion is made to improve presentation clarity and re-duce computational requirements. The actual numberof spatially aggregated areas can be de�ned by theend-user. Fig. 8 shows the corresponding membershipfunctions for (a) sensitivity, (b) predictability, and(c) synchronicity of soil water variability as functionsof semantic error in the water table position for thesix zones. In these relations the x-axis represents se-

mantic error as computed by Eq. (12), with positivevalues corresponding to locations where the water ta-ble depth is over-predicted by the model and negativevalues corresponding to where the water table is pre-dicted to be too close to the surface. The plots weregenerated by �tting a weighted, localized robust line[39] through the points representing the lower enve-lope of the scatter of points representing the respec-tive relationships. The lower envelope is presented asthe most conservative of memberships. In general, thememberships tend to be larger at more positive levelsof semantic error. This is expected to occur in higherrecharge areas, which are typically saturated duringthe snow melt period regardless of the semantic errorand hence do not in uence plant available soil watergreatly. This is clearly shown in the �nal membershipfunctions for agreement on soil moisture for each ofthe six aggregated spatial areas (Fig. 9). The �nalmembership functions were plotted using the same�tting method as described above [39]. The highestmemberships tend to occur in areas around streams,where soil moisture is near saturation with or withoutthe semantic error removed. Fig. 8a shows that drier(lowest TSI) areas tend to be more sensitive to smallamounts of error in predicting the water table posi-tion. This suggests that the integrated model providessemantically reasonable plant available soil moistureprediction in valley bottom areas and poor predictionof soil moisture in drier areas. This is not surprisinggiven that TOPMODEL was designed for predict-ing stream ow response, not providing reasonableestimates of plant available soil water.Mackay [9] showed that semantic error in the water

table position approaches zero everywhere at the endof snow melt. This period of time corresponds withthe beginning of the vegetation growing season. Thus,an evaluation of the memberships for each aggregatedarea when semantic error of the water table positionis zero was used to evaluate the results of the query.Fig. 10a shows the fuzzy memberships in spatial setagreed soil water, which were produced by mappingthe fuzzy memberships from Fig. 9 at a point wheresemantic error in the water table position (��) is zero,for each of the six respective zones. Areas of low soilmoisture agreement membership occur where soilsshould normally have low water content, but haveincreased water content due to error in the water tabledepth redistribution. Also shown (Fig. 10b) is the crisp


Fig. 6. Partitioning of Onion Creek into subcatchment areas, elevation bands or zones, and TSI patches for spatial representation of inputto RHESSysD.

Fig. 7. Spatially aggregated TSI zones used for model agreement analysis and presentation of �nal memberships.


Fig. 8. Plots of (a) sensitivity memberships, (b) predictability memberships, and (c) synchronicity memberships for soil moisture versussemantic error in the water table, for each of the respective aggregate TSI zones.


Fig. 9. Overall agreed soil water memberships for the 6 aggregate TSI zones.

Fig. 10. Final (a) fuzzy and (b) crisp maps showing where the sub-models within RHESSysD agree=disagree on soil moisture at the startof the growing season. In (a) brighter areas have higher fuzzy memberships, while in (b) the black areas represent where the models agree.

spatial set derived from the fuzzy spatial set by map-ping only the areas that have fuzzy memberships of atleast 0.5. The areas mapped in black correspond to ar-eas of greatest likelihood of soil saturation and henceruno� generation. The results are consistent with

the need for a resetting event, such as a soil surfacesaturation, to establish a reasonable set of initialconditions for simulation [8,25]. For the purposes ofdecision making the areas mapped in black are wherethe integrated model successfully answers the query.


These are areas where the integrated model gives soilmoisture estimates that are not adversely a�ected bymodel structure.

5. Discussion

Simulation models typically generate large quanti-ties of output, which usually have to be analyzed by themodel end-user in order to assess model quality. Forinstance, simulations of the watershed used here have1450 TSI patches, which are simulated for each of 365days. This represents over a half-million soil moistureoutput values, which may or may not be trustworthygiven the structure of the integrated model and thecharacteristics of the input data. Model self-evaluationprovides an initial basis for evaluating the model bymeasuring model semantic error in computing theposition of the water table. How goal variables respondto this error provides deeper insight on model agree-ment. The signi�cance of transient disagreementsbetween sub-models is that for certain queries tothe integrated model (e.g., soil moisture), sub-modelincompatibility may or may not preclude the useof these models. The semantic analysis presentedhere allows for the identi�cation of areas within thespatial and temporal domains where sub-models pro-vide agreed upon answers to speci�c queries. Thisapproach di�ers from more commonly used modeltesting tools such as sensitivity analysis (e.g., MonteCarlo analysis in which many realizations of equallikelihood are run) for a model treated as a black box[1,17,13]. Sensitivity analysis cannot distinguish be-tween uncertainty due to data quality and uncertaintydue to model structure [17], whereas the approachpresented here focuses on model structure problemsrather than comparison of model output to data.Here fuzzy sets are used as a method of combining

several criteria for evaluating model performance.This di�ers from other applications of fuzzy sets inthe GIS community. For instance, systems such asSRAS [37] and SOLIM [2] use fuzzy set theory to lin-guistically express spatial or co-spatial relations usingexperience of expert subjects to produce membershipfunctions. Here, membership functions are generatedby a simulation model using a �xed set of criteriafor evaluation, for the purpose of controlling theselection of sub-models to be combined. Once a deci-

sion is made on model agreement for a given query,a speci�c set of model alternates can be selected.For instance, a model that represents the water ta-ble in a mathematically more explicit way, but iscomputationally intractable for large spatial areas[31], may be preferred for certain types of ques-tions over small spatial areas. However, for manyqueries aggregating to a level appropriate for cap-turing the dominant spatial variability of processesavoids the need for extremely accurate, �ne resolu-tion data sets [11]. Since most data cannot alwaysbe obtained to support detailed physically-basedmodels, more conceptually-based models, which arebased on simpli�ed relationships that use more read-ily available data, will remain in widespread use[6,16]. Thus, model self-evaluation is an importantstep towards providing general tools for determiningthe range of acceptable queries for integrated modelsconstructed from conceptual models.

6. Conclusions

The most signi�cant result of this work is the in-troduction of self-evaluation into spatially distributedprocess models, and its application as a decision-making tool for selecting or rejecting a combinationof sub-models for a given application. The model,RHESSysD, incorporates this self-evaluation in thesense that it identi�es and keeps track of semanticerror in the water table position for every modeledpatch. In essence, two parallel simulations, one usingEq. (9) (uncorrected) and the other using Eq. (13)(corrected), are maintained by the model. All otheraspects of the model remained unchanged betweenthe parallel versions, making the comparative analysisbetween the two simulated results feasible. The resultsof these simulations are then passed to the decisionsupport system for comparative analysis using thethree criteria. The ability to manage the interactionsbetween sub-models during simulation allows for theidenti�cation of model semantic inconsistencies thatcould be used to trigger the replacement of particularsub-models as the simulation model executes. The useof fuzzy set theory allows the decision to be based ona combination of criteria allowing for di�erent indi-cators of uncertainty in the model response variablesto be considered. Fuzzy sets theory thus facilitates


the analysis of a large quantity of simulation outputand its synthesis more decision making. The currentsoftware system is only a prototype and is gearedtowards analysis of output from RHESSysD. Futurework should consider how alternative models can beselected and used based on the decision processes de-scribed in this paper. In addition, future work with themethod must consider how queries involving multipleresponse variables can be addressed, and an integra-tion of memberships made across a suite of variables.

Acknowledgements

This work was partially supported by a start-upgrant from the Graduate School of the University ofWisconsin-Madison to the �rst author. Data for OnionCreek were provided by the U.S. Forest Service andby NTSG, University of Montana. The authors grate-fully acknowledge Pauline van Gaans and an anony-mous reviewer for their careful and thoughtful reviewof this manuscript.

References

[1] A.M. Binley, K.J. Beven, A. Calver, L.G. Watts, Changingresponses in hydrology: assessing the uncertainty inphysically based model predictions, Water Resources Res. 27(6) (1991) 1253–1261.

[2] A.X. Zhu, L.E. Band, B. Dutton, T. Nimlos, Automatedsoil inference under fuzzy logic, Ecol. Modeling 90 (1996)123–145.

[3] B.P. Zeigler, Object-Oriented Simulation with Hierarchical,Modular Models: Intelligent Agents and EndomorphicSystems, Academic Press, San Diego, CA, 1990.

[4] C. Loehle, Applying arti�cial intelligence techniques to Ecol.Modelling 38 (1987) 191–212.

[5] C.G. Wesseling, D. Karssenberg, P.A. Burrough, W.P.A. vanDeursen, Integrating dynamic environmental models in GIS:the development of a dynamic modelling language, Trans.GIS 1 (1996) 40–48.

[6] D.A. Bennett, M.P. Armstrong, F. Weirich, An object-oriented modelbase management system for environmentalsimulation, in: M.F. Goodchild, B.O. Parks, L.T. Steyaert(Eds.), GIS and Environmental Modelling: Progress andResearch Issues, GIS World Books, Fort Collins, CO, 1996,pp. 439–444.

[7] D.H. Jamieson, K. Fedra, The ‘WaterWare’ decision-supportsystem for river-basin planning. 1. conceptual design,J. Hydrol. 177 (1996) 163–175.

[8] D.S. Mackay, L.E. Band, Forest ecosystem processes at thewatershed scale: dynamic coupling of distributed hydrologyand canopy growth, Hydrol. Processes 11 (1997) 1197–1218.

[9] D.S. Mackay, Integrating self-evaluating hydrological andecological models of di�erent spatial scales, in: Proc.GIS=LIS’97, American Society of Photogrammetry andRemote Sensing and American Congress on Surveying andMapping, Falls Church, VA, 1997, pp. 486–498.

[10] D.S. Mackay, V.B. Robinson, L.E. Band, A knowledge-based approach to the management of geographic informationsystems for simulation of forested ecosystems, in:W.K. Michener, J.W. Brunt, S.G. Sta�ord (Eds.), Environ-mental Information Management and Analysis: Ecosystemsto Global Scales, Taylor & Francis, London, 1984, pp.515–538.

[11] E.F. Wood, M. Sivapalan, K. Beven, L. Band, E�ects ofspatial variability and scale with implications to hydrologicmodeling, J. Hydrol. 102 (1988) 29–47.

[12] F. Hayes-Roth, N. Jacobstein, The state of knowledge-basedsystems, Comm. ACM 37 (3) (1994) 27–39.

[13] G.B.M. Heuvelink, Error propagation in quantitative spatialmodelling: applications in geographical information systems,Thesis, University of Utrecht, 1993.

[14] H. Abelson, M. Eisenberg, M. Halfant, J. Katzenelson,E. Sacks, G.J. Sussman, J. Wisdom, K. Yip, Intelligence inscienti�c computing, Comm. ACM 32 (5) (1989) 546–562.

[15] H. Caswell, The validation problem, in: B.C. Patten (Ed.),Systems Analysis and Simulation in Ecology, AcademicPress, New York, 1976, pp. 313–325.

[16] I.D. Moore, T.W. Norton, J.E. Williams, Modellingenvironmental heterogeneity in forested landscapes, J. Hydrol.150 (1993) 717–747.

[17] J. Freer, K. Beven, B. Ambroise, Bayesian estimation ofuncertainty in runo� prediction and the value of data: anapplication of the GLUE approach, Water Resources Res. 32(7) (1996) 2161–2173.

[18] J.D. Ullman, Principles of Database and Knowledge-BaseSystems, Computer Science Press, Rockville, MD, 1988.

[19] J.W. Rozenblit, P.L. Janowski, An integrated framework forknowledge-based modeling and simulation of natural systems,Simulation 57 (1991) 152–165.

[20] K. Beven, Changing ideas in hydrology – the case ofphysically-based models, J. Hydrol. 105 (1989) 157–172.

[21] K. Fedra, Distributed models and embedded GIS: strategiesand case studies of integration, in: M.F. Goodchild,B.O. Parks, L.T. Steyaert (Eds.), GIS and EnvironmentalModeling: Progress and Research Issues, GIS World Books,Fort Collins, CO, 1996, pp. 413–418.

[22] K.J. Beven, M.J. Kirkby, A physically based, variablecontributing area model of basin hydrology, Hydrol. Sci. Bull.24 (1) (1979) 43–69.

[23] L. Zadeh, Fuzzy sets, Inform. and Control 8 (1965) 339–353.[24] L.E. Band, A terrain-based watershed information system,

Hydrol. Processes 3 (1989) 151–162.[25] L.E. Band, P. Patterson, R. Nemani, S.W. Running, Forest

ecosystem processes at the watershed scale: incorporatinghillslope hydrology, Agri. Forest Meteorol. 63 (1993)93–126.


[26] M. Sivapalan, K. Beven, E.F. Wood, On hydrologic similarity2. A scaled model of storm runo� production, WaterResources Res. 30 (1987) 1665–1679.

[27] M.F. Worboys, S.M. Deen, Semantic heterogeneity indistributed geographic databases, ACM SIGMOD Record 20(4) (1991) 30–34.

[28] N. Oreskes, K. Shrader-Frechette, K. Belitz, Veri�cation,validation, and con�rmation of numerical models in the earthsciences, Science 263 (1994) 641–646.

[29] P. Quinn, K. Beven, P. Chevalier, O. Planchon, The predictionof hillslope ow paths for distributed hydrological modellingusing digital terrain models, Hydrol. Processes 5 (1991)59–79.

[30] R. Ford, S. Running, R. Nemani, A modular system forscalable ecological modeling, Comput. Sci. Eng. (Fall) (1994)32–44.

[31] R.B. Grayson, I.D. Moore, T.A. McMahon, Physically basedhydrologic modeling 2. Is the concept realistic?, WaterResources Res. 28 (10) (1992) 2659–2666.

[32] R.D. Moore, J.C. Thompson, Are water table variations in ashallow forest soil consistent with the TOPMODEL concept?,Water Resources Res. 32 (3) (1996) 663–669.

[33] R.M. Keller, M. Rimin, A. Das, A knowledge-basedprototyping environment for construction of scienti�cmodeling software, Autom. Software Eng. 1 (1994) 79–128.

[34] S.M. Dunn, R. Mackay, R. Adams, D.R. Oglethorpe, Thehydrological component of the NELUP decision-supportsystem: an appraisal, J. Hydrol. 177 (1996) 213–235.

[35] V.B. Robinson, A.U. Frank, About di�erent kinds ofuncertainty in collections of spatial data, Proc. Auto-Carto 7,American Society for Photogrammetry and Remote Sensingand American Congress on Surveying and Mapping, FallsChurch, VA, 1985, pp. 440–449.

[36] V.B. Robinson, D.S. Mackay, Semantic modeling forthe integration of geographic information and regionalhydroecological simulation management, Comput. Environ.and Urban Systems 19 (1996) 321–339.

[37] V.B. Robinson, Interactive machine acquisition of a fuzzyspatial relation, Comput. Geosci. 16 (1990) 857–872.

[38] W.P.A. van Deursen, Geographical information systems anddynamic models, Thesis, University of Ultrecht, 1995.

[39] W.S. Cleveland, Robust locally weighted regression andsmoothing scatterplots, J. Amer. Statist. Soc. Assoc. 74(1979) 829–836.

a multiple criteria decision support system for testing integrated

Documents