hydrologic data and modeling: towards hydrologic information science

Hydrologic Data and Modeling: Towards Hydrologic Information Science

David R. MaidmentCenter for Research in Water Resources

University of Texas at Austin

Hydrologic Data and Modeling

• New knowledge in hydrology

• Hydrologic data

• Hydrologic modeling

• Hydrologic information systems

Hydrologic Data and Modeling

• New knowledge in hydrology

• Hydrologic data

• Hydrologic modeling

• Hydrologic information systems

How is new knowledge discovered?

• By deduction from existing knowledge

• By experiment in a laboratory

• By observation of the natural environment

After completing the Handbook of Hydrology in 1993, I asked myself the question: how is new knowledge discovered in hydrology?

I concluded:

Deduction – Isaac Newton

• Deduction is the classical path of mathematical physics– Given a set of axioms– Then by a logical process– Derive a new principle or

equation

• In hydrology, the St Venant equations for open channel flow and Richard’s equation for unsaturated flow in soils were derived in this way.

(1687)Three laws of motion and law of gravitation

http://en.wikipedia.org/wiki/Isaac_Newton

Experiment – Louis Pasteur

• Experiment is the classical path of laboratory science – a simplified view of the natural world is replicated under controlled conditions

• In hydrology, Darcy’s law for flow in a porous medium was found this way.

Pasteur showed that microorganisms cause disease & discovered vaccinationFoundations of scientific medicine http://en.wikipedia.org/wiki/Louis_Pasteur

Observation – Charles Darwin

• Observation – direct viewing and characterization of patterns and phenomena in the natural environment

• In hydrology, Horton discovered stream scaling laws by interpretation of stream maps

Published Nov 24, 1859Most accessible book of great

scientific imagination ever written

Conclusion for Hydrology

• Deduction and experiment are important, but hydrology is primarily an observational science

• discharge, water quality, groundwater, measurement data collected to support this.

Great Eras of Synthesis

• Scientific progress occurs continuously, but there are great eras of synthesis – many developments happening at once that fuse into knowledge and fundamentally change the science

1900

1960

1940

1920

1980

2000

Physics (relativity, structure of the atom, quantum mechanics)

Geology (observations of seafloor magnetism lead to plate tectonics)

Hydrology (synthesis of water observations leads to knowledge synthesis)

2020

Hydrologic Science

Hydrologic conditions(Fluxes, flows, concentrations)

Hydrologic Process Science(Equations, simulation models, prediction)

Hydrologic Information Science(Observations, data models, visualization

Hydrologic environment(Physical earth)

Physical laws and principles(Mass, momentum, energy, chemistry)

It is as important to represent hydrologic environments precisely with

data as it is to represent hydrologic processes with equations

Hydrologic Data and Modeling

• New knowledge in hydrology

• Hydrologic data

• Hydrologic modeling

• Hydrologic information systems

CUAHSI Member Institutions

122 Universities as of July 2008 (and CSIRO!)

HIS Team and Collaborators

• University of Texas at Austin – David Maidment, Tim Whiteaker, Ernest To, Bryan Enslein, Kate Marney

• San Diego Supercomputer Center – Ilya Zaslavsky, David Valentine, Tom Whitenack

• Utah State University – David Tarboton, Jeff Horsburgh, Kim Schreuders, Justin Berger

• Drexel University – Michael Piasecki, Yoori Choi• University of South Carolina – Jon Goodall, Tony

Castronova• CUAHSI Program Office – Rick Hooper, David

Kirschtel, Conrad Matiuk• National Science Foundation Grant EAR-0413265

HIS Goals

• Data Access – providing better access to a large volume of high quality hydrologic data;

• Hydrologic Observatories – storing and synthesizing hydrologic data for a region;

• Hydrologic Science – providing a stronger hydrologic information infrastructure;

• Hydrologic Education – bringing more hydrologic data into the classroom.

HIS Overview Report

• Summarizes the conceptual framework, methodology, and application tools for HIS version 1.1

• Shows how to develop and publish a CUAHSI Water Data Service

• Available at:

http://his.cuahsi.org/documents/HISOverview.pdf

Rainfall & SnowWater quantity

and quality

Remote sensing

Water Data

Modeling Meteorology

Soil water

Water Data Web Sites

HTML as a Web Language

Text and Picturesin Web Browser

<head><meta http-equiv="content-type" content="text/html; charset=utf-8" /><title>Vermont EPSCoR</title><link rel="stylesheet" href="epscor.css" type="text/css" media="all" /><!-- <script type='text/javascript' language='javascript‘ src='Presets.inc.php'>--></head>

HyperText Markup Language

WaterML as a Web LanguageDischarge of the San Marcos River at Luling, TX June 28 - July 18, 2002

Streamflow data in WaterML language

Services-Oriented Architecture for Water Data

• Links geographically distributed information servers through internet

• Web Services Description Language (WSDL from W3C)

• We designed WaterML as a web services language for water data

• Functions for computer to computer interaction

HIS Servers in the WATERS Network

HIS Central at San Diego Supercomputer Center

Web Services

HIS Central

National Water Metadata Catalog

WaterML

Get Data

Get Metadata

CUAHSI Point Observation Data Services

1. Data Loading– Put data into the CUAHSI Observations Data

Model

2. Data Publishing– Provide web services access to the data

3. Data Indexing– Summarize the data in a centralized

cataloging system

CUAHSI Point Observation Data Services

1. Data Loading– Put data into the CUAHSI Observations Data

Model

2. Data Publishing– Provide web services access to the data

3. Data Indexing– Summarize the data in a centralized

cataloging system

Data Values – indexed by “What-where-when”

Space, S

Time, T

Variables, V

s

t

Vi

vi (s,t)

“Where”

“What”

“When”A data value

Data Values Table

Space, S

Time, T

Variables, V

s

t

Vi

vi (s,t)

Observations Data Model

Horsburgh, J. S., D. G. Tarboton, D. R. Maidment and I. Zaslavsky, (2008), "A Relational Model for Environmental and Water Resources Data," Water Resour. Res., 44: W05406, doi:10.1029/2007WR006392.

• 11 WATERS Network test bed projects• 16 ODM instances (some test beds have more than one ODM

instance)• Data from 1246 sites, of these, 167 sites are operated by WATERS

investigators

National Hydrologic Information Server

San Diego Supercomputer Center

HIS Implementation in WATERS Network Information System

CUAHSI Point Observation Data Services

1. Data Loading– Put data into the CUAHSI Observations Data

Model

2. Data Publishing– Provide web services access to the data

3. Data Indexing– Summarize the data in a centralized

cataloging system

Point Observations Information Model

Data Source

Network

Sites

Variables

Values

{Value, Time, Metadata}

Utah State Univ

Little Bear River

Little Bear River at Mendon Rd

Dissolved Oxygen

9.78 mg/L, 1 October 2007, 5PM

• A data source operates an observation network• A network is a set of observation sites• A site is a point location where one or more variables are measured• A variable is a property describing the flow or quality of water• A value is an observation of a variable at a particular time• A metadata quantity provides additional information about the value

GetSites

GetSiteInfo

GetVariableInfo

GetValues

Assemble Data From Different Sources

Ingest data using ODM Data Loader

Load Newly Formatted Data into ODM Tables in MS SQL/Server

Wrap ODM with WaterML Web Services for Online Publication

Utah State University

University of Florida

Texas A&MCorpusChristi

Publishing an ODM Water Data Service

USU ODM

UFL ODM

TAMUCC ODM

Observations Data Model (ODM)

WaterML

SnotelDataValues

SnotelMETADATA

ODM

WaterML

Metadata From:ODM Database in

San Diego, CA

Snotel Web Site in Portland, OR

SnotelWater Data

Service

Publishing a Hybrid Water Data Service Snotel Metadata are

Transferred to the ODM

Web Services can both Query the ODM for Metadata and use a Web Scraper for Data Values

Calling the WSDL Returns Metadata and Data Values as if from the same Database

Get Values from:

http://river.sdsc.edu/snotel/cuahsi_1_0.asmx?WSDL

Locations

Variable Codes

Date Ranges

WaterML and WaterOneFlow

GetSiteInfoGetVariableInfoGetValues

WaterOneFlowWeb Service

Client

Penn State

Utah StateNWIS

DataRepositories

Data

DataData

EXTRACTTRANSFORMLOAD

WaterML

WaterML is an XML language for communicating water dataWaterOneFlow is a set of web services based on WaterML

WaterOneFlow• Set of query functions • Returns data in WaterML

CUAHSI Point Observation Data Services

1. Data Loading– Put data into the CUAHSI Observations Data

Model

2. Data Publishing– Provide web services access to the data

3. Data Indexing– Summarize the data in a centralized

cataloging system

Data Series – Metadata description

Space

Variable, Vi

Site, Sj

End Date Time, t2

Begin Date Time, t1

Time

Variables

Count, C

There are C measurements of Variable Vi at Site Sj from time t1 to time t2

Series Catalog

Space

Variable, Vi

Site, Sj

End Date Time, t2

Begin Date Time, t1

Time

Variables

Count, C

Vi

Sj

t2

t1

C

Texas Hydrologic Information System

Sponsored by the Texas Water Development Board and using CUAHSI technology for state and local data sources (using state funding)

CUAHSI National Water Metadata CatalogIndexes:• 50 observation networks• 1.75 million sites• 8.38 million time series• 342 million data values

NWIS

STORET

TCEQ

• Search multiple heterogeneous data sources simultaneously regardless of semantic or structural differences between them

Data Searching

NWIS

NARR

NAWQANAM-12

request

request

request

request

request

requestrequest

request

request

return

return

return

return

return

returnreturn

return

return

Searching each data source separately

Michael PiaseckiDrexel University

Semantic Mediation Searching all data

sources collectively

NWIS

NAWQA

NARR

generic

request

GetValues

GetValues

GetValues

GetValues

GetValues

GetValuesGetValues

GetValues

GetValues HODM

Michael PiaseckiDrexel University

Hydroseekhttp://www.hydroseek.org

Supports search by location and type of data across multiple observation networks including NWIS and Storet

Bora Beran, Drexel

HydroTaggerOntology: A hierarchy of concepts

Each Variable in your data is connected to a corresponding Concept

NWISNWIS

ArcGISArcGIS

ExcelExcel

AcademicAcademic

UnidataUnidata

NASANASAStoretStoret

NCDCNCDC

SnotelSnotel

MatlabMatlab

JavaJava

Visual BasicVisual Basic

Operational services

CUAHSI Web ServicesCUAHSI Web Services

Data SourcesData Sources

ApplicationsApplications

Extract

Transform

Load

http://www.cuahsi.org/his/

HydroExcel

HydroGET: An ArcGIS Web Service Client

http://his.cuahsi.org/hydroget.html

Direct analysis from your favorite analysis environment. e.g. Matlab% create NWIS Class and an instance of the class

createClassFromWsdl('http://water.sdsc.edu/wateroneflow/NWIS/DailyValues.asmx?WSDL');WS = NWISDailyValues;% GetValues to get the datasiteid='NWIS:02087500';bdate='2002-09-30T00:00:00';edate='2006-10-16T00:00:00';variable='NWIS:00060';valuesxml=GetValues(WS,siteid,variable,bdate,edate,'');

1920 1930 1940 1950 1960 1970 1980 1990 2000 20100

0.5

1

1.5

2

2.5x 10

4

cfs

Daily Discharge NEUSE RIVER NEAR CLAYTON, NC

National Water Metadata Catalog

Synthesis and communication of the nation’s water data http://his.cuahsi.org

Hydroseek WaterML

Government Water Data

Academic Water Data

Hydrologic Data and Modeling

• New knowledge in hydrology

• Hydrologic data

• Hydrologic modeling

• Hydrologic information systems

• Project sponsored by the European Commission to promote integration of water models within the Water Framework Directive

• Software standards for model linking• Uses model core as an “engine”• http://www.openMI.org

OpenMI – Links Data and Simulation Models

CUAHSI Observations Data Model as an OpenMI component

Simple River Model

Trigger (identifies what value should be calculated)

Typical model architectureApplication

User interface + engineEngine

Simulates a process – flow in a channelAccepts inputProvides output

ModelAn engine set up to represent a particular location e.g. a reach of the Thames

Engine

Output data

Input data

Model application

Run

Write

Write

Read

User interface

Accepts Provides

Rainfall

(mm)

Runoff

(m3/s)

Temperature

(Deg C)

Evaporation

(mm)

Accepts Provides

Upstream Inflow

(m3/s)

Outflow

(m3/s)

Lateral inflow

(m3/s)

Abstractions

(m3/s)

Discharges

(m3/s)

River Model

Linking modelled quantities

Rainfall Runoff Model

Data transfer at run time

Rainfall runoff

Output data

Input data

User interface

River

Output data

Input data

User interface

GetValues(..)

Models for the processes

River(InfoWorks RS)

Rainfall(database)

Sewer(Mouse)

RR(Sobek-Rainfall

-Runoff)

Data exchange3 Rainfall.GetValues

River(InfoWorks-RS)

Rainfall(database)

Sewer(Mouse)

2 RR.GetValues

7 RR.GetValues

RR(Sobek-Rainfall

-Runoff)

1 Trigger.GetValues

6 Sewer.GetValues

call

data

4

5 8

9

Hydrologic Data and Modeling

• New knowledge in hydrology

• Hydrologic data

• Hydrologic modeling

• Hydrologic information systems

Space, L

Time, T

Variable, V

D

Data Cube – What, Where, When

“What”

“Where”

“When”

A data value

Continuous Space-Time Data Model -- NetCDF

Space, L

Time, T

Variables, V

D

Coordinate dimensions

{X}

Variable dimensions{Y}

Space, FeatureID

Time, TSDateTime

Variables, TSTypeID

TSValue

Discrete Space-Time Data Model

Geostatistics

Time Series Analysis

Multivariate analysis

Hydrologic Statistics

How do we understand space-time correlation fields of many variables?

CUAHSI Hydrologic Information Systems

• A system for integrating water data and models

• CUAHSI HIS team invites EPSCoR scientists to publish their data using CUAHSI Water Data Services and to help us build HIS Desktop during 2009

Observations

ModelsClimate

GIS