cuahsi his service oriented architecture
DESCRIPTION
CUAHSI HIS Service Oriented Architecture. Ilya Zaslavsky, David R. Maidment, David G. Tarboton, Michael Piasecki, Jon Goodall , David Valentine, Thomas Whitenack, Jeffery S. Horsburgh, Tim Whiteaker and the entire CUAHSI HIS Team. http://his.cuahsi.org/. CUAHSI HIS - PowerPoint PPT PresentationTRANSCRIPT
CUAHSI HIS Service Oriented Architecture
SupportEAR 0622374
CUAHSI
HISSharing hydrologic data
http://his.cuahsi.org/
Ilya Zaslavsky, David R. Maidment, David G. Tarboton, Michael Piasecki, Jon Goodall, David Valentine, Thomas
Whitenack, Jeffery S. Horsburgh, Tim Whiteaker
and the entire CUAHSI HIS Team
Data Discovery and Integration
Data Publication Data Analysis and Synthesis
HydroCatalog
HydroDesktopHydroServer
ODM Geo Data
CUAHSI Hydrologic Information System Services-Oriented Architecture
Data Services
Metadata Services Search ServicesWaterML, Other OGC Standards
Information Model and Community Support Infrastructure
What is a “service oriented architecture”?
• A design strategy for informationsystems that enables loose coupling among components
• Essential relationships and dependencies shall be preserved, non-essential can be discarded
• Service == unit of work, performed based on a contract between service provider and service consumer– Hides the internal workings of service– Implementation/platform-independent– Presents a relatively simple interface– Can be published, discovered and invoked using this interface
• Everything is a service: data, models, visualization, ……
"Things should be made as simple as possible, but no simpler."
What makes an open community-driven hydrologic information system• Agreeing on standards for information models and
services: WaterML, WaterOneFlow services, OGC specs
• Making the services easily discoverable, sharing and indexing a lot of quality data: HISCentral
• Reliable core services: monitoring; logging/reporting; user support; high availability
• Sharing code: Codeplex, etc.
4
WaterML as a Web LanguageDischarge of the San Marcos River at Luling, June 28 - July 18, 2002Streamflow data in WaterML language
First presented as an OGC Discussion Paper in 2007
Adopted by USGS, NCDC, multiple academic groups, internationally
OGC®
6
M-WRIIMs System Implementation
WRIIMs
WaterML(Water Markup Language)
Site n
On site sensor query interface and results
water quality, real time and historic data
requesting
responding
Feng-Chia University, Taiwan– Presented 6/16/2011, HydroDWG
HIS Central Catalog
GetSitesGetSiteInfoGetVariableInfoGetValues
WaterOneFlowWeb Service
WaterML
Discovery and Access
Hydro Desktop
Water Metadata Catalog
Harvester
Service Registry Hydrotagger
Search Services
http://hiscentral.cuahsi.org
• Integrates data services from multiple sources
•Supports concept based data discovery
CUAHSI Data
Server
3rd Party Server
e.g. USGS
Map integrating NWIS, STORET, & Climatic Sites
69 public services18,000+ variables1.96+ million sites23.3 million seriesReferencing 5.2 billion data values
HIS Central Content
Available via HISCentraldiscovery services
Available via GetValues requests
2008 2009 2010 20110
1020304050607080
2839
5669
Public Services
Growth in GetValues calls for all services reporting to HIS Central
May-June2011
Federal Agency Water Data Services at HISCentral
Network Name Site Count Value Count(thousands) Earliest Observation Notes
NWISDV 31,800 304,000 10/18/1847 WaterML-compliant GetValues service from NWIS, catalog ingested
EPA 236,000 78,000 01/11/1900 SOAP wrapper over WQX services, catalog ingested
NWISUV 11,800 169,000 120 DAYS WaterML-compliant GetValues Service, catalog ingested
NCDC ISH 11,600 3,000* 1/1/2005 WaterML-compliant GetValues service from NCDC
NCDC ISD 24,800 18,200 1/1/1892 WaterML-compliant GetValues service from NCDC
NWISIID 376,000 86,500 9/1/1867 SOAP wrapper over NWIS web site, catalog ingested
NWISGW 834,000 8,490 1/1/1800 SOAP wrapper over NWIS web site, catalog ingested
RIVERGAGES 1,300 264,000 1/1/2000 WaterML compliant REST services from the Army Corps of Engineers
* Estimated
Hydrologic Ontologyhttp://hiscentral.cuahsi.org/startree.aspx
Semantic heterogeneity: water data source use their own vocabularies, which makes it difficult to discover and interpret dataSolutions: controlled vocabularies community vocabulary of hydrologic parameters, semantic tagging, and semantic query rewriting
acre feet acre-feet
micrograms per kilogram
micrograms per kilgram
FTU NTU
mho Siemens
ppm mg/kg
Dissloved oxygen
HydroTagger
Each Variable is connected to a corresponding Concept
http://water.sdsc.edu/hiscentral/startree.aspx
HISCentral Hosting Facility
Water.sdsc.edu
WebServiceR-U-On Server Monitor
Disrupter
DataStoreR-U-On Process Monitor
R-U-On Server Monitor
River
CUAHSIR-U-On
HIS Central Montior
Server
R-U-On Server Monitor
HIS Central Client Code
R-U-On Process Monitor
R-U-On Service
Server
R-U-OnNet Montiors
Server Monitor
Hiscentral.cuahsi.org
WebServiceR-U-On Server Monitor
DataStor
Kyle.ucsd.edu
DataStoreR-U-On Process Monitor
R-U-On Server Monitor
Mirroring
Usage Logger
Monitored WebsitesAnd HydroServers
Monitored REST EndpointsWaterdata.usgs.gov
• Redundant • Continuously
monitored (R-U-On)• Synchronized
databases• Fail over
management• Monitoring of
external servers• Usage reporting
Service Monitors
14
Local CZO DB
CZO Data Publication System
Spatial, hydrologic, geophysical, geochemical, imagery, spectral…
Local CZO DB Local CZO DB
Web site Web site Web site
Standard CZO Services
Shar
ed
voca
bula
ries
CZO
M
etad
ata
Ont
olog
y
Arc
hive
Har
vest
er
Standard CZO data display formats
CZO
Desktop
Matlab
R
Excel
ArcGIS
Modeling
CZO DesktopApplications
CZOData Products
CZO Web-based Data Discovery
System
External cross-project registries
DataNet
CZO Data Repository and Indexing (CZO Central)
International Standardization of WaterML
Hydrology Domain Working Group- working on WaterML 2.0- organizing Interoperability Experiments focused on different sub-domains of water- towards an agreed upon feature model, observation model, semantics and service stack
http://external.opengis.org/twiki_public/bin/view/HydrologyDWG/WebHome
Iterative DevelopmentTimelineGroundwater IE
– GSC+USGS– Dec 09 – Dec 10
Surface Water IE– CSIRO+many – Jun 10 – Sep 11
Forecasting IE– NWS+Deltares?– Sep 11 – Sep 12?
Water Quality IEWater Use IE
WaterML 2 SWG(Mar 2011)
June’11
New requirements, and the path forward• Transition to OGC model – for better interoperability, including
international: what are new service interfaces; how we transition an operational system?
• Federation of catalogs – since many data providers stand up catalogs, also better scalability: what is the suggested combination of catalog technologies and interfaces?
• Recognition that we don’t need to search over all services: what are the better search patterns (e.g. 3-step data access: identify services, then extract time series metadata, and then request data content for the time series)?
• Recognition that we can (and need to) rely on common implementations of mature, modular standard specifications: what is an appropriate operational governance model for distribution of roles and responsibilities within such a modular system?
The Migration Path (1)• Step 1: Prototyping a new infrastructure and assimilating results of
international validation of new OGC specifications:
A client developed at UT-Austin that implements the Who (data service providers) – What (variables) - Where (locations) search pattern using OGC CSW and WFS services. The CSW interface provides federation of catalog services, while WFS is used to relay time series catalogs
A Kisters WISKI-based client demonstrating access to WFS (for locations of sampling features) and SOS (for observational data encoded in WaterML 2.0), developed as part of Hydrology DWG’s Surface Water IE
The Groundwater (2009-2010) and Surface Water (2010-2011) Interoperability Experiments of the OGC/WMO Hydrology Domain Working Group have demonstrated serving water data encoded in WaterML2 using SOS1 and SOS2 services.
Goal: smooth transition of the operational HIS
The Migration Path (2)• Step 2: Settle on a time series catalog information model that can
be relayed via common WFS implementations• Step 3: Create WFS interfaces over observation networks in the HIS
Central catalog, integrated with HIS Central administration interface
http://hiscentral.cuahsi.org/wfs/52/cuahsi.wfs?request=getCapabilities
http://hiscentral.cuahsi.org/pub_network.aspx?n=52
An additional WFS endpoint for this network
An observation network page in HISCentral administration interface for network #52 (Little Bear River)
The Migration Path (3)• Step 4: Make the networks registry in HISCentral CSW compatible• Step 5: Establish a distributed system of federated hydrologic
catalogs, using the CSW standard
The Migration Path (4)• Step 6: Create WaterML2/SOS endpoints, initially for networks already
registered in the HIS Central Metadata Catalog at SDSC:
The Migration Path (5)• Step 7: Integrate the WaterML2/SOS2 endpoints in
HydroServer software stack• Step 8: Integrate WFS-based series catalog in
HydroServer software stack• Step 9: Update HISCentral harvesting routines to rely on
WFS services• Step 10: Update HydroDesktop client to interact with
CSW and WFS services–These are to be completed
Conclusions• HISCentral maintains a large collection
of hydrologic time series from distributed data sources, both academic and government– Supports data discovery queries and
vocabulary queries– Monitors and validates services– Regular harvesting of registered services– Supports variety of clients– High-availability setup
• Water data exchange standards are the backbone of HIS SOA: – The specifications have seen wide adoption
• One of the benefits of SOA: smooth migration to a new set of standards (OGC)
• Building a community hydrologic information system:– Sharing data and code; reliable core services; access to large volumes of quality data
Catalog
Server Desktop
Metadata
Services
Catalog Services
Data Services