Open Solutions to Regional Observing Systems

Outline

Recent near real-time in-situ observations are aggregated to a Recent near real-time in-situ observations are aggregated to a ‘Xenia’ schema relational database(RDB) via its import XML ‘Xenia’ schema relational database(RDB) via its import XML format ‘ObsKML’ schema. From this database of recent format ‘ObsKML’ schema. From this database of recent observations a variety of file formats, web services and observations a variety of file formats, web services and applications can be driven. By suggesting a minimally applications can be driven. By suggesting a minimally common observation oriented XML and RDB schema, common observation oriented XML and RDB schema, developed scripts and products can benefit from and build developed scripts and products can benefit from and build around these shared schema. For more info contact around these shared schema. For more info contact jeremy.cothran@gmail.comjeremy.cothran@gmail.com

KeywordsKeywordsScripting languages Scripting languages Perl PHPPerl PHPRelational Database Relational Database Sqlite PostgresSqlite PostgresMapping Engine/Interface Mapping Engine/Interface Mapserver OpenLayers TileCacheMapserver OpenLayers TileCacheOpenGeospatial Consortium(OGC) services OpenGeospatial Consortium(OGC) services WMS/WFS/SOSWMS/WFS/SOSGoogle Earth/Maps Google Earth/Maps KML/KMZKML/KMZGoogle Code/AnalyticsGoogle Code/Analytics

Problems Xenia intended to address

• Grants for research instrumentation which will be collecting observation data while lacking a data management/sharing component beyond basic file storage

• Low-volume data(< 100,000 records per hour) in-situ observational platforms or system arrays (e.g. 1 to 1000 platforms collecting 10-20 observations per hour) collecting data at any geographic scale (local,regional,national,etc)

• Bridging the gap between raw data collection and the organization and sharing of data using previously developed products, services and standards(leveraging earlier work against new data providers)

• Fostering a standardization of products and services via a common openly shared technical infrastructure(common database schema and product support scripts)

Problems Xenia not intended to address

High-volume data (millions of records per hour) such as gridded model outputs, hf radar, etc. High-volume data problems at this time are better addressed using traditional file processing techniques where data management can suggest output file formats(such as images, shapefiles, etc) and metadata that are conducive to search and usage needs.

Xenia Relational Database Schema

Hierarchy = organization->platform->sensor->multi_obs(observation data)

ObsKML XML schema for import/export

Data flows – general

Data flows – ObsKML

ObsKML (Observations KML)

A simple XML encoding of observation metadataassociated with a KMLPlacemark.

Default XML import/exportformat for Xenia databaseInstances.

Postgres or Sqlite database paths

Xenia data organized by time interval

Latestpast several hours

New Data

Recent0-6 weeks

Archival3+ weeks to 1-2 years

Possibly table separated by year,month,etc

Archival file1-2+ years

Files separated by product/year/month

Julian weekly divided database files(sqlite) of all observations are available http://tinyurl.com/6ctz66

Xenia aggregation, replication, redundancy

XeniaA,B,C,D,E,F

XeniaA,B,C

XeniaD,E,F

XeniaA

XeniaB

XeniaC

XeniaD

XeniaE

XeniaF

Xenia BackupA,B,C,D,E,F

Xenia BackupD,E,F

Overlapping/redundant systems could aggregate/replicate via SQL or XML

Flow/event monitoring

Around 10,000 observations are collected each hour from a variety of data providers. The data flows are monitored for both successful import and towards provided report metrics or notifications when problems or events occur.

Sensor count graph showing the hourly number of successful sensor imports per provider

More detailed platform/observation table showing daily counts color coded to highlight specific sensor issues

Applications – National Weather Service

Latest regional platform observations from Xenia database are styled to html tables and presented in fixed map interface alongside NWS forecast warnings/advisories and other map layers of interest

http://forecast.weather.gov/mwp/

Applications – SECOORA interactive map

Latest regional observation maps are accessed via WMS(Web Mapping Service) and merged into an OpenLayers(browser javascript) map interface. Observation database requests are supported via WMS GFI(Get Feature Info)

http://secoora.org/maps/dynamic

Applications - CarolinasRCOOS

Map image layers supported via MapServer WMS with increased map response time via TileCache which tiles and caches image requests for repeated reference. Site feature info supported via Xenia database instance styled output html table accessed via WMS GFI.

http://carolinasrcoos.org

KML/KMZ – Latest obs by platform

KML (Keyhole Markup Language) which is the XML format used to visualize data in Google Earth/Maps and potentially other globes/maps with KML support such as NASA WorldWind and ESRI ArcExplorer

Latest observation data organized by all observations per platform

http://tinyurl.com/664wtx

KML/KMZ – latest obs by obsType

KML (Keyhole Markup Language) which is the XML format used to visualize data in Google Earth/Maps and potentially other globes/maps with KML support such as NASA WorldWind and ESRI ArcExplorer

Latest observation data organized by observation type for all platforms

http://tinyurl.com/664wtx

All observations carry

* observation type(obsType)

* unit of measure(uom)

* measurement value

obsType+uom = measurement type(m_type)

Color styled low to high value blue/green/red

Development/documentation support

Google Code provides very simple, free browser based access to the basic project support tools such as a subversion(svn) code/documentation sharing/versioning, wiki documentation and issue tickets.

Google Analytics provides free browser based access to website usage statistics via a javascript project token included in webpage access.

Lessons Learned

• Web GIS interface: php/mapscript -> openlayers/javascript/ajax, tilecache • Latest in-situ obs: RDB – sensor/obs hard-coded/table approach -> obs non-specific infrastructure(xenia,obskml) • Sensor/observation specific metadata: loosely link a variety of more specific metadata files/schemas to common infrastructure• Hourly gridded data: RDB quickscat, model layers -> ogr/sql+shapefiles • Full RDB to file-based RDB: postgresql+postgis -> sqlite+ogr, sqlitegeo? • Self hosted to third party hosted: third party website hosting for static content/context, common processing/storage. Google code/analytics for basic shared development

support/documentation.• ROA(Resource Oriented Architecture)/REST style simpler declarative alternatives to complex query SOA(Service Oriented Architecture)

File formats and web services• File formats (CSV,shapefile,netCDF,KML/KMZ,Sqlite DB)

http://secoora.org/data

Free desktop analysis tools

MapWindows(shapefile/GIS) http://www.mapwindow.com

ncBrowse(netCDF) http://www.epic.noaa.gov/java/ncBrowse

ODV(ctd/csv) http://odv.awi.de

Sqlite DB http://sqlite.org

• Web services (OGC WMS/WFS/SOS)

http://secoora.org/data

http://code.google.com/p/xenia/wiki/XeniaSOS

Credits

2C.Calloway,2J.Cleary,1J.Cothran,4J.Donovan,3J.Dorton,1M.Fletcher,C.Galvarino,1S.King,2S.Haines,3L.Leonard,1D.Porter,3X.Qi,1D.Ramage,2H.Seim,4V.Subramanian,1S.Walker,4

R.Weisberg

1University of South Carolina, 2University of North Carolina at Chapel Hill, 3University of North Carolina at Wilmington, 4University of South Florida

2C.Calloway,2J.Cleary,1J.Cothran,4J.Donovan,3J.Dorton,1M.Fletcher,C.Galvarino,1S.King,2S.Haines,3L.Leonard,1D.Porter,3X.Qi,1D.Ramage,2H.Seim,4V.Subramanian,1S.Walker,4

R.Weisberg

1University of South Carolina, 2University of North Carolina at Chapel Hill, 3University of North Carolina at Wilmington, 4University of South Florida