Xenia PackageXenia Package

http://nautilus.baruch.sc.edu/twiki_dmcc/bin/vihttp://nautilus.baruch.sc.edu/twiki_dmcc/bin/view/Main/XeniaPackageew/Main/XeniaPackage

http://nautilus.baruch.sc.edu/twiki_dmcc/bin/vihttp://nautilus.baruch.sc.edu/twiki_dmcc/bin/view/Main/XeniaPackageV2ew/Main/XeniaPackageV2

Problems Xenia intended to addressProblems Xenia intended to address

Grants for research instrumentation which will be Grants for research instrumentation which will be collecting observation data while lacking a data collecting observation data while lacking a data management/sharing component beyond archiving management/sharing component beyond archiving datalogger filesdatalogger files

Low-volume data(< 100,000 records per hour) in-situ Low-volume data(< 100,000 records per hour) in-situ observational platforms or system arrays (e.g. 1 to 1000 observational platforms or system arrays (e.g. 1 to 1000 platforms collecting 10-20 observations per hour) platforms collecting 10-20 observations per hour) collecting data at any geographic scale collecting data at any geographic scale (local,regional,national,etc)(local,regional,national,etc)

Bridging the gap between raw data collection and the Bridging the gap between raw data collection and the organization and sharing of data using previously organization and sharing of data using previously developed products, services and standards(leveraging developed products, services and standards(leveraging earlier work against new data providers)earlier work against new data providers)

Fostering a standardization of products and services via a Fostering a standardization of products and services via a common openly shared technical common openly shared technical infrastructure(common database schema and product infrastructure(common database schema and product support scripts)support scripts)

Problems Xenia not intended to addressProblems Xenia not intended to address

High-volume data (millions of records per High-volume data (millions of records per hour) such as gridded model outputs, hf hour) such as gridded model outputs, hf radar, etc. High-volume data problems at radar, etc. High-volume data problems at this time are better addressed using this time are better addressed using traditional file processing techniques where traditional file processing techniques where data management can suggest output file data management can suggest output file formats(such as images, shapefiles, etc) and formats(such as images, shapefiles, etc) and metadata that are conducive to search and metadata that are conducive to search and usage needs.usage needs.

Table SchemaTable Schema

Basic tablesBasic tables Extended, Support tables Extended, Support tables

Table Schema – BasicTable Schema – Basic

Main tables used for storingMain tables used for storing

organization->platform->sensor->observation dataorganization->platform->sensor->observation data

Not using geospatial indexing initially(can be added) to keep things Not using geospatial indexing initially(can be added) to keep things simplesimple

Current database implementation is in Current database implementation is in PostgreSQL, but should be portable to MySQL, PostgreSQL, but should be portable to MySQL, etc later. Output products developed on Linux etc later. Output products developed on Linux system using mostly perl scripts.system using mostly perl scripts.

Data dictionary captured from earlier Data dictionary captured from earlier development in the lookup tables for development in the lookup tables for m_type_id (m_* = measurement) which can m_type_id (m_* = measurement) which can vary by their standard vary by their standard name(sea_water_temperature,sea_water_salinname(sea_water_temperature,sea_water_salinity) and unit of measure(celsius, fahrenheit, ity) and unit of measure(celsius, fahrenheit, psu)psu)

All measurements stored in multi_obs table with All measurements stored in multi_obs table with their corresponding timestamp, location and their corresponding timestamp, location and qc. Multiple observation types stored similarly qc. Multiple observation types stored similarly varying by their m_type_id index. Each varying by their m_type_id index. Each measurement can/will provide a lookup for measurement can/will provide a lookup for sensor id and possibly collection id.sensor id and possibly collection id.

Table Schema – ExtendedTable Schema – Extended

Additional tables used for supporting quality control tests and Additional tables used for supporting quality control tests and user/group notificationuser/group notification

Additional support tables for collections, quality control will be addedAdditional support tables for collections, quality control will be added

Format Convention

No Convention

Xenia Relational Database

SQL

Web Screen-Scrape

ASCII Fields

ASCII Fields+ Key File

SEACOOS netCDF

XML

SQL conversion script

Time Series Graphs

Maps/WMSAnimations

Archival filesby Obs/Platform

CSVnetCDF,shapefile,etc

Latest Databy Obs/Platform

KML/Google Earth,etcXML/RSS/WFS?

Quality ControlNotification

Products

Quality Control and NotificationQuality Control and Notification

Initial quality control tests are intended to Initial quality control tests are intended to flag/notify on observations by:flag/notify on observations by:

Range tests - values outside of acceptable Range tests - values outside of acceptable range low, range highrange low, range high

Continuity tests – values change too much Continuity tests – values change too much within a specific time intervalwithin a specific time interval

Optional email notification of users or user Optional email notification of users or user groups when qc tests failgroups when qc tests fail

Time Series Graphs/DataTime Series Graphs/Data

Web request for graph only(can be placed as needed in other Web request for graph only(can be placed as needed in other website contexts), webpage(graph+data) or download of website contexts), webpage(graph+data) or download of time series data at specific platform sensorstime series data at specific platform sensors

Maps/WMS(Web Mapping Service) via MapServerMaps/WMS(Web Mapping Service) via MapServer

Map animations via ImageMagick,Gifsicle, AniSMap animations via ImageMagick,Gifsicle, AniS

DODS/OPeNDAP access to basic tables DODS/OPeNDAP access to basic tables (organization, platform, sensor, multi_obs)(organization, platform, sensor, multi_obs)

Latest and Archival productsLatest and Archival products

Guiding concept is to make products available at Guiding concept is to make products available at both regional scale(same observation/product both regional scale(same observation/product across all platforms) and local scale(same across all platforms) and local scale(same platform across all observations/products)platform across all observations/products)

Often a regional product can tie into a local one – a Often a regional product can tie into a local one – a regional water temperature map allows a user to regional water temperature map allows a user to select a water temperature graph at a specific select a water temperature graph at a specific platform listed on the mapplatform listed on the map

Products and design divided temporally between Products and design divided temporally between latestlatest, , recentrecent(0-6 weeks), (0-6 weeks), archivalarchival(3+ weeks and (3+ weeks and older). Latest products continually generated with older). Latest products continually generated with new data(hourly) where recent and archival new data(hourly) where recent and archival products may be generated at periodic products may be generated at periodic intervals(daily, weekly).intervals(daily, weekly).

Xenia latest, recent, archival table structure for Xenia latest, recent, archival table structure for observations. Oldest observations stored to files.observations. Oldest observations stored to files.

Latestpast several hours

New Data

Recent0-6 weeks

Archival3+ weeks to 1-2 years

Possibly table separated by year,month,etc

Archival file1-2+ years

Files separated by product/year/month

Latest data productsLatest data products

XML schema convention (ObsKML – my term/schema)XML schema convention (ObsKML – my term/schema)

Regulary(hourly) produced xml file containing all latest Regulary(hourly) produced xml file containing all latest measurements organized by organization->platform-measurements organized by organization->platform->observations. Designed for cross-system >observations. Designed for cross-system aggregation needs.aggregation needs.

Regulary(hourly) produced xml files (1 per platform) Regulary(hourly) produced xml files (1 per platform) containing all latest measurements within that containing all latest measurements within that platform. Designed for local use similar to a RSS feed platform. Designed for local use similar to a RSS feed for each platform.for each platform.

Regulary(hourly) produced xml files (1 per observation) Regulary(hourly) produced xml files (1 per observation) containing all latest measurements of the same containing all latest measurements of the same observation type. Designed for cross-system observation type. Designed for cross-system aggregation needs focusing on a specific observation.aggregation needs focusing on a specific observation.

Latest data productsLatest data products

Example of latest XML feed used to populate Carolinas Coast application Example of latest XML feed used to populate Carolinas Coast application and potentially further systems or Xenia instancesand potentially further systems or Xenia instances

Latest data productsLatest data products

KML (Keyhole Markup Language) which is the XML format used to KML (Keyhole Markup Language) which is the XML format used to visualize data in Google Earth and potentially other 3D Globes visualize data in Google Earth and potentially other 3D Globes such as NASA WorldWind and ESRI ArcExplorersuch as NASA WorldWind and ESRI ArcExplorer

Archival data productsArchival data products

CSV (Comma Separated Value) files viewable CSV (Comma Separated Value) files viewable using Excelusing Excel

Archival folder/file separated byArchival folder/file separated by

observation type or platformobservation type or platform

month(or some manageable regular month(or some manageable regular timestep)timestep)

for file download according to user regional/local for file download according to user regional/local interestinterest

Other output file formats(netCDF, shapefiles, etc) Other output file formats(netCDF, shapefiles, etc) archives similarly folder/file organizedarchives similarly folder/file organized

Archival data productsArchival data products

CSV (Comma Separated Value) files(exchange format) viewable CSV (Comma Separated Value) files(exchange format) viewable usingusing

ODV(Ocean Data View) for CTD/Bottle analysisODV(Ocean Data View) for CTD/Bottle analysis

Archival data productsArchival data products

netCDF for analysis using ncBrowsenetCDF for analysis using ncBrowse

Xenia aggregation, replication, redundancyXenia aggregation, replication, redundancy

With several distributed Xenia systems, these systems could With several distributed Xenia systems, these systems could feed each other using either the same feed each other using either the same latest XML feedlatest XML feed or or a a direct copy of table datadirect copy of table data offered by each Xenia offered by each Xenia instanceinstance

XeniaA,B,C,D,E,F

XeniaA,B,C

XeniaD,E,F

XeniaA

XeniaB

XeniaC

XeniaD

XeniaE

XeniaF

Xenia BackupA,B,C,D,E,F

Xenia BackupD,E,F