implementing marine xml for noaa observing data

Implementing Marine XML for NOAA Observing Data

Nazila Merati and Eugene Burger

NOAA/Pacific Marine Environmental Laboratory

Seattle, WA

Ocean observation systems (OOS)and data transformations

• PMEL and its partners carry out several large scale ocean monitoring programs (TAO, FOCI, Tsunami, Argo) with hundreds of instruments and thousands of observations.

Data transformations and data sharing

• Scientists spend a great majority of their time formatting data, transferring data and developing transfer protocols before starting data analysis and sharing.

History of the project• In 2003, we went to the ESRI User Conference and heard that XML

would play an important role in the new ArcGIS architecture.• Also in 2003, NOAA IT and web developers began to discuss the

need for standards when using XML.• We found that others had started looking at XML standards and

defining XML standards for observational Marine data – XBTs and meteorological observations

• We identified a project at PMEL and NODC that we could use to test XML – Argo Profiling Floats.

Test case– Argo Profiling Float Data• The Argo program is broad-scale

global ocean observing system of temperature and salinity

• Floats are designed to drift at a fixed pressure for a fixed period and then move to a profiling pressure. As they rise, they collect profiles of pressure, temperature and salinity on their way to the surface.

• Each float can produce up to 150 profiles during its life time

Current Argo Profiling Float deployments

Why use Argo Profiling Floats?

• Argo Profiling Floats were selected because they are a good example of a profiling data type, have only a few attributes associated with each profile and are integral to the ocean observing system architecture

• NODC’s data manager is interested in using XML as a method of data sharing between Argo Data Centers

• Integration of Argo Profile data into GIS will introduce the data to a new group of users that are more GIS savvy

Project objectives• To work with NODC to identify key data sets and define

which file formats to work with • Using data specifications from example libraries, to

define optimal parameters for a MarineXML dictionary • To build schema that works the best with Floats• To test the schema, assess bandwidth versus file size

issues, and test feasibility of web transfers• To load data into ArcGIS 9.x and personal geodatabase • To test the metadata creator in ArcGIS for compatibility

with MarineXML schema

Why use XML?

• Platform independent tool that makes data exchange and communication between organizations easier – its not just for science

• Extensible

• Data management - allows preservation of data, metadata, quality flags and edited data

• It can serve as the “basis of a data management framework”

Different types of XML• Marine XML – different flavors – strong community component, been in

use since 2001• ESML – Earth Sciences Markup Language – works with netCDF, HDF,

good community backing• Borehole XML - good as it has a z-value, may be better as a descriptor of

actual location rather than data – boreholes are also more shallow that the float data, not quite sure what borehole data are stored as, mostly text files

• NcML – good open source standard, data already are stored as netCDF, it’s the marine standard for data storage, most translators are for gridded data

• Sensor ML – still written within the OGC specs, again more for descriptor, but not necessarily for data transfer

How does this fit in with GIS?• Currently, PMEL has

ways of getting data into GIS that are great, but this still requires data formatting and the use of several different packages.

• The output is a shapefile to be used by ESRI products.

ESRI and the geodatabase• Changes are happening in

the way GIS data are managed. We are moving from shapefiles and coverages to geodatabases for data management and data storage.

• Geodatabases can be useful if you are building data models for a specific industry.

Geodatabases and data transformations• ArcGIS 9 has the ability to

take XML encapsulated data and transfer the data into the geodatabase.

• The Geodatabase XML format will allow you to import and export items and data to/from the geodatabase. Transfers include domains, rules and topologies and adding behaviors

Proposed solution

Potential problems

• netCDF to ncML or to MarineXML – no need to do a double jump

• Bandwidth and compression, XML may supersize data

• What if the geodatabase goes away?• Is MarineXML better than ESML?• Can it be applied to legacy data?

Next steps

• Look at MarineXML specifications and meet with AODC folks next week. Identify key data sets to test and talk to ESRI about netCDF translators.

• Determine if the existing EPIC in-situ XML DTD and schemas can be modified to work with Argo Profiling Floats and can handle data.

• Test the new XSLT tools from ESRI and test transfer times – is this just as clunky as before?

Questions and suggestions?