integrated model data management s.hankin esmf july ‘04 integrated data management in the esmf...

S.Hankin ESMF July ‘04S.Hankin ESMF July ‘04 Integrated ModelData Management

Integrated data Integrated data management in the ESMF management in the ESMF

(ESME)(ESME)

Steve HankinSteve Hankin(NOAA/PMEL & IOOS/DMAC)(NOAA/PMEL & IOOS/DMAC)

ESMF Team meetingESMF Team meetingJuly 2004July 2004


The growing importance The growing importance of data integration to of data integration to modelers:modelers:

We no longer fund “modeling”. We no longer fund “modeling”.

Today we fund “climate prediction” or Today we fund “climate prediction” or “coastal processes” … the science “coastal processes” … the science

topics. Modeling is just a topics. Modeling is just a component.component.

a program manager (anonymous)a program manager (anonymous)


Model outputs need to be Model outputs need to be made useful to many classes of made useful to many classes of usersusers

model runmodel run

projectproject

modeling communitymodeling community

research communityresearch community

educationeducation


Data and products need to be Data and products need to be made more usable for modelersmade more usable for modelers

modelingmodeling

validatiovalidationn

assimilatioassimilationn

Real time andReal time anddelayed-modedelayed-modeobservationsobservations data data

productsproducts

comparisocomparisonn

boundary &boundary &initial initial

conditionsconditions


An ESME must include a An ESME must include a plan for data plan for data management.management.

But, how ?But, how ?

Funds are limited …Funds are limited …


Partnership: A community Partnership: A community of data managers has of data managers has formed …formed …GO-ESSP – Global Organization of GO-ESSP – Global Organization of

Earth System Science PortalsEarth System Science Portals(http://esportal.gfdl.noaa.gov)(http://esportal.gfdl.noaa.gov)

• UnidataUnidata• ESG (NCAR, LLNL)ESG (NCAR, LLNL)• OPeNDAP (a.k.a. ‘DODS’) OPeNDAP (a.k.a. ‘DODS’) • COLACOLA• NOMADS (GFDL, PMEL, NCDC, NCEP)NOMADS (GFDL, PMEL, NCDC, NCEP)• NASA/GCMDNASA/GCMD• BADC, BODCBADC, BODC• WMOWMO• ……


Ocean data systems Ocean data systems following similar following similar approaches … approaches …

• National Virtual Ocean Data System National Virtual Ocean Data System (NVODS)(NVODS)

• US Integrated Ocean Observing US Integrated Ocean Observing SystemSystem

• GODAE (US and International)GODAE (US and International)

• OCMIP, AOMIP, … OCMIP, AOMIP, …


Data portal components:Data portal components:W

ork

flow

Work

flow • Data discoveryData discovery

• Data access/transportData access/transport

• On-line browse and On-line browse and comparisoncomparison

• (Segue to analysis)(Segue to analysis)


Data discovery == Data discovery == MetadataMetadata + + searchsearch

Mature standards do not exist Mature standards do not exist today. today.

A task for ESMF:A task for ESMF: define and utilize define and utilize metadata standards for modelersmetadata standards for modelers


Metadata for modelersMetadata for modelers

• Reviewed & “published” (a “standard”)Reviewed & “published” (a “standard”)

• Structured (XML)Structured (XML)

• Generated automatically in conjunction Generated automatically in conjunction with setting up model runswith setting up model runs

• Standardized parameter names Standardized parameter names (“controlled vocabularies”)(“controlled vocabularies”)

• HierarchicalHierarchical– components, grids, fields and attributes components, grids, fields and attributes


Others are working hard on search:Others are working hard on search:

1.1. Traditional metadata partnersTraditional metadata partners(e.g. GCMD)(e.g. GCMD)

2.2. Semantic Web (Google on Semantic Web (Google on steroids)steroids)(3 years off?)(3 years off?)

Data discovery == Metadata + Data discovery == Metadata + searchsearch


Data portal components:Data portal components:

• Data discoveryData discovery




Work

flow

Work

flow


Data transportData transport

OPeNDAP (a.k.a. “DODS”)OPeNDAP (a.k.a. “DODS”)– Network data accessNetwork data access– Format-independenceFormat-independence– SubsettingSubsetting– Aggregation (GDS, Unidata)Aggregation (GDS, Unidata)– CompressionCompression– Security: Grid-enabled Security: Grid-enabled

OPeNDAPgOPeNDAPg

OPeNDAP: distributed OPeNDAP: distributed access to data and access to data and semantic metadatasemantic metadata


CF CF (“climate and forecast”)(“climate and forecast”)

CF 1.0 is now a “standard”CF 1.0 is now a “standard”• ““use” metadata – e.g. units, coords.use” metadata – e.g. units, coords.• curvilinear, hybrid-Z, time-dependentcurvilinear, hybrid-Z, time-dependent• great applicability beyond modeling, toogreat applicability beyond modeling, too

Discussion question:Discussion question:As the use of the CF standard widens how should As the use of the CF standard widens how should

the community support it?the community support it?Not enough to “endorse” it. Need a Not enough to “endorse” it. Need a partnership.partnership.







Work

flow

Work

flow

Live Access Live Access ServerServer

netCDnetCDFF


• Metadata (XML) contains the “intelligence”• Back end applications do the real work• OPeNDAP provides remote data access

LAS -- an “Information Product Server”

Internet

2

3

LocalData

LAS productserver(perl)

NetScape

Temp

Dep

th

4

Internet(OPenDAP)

RemoteData

Meta-data

LASUI

1

LAS UIserver(java)

workapplication

Ferret,Ferret,CDATCDAT

or otheror other

InformatioInformationaccessnaccess

Uniform Uniform data data accessaccess


The UI talks to LAS through an XML “web service”

Internet

LocalData

LAS core(perl)

legacyapplication

(Ferret)

Standard

Multiple User Interfaces

Batch

K-12

Meta-data

LASUI

For example …For example …

On-line On-line comparisoncomparison

Example:Example:Average over a lat-long Average over a lat-long boxbox







Work

flow

Work

flow

discover discover browse browse accessaccess

NASA Global Change NASA Global Change Master Directory (GCMD)Master Directory (GCMD)

Metadata Metadata StandardStandard

ss


LAS “sisters” share metadata to form a unified (virtual) site.OPeNDAP allows LAS to difference distributed fields.

Collaborating groups of modelers

LASUI

NetScape

Data

LAS

site 1

Meta

Meta

VIRTUAL server

Data

LAS

Meta

Data

LAS

site 2

Meta

site 4

Data

LAS

Meta

site 3

LASuser

interface

Meta Meta

Meta

A wealth of data products are A wealth of data products are available through the National available through the National Virtual Ocean Data System Virtual Ocean Data System (NVODS)(NVODS)

Example: Zebiac model Example: Zebiac model outputoutput“live” from IRI/LDEO“live” from IRI/LDEO

Access to Access to observations observations

(WODB -- 9 million ocean (WODB -- 9 million ocean profiles)profiles)

configurable configurable constraintsconstraints


Query available data sets:Query available data sets:

Query variables in data set “model_1”:Query variables in data set “model_1”:

Query space-time domain:Query space-time domain:

Request a subset of data as a file:Request a subset of data as a file: (“asc” for ASCII format) (“asc” for ASCII format)

““Batch” access to productsBatch” access to products

>lasls http://cpu/LAS>lasls http://cpu/LAS

>lasls http://cpu/LAS model_1>lasls http://cpu/LAS model_1

>lasls http://cpu/LAS model_1 sst>lasls http://cpu/LAS model_1 sst

>lasget -x 20:60 -y 20:60 -t 11-Dec-2000 -f asc >lasget -x 20:60 -y 20:60 -t 11-Dec-2000 -f asc http://cpu/LAS model_1 sst http://cpu/LAS model_1 sst


IOOSIOOSUS Integrated Ocean Observing US Integrated Ocean Observing SystemSystem

1.1. Detect and Predict Change Detect and Predict Change

2.2. Mitigate natural hazardsMitigate natural hazards

3.3. Improve safety and efficiency of Improve safety and efficiency of marine opsmarine ops

4.4. Ensure national securityEnsure national security

5.5. Reduce public health risksReduce public health risks

6.6. Protect and restore marine ecosystemsProtect and restore marine ecosystems

7.7. Sustain marine resourcesSustain marine resources

IOOS Data Management IOOS Data Management andand

Communications Communications SubsystemSubsystem

UsersUsers

Archive Archive CentersCenters

ModelingModeling

On-line On-line BrowseBrowse

ProductsProducts

ForecastsForecastsMapsMaps

RegionalRegionalData Data

ManagementManagementSystemsSystems

InternationalInternationalData Data

ManagementManagementSystemsSystems

Terrestrial and Terrestrial and AtmosphericAtmospheric

Data ManagementData ManagementSystemsSystems

Metadata, Data DiscoveryMetadata, Data Discoveryand Data Transportand Data Transport

Standards and ProtocolsStandards and Protocols

SatellitesSatellitesShipsShips

Primary DataPrimary DataAssembly & Assembly &

QCQC

Hand Hand MeasurementsMeasurements

MooringsMoorings

FloatsFloats

Recommendations for ESMF

1.1. Define ESMF metadata Define ESMF metadata standard and use itstandard and use it

• Consider a GCMD modelers’ portal, tooConsider a GCMD modelers’ portal, too

2.2. Endorse partnership with GO-Endorse partnership with GO-ESSP and the emerging toolsESSP and the emerging tools

• Data available through OPeNDAP(g)Data available through OPeNDAP(g)

• Live Access Servers for on-line Live Access Servers for on-line collaborationscollaborations

Questions?

NVODS LAS:NVODS LAS:http://www.ferret.noaa.gov/nvods

GCMD “DODS” Portal:GCMD “DODS” Portal:http://gcmd.gsfc.nasa.gov/Data/portals/dodhttp://gcmd.gsfc.nasa.gov/Data/portals/dod

ss

integrated model data management s.hankin esmf july ‘04 integrated data management in the esmf...

Documents