Download - SURA IT Committee Meeting March 22, 2005
SU
RA
IT M
eetin
g –
22 M
arch
200
5
SURA IT Committee MeetingMarch 22, 2005
Sara J. Graves, Ph.D.
Director, Information Technology and Systems Center
University Professor, Computer Science Department
University of Alabama in Huntsville
Director, Information Technology Research Center
National Space Science and Technology Center
256-824-6064
SCOOP Status
SU
RA
IT M
eetin
g –
22 M
arch
200
5
Whereas, the Southeastern Universities Research Association has proposed the creation of an open-access network of distributed sensors, linked via an ultra-fast network to state-of-the-art computing systems that track and model the southeastern coastal zone in real time and provide components of a more comprehensive coastal security infrastructure — known as Southeastern Coastal Ocean Observing Program (SCOOP); now, therefore, be it
Resolved, That the Southern Governors’ Association supports SURA’s Southeastern Coastal Ocean Observing Program to bring more effective protection of lifeand property to the increasingly developed coastal zone, to offer a vehicle for bringingthe extensive and widely dispersed intellectual talent of the ocean sciences community toaddress program of homeland security via an integrated and spatially distributed program,and to aid in addressing the ecological and environmental concerns endangering healthand safety of inhabitants and marine resources.
SU
RA
IT M
eetin
g –
22 M
arch
200
5
SURA’s Southeastern Coastal Ocean Observing Program (SCOOP) will facilitate the assimilation of observational data into community models and provide a distributed data ingestion and support grid with broad band connectivity. This is expected to become a coastal counterpart to the Global Ocean Data Assimilation Experiment (GODAE) with emphasis on the southeast.
SU
RA
IT M
eetin
g –
22 M
arch
200
5Board of Trustees Meeting
Nov 2002
• Data fusion is critical• Modeled and observed fields must have equal
representation• Use GODAE (Global Data Assimilation Experiment)
as a guide for CODAE (Coastal Ocean Data Assimilation Experiment)
• SURA is a strong brand (we should use it)• Focused sub-regional efforts with specified
deliverables which would be new and exciting• Broad SURA effort targeted on building a culture
supporting region-wide collaboration in shared scientific goals
SU
RA
IT M
eetin
g –
22 M
arch
200
5
Integrated Ocean Observing System (IOOS)
Serves national needs for:• Detecting and forecasting oceanic components of
climate variability • Facilitating safe and efficient marine operations • Ensuring national security • Managing resources for sustainable use • Preserving and restoring healthy marine ecosystems • Mitigating natural hazards • Ensuring public health
SU
RA
IT M
eetin
g –
22 M
arch
200
5National Federation of Regional
Systems National Backbone• Satellite remote sensing• In situ sensing reference & sentinel station-network• Link to global ocean component• Data standards & exchange protocols
Regional Systems• Regional priorities• Effects of climate change & Effects of land-based sources Resolution, Variables
SU
RA
IT M
eetin
g –
22 M
arch
200
5
1. A national coastal observing program will necessarily consist of regional and sub-regional components.
2. National, regional and sub-regional observing
systems must consist of three interconnected aspects: (i) spatially distributed sensor arrays; (ii) data management and dissemination hubs; and (iii) nowcasting and forecasting models that are fused with assimilated observational data.
3. The creation and long-term viability of nested integrated and sustained coastal observing systems will depend on a high level of interagency coordination.
Overarching Principles for Coastal Observing Programs
SU
RA
IT M
eetin
g –
22 M
arch
200
5
The SURA Coastal Ocean Observing and Prediction (SCOOP) program is an initiative to create a open-access, distributed national laboratory for scientific research and coastal operations. SCOOP is designed to complement the efforts of both Ocean.US - the organization responsible for implementing the national Integrated Ocean Observing System (IOOS)- and the coastal component of NSF’s Ocean Research Interactive Observatory Networks (ORION) project. The SCOOP emphasis is on interoperability in order to create a real-time observations system for both monitoring and prediction. Through SURA Universities, SCOOP will provide the expertise and IT infrastructure to integrate observing systems that currently exist, and incorporate emerging systems. This will promote the effective and rapid fusion of observed data with numerical models, and facilitate the rapid dissemination of information to operational, scientific, and public and private users.
SCOOP Vision Statement
SU
RA
IT M
eetin
g –
22 M
arch
200
5
1. System of Systems• Ocean Observing = IOOS & OOI & ORION • Coastal Ocean Component of the Global Earth Observing
System of Systems (GEOSS)• Components: (i) Sensor arrays, (ii) Data management &
communication, (iii) Predictive models 2. Distributed National Lab for Research & Applications
• IT Glue...Bricks & Mortar• Research to Operations• Academic & Federal Agency & Industry partnership
3. IT Enabling Big Science• Environmental prediction• Standards enable innovation• Interoperable community solving the really big problems
Overarching Goals for SCOOP
SU
RA
IT M
eetin
g –
22 M
arch
200
5
Planned Capabilities
• Validate accurate and timely short and long-term predictions
• Simultaneous measurements of winds, waves, currents, water density, nutrients, water quality, biological indices, and fish stocks under all conditions
• Focus on storm surge, wind waves, and surface currents, with special attention to predicting and visualizing phenomena that cause damage and inundation of coastal regions during severe storms, hurricanes and possibly tsunamis
• Bridge the gap between scientific research and coastal operations
SU
RA
IT M
eetin
g –
22 M
arch
200
5
SCOOP Science Goals
• Assess and predict the coastal response to extreme atmospheric events – focus on storm surge, flooding & waves
• Modular modeling tools for regional issues (wave coupling, sediment suspension, etc.)
• Standardized interfaces for data and (coupled) model interoperability
• Ensemble prediction – forecasts based on many independent models runs
SU
RA
IT M
eetin
g –
22 M
arch
200
5
SCOOP Research Goals
• Measure, understand and predict environmental conditions
• Provide R&D support for operational agencies including NOAA, the U.S. Navy, and others
• Include outreach and education components that assure relevance of their observing activities
SU
RA
IT M
eetin
g –
22 M
arch
200
5
SCOOP Objectives
• Develop and deploy standards and protocols for data management, exchange, translation and transport
• Implementation of existing standards and protocols (e.g. FGDC, OGC, web services, etc.)
• Application of Grid Technologies• Deployment of the communications infrastructure to
link ocean sensors operating in extreme environmental conditions to people who need timely information
• Cultivation of industry partners
SU
RA
IT M
eetin
g –
22 M
arch
200
5
Coordination is Key
• Ocean.US - National Office for Integrated and Sustained Ocean Observations coordinates development of an operational, integrated and sustained Ocean Observing System (created by NOPP) http://www.ocean.us/
• Integrated Ocean Observation System (IOOS) a national effort to create an Integrated Ocean Observing System http://www.openioos.org/
• National Oceanographic Partnership Program (NOPP) 15 federal agencies providing leadership and coordination of national research and education programs http://www.nopp.org/
• National Federation of Regional Associations provide a framework for orchestrating regional collaborations http://www.usnfra.org/
• NSF Ocean Research Interactive Observatory Networks (ORION) an emerging network of science-driven ocean observing systems http://www.orionprogram.org/default.html
SU
RA
IT M
eetin
g –
22 M
arch
200
5
Interoperability is Key
• Ocean.US Data Management and Communications (DMAC) Plan provides the framework for interoperability http://dmac.ocean.us/dacsc/imp_plan.jsp
• Open Geographic Information Systems (GIS) Consortium (OGC) an open consortium of industry, government, and academia developing interface specifications to support interoperability http://www.opengis.org
• Marine Metadata Interoperability a community effort to make marine metadata easier to find, access and use http://www.marinemetadata.org/
SU
RA
IT M
eetin
g –
22 M
arch
200
5
Interoperability Demonstration
www.openioos.orgwww.openioos.orgwww.openioos.orgwww.openioos.org
NOAA and ONR grant recipients collaborationNOAA and ONR grant recipients collaborationNOAA and ONR grant recipients collaborationNOAA and ONR grant recipients collaboration
SU
RA
IT M
eetin
g –
22 M
arch
200
5
• Funding provided by ONR, NOAA• 2004 List of SCOOP Partners:
SCOOP System Development
• Consortium for Oceanographic Research and Education• Gulf of Maine Ocean Observation System (GoMOOS) • Louisiana State University, Center for Computation &Technology• Louisiana State University, Coastal Studies Institute• Southeast Atlantic Coastal Ocean Observing System (SEACOOS)• Southeast Coastal Ocean Observations Regional Association (SECOORA)• Texas A&M University & Gulf Coast Ocean Observing System (GCOOS)• University of Alabama in Huntsville• University of Delaware (Mid-Atlantic Regional Association (MACOORA)• University of Florida • University of Miami, Center for Southeastern Tropical Advanced Remote Sensing• University of North Carolina• Virginia Institute of Marine Science
SU
RA
IT M
eetin
g –
22 M
arch
200
5
SCOOP Program Elements
1. Data StandardsMetadata standards – compliant with existing and
emerging standards
Standard data models – to facilitate aggregation
2. Data GridOGC Web services – for distributed maps and data
Augmenting with new data, e.g., surface currents
3. Model GridStorm surge & wave prediction
Modular, standardized prediction system
SU
RA
IT M
eetin
g –
22 M
arch
200
5
SCOOP Data Architecture (high level)
NOAA
Web Browsers
GIS Clients
NDBC
Regional Association Data Center (Archive)
OGC Protocols
Other Regional
Association Data Centers
HTTP & HTML
LDM…???
NDBC MODEM
HTTP & HTML
SCOOP Modeling Partners
TBD…???Regional Data
Provider #1
Regional Data
Provider #2
Regional Data
Provider #N
SCOOP Modeling Partners
TBD…???
Transport Mediums
SU
RA
IT M
eetin
g –
22 M
arch
200
5SCOOP Prediction System
All Versions
1. Standard naming conventions – Adopt existing community standards where appropriate (e.g., CF or NCEP) and add our own conventions only when necessary.
2. Mechanisms for tracking metadata, e.g., provenance, forcing, source of OBCs, forcing used to create OBCs, etc.
3. Portals – entry point for access to models & model output. Deals with authentication & authorization.
SU
RA
IT M
eetin
g –
22 M
arch
200
5
SCOOP Prediction System, Version 1.0
• Modular wind forcing • Modular embedded regional models• Coupled models – for existing groups• Using existing computational resources• Verification – real time model-data comparisons• Model-GIS interface & OGC Web services• Web mapping with roads, etc.• Web mapping with time sequences (WMS)• Standardized time-series verification• Openioos.org for displaying results• Other…?
SU
RA
IT M
eetin
g –
22 M
arch
200
5
SCOOP Operational Prediction System Version 1.0
Regional Model Center #1
Operational Wave Predictions
(BIO/GoMOOS)
Operational Tide/Surge Predictions (SABLAM)
Coupled Wave-Surge Predictions
(Miami)
Large Scale Response
NOAA/NCEP
(ETA)
NOAA/NCEP/UNC?
(EDAS) Archive
Enhanced Winds
(Miami)
Forcing
Regional Response
Standardize model interfaces
Regional Model Center #2
Standardize Transport/Encapsulation XML, FTP, LDM,
OPeNDAP…?
SU
RA
IT M
eetin
g –
22 M
arch
200
5
SCOOP Verification & Visualization
Prediction System
Standardize model-GIS interfaces
Regional Web Server #1
Modeling Center #1 (Regional or otherwise)
Modeling Center #2 (ditto)
Regional Data Center #1
Information Providers
Regional Web Server #2
OGC, RSS…?
Regional Data Center #2
Standardize verification tools & data
Data System
SU
RA
IT M
eetin
g –
22 M
arch
200
5
Prediction System Task Elements for Version 1.0
Task: Lead Partner:Data standards TAMUData transport UAHData translation & mgmt UAHCoupled modeling MiamiNested Modeling VIMSCustomized configuration TAMUVisualization Services LSUVerification & validation MiamiComputing & storage resources LSUSecurity TAMUGrid management middleware LSUWeb Mapping Demonstration GoMOOS
SU
RA
IT M
eetin
g –
22 M
arch
200
5
Users, Modeling Partners, other Data Centers, etc.Regional Association Data CentersRegional Data Providers
Data Provider
Data Translation
Services
Data Provider
SCOOP Data Architecture:High level Services
Metadataonly
dataData andMetadata
data
data
Modeler /Data Provider
ObservationData
Model
Data Access Services
Metadata Services
SCOOPCatalog
data
Archive/Repository Broker
Data Access Services
Model orApplication
User InterfaceGeoSpatial OneStop/ FGDC
Clearinghouse
SU
RA
IT M
eetin
g –
22 M
arch
200
5
Regional Association Data CentersRegional Data Providers
Data Provider
Data Provider
SCOOP Data Architecture Specifics:Data Acquisition – example technologies to support dynamic transport and metadata cataloging
Metadataonly
dataMetadataand Data
data
data
Modeler /Data Provider
ObservationData
Model
Data Access Services
Metadata Services
data
Archive/Repository Broker
Data Access Services
Data TransportMetadata Cataloging
LDM
e.g., LDM
e.g., XML, Metadata Harvest
SCOOPCatalog
SU
RA
IT M
eetin
g –
22 M
arch
200
5
Users, Modeling Partners, other Data Centers, etc.
Data Translation
Services
data
Modeler /Data Provider
FTP OPeNDAP OGC
Model orApplication
User Interface
GeoSpatial OneStop/ FGDC
Clearinghouse
SCOOP Data Architecture Specifics:Data Discovery & Access – example technologies to support dynamic transport, analysis and
visualization
Metadata Query Services
data
Archive/Repository Broker
FTP OPeNDAP OGC
ESML
IOOS Interoperability
Demo
Z39.50
SOAP
OGC WMS
XML
HTTP
FTP
DataDiscovery
DataAccess
Regional Association Data Centers
RegionalData Providers
SCOOPCatalog
SU
RA
IT M
eetin
g –
22 M
arch
200
5
data
Modeler /Data Provider
User
GeoSpatial OneStop
SCOOP Information Architecture:Example metadata exchange technologies to support Data Discovery
IOOS Interoperability Demo
Z39.50XML
Regional Association Data/Service Centers
RegionalData Providers
Data Provider
data
LocalMetadata
SCOOPData Dictionary
SCOOP Catalog
SOAP
OGC
Get Capabilities
MetadataHarvest
MetadataHarvest
SCOOP Interactive Search U/I Model or
Application
FGDC Records
HTTPWMS data list
& metadata
Automated Data Discovery
SOAP
SOAP
SOAP
XML ?
MetadataHarvest
Metadata Services
DataDiscovery
Metadata Population
Users, Modeling Partners, other Data Centers, etc.
Ingest Svcs Query SvcsManual Updates
SCORE
SU
RA
IT M
eetin
g –
22 M
arch
200
5
SCORE: Accomplishments & Plans
• SCORE is the catalogs and services infrastructure for SCOOP data management
• Data & Model Survey provided initial snapshot of partners’ data (observations and model results)
• Developed database schema for SCORE to support– Strawman SCOOP Catalog: requesting input on improved capabilities – IOOS demo: working with GoMOOS team to integrate catalog with
demo– FGDC Clearinghouse to support Geospatial One-Stop: plan to create
FGDC metadata records from SCORE
• Issues– What data management functionality is needed within SCOOP?
• Metadata services for data collections, data files/streams, general model information, information on specific model runs,…
– How to coordinate metadata and data management across sites?– How to automate population of SCORE?
SU
RA
IT M
eetin
g –
22 M
arch
200
5
1. Environmental Prediction• Prediction systems fuse models & observations • Nonlinear dynamics limits predictability – Lorentz’s seagull• Probability and statistics – ensemble modeling
2. Hurricane Surge & Waves• Biggest uncertainty in the winds• Ensemble of winds: different models or different simulations• New paradigm & new metrics for skill assessment
3. Research to Operations• Improving upon SLOSH – a good idea 30 years ago• GIS compatibility enabling application & visualization• OpenIOOS.org is the high visibility “front end”
Science Goals for Version 2.0
SU
RA
IT M
eetin
g –
22 M
arch
200
5
RegionalArchives
Ensemble wind fields from varied and distributed
sources
ADCirc
ElCirc
WAM/SWAN
Ensemble of models run
across distributed resources
Archive
Verification
Visualization
Analysis, storage and cataloguing of output data
Select region and time range
Transform and
transport data
Wind Forcing
Wave and/or Surge Models
Result Dissemination
Synthetic Wind Ensemble
NCEP
MM5
NCARor
orOpenIOOS
Version 2.0
SU
RA
IT M
eetin
g –
22 M
arch
200
5SCOOP: Data-to-Model (D2M) Realtime
Transport and Translation (Nested Model) Scenario
UNCTAMU
VIMSTAMU, UF,
Others?
ADCIRC
MM5Translation Services• subset• subsample• re-format• re-grid
ELCIRCModel X
LDM, OPeNDAP, FTPPush/pull
POC: Rick Luettich, Brian Blanton
POC: Gerry Creager
POC: Harry Wang
LDM-push
POC: Matt Smith, Ken Keiser (UAH)
LDM-push
Water levels
NCEP (NAM)Wind Forecasts
Atmospheric Models Regional Oceanic/Coastal Models
Localized Models, Users and Archives
High-Res Wind Forecasts LDM-push
D2M Node
TranslatedWater levels
(1)
(2)
(3)
(4)
(1a)WRF(future)
(1) Atmospheric Model products are “translated” through D2M to the form requested by the client model. Currently, using ftp-pull, all NAM grids 0-84h for the 4 runs (00, 06, 12, 18 UTC) of AWIP12 and AWIP32 are sent to a D2M node and translated.
(2) Via LDM, UNC, TAMU, & UF have access to the raw and translated model data.
(3) Partners use translated ob/model data in their models. Then push their results to a D2M node. Currently, ADCIRC output files (text and netCDF) are being pushed to a D2M node (for translation) and other modeling partners via LDM.
(4) Resulting translated data products area pushed to a client model’s site and made available for other transport vehicles (FTP, OPeNDAP, OGC, etc) for use in retrospective studies and other applications. Likewise the output of other models can be processed through D2M for translation steps requested by other client models.
TranslatedWinds and fluxes
Translation Services• subset• subsample• re-format• re-grid
D2M Node
ESML
ESML
LDM-push
LDM-push (FTP-pull)
Alternate
SU
RA
IT M
eetin
g –
22 M
arch
200
5
Data Management Goals
Version 1• Provided a high-level data catalog for SCOOP data discovery, providing
descriptions of partner data holdings and pointers to partner data access points (web, ftp, OPeNDAP, etc.)
– Based initial catalog on Data & Model Survey results• Coordinated with Data Transport (Task 2) to develop initial LDM network to
exchange data in near real time among SCOOP partners.• Coordinated with Data Standards (Task 1) on development of metadata
keywords for SCOOP
Version 2• Expand SCOOP data discovery capabilities based on evolving data
management practices of SCOOP partners. – Support IOOS Demo – Field an FGDC Clearinghouse node for SCOOP
• Monitor Marine Metadata Interoperability activities and their potential interaction with SCOOP
– Assist SCOOP partners in developing standard metadata to describe their data collections
• Continue coordination with all partners on data management issues
SU
RA
IT M
eetin
g –
22 M
arch
200
5
GLOBE AMSU-A KnowledgeBase
ITSC
Coastlines
Countries
MCS Events
Cyclone EventsAMSU-A Channel 01
AMSU-A data overlaid with MCS and Cyclone events, merged with world boundaries from GLOBE.
Merged data product for on-demand visualization
Distributed Data Integration
SU
RA
IT M
eetin
g –
22 M
arch
200
5Heterogeneity Leads to Data Usability
Problems
Science Data Characteristics• Many different
formats, types and structures (18 and counting for atmospheric science alone!)
• Different states of processing (raw, calibrated, derived, modeled or interpreted)
• Enormous volumes
SU
RA
IT M
eetin
g –
22 M
arch
200
5
Interoperability: Accessing Heterogeneous Data
The Problem
DATA FORMAT 1
DATA FORMAT 1
DATA FORMAT 2
DATA FORMAT 2
DATA FORMAT 3
DATA FORMAT 3
READER 1 READER 2
FORMATCONVERTER
APPLICATION
ESML LIBRARY
APPLICATION
DATA FORMAT 1
DATA FORMAT 1
DATA FORMAT 2
DATA FORMAT 2
DATA FORMAT 3
DATA FORMAT 3
The Solution
ESMLFILEESMLFILE
ESMLFILEESMLFILE
ESMLFILEESMLFILE
One approach: Enforce a standard data format, but…
• Difficult to implement and enforce• Can’t anticipate all needs• Some data can’t be modeled or is lost in
translation• Converting legacy data is costlyA better approach: Interchange Technologies• Earth Science Markup Language
SU
RA
IT M
eetin
g –
22 M
arch
200
5
What is ESML?
• It is a specialized markup language for Earth Science metadata based on XML - NOT another data format.
• It is a machine-readable and -interpretable representation of the structure, semantics and content of any data file, regardless of data format
• ESML description files contain external metadata that can be generated by either data producer or data consumer (at collection, data set, and/or granule level)
• ESML provides the benefits of a standard, self-describing data format (like HDF, HDF-EOS, netCDF, geoTIFF, …) without the cost of data conversion
• ESML is the basis for core Interchange Technology that allows data/application interoperability
• ESML complements and extends data catalogs such as FGDC and GCMD by providing the use/access information those directories lack.
http://esml.itsc.uah.edu
SU
RA
IT M
eetin
g –
22 M
arch
200
5ESML IN ACTION:
Ingest surface skin temperature data in Numerical Models
ReanalysisGRIB filesReanalysisGRIB files
MM5MM5 GOESGOES
ESML file
ESMLfile
ESMLfile
http://vortex.nsstc.uah.edu/~sud/web/default.htm
ESML LibraryNUMERICAL WEATHER
MODELS (MM5, ETA, RAMS)
Scientists can:• Select remote files
across the network• Select different
observational data to increase the model prediction accuracy
Purpose:• Use ESML to incorporate
observational data into the numerical models for simulation
• Skin temperatures come in a variety of data formats
• GOES – McIDAS• Reanalysis Data - GRIB • MM5 Model - Binary • AVHRR – HDF• MODIS - EOS-HDF