a distributed approach to ocean, atmosphere & climate model data interoperability rich signell...
TRANSCRIPT
A Distributed Approach to A Distributed Approach to Ocean, Atmosphere & Climate Ocean, Atmosphere & Climate
Model Data InteroperabilityModel Data Interoperability
A Distributed Approach to A Distributed Approach to Ocean, Atmosphere & Climate Ocean, Atmosphere & Climate
Model Data InteroperabilityModel Data InteroperabilityRich Signell
USGS Woods Hole, MACOAWST Training Workshop
Woods Hole, MA: Aug 25-28, 2014
Rich SignellUSGS Woods Hole, MA
COAWST Training Workshop
Woods Hole, MA: Aug 25-28, 2014
22
US Integrated Ocean Observing System (IOOS®)
IOOSIOOS®® Plan defines: Plan defines:• Global ComponentGlobal Component• Coastal ComponentCoastal Component
17 Federal Agencies17 Federal Agencies 11 Regional Associations 11 Regional Associations
The problem: lots of unique (stovepipe) web sites
The GoMOOS Nowcast/Forecast Circulation Model (University of Maine)
IOOS Core PrinciplesIOOS Core Principles
• Adopt open standards & practices
• Avoid customer-specific stovepipes
• Standardized access services implemented at data providers
• Adopt open standards & practices
• Avoid customer-specific stovepipes
• Standardized access services implemented at data providers
55
CustomerCustomer
Web accessWeb accessserviceservice
DataDataProviderProviderObservationsObservations
ModelsModels
Issue: Ocean grids are not regularly spaced!Issue: Ocean grids are not regularly spaced!
Stretched surface and terrain following vertical coordinates
Curvilinear orthogonal horizontal coordinates
NetCDF Climate and Forecast (CF) Conventions provide a solution
NetCDF Climate and Forecast (CF) Conventions provide a solution
Groups using CF:
GO-ESSP: Global Organization for Earth System Science Portal
IOOS: Integrated Ocean Observing System
ESMF: Earth System Modeling Framework
OGC: Open Geospatial Consortium (GALEON: WCS profile)
CF-Compliant ROMS fileCF-Compliant ROMS file
double s_rho(s_rho) ; s_rho:long_name = "S-coordinate at RHO-points" ; s_rho:valid_min = -1. ; s_rho:valid_max = 0. ; s_rho:positive = "up" ; s_rho:standard_name = "ocean_s_coordinate_g1" ; s_rho:formula_terms = "s: s_rho C: Cs_r eta: zeta depth: h depth_c: hc" ; s_rho:field = "s_rho, scalar" ;
float temp(ocean_time, s_rho, eta_rho, xi_rho) ; temp:_FillValue = 1.e+37f ; temp:long_name = "potential temperature" ; temp:units = "Celsius" ; temp:time = "ocean_time" ; temp:coordinates = "lon_rho lat_rho s_rho ocean_time" ;
IOOS Model Data Interoperability DesignIOOS Model Data Interoperability Design
ROMS
ADCIRC
HYCOM
SELFE
NCOM NcML
NcML
NcML
NcML
NcML
Common Data Model
OPeNDAP+CF
WCS
NetCDF Subset
THREDDS Data Server
Standardized (CF-1.6, UGRID-0.9) Virtual Datasets
NonstandardModel Output Data Files
Web Services Matlab
Panoply
IDV
Clients
NetCDF-Java
Libraryor Broker
WMS
ncISO
ArcGIS
NetCDF4-Python
FVCOM
PythonERDDAP
NetCDF-Java
SOS
Geoportal Server
GeoNetwork
GI-CAT
Observed data (buoy, gauge, ADCP, glider)
Godiva2
CKANNcML
GridUgridTimeSeriesProfileTrajectoryTimeSeriesProfile
NonstandardData Files
CatalogServices
WMS Browsing with THREDDS/ncWMSWMS Browsing with THREDDS/ncWMS
Matlab NCTOOLBOXhttps://github.com/nctoolbox
Matlab NCTOOLBOXhttps://github.com/nctoolbox
• Objective: Make it simple to access CF data• Example function:• [t, geo]=nj_tslice(URL,’temp’,1);• t = 22x120x180 single• geo =
– lat: [120x180 single] – lon: [120x180 single]– z: [22x120x180 double]– time: 733582 (matlab datenum)
• nj_tslice works identically for ROMS, POM, ECOM, WRF, Wavewatch3
• URL can be: local NetCDF, remote NetCDF, NcML, OpenDAP Data URL
• Objective: Make it simple to access CF data• Example function:• [t, geo]=nj_tslice(URL,’temp’,1);• t = 22x120x180 single• geo =
– lat: [120x180 single] – lon: [120x180 single]– z: [22x120x180 double]– time: 733582 (matlab datenum)
• nj_tslice works identically for ROMS, POM, ECOM, WRF, Wavewatch3
• URL can be: local NetCDF, remote NetCDF, NcML, OpenDAP Data URL
Comparing Models with Data in MatlabComparing Models with Data in Matlab
Model 1: UMASS-ECOMModel 1: UMASS-ECOM Model 2: UMAINE-POMModel 2: UMAINE-POM Data: SST 2008-Sep-08 07:32Data: SST 2008-Sep-08 07:32
Skidaway modena glider (Sep 11-23)Skidaway modena glider (Sep 11-23)
compare_secoora_model_sections.m(using nc_genslice.m)
compare_secoora_model_sections.m(using nc_genslice.m)
Buoy_comp.mBuoy_comp.m
3D visualization of data with IDV3D visualization of data with IDV
NetCDF Subset Service to CSVNetCDF Subset Service to CSV
CSV Time Series from Subset ServiceCSV Time Series from Subset Service
CSV request is a “RESTful” URL:http://geoport.whoi.edu/thredds/ncss/grid/coawst_4/use/fmrc/coawst_4_use_best.ncd?var=Hwave&latitude=39.5&longitude=-69.5&time_start=2014-08-27T01:00:00Z&time_end=2014-08-28T00:00:00Z&vertCoord=&accept=csv
Accessing the CSV data in ExcelAccessing the CSV data in Excel
Access the CSV data in Python, R, etcAccess the CSV data in Python, R, etc
Iris Python tools from the UK Met OfficeIris Python tools from the UK Met Office
ESPreSSO model (Rutgers)ESPreSSO model (Rutgers)
Global RTOFS (NOAA)Global RTOFS (NOAA)
IOOS Models Notebook on WakariIOOS Models Notebook on Wakari
60 minutes to serving your data60 minutes to serving your data
1. Install Sun/Oracle Java (10 min)2. Install/configure Tomcat (15 min)3. Install/configure the Thredds Data
Server (15 min)4.Put NetCDF (or HDF4, Grib1, Grib2,
HDF5 files) in a directory (5 min)5.Add NcML (XML) files for CF
compliance and aggregation (15 min)
Adding data to THREDDSAdding data to THREDDS
rsignell@gam:/usgs/data0/bbleh/tidal$ ls *wide*…his_bbleh_wide_0048.nc his_bbleh_wide_0100.nc his_bbleh_wide_0152.nchis_bbleh_wide_0049.nc his_bbleh_wide_0101.nc his_bbleh_wide_0153.nchis_bbleh_wide_0050.nc his_bbleh_wide_0102.nc his_bbleh_wide_0154.nchis_bbleh_wide_0051.nc his_bbleh_wide_0103.nc wide.ncml
$more wide.ncml
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2"> <aggregation dimName="ocean_time" type="joinExisting"> <scan location="." regExp=".*wide.*_[0-9]{4}\.nc$"/> </aggregation></netcdf>
Browsing WMS with Godiva2Browsing WMS with Godiva2
GI-CAT catalog broker service architectureGI-CAT catalog broker service architecture
Searching for DataSearching for Data
USGS COAWST East Coast ForecastUSGS COAWST East Coast Forecast
Automated model comparisonAutomated model comparison
Summary (1 of 2)Summary (1 of 2)
• Common data models for “feature types” (structured and unstructured grids, time series, profiles, swaths) (Unidata CDM)
• Standard web services for delivering these data and metadata (OGC, Unidata)
• Tools to access and process these services in common analysis environments: R, Matlab, Python, ArcGIS, JavaScript
Summary (2 of 2)Summary (2 of 2)
• It’s easy (1 hour) to deploy free, supported systems that allow for standards-based delivery of aggregated data from native model grids that put little effort on the data provider
• What do you get? – Lots of choices for data access (Browser, Matlab, Python, Excel, IDV,
R, IDL)– More usage of model results by more people– Faster feedback to modelers, leading to improved models– Shared code base in the community– Increased community support for standards-based access– Less time wasted messing with data, more time spent on ecosystem
based management• What should you do? Encourage providers to use these
standards, and develop tools that use standardized access
• It’s easy (1 hour) to deploy free, supported systems that allow for standards-based delivery of aggregated data from native model grids that put little effort on the data provider
• What do you get? – Lots of choices for data access (Browser, Matlab, Python, Excel, IDV,
R, IDL)– More usage of model results by more people– Faster feedback to modelers, leading to improved models– Shared code base in the community– Increased community support for standards-based access– Less time wasted messing with data, more time spent on ecosystem
based management• What should you do? Encourage providers to use these
standards, and develop tools that use standardized access
It’s all on GitHub + rsignellIt’s all on GitHub + rsignell
Glossary (1/6)Glossary (1/6)
• CF Conventions: NetCDF Climate and Forecast Metadata Conventionshttp://cf-pcmdi.llnl.gov/(Standards for identifying geospatial coordinates in NetCDF files andgrid relationships)
• DMAC: Data Management And Communications subsystem http://www.ioos.gov/data/dmac/welcome.html (One of three subsystems of IOOS, providing infrastructure for the Observing and the Modeling and Applications subsystems)
• ECOM: Estuarine, Coastal, and Ocean Model. http://www.hydroqual.com/ehst_env_hyd.html(3D ocean circulation model developed principally by Alan Blumberg)
• ERDDAP: Environmental Research Division's Data Access Programhttp://coastwatch.pfeg.noaa.gov/erddap/index.html(A web service that aggregates data from diverse remote sources and offers a simple, consistent way to access the data)
• CF Conventions: NetCDF Climate and Forecast Metadata Conventionshttp://cf-pcmdi.llnl.gov/(Standards for identifying geospatial coordinates in NetCDF files andgrid relationships)
• DMAC: Data Management And Communications subsystem http://www.ioos.gov/data/dmac/welcome.html (One of three subsystems of IOOS, providing infrastructure for the Observing and the Modeling and Applications subsystems)
• ECOM: Estuarine, Coastal, and Ocean Model. http://www.hydroqual.com/ehst_env_hyd.html(3D ocean circulation model developed principally by Alan Blumberg)
• ERDDAP: Environmental Research Division's Data Access Programhttp://coastwatch.pfeg.noaa.gov/erddap/index.html(A web service that aggregates data from diverse remote sources and offers a simple, consistent way to access the data)
Glossary (2/6)Glossary (2/6)
• ESMF: Earth System Modeling Frameworkhttp://www.esmf.ucar.edu/
(Software for building and coupling weather, climate, and related models)• GO-ESSP: Global Organization for Earth System Science Portal
http://go-essp.gfdl.noaa.gov/(Collaboration designed to develop a new generation of software infrastructure that will provide distributed access to observed and simulated data from the climate and weather communities)
• IDV: Integrated Data Viewerhttp://www.unidata.ucar.edu/software/idv/(Java-based software for analyzing and visualizing geoscience data)
• IOOS: Integrated Ocean Observing System http://ioos.noaa.gov/(Multidisciplinary system designed to collect, deliver, and use ocean information)
• IOOC: Interagency Ocean Observing Committeehttp://www.iooc.us/
• ESMF: Earth System Modeling Frameworkhttp://www.esmf.ucar.edu/
(Software for building and coupling weather, climate, and related models)• GO-ESSP: Global Organization for Earth System Science Portal
http://go-essp.gfdl.noaa.gov/(Collaboration designed to develop a new generation of software infrastructure that will provide distributed access to observed and simulated data from the climate and weather communities)
• IDV: Integrated Data Viewerhttp://www.unidata.ucar.edu/software/idv/(Java-based software for analyzing and visualizing geoscience data)
• IOOS: Integrated Ocean Observing System http://ioos.noaa.gov/(Multidisciplinary system designed to collect, deliver, and use ocean information)
• IOOC: Interagency Ocean Observing Committeehttp://www.iooc.us/
Glossary (3/6)Glossary (3/6)
• NcWMS: Web Map Servicehttp://www.resc.rdg.ac.uk/trac/ncWMS/(ncWMS is a Web Map Service for geospatial data that are stored in CF-compliant NetCDF files. )
• NcML: NetCDF Markup Languagehttp://www.unidata.ucar.edu/software/netcdf/ncml/(Used to standardize and aggregate non-standard NetCDF files)
• NetCDF-Java: NetCDF-Java Libraryhttp://www.unidata.ucar.edu/software/netcdf-java/(Library that takes advantage of CF conventions, NcML to facilitatebuilding standards-based clients)
• NcWMS: Web Map Servicehttp://www.resc.rdg.ac.uk/trac/ncWMS/(ncWMS is a Web Map Service for geospatial data that are stored in CF-compliant NetCDF files. )
• NcML: NetCDF Markup Languagehttp://www.unidata.ucar.edu/software/netcdf/ncml/(Used to standardize and aggregate non-standard NetCDF files)
• NetCDF-Java: NetCDF-Java Libraryhttp://www.unidata.ucar.edu/software/netcdf-java/(Library that takes advantage of CF conventions, NcML to facilitatebuilding standards-based clients)
Glossary (4/6)Glossary (4/6)
• OPeNDAP: Open-source Project for a Network Data Access Protocolhttp://opendap.org(allows efficient hyperslabbing of geospatial data)
• POM: Princeton Ocean Modelhttp://www.aos.princeton.edu/WWWPUBLIC/htdocs.pom/(Sigma coordinate, free surface ocean model with embedded turbulence and wave sub-models, and wet-dry capability)
• RCOOS: Regional Coastal Ocean Observing System• http://www.ioosassociation.org/
(Generic term for one of the 11 regional observing systems in IOOS)
• OPeNDAP: Open-source Project for a Network Data Access Protocolhttp://opendap.org(allows efficient hyperslabbing of geospatial data)
• POM: Princeton Ocean Modelhttp://www.aos.princeton.edu/WWWPUBLIC/htdocs.pom/(Sigma coordinate, free surface ocean model with embedded turbulence and wave sub-models, and wet-dry capability)
• RCOOS: Regional Coastal Ocean Observing System• http://www.ioosassociation.org/
(Generic term for one of the 11 regional observing systems in IOOS)
Glossary (5/6)Glossary (5/6)
• ROMS: Regional Ocean Modeling System
www.myroms.org
(A free-surface, terrain-following, primitive equations ocean model widely used by the scientific community for a diverse range of applications)
• SCCOOS: Southern California Coastal Ocean Observing System
http://www.sccoos.org/
(Brings together coastal observations in the Southern California Bight to provide information necessary to address issues in climate change, ecosystem preservation and management, coastal water quality, maritime operations, coastal hazards and national security. One of the 11 IOOS regions.)
• THREDDS (Thematic Realtime Environmental Distributed Data Services)
http://www.unidata.ucar.edu/projects/THREDDS
(Services for delivering standards-based geospatial data via OpenDAP,
WCS and more, works with NcML)
• ROMS: Regional Ocean Modeling System
www.myroms.org
(A free-surface, terrain-following, primitive equations ocean model widely used by the scientific community for a diverse range of applications)
• SCCOOS: Southern California Coastal Ocean Observing System
http://www.sccoos.org/
(Brings together coastal observations in the Southern California Bight to provide information necessary to address issues in climate change, ecosystem preservation and management, coastal water quality, maritime operations, coastal hazards and national security. One of the 11 IOOS regions.)
• THREDDS (Thematic Realtime Environmental Distributed Data Services)
http://www.unidata.ucar.edu/projects/THREDDS
(Services for delivering standards-based geospatial data via OpenDAP,
WCS and more, works with NcML)
Glossary (6/6)Glossary (6/6)
• WCS: Web Coverage Servicehttp://www.opengeospatial.org/standards/wcs(OGC standard for serving gridded information: time series, profiles,hyperslabs of remote sensing data, bathymetry, model output)
• WW3: WaveWatch III
http://polar.ncep.noaa.gov/waves/wavewatch/wavewatch.html
(Ocean wave model developed by Hendrik Tolman)• WRF: Weather Research and Forecasting Model
http://www.wrf-model.org/index.php
(Next-generation mesocale numerical weather prediction system designed to serve both operational forecasting and atmospheric research needs)
• WCS: Web Coverage Servicehttp://www.opengeospatial.org/standards/wcs(OGC standard for serving gridded information: time series, profiles,hyperslabs of remote sensing data, bathymetry, model output)
• WW3: WaveWatch III
http://polar.ncep.noaa.gov/waves/wavewatch/wavewatch.html
(Ocean wave model developed by Hendrik Tolman)• WRF: Weather Research and Forecasting Model
http://www.wrf-model.org/index.php
(Next-generation mesocale numerical weather prediction system designed to serve both operational forecasting and atmospheric research needs)