the hdf product designer – interoperability in the first mile
TRANSCRIPT
The HDF Group
www.hdfgroup.orgDecember 17, 2014 American Geophysical Union Fall Meeting
HDF Product Designer: Interoperability in the First Mile
H. Joe Lee ([email protected]),Aleksandar Jelenak, and Ted HabermannThe HDF Group
www.hdfgroup.orgAmerican Geophysical Union Fall Meeting
Data Life Cycle – First and Last Miles
Archive Discovery AnalysisQuestion Processing Distribution
Repurposing
DataCollection
www.hdfgroup.orgAmerican Geophysical Union Fall Meeting
Data Life Cycle – First and Last Miles
Archive Discovery AnalysisQuestion Processing Distribution
Principal Investigator Someone Else
Repurposing
DataCollection
www.hdfgroup.orgAmerican Geophysical Union Fall Meeting
Data Life Cycle – First and Last Miles
Archive Discovery AnalysisQuestion Processing Distribution
Principal Investigator Someone Else
Repurposing
DataCollection
Experts Non-Experts
www.hdfgroup.orgAmerican Geophysical Union Fall Meeting
Data Life Cycle – First and Last Miles
Archive Discovery AnalysisQuestion Processing Distribution
Principal Investigator Someone Else
Repurposing
DataCollection
# Users
Experts Non-Experts
www.hdfgroup.orgAmerican Geophysical Union Fall Meeting
Data Life Cycle – First and Last Miles
Archive Discovery AnalysisQuestion Processing Distribution
Principal Investigator Someone Else
Repurposing
DataCollection
# Users
Experts Non-Experts
Standards and Conventions
www.hdfgroup.orgAmerican Geophysical Union Fall Meeting
Data Life Cycle – First and Last Miles
Archive Discovery AnalysisQuestion Processing Distribution
Principal Investigator Someone Else
Repurposing
DataCollection
# Users
Experts Non-Experts
? Standards and Conventions
www.hdfgroup.orgAmerican Geophysical Union Fall Meeting
Data Life Cycle – First and Last Miles
Archive Discovery AnalysisQuestion Processing Distribution
Principal Investigator Someone Else
Repurposing
DataCollection
To facilitate collaborative design of interoperable and standards-compliant data products in HDF5 as early as possible in the mission development process.
www.hdfgroup.orgAmerican Geophysical Union Fall Meeting
Data Life Cycle – First and Last Miles
Archive Discovery AnalysisQuestion Processing Distribution
Principal Investigator Someone Else
Repurposing
DataCollection
To facilitate collaborative design of interoperable and standards-compliant data products in HDF5 as early as possible in the mission development process.
Interoperability in the First Mile
www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
Mission Data Producer’s Conundrum
Mission Requirements
• Science objectives• Data processing• Data discovery & distribution• Data documentation• User engagement,
preparedness, feedback
www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
Mission Data Producer’s Conundrum
Interoperability
Mission Requirements
• Standards• Conventions• Best Practices• Metadata• Software Tools• netCDF4, CF
• Science objectives• Data processing• Data discovery & distribution• Data documentation• User engagement,
preparedness, feedback
www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
Mission Data Producer’s Conundrum
HDF FeaturesInteroperability
Mission Requirements
• Datatypes• Groups
• Attributes• Scale/offset
• Dimension scales• Compression
• Chunking
• Standards• Conventions• Best Practices• Metadata• Software Tools• netCDF4, CF
• Science objectives• Data processing• Data discovery & distribution• Data documentation• User engagement,
preparedness, feedback
www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
Mission Data Producer’s Conundrum
HDF FeaturesInteroperability
Mission Requirements
HDF Product Designer
• Datatypes• Groups
• Attributes• Scale/offset
• Dimension scales• Compression
• Chunking
• Standards• Conventions• Best Practices• Metadata• Software Tools• netCDF4, CF
• Science objectives• Data processing• Data discovery & distribution• Data documentation• User engagement,
preparedness, feedback
www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
Collaboration
Individuals
www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
Collaboration
IndividualsTeams
www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
Collaboration
IndividualsTeamsProjectsPrograms
www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
HDF5 Product Design Architecture
Desktop Client
www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
HDF5 Product Design Architecture
Desktop Client
Restful Service
(Tornado/Python)
www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
HDF5 Product Design Architecture
Data Store(PostgreSQL)
Desktop Client
Restful Service
(Tornado/Python)
HDF5Server
www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
HDF5 Product Design Architecture
Data Store(PostgreSQL)
Desktop Client
Restful Service
(Tornado/Python)
HDF5JSON
HDF4 MAPXML
NcML HDF5Server
FlexibleInput
www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
HDF5 Product Design Architecture
Data Store(PostgreSQL)
Desktop Client
HDF5JSON
Fortran IDL MATLAB Python
Restful Service
(Tornado/Python)
CSV (Excel)
HDF5JSON
HDF4 MAPXML
NcML HDF5Server
HDF5 File Template
FlexibleInput
Flexible Output
www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
Convention Support
• Initial:• NetCDF User’s Guide (NUG)• Attribute Convention for Data Discovery (ACDD)• Object Convention for Data Discovery (OCDD)• Climate and Forecast (CF)• HDF-EOS
• Implementation:• Conventions for groups and variables• Convention for connected variables (e.g.
coordinate dimensions)• Compliance checkers on entire file
• Support for community components
www.hdfgroup.orgAmerican Geophysical Union Fall MeetingDecember 17, 2014
Conclusion
• enable individuals and mission teams to design products quickly and easily
• enable collaboration at many levels• promote data management best practices
HDF Product Designer is being built using a flexible architecture to support multiple front and back ends. It will: