hdf5 for npoess data products

21
HDF5 for NPOESS Data Products Alan M. Goldberg The MITRE Corporation [email protected] Organization: W803 Project: 1400NT01-SE This work was performed under NOAA contract 50-SPNA-9-00010. Opinions expressed are those of the author, and do not represent the United States Government. © 2002 The MITRE Corporation. All rights reserved.

Upload: the-hdf-eos-tools-and-information-center

Post on 12-May-2015

60 views

Category:

Technology


2 download

DESCRIPTION

HDF-EOS Workshop VI (2002) Source: http://hdfeos.org/workshops/ws06/presentations/Goldberg/HDF5_for_NPOESS.ppt

TRANSCRIPT

Page 1: HDF5 for NPOESS Data Products

HDF5 for NPOESS Data Products

Alan M. GoldbergThe MITRE Corporation [email protected]

Organization: W803Project: 1400NT01-SE

This work was performed under NOAA contract 50-SPNA-9-00010.Opinions expressed are those of the author, and do not represent the

United States Government.

© 2002 The MITRE Corporation. All rights reserved.

Page 2: HDF5 for NPOESS Data Products

The NPOESS Program

● Products● Requirements● HDF Implementation (Presentation by Chad Fox / Raytheon)

Page 3: HDF5 for NPOESS Data Products

Data Flow OverviewField Terminals and Centrals

Application Processing,Display, & Dissemination

FT ProcessorElement

CentralCentral Element

NPOESS-UniqueProcessing

RFElectronics

Demodulator

C3S/DRR

SpaceSegment

NPOESS-UniqueCollection

Antenna

UserTerminal

IDPS External I/FOtherData

Page 4: HDF5 for NPOESS Data Products

Data Delivery to Users

● Centrals *- National Environmental Satellite, Data, and Information

Service- Naval Oceanography Command- Fleet Numerical Meteorology & Oceanography Command- Air Force Weather Agency

● Field terminals *- US government- Private- Worldwide users

● NESDIS Archives *● Misc- NASA (for NPP), SARSat, DCS, etc.

Page 5: HDF5 for NPOESS Data Products

Major Product Relationships

Environmental Phenomena

Propagation Sensor

Sensor Scene

Environmental Data Record

EnvironmentModel

Sensor Model

Sensor Data Record

A0569F223B365CC369B707070FFF

Comm

Raw Data

Record

PacketizedData

A0569F223B365CC369B707070FFF

100110110011100

100110110011100

Page 6: HDF5 for NPOESS Data Products

Product Types

● Major products- Raw data records *- Sensor Data Records *- Temperature Data Records *- Environmental data records *

● Other- Raw data- Auxiliary data- Ancillary Data- Calibration & correction data- Telemetry & housekeeping- Memory dumps and diagnostics

Page 7: HDF5 for NPOESS Data Products

Delivery Requirements

● Operational *; platform independence; suitable for transmission, efficient conversion, and storage

● Standards based: Interoperability*, consistent with Joint Technical Architecture and National Spatial Data Infrastructure

● Rapid & available*: >95% delivered within 30 min. at Centrals; 15 min. goal

● Self-documenting● Flexible for over 100 initial products● Flexible for evolutionary or new sensors and algorithms● User selectable aggregation (from one granule up to full orbit)● Delivery: Push or pull; Assured; Secure*● Efficient: 3 TB per day at each Central● Consistent interface for delivery to Centrals*, Field terminals*,

& long term archives*

* key requirement

Page 8: HDF5 for NPOESS Data Products

Schedule

● Interface with Centrals to be published (draft) in spring 2003.● First deliverable version of IDPS to be ready in spring 2005

for NPP risk-reduction mission in spring 2006● Hardware specification for software support at field

terminals to be published by Oct. 2005● Operational system to be ready in mid 2008 for first NPOESS

launch in mid 2009.

Page 9: HDF5 for NPOESS Data Products

Issues

● How much EOSDIS heritage to retain, if any, e.g.– whether to use EOS swath construct

● Develop an NPOESS profile to handle particular attributes of NPOESS data, e.g.– variable length compressed packets in RDRs– conical scan geometry

● Assure long-term stability of the standard● Provide user support● Suitability for archival use– HDF5 is primarily defined by its API, not the format

Page 10: HDF5 for NPOESS Data Products

Backup

Page 11: HDF5 for NPOESS Data Products

Data Processing Flow in IDPS

MetadataMetadata

Converted Mission

Data

Converted Mission

Data

MetadataMetadata

Mission Data Packets

Mission Data Packets

Corr. Data Packets

Corr. Data Packets

Auxiliary/Telemtry/

Hskpng DataPackets

Auxiliary/Telemtry/

Hskpng DataPackets

MetadataMetadata

Envi-ronment

Data

Envi-ronment

Data

Ancillary Data

Ancillary Data

RDR EDR

Ancillary Data Packets

Ancillary Data Packets

LRD Only

ConvertedTelemetry

TDR/ SDR

Format Format Format Format

EDR Process

Environment

Model

SDR Process

Sensor Model

Delivered Raw DataDelivered Raw Data

RDRProcess

Communi-cationsModel

SDR-like

Process

Product Interface to Users

Local

Page 12: HDF5 for NPOESS Data Products

Raw Data Granule Contents

● CCSDS Application Packets, unprocessed, generally from one sensor over a short time interval

● Mission data as produced by the sensor or payload● Telemetry● Calibration data & Correction parameters● Auxiliary data: spacecraft location & attitude

Page 13: HDF5 for NPOESS Data Products

Sensor/Temperature Granule Contents

● Radiometrically corrected data, which is an estimate of the flux at the sensor aperture.

● Mission data by in-track position, cross-track position, channel, detector, etc.

● Georeferenced in spacecraft coordinates, & intersection with the ellipsoid

● Generally, not filtered, resampled, etc.

● TDRs are microwave SDRs without antenna pattern removed

Page 14: HDF5 for NPOESS Data Products

Environmental Data Granule Contents

● SDRs processed with environmental models and ancillary data (by others) to produce estimates of environmental parameters

● Reported as cross-track / in-track values identified in earth coordinates- unless resampling is specifically required

Page 15: HDF5 for NPOESS Data Products

Metadata - Scope● Identifications: - Name / Level / Type-subtype / Sequential number

● Description: - Start-end date-time; Start-end orbit- Subsatellite coordinates; Corner coordinates

● History: - Space segment components / DRR components / Source

files / Algorithm versions / Processing facility / Processing date-time

● Contents: - Size / Granules- Predecessor quality / Missing data / Out-of-bounds /

Processing errors / Quality free text● Formats: - Data elements / Structures / Formats

Page 16: HDF5 for NPOESS Data Products

Metadata - Structure

● As goal, all metadata will be compliant with NSDI standards- <tag> = “parameter”, where <tag> is defined by FGDC or the

NPOESS extension profile.● Metadata will be hierarchical, as appropriate:- file-metadata

● granule_metadata– scan_metadata

» point_metadata● Metadata should map to the file naming convention● Metadata will be included with associated data files, and

accessible as a stand-alone file.

Page 17: HDF5 for NPOESS Data Products

Data

● Hierarchical● Sensor specific format● “Internal” metadata, such as detailed geolocation, quality,

annotation, illumination & view geometry

Page 18: HDF5 for NPOESS Data Products

Why HDF5?

● Familiarity -- Environmental scientists already have experience with the standard, most recently from EOS products.

● Maturity -- HDF has shown its "staying power", and has been available long enough to have matured from user experiences. NASA, DOE, and others invested heavily in its development.

● Capability -- HDF was designed to manage large, compound data sets within high performance computing environments. HDF5 incorporates new features that are important for NPOESS.

● Compatibility -- HDF operates on multiple appropriate operating systems and languages: C, C++, Java and Fortran.

Page 19: HDF5 for NPOESS Data Products

Why HDF5? (concluded)

● Availability -- HDF was developed in the public interest at NCSA, and is freely available. HDF also has many free, share and commercial supports tools available.

● Interoperability -- The DoD Joint Technical Architecture is in the process of accepting HDF as a standard for interoperability among DoD systems.

● Efficiency – Low overhead compared with legacy formats: GRIB and BUFR. Efficient indexing and subsetting. Support for data compression.

Page 20: HDF5 for NPOESS Data Products

Notional Structure -- external metadata

granule_ngranule_2

file_metadata summary_ metadata

granule_ metadata

granule_1

UniqueFileName.txt

Page 21: HDF5 for NPOESS Data Products

Notional Structure -- HDF portion

granule_ngranule_2

metadata data

granule_1

UniqueFileName.hdfroot

otherlocal_metadata

mission_data