harmonization of hdf with national imagery transfer format (nitf) 2.1
DESCRIPTION
HDF5 has already demonstrated its ability to adapt to diverse applications, and to integrate with other standards, e.g., netCDF. The National Imagery Transfer Format (NITF) is another format which might benefit from HDF5 as it evolves. NITF is the mandated standard for formatting digital imagery and imagery-related products, and exchanging them among the DoD and a number of US government agencies. Although NITF has been improved over the years, and although designed to be extensible, there are technical and conceptual limits to its original paradigm: mono- and polychromatic images, symbols, text and associated data. NITF has been a mandated standard for many years, and enterprise architectures have been built around it. There are important reasons why it should be retained. New sensors and algorithms are at the verge of stressing the standard with multispectral, hyperspectral, extended response, variable scale, time series, radar, video, and multisensor fusion products. Metadata are becoming more complex as the need for annotation and supporting data grows. Some imagery-like products are already originated in HDF, and others would benefit from a flexible format such as HDF. Portions of the NITF user community are exploring ways to move beyond its limits, to incorporate enhanced metadata, and to set procedures for suitable product profiles. This presentation develops a mapping between HDF5 and NITF structures and features. It concludes with some ideas on how NITF could be extended by harmonization with HDF5, while affording minimal disruption to operational uses.TRANSCRIPT
Harmonizing HDF5 and NITF
Alan M. GoldbergHDF Workshop XNovember 2006
The opinions expressed in this presentation are those of the author. They do not represent any organization or agency.
Copyright ©2006. All rights reserved.
Outline
Background on standards and NITF How HDF might help Issues & Conclusions
Purpose of data standardization Standards typically help in three ways:
supported data formats and methods of organization
annotation of the data profiles which limit the way in which data is used
Standards may give fixed specifications, permit variable specifications, or provide preferred implementations
Different standards provide different levels of detail, and provide different options to the user.
New standards need to find a niche among existing standards
Other Standards
Specific Instance
Profile
My Standard
Foundation Standards
Other Standards
Other Standards
Other Standards
Any standard often has a complicated relationship with other standards and implementations
Meaning
Identification
Structure
Elements
Physical Representation
National Imagery Transfer Format 10USC467. Definitions: The term “imagery” means … a
likeness or presentation of any natural or manmade feature or related object or activity and the positional data acquired at the same time the likeness or representation was acquired, including [air- or space-borne systems] or other similar means.
MIL-STD-2500C. NITF Ver. 2.1 for the NITF Standard “The NITF provides a common basis for storage and interchange of images and associated data among existing and future systems. The NITF can be used to support interoperability by providing a data format for shared access applications, while also serving as a standard file format for dissemination of images, graphics, text, and associated data.”
Also used in SDTS & known as ISO/IEC 12087-5:BIIF and NATO Secondary Imagery Format (STANAG 4545)
Essence of NITF NITF is based on the concept of an image with
annotation. It is strongly tied to the concept of a regular gridded 2-D data array, and often explicitly to a map projection. Defines metadata which is required to geolocate the
image and to understand its provenance and other, similar metadata attributes
Difficult to associate gridded ancillary data with the gridded image.
Requires much predefinition and can be inflexible. Recently extended by incorporating XML, but this
requires two levels of interpretations Provides other features, eg. compression, color space
A NITF image
Complex displays are a NITF strength
Implementation of a multi-level NITF display in common display coordinates
Source this page and previous: ITT, ENVI Certified NITF/NSIF Module Guide, October 2003, http://www.ittvis.com/envi/pdfs/NITFguide.pdf
NITF file structure
Segments establish the basic file contents Additional major data elements can be incorporated in DES Additional descriptive information can be incorporated in Tagged
Record Extensions (TREs) A TRE is a deterministic structure of text elements Recent TREs permit XML
DES and TRE content is defined in reference documents; not self-describing
Multiple Image Segments are overlays or complex data sets, as appropriate
Not our father’s ‘Imagery’ anymore
Imagery
ImageryData File
ImageryDisplay
can be can be
processes into
Topical data arraysMetadataSupporting data
RGB 2D regular arrayStill or motionAnnotations
built of built of
When imagery was 10b 1- or 3-color, data set and display were the same.New imagery is arrays over 2D space, time, & color; more bits; RF; mosaics…
Derived dataCollections
Some current constraints NITF is a mandated standard NITF has large installed operational
base: users, libraries, industry NITF has conceptual limitations
Fixed format components Limited data model Display paradigm Size limits (10GB IS, 1GB DES)
Original data arriving in other formats
Potential georeferenced standards to supplement NITF XML is more flexible, but only incorporates ASCII data;
requires strong profile. Advanced Authorizing Format is based on the concept of
multiple time series, primarily video or audio. geoTIFF seems to have functionality similar to NITF HDF-EOS is based on geolocated 1D-2D arrays and other
ad hoc structures. netCDF is based on n-D arrays, with strong metadata
requirements and even stronger conventions. FGDC has developed metadata content standards,
related to this issue. HDF is extremely flexible, enabling all of the above.
However, it also requires a strong profile to avoid becoming uncontrolled.
Approaches to harmonization There are several ways to go:
One way is to use the best portions of NITF and other standards to establish a strong profile for HDF. E.g., NITF tags can be directly mapped to NITF attributes.
Another is to use XML to establish a framework within which data arrays can be created in HDF or some other data-oriented standard.
Example of NITF to HDF mapping File header mostly attributes Image Seg subhead mostly attributes, some data space Image Seg TREs mostly attributes Image Seg data array(s) Graphics Seg array wraping a shapefile Text Seg array of CHAR, or atttributes Ext Seg n/a Reserved Seg n/a
Some options for HDF in a NITF environment
HDFFile
FileHdr.
NITF 2.1 File
DESSubhd
HDF DES
HDFFile
HDFRef.
FileHdr.
NITF 2.1 File
DESSubhd
HDF DES
HDFFile
NITFAttrib.
ImageFileHdr.
NITF 2.1 File
ISSubhd
Image Seg.
HDFFile
File Hdr.
NITF 2.1 File
User Block
1. NITF wraps HDF
2. NITF points to HDF
3. NITF coexists with HDF
4. NITF in the HDF user block
Pervasive problem is difference between NITF fixed structure, and HDF API paradigms.
* ** Possible XML aid to HDF
Conclusions HDF5 was designed with sufficient flexibility
to handle the data engineering needs of NITF users.
An appropriate profile would permit NITF data content to be stored in HDF5.
HDF5 would permit emerging & related extensions which are difficult for NITF.
NITF is a mandated, interoperable standard. Converting operational users will not be easy.
Are there creative solutions out there?
Backup
New standards need to find a niche among existing standards
Other Standards
Specific Instance
Profile
My Standard
Foundation Standards
Other Standards
Other Standards
Other Standards
Any standard often has a complicated relationship with other standards and implementations
Meaning
Identification
Structure
Elements
Physical Representation
HDF NITF
AbstractThe National Imagery Transfer Format (NITF) is the mandated standard for formatting digital
imagery and imagery-related products, and exchanging them among the DoD and a number of US government agencies.
Although NITF has been improved over the years, and although designed to be extensible, there are technical and conceptual limits to its original paradigm: mono- and polychromatic images, symbols, text and associated data. NITF has been a mandated standard for many years, and enterprise architectures have been built around it. There are important reasons why it should be retained.
New sensors and algortihms are at the verge of stressing the standard with multispectral, hyperspectral, extended response, variable scale, time series, radar, video, and multisensor fusion products. Metadata are becoming more complex as the need for annotation and supporting data grows. Some products are arriving from sensors as HDF files, and others would benefit from a flexible format such as HDF.
Portions of the NITF user community are exploring ways to move beyond its limits, to incorporate enhanced metadata, and to set procedures for suitable product profiles. This presentation develops a mapping between HDF5 and NITF structures and features. It concludes with some ideas on how NITF could be extended by harmonization with HDF5, while affording minimal disruption to operational uses.
New standards need to find a niche among existing standards
My Standard
Basic Standards
Peer Standards
Applied Standards
Profiles
Hardware
Application
Problems
NITF has a large embedded base supported by industry. Difficult to reorient
NITF has a fixed data layout and structure. HDF is defined by its API.
There are conceptual differences between the standards, not just data engineering differences.