![Page 1: The Python Programming Language and HDF5: H5py](https://reader036.vdocument.in/reader036/viewer/2022062314/56813e60550346895da866c0/html5/thumbnails/1.jpg)
The Python Programming Language and HDF5: H5py.
Introduction to some Python elements.
Introduction to Numerical Python with Numpy.
Tying them together with H5py.
Needs of satellite remote sensing data processing and programming languages.
Daniel Kahn, Science Systems and Applications Inc.
HDF/HDF-EOS Workshop XIII, Riverdale, MD3 Nov 2009
Application to Satellite Remote Sensing Data Processing.
Examples.
![Page 2: The Python Programming Language and HDF5: H5py](https://reader036.vdocument.in/reader036/viewer/2022062314/56813e60550346895da866c0/html5/thumbnails/2.jpg)
Where credit is due:
Andrew Collette, UCLA, author of h5py
Francesc Alted, author of PyTables and the HDF5 1.6 Pyrex definitions (i.e Python-C defs used in H5py)
Andrew also requested acknowledgment for Darren Dale (Cornell) and Armando Sole (ESRF) for their suggestions, patches and encouragement.
![Page 3: The Python Programming Language and HDF5: H5py](https://reader036.vdocument.in/reader036/viewer/2022062314/56813e60550346895da866c0/html5/thumbnails/3.jpg)
Modern remote sensing satellite data and data processing needs have several characteristics.
1) The data need to be subjected to numerical algorithms. The algorithms vary in their sophistication but few, if any, require supercomputer architectures. E.g. Computing a zonal mean. (Fortran, C, C++)
2) The data need to be subjected to non-numerical “bookkeeping” algorithms. Associated metadata needs to be read, matched, sliced, diced, acted upon and committed to databases. (Perl)
3) Modern systems automatically the schedule processing on to lots of inexpensive, commodity CPUs. This complicates using managed (i.e. licensed) commercial runtime environments, e.g. IDL, Matlab, Mathematica.
4) Satellite processing systems are dynamic. They need customization several times over the course of each instrument's life. Development time needs to be short. (Perl, IDL, Matlab, Mathematica)
5) Data processing systems need to track data provenance.
![Page 4: The Python Programming Language and HDF5: H5py](https://reader036.vdocument.in/reader036/viewer/2022062314/56813e60550346895da866c0/html5/thumbnails/4.jpg)
Goal:
Find a programming language which allows fast development and is adept at numerical processing as well as non-numerical processing. “Adept” doesn't mean “best”, just pretty good.
We lack a programming language which spans the breath of our remote sensing data processing needs, i.e. Which can handle both numerical and non-numerical processing (including HDF5) and that has a short development cycle.
A Remote Sensing Data Processing Challenge...
Problem:
![Page 5: The Python Programming Language and HDF5: H5py](https://reader036.vdocument.in/reader036/viewer/2022062314/56813e60550346895da866c0/html5/thumbnails/5.jpg)
Why Python fits the bill better than:
As an interpreted language with dynamic variable bindings Python offers more flexibility during development. The multidimensional array extension (numpy) provides efficient and competitive array operations. It is more similar to IDL or Matlab...
Fortran
IDL or Matlab
No license required which can save money on several levels.
Python language can be more well suited to non-numerical data processing tasks such as are encountered in production processing of remote sensing. It is more similar to Perl...
Perl/PDL
Nicer syntax for array operations.
More complete HDF5 support (h5py vs PDL HDF5).
...and the Python Programming Language
![Page 6: The Python Programming Language and HDF5: H5py](https://reader036.vdocument.in/reader036/viewer/2022062314/56813e60550346895da866c0/html5/thumbnails/6.jpg)
Python has many of the same elements as programming languages used for numerical processing (I.e. loops, conditionals, etc) plus...
Some elements of Python
Name spaces
Lists
Dictionaries
Python is an interpreted language. Python statements can easily be run from an interactive prompt. Python programs are written in ASCII files.
![Page 7: The Python Programming Language and HDF5: H5py](https://reader036.vdocument.in/reader036/viewer/2022062314/56813e60550346895da866c0/html5/thumbnails/7.jpg)
Python has name spaces
Name spaces allow related functions and data to be grouped together in a way that will not conflict with unrelated functions and data which happens to be named the same.
Name space are very useful, but are only relevant here because Python name spaces correspond to Python modules. The numerical and HDF5 routines we will see later are implemented as Python modules.
For example, to open an HDF5 file in Python we first import the h5py module and then we open the file using the File function of the h5py module.
import h5py # First import the module.FileID = h5py.File('hdf5_test.h5','r')#Now use the File function, #note the module name in red....
We could import other modules simultaneously which have an unrelated function called File and there would be no conflict or error.
![Page 8: The Python Programming Language and HDF5: H5py](https://reader036.vdocument.in/reader036/viewer/2022062314/56813e60550346895da866c0/html5/thumbnails/8.jpg)
These are 1D arrays which are useful for solving many problems.List index notation looks like 1D array notation in many languages.
(Aka hash tables) These map symbols to objects in analogy to how a 1D list maps an integer index value to an object.
>>> MyList = [1.0,70] # initialize variable to 2 element list>>> MyList.append("Demo Data!") # Append an element, NOT like fortran>>> MyList.append(MyList[0] + MyList[1]) # Append sum of first 2 elements>>> print MyList[1.0, 70, 'Demo Data!', 71.0]
>>> MyDictionary = {} # Initialize empty Dictionary>>> MyDictionary[0] = 1>>> MyDictionary['Gadzooks'] = 70>>> MyDictionary['Kind of Data'] = 'Demo Data!'>>> MyDictionary['Result'] = MyDictionary[0] + MyDictionary['Gadzooks']>>> print MyDictionary{0: 1, 'Kind of Data': 'Demo Data!', 'Result': 71, 'Gadzooks': 70}
Python has Lists:
Python has Dictionaries:
![Page 9: The Python Programming Language and HDF5: H5py](https://reader036.vdocument.in/reader036/viewer/2022062314/56813e60550346895da866c0/html5/thumbnails/9.jpg)
The objects referenced by a List or Dictionary can be any Python object. This is in contrast to more traditional languages, eg. The elements Fortran array must be of a particular type such as real*4.
Python Lists vs Python Dictionaries vs Arrays in Fortran
Index to a List must be an integer, but a Dictionary can be indexed by an integer or string.
MyList[Integer] vs. MyDictionary[Integer or String]
MyList = [34,”String Datum”]orMyDictionary={'FirstKey':34,'SecondKey':”String Datum”}
![Page 10: The Python Programming Language and HDF5: H5py](https://reader036.vdocument.in/reader036/viewer/2022062314/56813e60550346895da866c0/html5/thumbnails/10.jpg)
The List and Dictionary data structures make it possible to write flexible programs very quickly. However, they are not good for numerical computation. The variety of objects (number, string, etc) which they can reference makes computation slow.
For example, adding array elements: MyList[0] + MyList[1]
Python must check at run time if MyList[0] and MyList[1] are objects that can be added together, and not, say, a number and a string. In a loop this check is performed at every iteration!
Enter Numpy
Numpy is a package for use with Python which provides multidimensional arrays of numeric (and other) types and extends Python syntax to provide a familiar interface to the arrays.
Numpy extends Python syntax to allow the expression of vector operations similar to those in IDL, Matlab, or Fortran (>=90).
What to do?
![Page 11: The Python Programming Language and HDF5: H5py](https://reader036.vdocument.in/reader036/viewer/2022062314/56813e60550346895da866c0/html5/thumbnails/11.jpg)
Example: Add two dimensional arrays.
Numpy Example: Create, Print, and add two 2D arrays
Build from Python lists of lists of elements (the module name is numpy)...>>> a = numpy.array([[1,2,3],[4,5,6]])>>> print a[[1 2 3] [4 5 6]]
Build from dimension sizes...>>> b = numpy.ones([2,3])>>> print b[[ 1. 1. 1.] [ 1. 1. 1.]]
>>> print a+b[[ 2. 3. 4.] [ 5. 6. 7.]]>>>
Print a selected element...>>> print a[1,2]6
![Page 12: The Python Programming Language and HDF5: H5py](https://reader036.vdocument.in/reader036/viewer/2022062314/56813e60550346895da866c0/html5/thumbnails/12.jpg)
The HDF5 connection: H5py
H5py is an Python-HDF5 interface is a Python module written by Andrew Collette. Its design allows the use of familiar Python structures for working with HDF5 files.
The interface maps Python syntax for familiar Python objects to similar objects in the HDF5 file.
![Page 13: The Python Programming Language and HDF5: H5py](https://reader036.vdocument.in/reader036/viewer/2022062314/56813e60550346895da866c0/html5/thumbnails/13.jpg)
Here is a familiar example HDF5 file from the HDFView distribution:
Here is how to read the 3D int array using h5py.
>>> import h5py>>> fid = h5py.File('hdf5_test.h5','r')>>> group = fid['arrays']>>> The3DArray = group['3D int array'].value>>> fid.close()>>> The3DArrayarray([[[ 174, 27, 0, ..., 102, 71, 100009], [ 171, 27, 0, ..., 194, 79, 100109], [ 172, 27, 0, ..., 102, 55, 100209], ...,
![Page 14: The Python Programming Language and HDF5: H5py](https://reader036.vdocument.in/reader036/viewer/2022062314/56813e60550346895da866c0/html5/thumbnails/14.jpg)
Equivalence of HDF5 Groups and Python Dictionaries
Print value of dictionary entry:
Print value of HDF5 file entry:
>>> MyDictionary = {'RandArray':numpy.random.random([2,2])}>>> print MyDictionary['RandArray'][[ 0.82066938 0.39219683] [ 0.86546443 0.91276533]]
>>> fid = h5py.File('RandomArray.h5','r')>>> print fid['RandArray'].value[[ 0.1 3.14152908] [ 2.71828008 0. ]]
![Page 15: The Python Programming Language and HDF5: H5py](https://reader036.vdocument.in/reader036/viewer/2022062314/56813e60550346895da866c0/html5/thumbnails/15.jpg)
Simple Real world example:
Goal: Retrieve HDF5 file from Configuration Management (CM) and insert CM metadata into HDF5 file.
We want to place the CM path and revision number inside the HDF5 file as an attribute.
![Page 16: The Python Programming Language and HDF5: H5py](https://reader036.vdocument.in/reader036/viewer/2022062314/56813e60550346895da866c0/html5/thumbnails/16.jpg)
#! /usr/bin/env python
import sysimport osimport h5py
Rev = sys.argv[1] # Specifiy CM path on command lineSVNFilepath = sys.argv[2] # Specify revision number on #comand line.
command = 'svn export -r ' + Rev + ' ' + SVNFilepath #Subversion # CommandInStream = os.popen(command,'r')ExportString = InStream.read()ExportReturnCode = InStream.close()Elements = SVNFilepath.split('/')
# HDF5 code
fid = h5py.File(Elements[-1]) # Elements[-1] is file name
fid.attrs['SVN Path and Revision'] = SVNFilepath + '@' + Rev
fid.close()
Python script to retrieve file from CM and store Rev number as attribute.
H5py code in red. Note the minimal effort coding HDF5 calls.
![Page 17: The Python Programming Language and HDF5: H5py](https://reader036.vdocument.in/reader036/viewer/2022062314/56813e60550346895da866c0/html5/thumbnails/17.jpg)
Another real world example (if NPOESS is your real world).
We have had several occasions to do data aggregation on HDF5 files for the OMPS Limb instrument.
Our retrieval code (Fortran) processes an orbit of data as 480 distinct pieces and places the results into 480 distinct HDF5 files. We wish to aggregate the files such that an N dimensional array in a unaggregated file becomes a N+1 dimensional array in the aggregated HDF5 file.
The Fortran code is (mostly) the product of the retrieval scientist, while the aggregation is a requirement of the production data processing system. It makes sense to aggregate as a post-processing step to the Fortran code so as to minimize changes to the original Fortran code.
![Page 18: The Python Programming Language and HDF5: H5py](https://reader036.vdocument.in/reader036/viewer/2022062314/56813e60550346895da866c0/html5/thumbnails/18.jpg)
Aggregation Algorithm
1) Input is list of HDF5 files2) Analyze structure of one file to generate list of fully qualified dataset names, dimensions, and type (In the code I use a Python Dictionary and not a list).3) Assume all files have that structure. Read corresponding datasets (of dim N) from each file into aggregation variable (of dim N+1).4) After corresponding datasets have been read from all files write aggregation variable to HDF5 file.5) Repeat 3 until all datasets have been aggregated.
+ + +
Schematic of Array Aggregation for One Dataset
File 1 Array File 2 Array File 3 Array File 4 Array
Array 1Array 2
Array 3Array 4
![Page 19: The Python Programming Language and HDF5: H5py](https://reader036.vdocument.in/reader036/viewer/2022062314/56813e60550346895da866c0/html5/thumbnails/19.jpg)
# Data Aggregator. This script takes a set of HDF5 files as input.# The files are expected to have identical struture. All the# corresponding arrays in the input files are combined into an array# which has dimensions N+1, where N is the number of dimensions in the# original, constituent arrays.
import sysimport h5pyimport numpy
Files = sys.argv[1:] # Get file names from command line
# First step is to select one file and create a map of the shape and# data type of all datasets. This is naturally done via a recursive# function, called VisitAllObjectsFirstFid = h5py.File(Files[0],'r') # Open first HDF5 file
FileInfo = {} # Initialize FileInfo to be a dictionary. We will use it to build a mapping from # dataset name to a tuple containing shape and type of dataset.# EVALUATE HDF5 HIERARCHY# Evaluating a a hierarchy is naturally a recursive process so we define a function....def VisitAllObjects(Group,Path): for i in Group.items(): if isinstance(i[1],h5py.Group): VisitAllObjects(i[1],Path + '/' + i[0]) else: DatasetName = Path + '/' + i[0] FileInfo[DatasetName] = (Group[DatasetName].shape, Group[DatasetName].dtype)
VisitAllObjects(FirstFid,'')FirstFid.close()
![Page 20: The Python Programming Language and HDF5: H5py](https://reader036.vdocument.in/reader036/viewer/2022062314/56813e60550346895da866c0/html5/thumbnails/20.jpg)
# Print dataset paths and info to screenfor (k,v) in FileInfo.items(): print k,v# AGGREGATE DATA # Now that we understand the file structure we can perform the aggregation.OutputFileID = h5py.File('AggregatedData.h5','w')NumberOfFiles = len(Files)
# Here is the meat of the code. The outer loop is over datasets, the inner over all files. for Dataset in FileInfo.keys(): AggregatedData = numpy.ndarray(FileInfo[Dataset][0]+(NumberOfFiles,),dtype = FileInfo[Dataset][1]) for FileNumber in range(NumberOfFiles): # Open file, read data into aggregation array, and close fid = h5py.File(Files[FileNumber],'r') AggregatedData[...,FileNumber] = fid[Dataset].value fid.close() Path = Dataset.split('/') map((lambda(x): OutputFileID.require_group(x)), Path[1:-1]) #OutputFileID[Dataset] = AggregatedData OutputFileID.create_dataset(Dataset,data=AggregatedData,compression=5,chunks=FileInfo[Dataset][0]+(1,))
OutputFileID.close()
![Page 21: The Python Programming Language and HDF5: H5py](https://reader036.vdocument.in/reader036/viewer/2022062314/56813e60550346895da866c0/html5/thumbnails/21.jpg)
$ h5dump -H -d /ANCILLARY_DATA/GeopotentialHeight_NCEP \ Data/OMPS_LP_SDR_20041121_55_146_-69_-119.h5HDF5 "Data/OMPS_LP_SDR_20041121_55_146_-69_-119.h5" {DATASET "/ANCILLARY_DATA/GeopotentialHeight_NCEP" { DATATYPE H5T_IEEE_F32LE DATASPACE SIMPLE { ( 5, 3, 21 ) / ( 5, 3, 21 ) } ATTRIBUTE "Title" { DATATYPE H5T_STRING { STRSIZE 25; } DATASPACE SCALAR } ATTRIBUTE "Units" { DATATYPE H5T_STRING { STRSIZE 2; } DATASPACE SCALAR } ATTRIBUTE "_FillValue" { DATATYPE H5T_IEEE_F32LE DATASPACE SIMPLE { ( 1 ) / ( 1 ) } }}
Original, Unaggregated Data Field
Note 3 dimensions
![Page 22: The Python Programming Language and HDF5: H5py](https://reader036.vdocument.in/reader036/viewer/2022062314/56813e60550346895da866c0/html5/thumbnails/22.jpg)
$ h5dump -H -d /ANCILLARY_DATA/GeopotentialHeight_NCEP \ AggregatedData.h5HDF5 "AggregatedData.h5" {DATASET "/ANCILLARY_DATA/GeopotentialHeight_NCEP" { DATATYPE H5T_IEEE_F32LE DATASPACE SIMPLE { ( 5, 3, 21, 4 ) / ( 5, 3, 21, 4 ) }}}
Aggregated Data Field
Now we have 4 dimensions. The new dimension has extent 4, corresponding to the number of input files.
Note that none of the attributes were copied. This is a bug, but easily fixed.
![Page 23: The Python Programming Language and HDF5: H5py](https://reader036.vdocument.in/reader036/viewer/2022062314/56813e60550346895da866c0/html5/thumbnails/23.jpg)
def VisitAllObjects(Group,Path): for i in Group.items(): if isinstance(i[1],h5py.Group): VisitAllObjects(i[1],Path + '/' + i[0]) else: DatasetName = Path + '/' + i[0] FileInfo[DatasetName] = (Group[DatasetName].shape, Group[DatasetName].dtype, Group[DatasetName].attrs.listitems())
And also....
DS=OutputFileID.create_dataset(Dataset,data=AggregatedData,compression=5,chunks=FileInfo[Dataset][0]+(1,)) [DS.attrs.__setitem__(Attribute[0],Attribute[1]) for Attribute in FileInfo[Dataset][2]]
To fix the bug, I assume attributes (like “Units”) are not aggregated, and I take attributes and values from first file.
New code consists of only 2+ additional lines, shown below in green.
![Page 24: The Python Programming Language and HDF5: H5py](https://reader036.vdocument.in/reader036/viewer/2022062314/56813e60550346895da866c0/html5/thumbnails/24.jpg)
Fixed output:$ h5dump -H -d /ANCILLARY_DATA/GeopotentialHeight_NCEP \ AggregatedDataWithAttributes.h5HDF5 "AggregatedDataWithAttributes.h5" {DATASET "/ANCILLARY_DATA/GeopotentialHeight_NCEP" { DATATYPE H5T_IEEE_F32LE DATASPACE SIMPLE { ( 5, 3, 21, 4 ) / ( 5, 3, 21, 4 ) } ATTRIBUTE "Title" { DATATYPE H5T_STRING { STRSIZE 25; CTYPE H5T_C_S1; } DATASPACE SCALAR } ATTRIBUTE "Units" { DATATYPE H5T_STRING { STRSIZE 2; CTYPE H5T_C_S1; } DATASPACE SCALAR } ATTRIBUTE "_FillValue" { DATATYPE H5T_IEEE_F32LE DATASPACE SIMPLE { ( 1 ) / ( 1 ) } }}}
Now we have attributes.
![Page 25: The Python Programming Language and HDF5: H5py](https://reader036.vdocument.in/reader036/viewer/2022062314/56813e60550346895da866c0/html5/thumbnails/25.jpg)
Summary:
Python offers a high degree of flexibility for code development, combined with the ability to do easy text, numerical array and HDF5 coding make it a good candidate for solving problems in satellite remote sensing data processing.
Few, if any, other computer languages offer this combination of benefits making it uniquely suited for this task.
Tracking module version provenance is likely one of the outstanding questions for Python use in production.
Future Work:
Acknowledgment:
Curt Tilmes at NASA Goddard funded this work via contract NNG06HX18C task 614.5-01-07