Using HDF5 and Python: The H5py module

Using HDF5 and Python: The H5py module

Daniel Kahn
Science Systems and Applications, Inc.
Acknowledgement: Thanks to Ed Masuoka, NASA Contract NNG06HX18C

HDF & HDF-EOS Workshop XV 17 April 2012

Python has lists:
>>> for elem in ['FirstItem','SecondItem','ThirdItem']:
...
print elem
...
FirstItem
SecondItem
ThirdItem
>>>

We can assign the list to a variable.
>>> MyList = ['FirstItem','SecondItem','ThirdItem']
>>> for elem in MyList:
...
print elem
...
FirstItem
SecondItem
ThirdItem
>>>

Lists can contain a mix of objects:
>>> MixedList = ['MyString',5,[72, 99.44]]
>>> for elem in MixedList:
...
print elem
...
MyString
A list inside a list
5
[72, 99.44]
Lists can be addressed by index:
>>> MixedList[0]
'MyString'
>>> MixedList[2]
[72, 99.44]

A note about Python lists:
Python lists are one dimensional.
Arithmetic operations don’t work on them.
Don’t be tempted to use them for scientific array
based data sets. More the ‘right way’ later...


Python has dictionaries.
Dictionaries are key,value pairs
>>> Dictionary =

{'FirstKey':'FirstValue',
'SecondKey':'SecondValue',
'ThirdKey':'ThirdValue'}

>>> Dictionary
{'SecondKey': 'SecondValue', 'ThirdKey': 'ThirdValue',
'FirstKey': 'FirstValue'}
>>>

Notice that Python prints the key,value pairs in a different
order than I typed them.
The Key,Value pairs in a dictionary are unordered.

Dictionaries are not lists, however we can easily create a list
of the dictionary keys:
>>> list(Dictionary)
['SecondKey', 'ThirdKey', 'FirstKey']
>>>

We can use a dictionary in a loop without additional
elaboration:
>>> for Key in Dictionary:
...
print Key,"---->",Dictionary[Key]
...
SecondKey ----> SecondValue
ThirdKey ----> ThirdValue
FirstKey ----> FirstValue
>>>

HDF5 is made of
“Dictionaries” a dataset
name is the key, and the
array is the value.
Keys
Value
HDFView is a tool which
shows use the keys
(TreeView) and the values
(TableView) of an HDF5 file.

Andrew Collette’s H5py module allows us to use Python and
HDF5 together.
We can use H5py to manipulate HDF5 files as if they were
Python Dictionaries
>>> import h5py
>>> in_fid = h5py.File('DansExample1.h5','r')
>>> for DS in in_fid:
...
print DS,"------->",in_fid[DS]
...
FirstDataset -------> <HDF5 dataset "FirstDataset": shape (25,), type "<i4">
SecondDataset -------> <HDF5 dataset "SecondDataset": shape (3, 3), type "<i4">
ThirdDataset -------> <HDF5 dataset "ThirdDataset": shape (5, 5), type "<i4">

>>>

Keys

Values

So What? We need to be able to manipulate the arrays, not
just the file.
The Numpy module by Travis Oliphant allows the manipulation
of arrays in Python.
We will see examples of writing arrays later, but to get arrays
from the H5py object we have the ellipses.
>>> import h5py
>>> fid = h5py.File('DansExample1.h5','r')
>>> fid['FirstDataset']
<HDF5 dataset "FirstDataset": shape (25,), type "<i4">
>>> fid['FirstDataset'][...]
array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
17, 18, 19, 20, 21, 22, 23, 24])
>>> type(fid['FirstDataset'][...])
<type 'numpy.ndarray'>
>>>

16,

Reasons to use Python and HDF5 instead of C or Fortran
The basic Python Dictionary object has a close similarity to
the HDF5 Group. The object oriented and dynamic nature of
Python allows the existing Dictionary syntax to be repurposed
for HDF5 manipulation.
In short, working with HDF5 in Python requires much less
code than C or Fortran which means faster development and
fewer errors.


Comparison to C, h5_gzip:

C
# Lines of code

106

Python from
THG site
37

Fewer lines of code means fewer places to make mistakes
The 37 line h5_gzip.py example is a “direct” translation of the
C version. Some more advanced techniques offer insight into
advantages of Python/H5py programming. Text in next
slides is color coded to help match code with same functionality.
First writing a file…

Original h5_gzip.py

Pythonic h5_gzip.py

# This example creates and writes GZIP compressed
dataset.
import h5py
import numpy as np
# Create gzip.h5 file.
#
file = h5py.File('gzip.h5','w')
#
# Create /DS1 dataset; in order to use compression,
dataset has to be chunked.
#
dataset = file.create_dataset('DS1',
(32,64),'i',chunks=(4,8),compression='gzip',compressi
on_opts=9)
#
# Initialize data.
#
data = np.zeros((32,64))
for i in range(32):
for j in range(64):
data[i][j]= i*j-j
# Write data.
print "Writing data..."
dataset[...] = datafile.close()

#!/usr/bin/env python
# It's a UNIX thing.....

from __future__ import print_function # Code will work
with python 3 as well....

# This example creates and writes GZIP compressed
dataset.
import h5py # load the HDF5 interface module
import numpy as np # Load the array processing
module
# Initialize data. Note the numbers 32 and 64 only
appear ONCE in the code!
LeftVector = np.arange(-1,32-1,dtype='int32')
RightVector = np.arange(64,dtype='int32')
DataArray = np.outer(LeftVector,RightVector) # create
32x64 array of i*j-j
# The _with_ construct will automatically create and
close the HDF5 file
with h5py.File('gzip-pythonic.h5','w') as h5_fid:
# Create and write /DS1 dataset; in order to use
compression, dataset has to be chunked.
h5_fid.create_dataset('DS1',data=DataArray,chunks=(4
,8),compression='gzip',compression_opts=9)

dataset[...] = data
file.close()
file.close()


Reading data….

# Read data back; display compression properties and
dataset max value.
#
file = h5py.File('gzip.h5','r')
dataset = file['DS1']
print "Compression method is", dataset.compression
print "Compression parameter is",
dataset.compression_opts
data = dataset[...]
print "Maximum value in", dataset.name, "is:",
max(data.ravel())
file.close()

# Read data back; display compression properties and
dataset max value.
#
with h5py.File('gzip-pythonic.h5','r') as h5_fid:
dataset = h5_fid['DS1']
print("Compression method is", dataset.compression)
print("Compression parameter is",
dataset.compression_opts)
print("Maximum value in", dataset.name, "is:",
dataset.value.max())


And finally, just to see what the file looks like…


Real world example: Table Comparison
Background:
For the OMPS Instruments we need to design binary
arrays to be uploaded to the satellite to sub-sample the
CCD to reduced data rate.
For ground processing use we store these arrays in
HDF5.
As part of the design process we want to be able to
compare arrays in two different files.


Here is an example of a Sample Table


Here is another example:


Here is the “difference” of the arrays. Red pixels are
unique to the first array.


The code: CompareST.py
""" Documentation """
from __future__ import print_function,division
import h5py
import numpy
import ViewFrame
def CompareST(ST1,ST2,IntTime):
with h5py.File(ST1,'r') as st1_fid,h5py.File(ST2,'r') as st2_fid:
ST1 = st1_fid['/DATA/'+IntTime+'/SampleTable'].value
ST2 = st2_fid['/DATA/'+IntTime+'/SampleTable'].value
ST1[ST1!=0] = 1
ST2[ST2!=0] = 1
Diff = (ST1 - ST2)
ST1[Diff == 1] = 2
ViewFrame.ViewFrame(ST1)


..and the command line argument parsing.

if __name__ == "__main__":
import argparse
OptParser = argparse.ArgumentParser(description = __doc__)
OptParser.add_argument("--ST1",help="SampleTableFile1")
OptParser.add_argument("--ST2",help="SampleTableFile2")
OptParser.add_argument("--IntTime",help="Integration Time",
default='Long')
options = OptParser.parse_args()
CompareST(options.ST1,options.ST2,options.IntTime)


Recursive descent into HDF5 file
Print group names, number of children and dataset names.
from __future__ import print_function
import h5py
def print_num_children(obj):
if isinstance(obj,h5py.highlevel.Group):
print(obj.name,"Number of Children:",len(obj))
for ObjName in obj: # ObjName will a string
print_num_children(obj[ObjName])
else:
print(obj.name,"Not a group")
with h5py.File('OMPS-NPP-NPP-LP_STB', 'r+') as f:
print_num_children(f)


The Result….
ssai-s01033@dkahn: ~/python % ./print_num_children.py
/ Number of Children: 1
/DATA Number of Children: 10
/DATA/AutoSplitLong Not a group
/DATA/AutoSplitShort Not a group
/DATA/AuxiliaryData Number of Children: 6
/DATA/AuxiliaryData/FeatureNames Not a group
/DATA/AuxiliaryData/InputSpecification Not a group
/DATA/AuxiliaryData/LongLowEndSaturationEstimate Not a group
/DATA/AuxiliaryData/ShortLowEndSaturationEstimate Not a group
/DATA/AuxiliaryData/Timings Number of Children: 2
/DATA/AuxiliaryData/Timings/Long Not a group
/DATA/AuxiliaryData/Timings/Short Not a group
/DATA/AuxiliaryData/dummy Not a group
/DATA/Long Number of Children: 14
/DATA/Long/BadPixelTable Not a group
/DATA/Long/BinTransitionTable Not a group
/DATA/Long/FeatureNamesIndexes Not a group
/DATA/Long/Gain Not a group
/DATA/Long/InverseOMPSColumns Not a group

Summary
Python with H5py and Numpy modules make developing
Programs to manipulate HDF5 files and perform calculations
With HDF5 arrays simpler which increase development
speed and reduces errors.


Using HDF5 and Python: The H5py module

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Using HDF5 and Python: The H5py module

Similar to Using HDF5 and Python: The H5py module (20)

More from The HDF-EOS Tools and Information Center

More from The HDF-EOS Tools and Information Center (20)

Recently uploaded

Recently uploaded (20)

Using HDF5 and Python: The H5py module