HDF and HDF-EOS Workshop XVII (2014) Aleksandar Jelenak

1. The HDF Group
www.hdfgroup.orgJuly 8, 2014 HDF Workshop @ 2014 ESIP Summer Meeting
HDF Product Designer
Aleksandar Jelenak, H. Joe Lee, Ted Habermann
The HDF Group
1

2. www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting
HDF Data Producer’s Conundrum
2
HDF Features Interoperability
Mission Requirements
HDF Product Designer
• Datatypes
• Groups
• Attributes
• Dimension scales
• Compression
• Chunking
• Scale/offset
• Etc.
• Conventions
• Metadata
• Software
• netCDF
• Science objectives
• Data processing, discovery &
distribution
• Data documentation
• User engagement, preparedness,
feedback

3. www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting
HDF Product Designer Background
• Originated from the HDF5 Earth Science
(H5ES) Builder/Creator toolset developed by
Jeffrey Lee for the ICESat-2 mission
• Built using the MySQL, Apache, PHP software
stack
• Runs as a desktop application
• Under review by The HDF Group to generalize
the concept
3

4. www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting
What is HDF Product Designer?
4
• Application for organizing HDF5 file content to
data and metadata
• Built-in support for conventions
• Create and re-use parts of file content
• Collaborative design process
• Generate HDF5 template file, or code that
produces the designed file for several
programming languages
• HDF5 template has structure and metadata

5. www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting
Design Process
5
Create and organize content:
• Files, groups, datasets, attributes
• Many parameters of datasets and attributes
can be set
• Re-use any part of content elsewhere in the
same or another file
• All the work is saved in a database

6. www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting
After a File is Designed…
6
• Export description of the file’s content
into a tab-delimited text file
• This description file can be opened with
Excel
• Content description can be edited
• Description file can be imported back
into the Product Designer

7. www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting
After a File is Designed…
7
• A template HDF5 file can be generated
• Code can be produced for reading and
writing designed file for:
• FORTRAN
• IDL
• The generated write code can be edited
to fill in real data

8. www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting
Typical HDF Product Designer Workflow
8
Design a
product file
Export
description
file
Stakeholders
review/edit
content
Import
description
file
Generate
template file
and/or code
Testing

9. www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting
Expected Benefits
• Easy way to design HDF file content
• Interoperability issues addressed during the
design phase
• End-to-end data flow testing can start earlier
• Reduced risk for costly redesign of processing
pipeline based on stakeholder feedback later
• Team collaboration
• Promotion of design best practices across
mission teams
9

10. www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting
Future Work
10
• Local (desktop) or central (web site) app?
• Reduce app’s required software footprint
• MySQL already replaced with SQLite
• Support Python, MATLAB for template file code
• Web service for generating HDF5 templates and
code
• Generate portable/free production code using
Python and h5py
• Re-use content info from old products:
• Import HDF4 Content Map.

11. www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting
Future Work
• Support for compound datatypes
• Vdata in HDF4 maps to attribute or dataset?
• Convention compliance checkers
• CF, HDF-EOS, etc.
• Interoperability checkers
• netCDF-4, visualization tools, etc.
11

12. www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting
Local vs. Central Application?
• Local:
• No need for Internet connection
• Richer/more responsive user interface
• No user accounts management
• Requires a developer’s computer
• Central:
• Only web browser required
• Better collaboration workflow
• Only one place to update the app
• Dual mode – like email software?
12

13. www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting
Looking for Partners
• New missions
• Projects upgrading from HDF4 to HDF5
• Projects reprocessing their data to increase
interoperability
• Use cases
Contact: eoshelp@hdfgroup.org
13

14. www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting
Acknowledgement
15
This work was supported by Subcontract number
114820 under Raytheon Contract number
NNG10HP02C, funded by the National
Aeronautics and Space Administration (NASA).
Any opinions, findings, conclusions, or
recommendations expressed in this material are
those of the authors and do not necessarily
reflect the views of Raytheon or the National
Aeronautics and Space Administration.