The Data Access Protocol(DAP) is a protocol for requesting and transporting data across the web. By using OPeNDAP servers data can be made accessible to remote locations regardless of local storage format. A prototyped HDF5-OPeNDAP server was implemented several years ago. With the HDF5 being selected as the primary data format of NASA Aura satellite, a robust, fully supported HDF5-OPeNDAP server is necessary. Recently NASA Headquarters Science Mission Directorate Advancing Collaborative Connections For Earth-Sun System Science (ACCESS) program has awarded a joint proposal to implement a fully supported HDF5-OPeNDAP server. This poster will discuss the design and implementation plan for this project.

1. Access HDF5 Datasets via
OPeNDAP’s Data Access
Protocol (DAP)
MuQun Yang
The HDF Group
02/18/14
HDF and HDF-EOS Workshop X, Landover, MD
1

2. OPeNDAP
•
•
•
•
•
•
A software framework that allows simple access to remote data
Client-server model
Servers make the local data accessible remotely
Clients make applications easily access to remote served data
Widely used by Earth Science Community
URL: http://www.opendap.org/
02/18/14
HDF and HDF-EOS Workshop X, Landover, MD
2

3. The Data Request
 Data are requested with a URL.
http://www.cdc.noaa.gov/cgi-bin/nph-nc/datasets/Reynolds_sst
?sst[10:10][0:90][0:180]
Protocol Machine name
OPeNDAP server Directory
File name
Constraint
 The user can impose a constraint on the data that s/he
would like to acquire from a data set by appending a
constraint expression to the end of the URL.
02/18/14
HDF and HDF-EOS Workshop X, Landover, MD
3

4. OPeNDAP Data Types
OPeNDAP provides three basic data objects:
 data descriptor structure (.dds) - syntactic
metadata - rigid
 data attribute structure (.das) - semantic
metadata - flexible
 data (.dods) - the actual data in a binary
structure
02/18/14
HDF and HDF-EOS Workshop X, Landover, MD
4

5. OPeNDAP Servers
CODAR
netCDF HDF4
Data
Data
CODAR
netCDF
Data
Matlab
DSP
Tables
SQL
FITS
CDF
Flat
Binary
Data
Data
Data
Data
Data
Data
Data
JGOFS
Matlab
HDF4
DSP
FITS
JDBC
CEDAR
General
Data
Data
ESML
FreeFrom
CDF
CEDAR
No fully-supported HDF5 server
02/18/14
HDF and HDF-EOS Workshop X, Landover, MD
5

6. OPeNDAP Clients
netCDF C
Ferret
GrADS
netCDF Java
IDV
VisAD
ncBrowse
Matlab
Client
IDL
Client
Matlab
IDL
Access
Excel
No HDF5 client
02/18/14
HDF and HDF-EOS Workshop X, Landover, MD
6

7. Why HDF5-OPeNDAP server and
client?
Users!
Users!!
Users!!!
02/18/14
HDF and HDF-EOS Workshop X, Landover, MD
7

8. A User Case
Task:
•10 elements of 17 MB HDF5 data
•200 elements of 10 MB NetCDF data
•150 elements of 5 MB HDF4 data
02/18/14
HDF and HDF-EOS Workshop X, Landover, MD
8

9. General approach
HDF5 data
FTP
HDF4 data
FTP
NetCDF data
FTP
02/18/14
HDF and HDF-EOS Workshop X, Landover, MD
9

10. Approach with OPeNDAP
HDF5 data
FTP
HDF4 data
server
client
AP
D
NetCDF data
DA
P server
02/18/14
HDF and HDF-EOS Workshop X, Landover, MD
10

11. Conclusion
• HDF5-DAP server is required
• HDF5-DAP client is strongly recommended
02/18/14
HDF and HDF-EOS Workshop X, Landover, MD
11

12. Previous work
- A Prototype of DAP2-HDF5 server
•
•
•
•
Incomplete mapping from HDF5 to OPeNDAP
No comprehensive test suite
Only tested a few examples on two platforms
Incomplete documentation
02/18/14
HDF and HDF-EOS Workshop X, Landover, MD
12

13. An example from SSMI
From HDF4
server at JPL
02/18/14
From HDF5
server at NCSA
HDF and HDF-EOS Workshop X, Landover, MD
13

14. Current HDF5-OPeNDAP Support
HDF5
DAP
No Client
02/18/14
Prototyped Server
HDF and HDF-EOS Workshop X, Landover, MD
14

15. HDF5-OPeNDAP Project
• Two-year project awarded by NASA ROSES NRA
• Funding much less than expected
• Original proposed work kept
• with significant contributions from ESDIS
• Starting date: Nov. 15th, 2006
02/18/14
HDF and HDF-EOS Workshop X, Landover, MD
15

16. Proposed Work – software development
•
Implementation
•
•
•
HDF5-DAP2 server
HDF5-DAP4 server
Deployment
•
02/18/14
Release servers to OPeNDAP community
HDF and HDF-EOS Workshop X, Landover, MD
16

17. HDF5-DAP2 server
•
•
Update and complete the prototype, including make
files, instructions, documentation, etc
Add a few selected features to handle NASA Aura data
and NPOESS data
-
•
•
•
Map HDF5 group to DAP(?)
Map Object/regional reference to DAP(?)
Implement a comprehensive test suite
Test on platforms supported by OPeNDAP
Demonstrate with sample NASA data
02/18/14
HDF and HDF-EOS Workshop X, Landover, MD
17

18. Fully-Supported HDF5 Server
HDF5 server HDF5
DAP
HDF5 server
Matlab/Ferret Client
HDF5
DAP
02/18/14
HDF and HDF-EOS Workshop X, Landover, MD
18

19. Proposed Work – software development
•
Implementation
•
•
DAP4 to HDF5 conversion utility
Deployment
•
•
Release utility to OPeNDAP community
Design study
•
02/18/14
Feasibility of DAP-aware HDF5 library
HDF and HDF-EOS Workshop X, Landover, MD
19

20. Proposed Work - DAP to HDF5 utility
• A command line tool that converts DAP to HDF5 file
•
•
individual DAP variable
subsetting
Conversion tool
HDF5 server
HDF5
DAP
02/18/14
HDF and HDF-EOS Workshop X, Landover, MD
20

21. Proposed work - documentations
•
•
•
•
•
DAP4 specification
HDF5-DAP4 mapping document
DAP4 to HDF5 specification
RFC for DAP-aware HDF5 library
Submit specifications to ESDSWG
02/18/14
HDF and HDF-EOS Workshop X, Landover, MD
21

22. Acknowledgement
This work was supported basing upon the Cooperative
Agreement with the National Aeronautics and Space
Administration (NASA) under NASA grant
NNG05GC60A. Any opinions, findings, and
conclusions or recommendations expressed in this
material are those of the author(s) and do not necessarily
reflect the views of NASA.
02/18/14
HDF and HDF-EOS Workshop X, Landover, MD
22