In this talk, we will give an update on the HDF5 OPeNDAP project. We will update the new features inside OPeNDAP HDF5 data handler. We will also introduce a new HDF5-Friendly OPeNDAP client library and demonstrate how it can help users to view and analyze remote HDF-EOS5 data served by OPeNDAP HDF5 handler. A demo will be presented with a customized OPeNDAP visualization client (GrADS) that uses the library.
2024: Domino Containers - The Next Step. News from the Domino Container commu...
HDF5 OPeNDAP project update and demo
1. HDF5 OPeNDAP Project
Update and Demo
MuQun Yang and Hyo-Kyung Lee
(The HDF Group)
James Gallagher
(OPeNDAP, Inc.)
10/17/2008
HDF and HDF-EOS Workshop XII
1
2. OPeNDAP Review
• A software framework that allows
simple access to remote data
• Data Access Protocol
• Client-server model
• Widely used by Earth Science
Community
10/17/2008
HDF and HDF-EOS Workshop XII
2
3. OPeNDAP Concept
Typical Remote Access
View Data
(User)
FTP/HTTP
Remote Data
(HDF4)
Visualization
Tools
tools
(e.g. grads)
(gradsdap)
Handler
(hdf4_handler)
Client Library
(libnc-dap)
Server
(apache)
DAP Protocol (via http)
10/17/2008
HDF and HDF-EOS Workshop XII
3
4. OPeNDAP HDF5 Handler
View Data
(User)
Remote HDF5
HDF5 Handler
Server
(apache)
DAP Protocol (via http)
10/17/2008
HDF and HDF-EOS Workshop XII
4
5. Challenges
Mapping HDF5 to DAP
• Compound Datatype
• Groups
• Object/Regional References
10/17/2008
HDF and HDF-EOS Workshop XII
5
6. OPeNDAP HDF5 Handler with HDF-EOS5
View Data
(User)
Remote
HDF-EOS5
HDF5
Visualization
tools
(grads)
HDF5 Handler
Client Library
(libnc-dap)
Server
(apache)
DAP Protocol (via http)
10/17/2008
HDF and HDF-EOS Workshop XII
6
7. Challenges: HDF- EOS5
• Grid with No Geolocation data
Clients expect Grid with Geolocation data
• Some Attributes stored as Extremely Long
String(s) (e.g. StructMetada.0)
Clients expect structured attributes
10/17/2008
HDF and HDF-EOS Workshop XII
7
8. Tweaks for HDF-EOS5
Added two default HDF5 handler
configuration options:
• --enable-eos-grid
• --enable-eos-meta
10/17/2008
HDF and HDF-EOS Workshop XII
8
9. Challenges: OPeNDAP Clients
• Need special attributes on dataset.
• Need shared geolocation variables.
10/17/2008
HDF and HDF-EOS Workshop XII
9
10. Tweaks for OPeNDAP Clients
Added two optional handler
configuration options:
• --enable-short-name
• --enable-CF
10/17/2008
HDF and HDF-EOS Workshop XII
10
11. OPeNDAP HDF5 Handler with HDF-EOS5
View Data
(User)
Remote
HDF-EOS5
HDF5
Visualization
tools
(grads)
HDF5 Handler
w/ CF options
Client Library
(libnc-dap)
Server
(apache)
DAP Protocol (via http)
10/17/2008
HDF and HDF-EOS Workshop XII
11
12. Day After Server Tweaks
• Finally, Happy Clients!
10/17/2008
HDF and HDF-EOS Workshop XII
12
13. Problems of Tweaking HDF5 Handler
Remember that we added two optional
HDF5 handler configuration options:
1. --enable-short-name
2. --enable-CF
Causes ambiguity among
variable names
(e.g. /GroupA/ozone vs.
/GroupB/ozone)
Drops some key attributes
(e.g. StructMetdata, HDF_ROOT_GROUP)
10/17/2008
HDF and HDF-EOS Workshop XII
13
14. HDF5-Friendly OPeNDAP Client Library
View???
Groups
HDF5 Groups
View Data
(User)
Remote
HDF5
Visualization
tools
(grads)
HDF5 Handler
w/ CF options
HDF5-Friendly
Client Library
(liboc-dap)
(libnc-dap)
Server
(apache)
DAP Protocol (via http)
10/17/2008
HDF and HDF-EOS Workshop XII
14
15. Traditional Groups
Example:OPeNDAP clientin HDF5
library: It’s an attribute that I
don’t understand. I’ll ignore it.
HDF5-Friendly
OPeNDAP client
library: I was waiting
for this key attribute to
re-construct HDF5
10/17/2008
HDF and HDF-EOS Workshop XII
15
16. Example: Reference in HDF5
• Important for NPOESS
• Object / Regional Reference
• Map to DAP URL
10/17/2008
HDF and HDF-EOS Workshop XII
16
17. Example: dap2h5
View Group/Ref.
Group/Ref.
View Data
(User)
Remote
HDF5
• HDF5-F. Client
A test application for the client library
Library construct HDF5 from DAP output
HDF5 Handler
• It canApp.
(dap2h5)
HDF5-Friendly
Client Library
(liboc-dap)
Server
(apache)
DAP Protocol (via http)
10/17/2008
HDF and HDF-EOS Workshop XII
17
19. Visualizing HDF-EOS5 Swath Problem
Visualizing HDF-EOS5 Grids
View Swath
???
Swath
View Data
(User)
Remote
HDF-EOS5
HDF5
Visualization
Tools
tools
(gradsoc)
(grads)
HDF5 Handler
w/ CF option
HDF5-Friendly
Client Library
(liboc-dap)
(libnc-dap)
Server
(apache)
DAP Protocol (via http)
10/17/2008
HDF and HDF-EOS Workshop XII
19
20. Demo: MLS swath via GrADS
• GrADS coupled with our client
library
• The client library provides grid
mapping from swath data
• GrADS displays swath through
HDF-EOS5 specific client library
API calls
21. Summary of Client Library (Prototype)
• Finished coding
• Tested with the GrADS visualization
client
• Documented
• Working on a demo DAP to HDF5
tool by using the client library
prototype
• Will test with NCL if time allows
10/17/2008
HDF and HDF-EOS Workshop XII
21
22. Caution
• Our Client Library is ONLY A PROTOTYPE!
It does NOT support all DAP data types
It does NOT support all AURA files
It does NOT support all Visualization clients
10/17/2008
HDF and HDF-EOS Workshop XII
22
23. Summary
• HDF5 Access via OPeNDAP
Is Easy
Is used by GES DISC to serve Aura
files
• HDF5-Friendly OPeNDAP Client Library
Serves HDF5 better(EOS swath)
Benefits visualization clients
10/17/2008
HDF and HDF-EOS Workshop XII
23
24. Future Work
• HDF5 to DAP2 Mapping Document
• Release HDF5-friendly OPeNDAP
Client Library (Prototype)
• URL: http://hdfgroup.org/projects/opendap
10/17/2008
HDF and HDF-EOS Workshop XII
24
25. Credits
Mike Folk (The HDF Group)
Robert McGrath (NCSA)
Peter Leonard, Daniel Kahn, Marghi Hopkins (ADNET)
Christopher Lynnes, James Johnson, Denis Nadeau, Fan Fang
(NASA)
Jennifer Adams (GrADS)
Dave Brown (UCAR)
10/17/2008
HDF and HDF-EOS Workshop XII
25
26. Acknowledgement
This work was supported basing upon the
Cooperative Agreement with the National
Aeronautics and Space Administration (NASA)
under NASA grant NNX06AC83A,
NNX08A077A and NNX06AG75A. Any opinions,
findings, and conclusions or recommendations
expressed in this material are those of the
author(s) and do not necessarily reflect the
views of NASA.
10/17/2008
HDF and HDF-EOS Workshop XII
26
Notes de l'éditeur
Here’s a summary of our effort regarding client library.