HDF and HDF-EOS Workshop XX (2017)

1. JL2-SESIP-0717
Data Are from Mars,
Tools Are from Venus
H. Joe Lee (hyoklee@hdfgroup.org)
The HDF Group
This work was supported by NASA/GSFC under
Raytheon Co. contract number NNG15HZ39C
All images used in this presentation are from
autodraw.com for public use.

2. JL2-SESIP-0717
2
No “Earth” in title?
“Men Are from Mars, Women are from Venus”
- John Gray

3. JL2-SESIP-0717
3
Are data from Mars?
Why can’t I use Earth tools?
Correct geo-referencing

4. JL2-SESIP-0717
4
Are tools from Venus?
Why can’t I open Earth data?

5. JL2-SESIP-0717
5
Data Producers from Mars
• I want my data slim and efficient.
• How can I save money in managing data?
Tool Developers from Venus
• I make my tool work for popular data first.
• Can I make money by supporting your data?

6. JL2-SESIP-0717
6
Result: Frustrated Users

7. JL2-SESIP-0717
7
We (HDFEOS.org) can help.

8. JL2-SESIP-0717
8
• Hierarchical Data Format (HDF)
• Network Common Data Format (netCDF)
• Geospatial Tagged Image File Format
(GeoTIFF)
• Keyhole Markup Language (KML) /
zipped KML (KMZ)
• Comma-separated values (CSV)
• etc.
We identify gaps in File Formats

9. JL2-SESIP-0717
9
• hdf
• netcdf-C
• netcdf-Java
• HDF – Earth Observing System (hdf-eos)
• Climate and Forecast Metadata (CF)
conventions
• Geospatial Data Abstraction Library (GDAL)
• etc.
We identify gaps in Libraries

10. JL2-SESIP-0717
10
• Microsoft Excel
• Esri ArcGIS
• Google Earth
• MATLAB
• Python
• Interactive Data Language (IDL)
• Panoply
• Integrated Data Viewer (IDV)
• HDFView
• h5dump
• Etc.
We identify gaps in Tools

11. JL2-SESIP-0717
11
• Open-source Project for a Network Data
Access Protocol (OPeNDAP)
• Web Map Service (WMS)
• Web Map Tile Service (WMTS)
• Web Coverage Service (WCS)
• etc.
We identify gaps in Services

12. JL2-SESIP-0717
12
• File conversion
• Libraries and tools usage
• NASA HDF product specific examples
• Demo services (e.g., Hyrax*, THREDDS**)
AND we provide Solutions…
*Hyrax is the data server from OPeNDAP.
**Thematic Real-time Environmental Distributed Data Services

13. JL2-SESIP-0717
13
• Make HDF5 data work with
– GDAL
– netCDF
– Hyrax/THREDDS
• Don’t forget a few key CF conventions.
• Follow DIWG* recommendations.
Suggestions for data producers
*Data Interoperability Working Group

14. JL2-SESIP-0717
14
• Download and test NASA HDF products.
• Support them natively.
• Support augmentation.
– VRT* in GDAL
– NcML** in netCDF
• Support 3D visualization for data in the air.
Suggestions for tool developers
*Virtual Dataset in XML format
**netCDF Markup Language

15. JL2-SESIP-0717
15
3D Visualization

16. JL2-SESIP-0717
16
• Try OPeNDAP first. CSV may be enough.
• Try netCDF conversion / augmentation.
• Correct metadata with NcML / VRT.
• Try GEE* instead of GDAL.
• Use CMR** wisely.
Suggestions for end-users
*GDAL Enhancement for ESDIS project
** Core Metadata Repository

17. JL2-SESIP-0717
17
• Hadoop / Spark (streaming) / Dask
• Parquet / Arrow
• Elastic Search / Kibana
How about Big (fast) data?

18. JL2-SESIP-0717
18
Streaming Swath
Hadoop + Spark Streaming + Elasticsearch & Kibana
S3 + AWS Lambda + Elasticsearch & Kibana

19. JL2-SESIP-0717
19
• scikit-learn / keras / h2o.ai
• Please contact us at
eoshelp@hdfgroup.org if you’d like to see
examples on machine learning.
Future: (Deep) Machine Learning?

20. JL2-SESIP-0717
20
Deep Learning: Amazon
Rekognition

21. JL2-SESIP-0717
21
Alexa, what does MODIS see now?
Alexa: I can recognize “Alps.”

22. JL2-SESIP-0717
22
This work was supported by
NASA/GSFC under Raytheon Co.
contract number NNG15HZ39C