1. Implementation of HDFEOS5 and HDF5 in NCL
Wei Huang, Dave Brown, Mary Haley,
Rick Brownrigg
CISL/NCAR
Dennis Shea, Adam Phillips
CGD/NCAR
huangwei@ucar.edu

2. What is NCL
◦ NCL stands for NCAR Command Language
 check http://www.ncl.ucar.edu for more information
2/17/2014
CISL/NCAR

3. What is NCL
◦ NCL stands for NCAR Command Language
 check http://www.ncl.ucar.edu for more information
◦ NCL is developed at CISL/NCAR, for scientific
data analysis and visualization
2/17/2014
CISL/NCAR

4. What is NCL
◦ NCL stands for NCAR Command Language
 check http://www.ncl.ucar.edu for more information
◦ NCL is developed at CISL/NCAR, for scientific
data analysis and visualization
◦ Free available as binaries, or open source
2/17/2014
CISL/NCAR

5. What is NCL
◦ NCL stands for NCAR Command Language
 check http://www.ncl.ucar.edu for more information
◦ NCL is developed at CISL/NCAR, for scientific
data analysis and visualization
◦ Free available as binaries, or open source
◦ Supports NetCDF (3 and 4), GRIB (1 and 2),
HDF, HDF-EOS (as end of 2008)
2/17/2014
CISL/NCAR

6. What is NCL
◦ NCL stands for NCAR Command Language
 check http://www.ncl.ucar.edu for more information
◦ NCL is developed at CISL/NCAR, for scientific
data analysis and visualization
◦ Free available as binaries, or open source
◦ Supports NetCDF (3 and 4), GRIB (1 and 2),
HDF, HDF-EOS (as end of 2008)
◦ Newly added HDF5 and HDF-EOS5 (in 2009)
2/17/2014
CISL/NCAR

7. What is NCL
◦ NCL stands for NCAR Command Language
 check http://www.ncl.ucar.edu for more information
◦ NCL is developed at CISL/NCAR, for scientific
data analysis and visualization
◦ Free available as binaries, or open source
◦ Supports NetCDF (3 and 4), GRIB (1 and 2),
HDF, HDF-EOS (as end of 2008)
◦ Newly added HDF5 and HDF-EOS5 (in 2009)
◦ High quality graphics
2/17/2014
CISL/NCAR

8. What is NCL
◦ NCL stands for NCAR Command Language
 check http://www.ncl.ucar.edu for more information
◦ NCL is developed at CISL/NCAR, for scientific
data analysis and visualization
◦ Free available as binaries, or open source
◦ Supports NetCDF (3 and 4), GRIB (1 and 2),
HDF, HDF-EOS (as end of 2008)
◦ Newly added HDF5 and HDF-EOS5 (in 2009)
◦ High quality graphics
◦ Hundreds of example scripts, lots of sample data
2/17/2014
CISL/NCAR

9. What is NCL
◦ NCL stands for NCAR Command Language
 check http://www.ncl.ucar.edu for more information
◦ NCL is developed at CISL/NCAR, for scientific
data analysis and visualization
◦ Free available as binaries, or open source
◦ Supports NetCDF (3 and 4), GRIB (1 and 2),
HDF, HDF-EOS (as end of 2008)
◦ Newly added HDF5 and HDF-EOS5 (in 2009)
◦ High quality graphics
◦ Hundreds of example scripts, lots of sample data
◦ PyNIO, Python interface for same NCL file I/O
2/17/2014
CISL/NCAR

10. HDF-EOS5 Implementation

Start with HDF-EOS Code
◦ Reconstruct function structure
◦ Create new HDF-EOS5 functions
◦ Variables follow the NetCDF data model
2/17/2014
CISL/NCAR

11. HDF-EOS5 Implementation

Start with HDF-EOS Code
◦ Reconstruct function structure
◦ Create new HDF-EOS5 functions
◦ Variables follow the NetCDF data model

How NCL open HDF-EOS5 files
◦ Command: f = addfile(filename, “r”)
 Filename has the form: body.suffix




NetCDF has suffix: nc, cdf, netcdf
Grib has: gr, gr1, grb, grb1, grib, gr2, grb2, grib2
HDF has: hdf, h4
HDF-EOS has: hdfeos, he2, he4
2/17/2014
CISL/NCAR

12. HDF-EOS5 Implementation
(continue)

HDF-EOS5 has suffix: he5
◦ Function addfile check file suffix, and when
suffix is he5, it knows to move function
pointers put to HDF-EOS5 functions
◦ HDF-EOS5 uses HDF5, but we do not need to
deal with HDF5 directly
2/17/2014
CISL/NCAR

13. HDF-EOS5 Sample
2/17/2014
CISL/NCAR

14. HDF-EOS5 Sample (continue)
2/17/2014
CISL/NCAR

15. HDF-EOS5 Sample (continue)
Reflectivity of the ground pixel
2/17/2014
CISL/NCAR

16. More HDF-EOS5 info

Visit NCL web site:
 http://www.ncl.ucar.edu/Applications/HDF.shtml

For Sample Plots and Sample Scripts
Currently at Beta Version
 May provide user binary on request

2/17/2014
CISL/NCAR

17. 2/17/2014
CISL/NCAR

18. HDF5 Implementation

HDF5 is handled differently in NCL from
HDF in many ways
◦ Not just simply copy/modify from existing
HDF code
2/17/2014
CISL/NCAR

19. HDF5 Implementation

HDF5 is handled differently in NCL from
HDF in many ways
◦ Not just simply copy/modify from existing
HDF code

HDF5 Tools are useful:
◦ h5dump, h5ls, hdfview
◦ Help us understand data
structural/attribute/value, and coding
2/17/2014
CISL/NCAR

20. HDF5 Implementation (continue)

Read Data on Demand
◦ Data structures are saved in memory, but not data
itself (sometimes, it is impossible)
◦ Data are read when in need
2/17/2014
CISL/NCAR

21. HDF5 Implementation (continue)

First, HDF5 Has a Group (Root Group)
2/17/2014
CISL/NCAR

22. HDF5 Implementation (continue)
First, HDF5 Has a Group (Root Group)
 Second, a Group Has:

◦ Attributes
 Name, Type, Value, ……
2/17/2014
CISL/NCAR

23. HDF5 Implementation (continue)
First, HDF5 Has a Group (Root Group)
 Second, a Group Has:

◦ Attributes
 Name, Type, Value, ……
◦ Datasets
 Attributes
 Name, Type, Dimensions, Value, ……
2/17/2014
CISL/NCAR

24. HDF5 Implementation (continue)
First, HDF5 Has a Group (Root Group)
 Second, a Group Has:

◦ Attributes
 Name, Type, Value, ……
◦ Datasets
 Attributes
 Name, Type, Dimensions, Value, ……
◦ Links
◦ Groups
2/17/2014
CISL/NCAR

25. HDF5 Implementation (continue)
First, HDF5 Has a Group (Root Group)
 Second, a Group Has:

◦ Attributes
 Name, Type, Value, ……
◦ Datasets
 Attributes
 Name, Type, Dimensions, Value, ……
◦ Links
◦ Groups

Let us have a visual view
2/17/2014
CISL/NCAR

26. HDF5 Implementation (continue)
2/17/2014
CISL/NCAR

27. HDF5 Implementation (continue)
2/17/2014
CISL/NCAR

28. HDF5 Implementation (continue)
2/17/2014
CISL/NCAR

29. HDF5 Implementation (continue)
2/17/2014
CISL/NCAR

30. HDF5 Implementation (continue)
2/17/2014
CISL/NCAR

31. HDF5 Implementation (continue)
2/17/2014
CISL/NCAR

32. Sample Script to Read HDF5 and
Some Print-out

f = addfile(“MOD13A2A.h5", "r”)
◦ ;print(f)
;NCL treats “;” as comments thereafter
◦ printVarSummary(f)
2/17/2014
CISL/NCAR

33. Sample Script to Read HDF5 and
Some Print-out (continue)
Variable: f
filename:
MOD13A2A
path: MOD13A2A.h5
file global attributes:
ArchiveMetadata.0_GLOSDS :
GROUP
= ARCHIVEDMETADATA
GROUPTYPE
= MASTERGROUP
….
2/17/2014
CISL/NCAR

34. Sample Script to Read HDF5 and
Some Print-out (continue)
dimensions:
YDim:MODIS_Grid_16DAY_1km_VI = 1200
XDim:MODIS_Grid_16DAY_1km_VI = 1200
2/17/2014
CISL/NCAR

35. Sample Script to Read HDF5 and
Some Print-out (continue)
variables:
group </HDF4_DIMGROUP>
group </MODIS_Grid_16DAY_1km_VI>
HDF4_OBJECT_NAME : MODIS_Grid_16DAY_1km_VI
HDF4_OBJECT_TYPE : Vgroup
HDF4_REF_NUM : 2
2/17/2014
CISL/NCAR

36. Sample Script to Read HDF5 and
Some Print-out (continue)
short 1_km_16_days_EVI ( YDim:MODIS_Grid_16DAY_1km_VI,
XDim:MODIS_Grid_16DAY_1km_VI )
HDF4_REF_NUM : 6
_FillValue : -3000
add_offset :
0
add_offset_err :
long_name :
0
1 km 16 days EVI
scale_factor : 10000
scale_factor_err :
units :
0
EVI
valid_range : ( -2000, 10000 )
…...
2/17/2014
CISL/NCAR

37. Sample Script to Read HDF5 and
Some Print-out (continue)
f = addfile(“MOD13A2A.h5", "r”)
printVarSummary(f)
Variable: f
(0)
filename:
MOD13A2A
path: MOD13A2A.h5
file global attributes:
ArchiveMetadata.0_GLOSDS :
GROUP
= ARCHIVEDMETADATA
GROUPTYPE
….
= MASTERGROUP
dimensions:
YDim:MODIS_Grid_16DAY_1km_VI = 1200
XDim:MODIS_Grid_16DAY_1km_VI = 1200
variables:
group </HDF4_DIMGROUP>
2/17/2014
CISL/NCAR

38. Sample Script to Read HDF5 and
Some Print-out (continue)
grps = getfilegroups(f, "/", 0)
print(grps)
2/17/2014
CISL/NCAR

39. Sample Script to Read HDF5 and
Some Print-out (continue)
grps = getfilegroups(f, "/", 0)
print(grps)
Variable: grps
Type: string
Total Size: 16 bytes
4 values
Number of Dimensions: 1
Dimensions and sizes: [4]
Coordinates:
(0)
/HDF4_DIMGROUP
(1)
/MODIS_Grid_16DAY_1km_VI
(2)
/MODIS_Grid_16DAY_1km_VI/Data Fields
(3)
/MODIS_Grid_16DAY_1km_VI/Grid Attributes
2/17/2014
CISL/NCAR

40. Sample Script to Read HDF5 and
Some Print-out (continue)
grps2 = getfilegroups(f, "/MODIS_Grid_16DAY_1km_VI", 0)
print(grps2)
2/17/2014
CISL/NCAR

41. Sample Script to Read HDF5 and
Some Print-out (continue)
grps2 = getfilegroups(f, "/MODIS_Grid_16DAY_1km_VI", 0)
print(grps2)
Variable: grps2
Type: string
Total Size: 8 bytes
2 values
Number of Dimensions: 1
Dimensions and sizes: [2]
Coordinates:
(0) /MODIS_Grid_16DAY_1km_VI/Data Fields
(1) /MODIS_Grid_16DAY_1km_VI/Grid Attributes
2/17/2014
CISL/NCAR

42. Sample Script to Read HDF5 and
Some Print-out (continue)
v1 = f->1_km_16_days_red_reflectance
printVarSummary(v1)
2/17/2014
CISL/NCAR

43. Sample Script to Read HDF5 and
Some Print-out (continue)
v1 = f->1_km_16_days_red_reflectance
printVarSummary(v1)
Variable: v1
Type: short
Total Size: 2880000 bytes
1440000 values
Number of Dimensions: 2
Dimensions and sizes:
[YDim:MODIS_Grid_16DAY_1km_VI | 1200] x[XDim:MODIS_Grid_16DAY_1km_VI | 1200]
Coordinates:
Number Of Attributes: 13
_FillValue : -1000
long_name : 1 km 16 days red reflectance
……
2/17/2014
CISL/NCAR

44. Sample Script to Read HDF5 and
Some Print-out (continue)
vn = grps2(0)+"/1 km 16 days blue reflectance”
v2 = f->$vn$
printVarSummary(v2)
2/17/2014
CISL/NCAR

45. Sample Script to Read HDF5 and
Some Print-out (continue)
vn = grps2(0)+"/1 km 16 days blue reflectance”
v2 = f->$vn$
printVarSummary(v2)
Variable: v2
Type: short
Total Size: 2880000 bytes
1440000 values
Number of Dimensions: 2
Dimensions and sizes: [YDim:MODIS_Grid_16DAY_1km_VI | 1200] x [XDim:MODIS_Grid_16DAY_1km_VI | 1200]
Coordinates:
Number Of Attributes: 13
……
_FillValue : -1000
long_name : 1 km 16 days blue reflectance
……
2/17/2014
CISL/NCAR

46. HDF5 Implementation (continue)
2/17/2014
CISL/NCAR

47. HDF5 Implementation (continue)
2/17/2014
CISL/NCAR

48. HDF5 Implementation (continue)
2/17/2014
CISL/NCAR

49. HDF5 Implementation (continue)
2/17/2014
CISL/NCAR

50. HDF5 Implementation (continue)
2/17/2014
CISL/NCAR

51. HDF5 Implementation (continue)
2/17/2014
CISL/NCAR

52. Another Sample Script to Read HDF5
fn = "K1VHR_23SEP2008_2330_L02_SST.h5”
f = addfile(fn, "r”)
print(f)
lon1d = f->/SST/SST_Composite.Longitude
lat1d = f->/SST/SST_Composite.Latitude
sst1d = f->/SST/SST_Dataset.SST
2/17/2014
CISL/NCAR

53. Print-out
filename:
K1VHR_23SEP2008_2330_L02_SST
file global attributes:
dimensions:
DIM_000 = 36381
variables:
group </PRODUCT_INFORMATION>
group </PRODUCT_METADATA>
group </PRODUCT_METADATA/PRODUCT_DETAILS>
group </PRODUCT_METADATA/PROJECTION_PARAMETERS>
group </SST>
compound <SST_Composite> (Latitude, Longitude, SST) (DIM_000)
compound <SST_Dataset>
(Latitude, Longitude, SST) (DIM_000)
group </SST/GP_PARAM_INFO>
2/17/2014
CISL/NCAR

54. Plot for Compound Dataset
2/17/2014
CISL/NCAR

55. Add This Lines to Generate above Plot
dimlon = dimsizes(lon1d)
res@mpLimitMode
= "Corners”
dimlat = dimsizes(lat1d)
res@mpLeftCornerLatF
= lat1d(0)
sst1d@long_name = "SST”
res@mpLeftCornerLonF = lon1d(0)
sst1d@units = "K”
res@mpRightCornerLatF = lat1d(dimlat(0) - 1)
sst1d@_FillValue = -999.
res@mpRightCornerLonF = lon1d(dimlon(0) - 1)
wks = gsn_open_wks ("x11",”K1VHR”)
res@sfXArray = lon1d
res
res@sfYArray = lat1d
= True
res@cnFillOn
= True
res@cnLinesOn
plot = gsn_csm_contour_map (wks, sst1d, res)
= False
res@cnLineLabelsOn = False
res@cnFillMode
= "RasterFill”
res@cnRasterSmoothingOn = True
res@gsnSpreadColors = True
res@gsnAddCyclic
= False
res@lbLabelAutoStride = True
res@tiMainString
= fn
2/17/2014
CISL/NCAR

56. Summary

No need to become an expert on HDFEOS5 or HDF5 interfaces
◦ User can “check” and “go”
2/17/2014
CISL/NCAR

57. Summary

No need to become an expert on HDFEOS5 or HDF5 interfaces
◦ User can “check” and “go”

NCL with HDF-EOS5 is in Beta Version
◦ Will be released early 2010
◦ User may ask for binary
2/17/2014
CISL/NCAR

58. Summary

No need to become an expert on HDFEOS5 or HDF5 interfaces
◦ User can “check” and “go”

NCL with HDF-EOS5 is in Beta Version
◦ Will be released early 2010
◦ User may ask for binary

NCL with HDF5 is in Alpha Version
◦ Will be released Later 2010
◦ Try to be a friendly user for early beta test
2/17/2014
CISL/NCAR

59. Conclusion

HDF-EOS5 is handled well in NCL
◦
◦
◦
◦
◦
Swath data tested
Grid data tested
Prototype for Point data
Prototype for Zonal-Average data
Currently in Beta Version
2/17/2014
CISL/NCAR

60. Conclusion

HDF-EOS5 is handled well in NCL
◦
◦
◦
◦
◦

Swath data tested
Grid data tested
Prototype for Point data
Prototype for Zonal-Average data
Currently in Beta Version
Lots of can be done with HDF5 in NCL
◦ Some satellite data tested
◦ Currently in Alpha Version
2/17/2014
CISL/NCAR

61. 2/17/2014
CISL/NCAR

62. 2/17/2014
CISL/NCAR

63. A Full NCL Script
load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl”
load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl”
f = addfile("NISESSMI.h5", "r”)
grps1 = getfilegroups(f, "/Southern Hemisphere/", 0)
vs = grps1(0)+"/Extent”
v1 = f->$vs$
grps2 = getfilegroups(f, "/Northern Hemisphere/", 0)
vn = grps2(0)+"/Extent”
v2 = f->$vn$
wks = gsn_open_wks ("ps","NISESSMI”)
gsn_define_colormap(wks,"rainbow”)
2/17/2014
CISL/NCAR

64. A Full NCL Script (Continue)
res
= True
res@tiMainString = "NISESSMI.h5”
res@cnFillOn
= True
res@cnLinesOn
= False
res@cnLineLabelsOn = False
res@gsnSpreadColors = True
res@lbLabelAutoStride= True
res@lbOrientation = "Vertical”
res@cnFillMode
= "RasterFill"
; faster
res@cnRasterSmoothingOn = True
res@tiMainString = vs
plot1 = gsn_csm_contour (wks, tofloat(v1), res)
res@tiMainString = vn
plot2 = gsn_csm_contour (wks, tofloat(v2), res)
2/17/2014
CISL/NCAR