1. Implementation of HDFEOS5 and HDF5 in NCL Wei Huang, Dave Brown, Mary Haley, Rick Brownrigg CISL/NCARDennis Shea, Adam Phillips CGD/NCARhuangwei@ucar.edu

2. What is NCL NCL stands for NCAR Command Language check http://www.ncl.ucar.edu for more information2/17/2014CISL/NCAR 3. What is NCL NCL stands for NCAR Command Language check http://www.ncl.ucar.edu for more information NCL is developed at CISL/NCAR, for scientific data analysis and visualization2/17/2014CISL/NCAR 4. What is NCL NCL stands for NCAR Command Language check http://www.ncl.ucar.edu for more information NCL is developed at CISL/NCAR, for scientific data analysis and visualization Free available as binaries, or open source2/17/2014CISL/NCAR 5. What is NCL NCL stands for NCAR Command Language check http://www.ncl.ucar.edu for more information NCL is developed at CISL/NCAR, for scientific data analysis and visualization Free available as binaries, or open source Supports NetCDF (3 and 4), GRIB (1 and 2), HDF, HDF-EOS (as end of 2008)2/17/2014CISL/NCAR 6. What is NCL NCL stands for NCAR Command Language check http://www.ncl.ucar.edu for more information NCL is developed at CISL/NCAR, for scientific data analysis and visualization Free available as binaries, or open source Supports NetCDF (3 and 4), GRIB (1 and 2), HDF, HDF-EOS (as end of 2008) Newly added HDF5 and HDF-EOS5 (in 2009)2/17/2014CISL/NCAR 7. What is NCL NCL stands for NCAR Command Language check http://www.ncl.ucar.edu for more information NCL is developed at CISL/NCAR, for scientific data analysis and visualization Free available as binaries, or open source Supports NetCDF (3 and 4), GRIB (1 and 2), HDF, HDF-EOS (as end of 2008) Newly added HDF5 and HDF-EOS5 (in 2009) High quality graphics2/17/2014CISL/NCAR 8. What is NCL NCL stands for NCAR Command Language check http://www.ncl.ucar.edu for more information NCL is developed at CISL/NCAR, for scientific data analysis and visualization Free available as binaries, or open source Supports NetCDF (3 and 4), GRIB (1 and 2), HDF, HDF-EOS (as end of 2008) Newly added HDF5 and HDF-EOS5 (in 2009) High quality graphics Hundreds of example scripts, lots of sample data 2/17/2014CISL/NCAR 9. What is NCL NCL stands for NCAR Command Language check http://www.ncl.ucar.edu for more information NCL is developed at CISL/NCAR, for scientific data analysis and visualization Free available as binaries, or open source Supports NetCDF (3 and 4), GRIB (1 and 2), HDF, HDF-EOS (as end of 2008) Newly added HDF5 and HDF-EOS5 (in 2009) High quality graphics Hundreds of example scripts, lots of sample data PyNIO, Python interface for same NCL file I/O 2/17/2014CISL/NCAR 10. HDF-EOS5 Implementation Start with HDF-EOS Code Reconstruct function structure Create new HDF-EOS5 functions Variables follow the NetCDF data model2/17/2014CISL/NCAR 11. HDF-EOS5 Implementation Start with HDF-EOS Code Reconstruct function structure Create new HDF-EOS5 functions Variables follow the NetCDF data modelHow NCL open HDF-EOS5 files Command: f = addfile(filename, r) Filename has the form: body.suffix NetCDF has suffix: nc, cdf, netcdf Grib has: gr, gr1, grb, grb1, grib, gr2, grb2, grib2 HDF has: hdf, h4 HDF-EOS has: hdfeos, he2, he4 2/17/2014CISL/NCAR 12. HDF-EOS5 Implementation (continue) HDF-EOS5 has suffix: he5 Function addfile check file suffix, and when suffix is he5, it knows to move function pointers put to HDF-EOS5 functions HDF-EOS5 uses HDF5, but we do not need to deal with HDF5 directly2/17/2014CISL/NCAR 13. HDF-EOS5 Sample2/17/2014CISL/NCAR 14. HDF-EOS5 Sample (continue)2/17/2014CISL/NCAR 15. HDF-EOS5 Sample (continue)Reflectivity of the ground pixel 2/17/2014CISL/NCAR 16. More HDF-EOS5 info Visit NCL web site: http://www.ncl.ucar.edu/Applications/HDF.shtmlFor Sample Plots and Sample ScriptsCurrently at Beta Version May provide user binary on request 2/17/2014CISL/NCAR 17. 2/17/2014CISL/NCAR 18. HDF5 Implementation HDF5 is handled differently in NCL from HDF in many ways Not just simply copy/modify from existing HDF code2/17/2014CISL/NCAR 19. HDF5 Implementation HDF5 is handled differently in NCL from HDF in many ways Not just simply copy/modify from existing HDF codeHDF5 Tools are useful: h5dump, h5ls, hdfview Help us understand data structural/attribute/value, and coding2/17/2014CISL/NCAR 20. HDF5 Implementation (continue) Read Data on Demand Data structures are saved in memory, but not data itself (sometimes, it is impossible) Data are read when in need2/17/2014CISL/NCAR 21. HDF5 Implementation (continue) First, HDF5 Has a Group (Root Group)2/17/2014CISL/NCAR 22. HDF5 Implementation (continue) First, HDF5 Has a Group (Root Group) Second, a Group Has: Attributes Name, Type, Value, 2/17/2014CISL/NCAR 23. HDF5 Implementation (continue) First, HDF5 Has a Group (Root Group) Second, a Group Has: Attributes Name, Type, Value, Datasets Attributes Name, Type, Dimensions, Value, 2/17/2014CISL/NCAR 24. HDF5 Implementation (continue) First, HDF5 Has a Group (Root Group) Second, a Group Has: Attributes Name, Type, Value, Datasets Attributes Name, Type, Dimensions, Value, Links Groups 2/17/2014CISL/NCAR 25. HDF5 Implementation (continue) First, HDF5 Has a Group (Root Group) Second, a Group Has: Attributes Name, Type, Value, Datasets Attributes Name, Type, Dimensions, Value, Links Groups Let us have a visual view 2/17/2014CISL/NCAR 26. HDF5 Implementation (continue)2/17/2014CISL/NCAR 27. HDF5 Implementation (continue)2/17/2014CISL/NCAR 28. HDF5 Implementation (continue)2/17/2014CISL/NCAR 29. HDF5 Implementation (continue)2/17/2014CISL/NCAR 30. HDF5 Implementation (continue)2/17/2014CISL/NCAR 31. HDF5 Implementation (continue)2/17/2014CISL/NCAR 32. Sample Script to Read HDF5 and Some Print-out f = addfile(MOD13A2A.h5", "r) ;print(f);NCL treats ; as comments thereafter printVarSummary(f)2/17/2014CISL/NCAR 33. Sample Script to Read HDF5 and Some Print-out (continue) Variable: f filename: MOD13A2A path: MOD13A2A.h5 file global attributes: ArchiveMetadata.0_GLOSDS : GROUP = ARCHIVEDMETADATA GROUPTYPE = MASTERGROUP .2/17/2014CISL/NCAR 34. Sample Script to Read HDF5 and Some Print-out (continue) dimensions: YDim:MODIS_Grid_16DAY_1km_VI = 1200 XDim:MODIS_Grid_16DAY_1km_VI = 12002/17/2014CISL/NCAR 35. Sample Script to Read HDF5 and Some Print-out (continue) variables: group group HDF4_OBJECT_NAME : MODIS_Grid_16DAY_1km_VI HDF4_OBJECT_TYPE : Vgroup HDF4_REF_NUM : 22/17/2014CISL/NCAR 36. Sample Script to Read HDF5 and Some Print-out (continue) short 1_km_16_days_EVI ( YDim:MODIS_Grid_16DAY_1km_VI, XDim:MODIS_Grid_16DAY_1km_VI ) HDF4_REF_NUM : 6 _FillValue : -3000 add_offset :0add_offset_err : long_name :01 km 16 days EVIscale_factor : 10000 scale_factor_err : units :0EVIvalid_range : ( -2000, 10000 ) ...2/17/2014CISL/NCAR 37. Sample Script to Read HDF5 and Some Print-out (continue) f = addfile(MOD13A2A.h5", "r) printVarSummary(f) Variable: f (0) filename:MOD13A2Apath: MOD13A2A.h5 file global attributes: ArchiveMetadata.0_GLOSDS : GROUP = ARCHIVEDMETADATAGROUPTYPE .= MASTERGROUPdimensions: YDim:MODIS_Grid_16DAY_1km_VI = 1200 XDim:MODIS_Grid_16DAY_1km_VI = 1200 variables: group 2/17/2014CISL/NCAR 38. Sample Script to Read HDF5 and Some Print-out (continue) grps = getfilegroups(f, "/", 0) print(grps)2/17/2014CISL/NCAR 39. Sample Script to Read HDF5 and Some Print-out (continue) grps = getfilegroups(f, "/", 0) print(grps)Variable: grps Type: string Total Size: 16 bytes4 values Number of Dimensions: 1 Dimensions and sizes: [4] Coordinates: (0)/HDF4_DIMGROUP(1)/MODIS_Grid_16DAY_1km_VI(2)/MODIS_Grid_16DAY_1km_VI/Data Fields(3)/MODIS_Grid_16DAY_1km_VI/Grid Attributes2/17/2014CISL/NCAR 40. Sample Script to Read HDF5 and Some Print-out (continue) grps2 = getfilegroups(f, "/MODIS_Grid_16DAY_1km_VI", 0) print(grps2)2/17/2014CISL/NCAR 41. Sample Script to Read HDF5 and Some Print-out (continue) grps2 = getfilegroups(f, "/MODIS_Grid_16DAY_1km_VI", 0) print(grps2) Variable: grps2 Type: string Total Size: 8 bytes 2 values Number of Dimensions: 1 Dimensions and sizes: [2] Coordinates: (0) /MODIS_Grid_16DAY_1km_VI/Data Fields (1) /MODIS_Grid_16DAY_1km_VI/Grid Attributes 2/17/2014CISL/NCAR 42. Sample Script to Read HDF5 and Some Print-out (continue) v1 = f->1_km_16_days_red_reflectance printVarSummary(v1)2/17/2014CISL/NCAR 43. Sample Script to Read HDF5 and Some Print-out (continue) v1 = f->1_km_16_days_red_reflectance printVarSummary(v1) Variable: v1 Type: short Total Size: 2880000 bytes 1440000 values Number of Dimensions: 2Dimensions and sizes:[YDim:MODIS_Grid_16DAY_1km_VI | 1200] x[XDim:MODIS_Grid_16DAY_1km_VI | 1200]Coordinates: Number Of Attributes: 13 _FillValue : -1000long_name : 1 km 16 days red reflectance 2/17/2014CISL/NCAR 44. Sample Script to Read HDF5 and Some Print-out (continue) vn = grps2(0)+"/1 km 16 days blue reflectance v2 = f->$vn$ printVarSummary(v2)2/17/2014CISL/NCAR 45. Sample Script to Read HDF5 and Some Print-out (continue) vn = grps2(0)+"/1 km 16 days blue reflectance v2 = f->$vn$ printVarSummary(v2) Variable: v2 Type: short Total Size: 2880000 bytes 1440000 values Number of Dimensions: 2 Dimensions and sizes: [YDim:MODIS_Grid_16DAY_1km_VI | 1200] x [XDim:MODIS_Grid_16DAY_1km_VI | 1200] Coordinates: Number Of Attributes: 13 _FillValue : -1000 long_name : 1 km 16 days blue reflectance 2/17/2014CISL/NCAR 46. HDF5 Implementation (continue)2/17/2014CISL/NCAR 47. HDF5 Implementation (continue)2/17/2014CISL/NCAR 48. HDF5 Implementation (continue)2/17/2014CISL/NCAR 49. HDF5 Implementation (continue)2/17/2014CISL/NCAR 50. HDF5 Implementation (continue)2/17/2014CISL/NCAR 51. HDF5 Implementation (continue)2/17/2014CISL/NCAR 52. Another Sample Script to Read HDF5 fn = "K1VHR_23SEP2008_2330_L02_SST.h5 f = addfile(fn, "r) print(f)lon1d = f->/SST/SST_Composite.Longitude lat1d = f->/SST/SST_Composite.Latitude sst1d = f->/SST/SST_Dataset.SST2/17/2014CISL/NCAR 53. Print-out filename: K1VHR_23SEP2008_2330_L02_SST file global attributes: dimensions: DIM_000 = 36381 variables: group group group /PRODUCT_DETAILS> group /PROJECTION_PARAMETERS> group compound (Latitude, Longitude, SST) (DIM_000) compound (Latitude, Longitude, SST) (DIM_000)group /GP_PARAM_INFO>2/17/2014CISL/NCAR 54. Plot for Compound Dataset2/17/2014CISL/NCAR 55. Add This Lines to Generate above Plot dimlon = dimsizes(lon1d)res@mpLimitMode= "Cornersdimlat = dimsizes(lat1d)res@mpLeftCornerLatF= lat1d(0)sst1d@long_name = "SSTres@mpLeftCornerLonF = lon1d(0)sst1d@units = "Kres@mpRightCornerLatF = lat1d(dimlat(0) - 1)sst1d@_FillValue = -999.res@mpRightCornerLonF = lon1d(dimlon(0) - 1)wks = gsn_open_wks ("x11",K1VHR)res@sfXArray = lon1dresres@sfYArray = lat1d= Trueres@cnFillOn= Trueres@cnLinesOnplot = gsn_csm_contour_map (wks, sst1d, res)= Falseres@cnLineLabelsOn = False res@cnFillMode= "RasterFillres@cnRasterSmoothingOn = True res@gsnSpreadColors = True res@gsnAddCyclic= Falseres@lbLabelAutoStride = True res@tiMainString= fn2/17/2014CISL/NCAR 56. Summary No need to become an expert on HDFEOS5 or HDF5 interfaces User can check and go2/17/2014CISL/NCAR 57. Summary No need to become an expert on HDFEOS5 or HDF5 interfaces User can check and goNCL with HDF-EOS5 is in Beta Version Will be released early 2010 User may ask for binary2/17/2014CISL/NCAR 58. Summary No need to become an expert on HDFEOS5 or HDF5 interfaces User can check and goNCL with HDF-EOS5 is in Beta Version Will be released early 2010 User may ask for binaryNCL with HDF5 is in Alpha Version Will be released Later 2010 Try to be a friendly user for early beta test 2/17/2014CISL/NCAR 59. Conclusion HDF-EOS5 is handled well in NCL Swath data tested Grid data tested Prototype for Point data Prototype for Zonal-Average data Currently in Beta Version2/17/2014CISL/NCAR 60. Conclusion HDF-EOS5 is handled well in NCL Swath data tested Grid data tested Prototype for Point data Prototype for Zonal-Average data Currently in Beta VersionLots of can be done with HDF5 in NCL Some satellite data tested Currently in Alpha Version 2/17/2014CISL/NCAR 61. 2/17/2014CISL/NCAR 62. 2/17/2014CISL/NCAR 63. A Full NCL Script load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl f = addfile("NISESSMI.h5", "r) grps1 = getfilegroups(f, "/Southern Hemisphere/", 0) vs = grps1(0)+"/Extent v1 = f->$vs$ grps2 = getfilegroups(f, "/Northern Hemisphere/", 0) vn = grps2(0)+"/Extent v2 = f->$vn$ wks = gsn_open_wks ("ps","NISESSMI) gsn_define_colormap(wks,"rainbow) 2/17/2014CISL/NCAR 64. A Full NCL Script (Continue) res = True res@tiMainString = "NISESSMI.h5 res@cnFillOn = True res@cnLinesOn = False res@cnLineLabelsOn = False res@gsnSpreadColors = True res@lbLabelAutoStride= True res@lbOrientation = "Vertical res@cnFillMode = "RasterFill" ; faster res@cnRasterSmoothingOn = True res@tiMainString = vs plot1 = gsn_csm_contour (wks, tofloat(v1), res) res@tiMainString = vn plot2 = gsn_csm_contour (wks, tofloat(v2), res) 2/17/2014CISL/NCAR