how to meet the cf conventions with ncml for nasa hdf/hdf-eos
TRANSCRIPT
The HDF Group
1 www.hdfgroup.orgJuly 8, 2014 2014 Summer ESIP Federation Meeting
How to Meet the CF Conventions with NcML for NASA HDF/HDF-EOS
Hyo-Kyung Joe Lee and Ted HabermannThe HDF Group
www.hdfgroup.org2014 Summer ESIP Federation Meeting 2
The CF Conventions
Big and complex!
Just focus on a few key conventions:• coordinate variables & attributes (bald)• valid_range / _FillValue (fat)• scale_factor / add_offset (short)• units ($ vs. ₵)
www.hdfgroup.org2014 Summer ESIP Federation Meeting 3
Why the key CF conventions matter
If a data product doesn’t follow them, your NetCDF visualization tool like IDV is useless.
www.hdfgroup.org2014 Summer ESIP Federation Meeting 4
But if you follow them carefully…
you can visualize data instantly & correctly.
www.hdfgroup.org2014 Summer ESIP Federation Meeting 5
OBPG L3 NetCDF-4 Example
The previous screenshots are created from a NASA OBPG beta product.
The product doesn’t work with IDV as is. But we made the product work by augmenting data with NcML.
www.hdfgroup.org2014 Summer ESIP Federation Meeting 6
NcML
NetCDF Markup Language
XML representation of netCDF metadata
It can be an input file for editing metadata.
HDF
www.hdfgroup.org2014 Summer ESIP Federation Meeting 7
Why NcML?
Isn’t CDL enough? No.
NcML has more features and flexibility.
Plus, NcML works with THREDDS Data Server.
www.hdfgroup.org2014 Summer ESIP Federation Meeting 8
What OBPG NC4 missed - #1
Coordinate variables are not provided.
Dataset { String palette[rgb = 3]; Int16 /geophysical_data/sst[rows = 4320][columns = 8640]; Byte /geophysical_data/sst_qual[rows = 4320][columns = 8640]; Float32 /geophysical_data/Lat[rows= 4320]; Float32 /geophysical_data/Lon[columns = 8640]; } testAll/A20030602003090.L3m_MO_AT109_SST_4.nc4;
The CF convention expects lat/lon variables.
www.hdfgroup.org2014 Summer ESIP Federation Meeting 9
Add new coordinate variables in NcML.
<netcdf location="c:/tomcat/webapps/thredds/share/testdata/cdmUnitTest/it/content/thredds/public/testdata/A20030602003090.L3m_MO_AT109_SST_4.nc4" xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2"><dimension name="_geophysical_data_Lat" length="4320"/> <dimension name="_geophysical_data_Lon" length="8640"/> <group name="geophysical_data"> <variable name="Lat" shape="_geophysical_data_Lat" type="float"> <values start="90.0" increment="-0.0416"/> <attribute type="string" name="units" value="degrees_north" /> </variable> <variable name="Lon" shape="_geophysical_data_Lon" type="float"> <values start="-180.0" increment="0.0416"/> <attribute type="string" name="units" value="degrees_east" /> </variable> …
www.hdfgroup.org2014 Summer ESIP Federation Meeting 10
Now IDV can visualize data but…
No data on Sea w/ Range = [-1, -1]?Examine units.
www.hdfgroup.org2014 Summer ESIP Federation Meeting 11
What OBPG NC4 missed - #2
“units” attribute has “degrees-C” value. Accepted values in the CF Convention:Celsius / celsius / degree_centigrade / degC /degreeC / degree_C / degree_c / deg_C / deg_c
Remove “s-” from “degrees-C”.
www.hdfgroup.org2014 Summer ESIP Federation Meeting 12
Modify attribute value with NcML.
… <variable name="_sst" shape="_geophysical_data_Lat _geophysical_data_Lon" orgName="sst"> <attribute type="string" name="units" value="degreesC" /> </variable>…
Before:
After:
www.hdfgroup.org2014 Summer ESIP Federation Meeting 13
What OBPG NC4 missed - #3
The actual fill value is -1 according to HDFView.
Attributes {/geophysical_data/sst { Int16 _FillValue -32767; Float64 valid_min -5.0; Float64 valid_max 50.0; }
HDFView
www.hdfgroup.org2014 Summer ESIP Federation Meeting 14
Modify _FillValue attribute with NcML.
… <variable name="_sst" shape="_geophysical_data_Lat _geophysical_data_Lon" orgName="sst"> <attribute type="string" name="units" value="degreesC" /> <attribute type=“short" name=“_FillValue" value=“-1" /> </variable>…
Now every value is a fill value – “missing” in IDV.
www.hdfgroup.org2014 Summer ESIP Federation Meeting 15
What OBPG NC4 missed - #4
valid_min / valid_max doesn’t match the range of actual data.
Attributes {/geophysical_data/sst { Int16 _FillValue -32767; Float64 valid_min -5.0; Float64 valid_max 50.0; }
No scale/offset attributes are defined either.
www.hdfgroup.org2014 Summer ESIP Federation Meeting 16
Add scale/offset attributes with NcML.
<variable name="_sst" shape="_geophysical_data_Lat _geophysical_data_Lon" orgName="sst"> <attribute type="string" name="units" value="degreesC" /> <attribute type="float" name="scale_factor" value="0.000717185"/> <attribute type="float" name="add_offset" value="-2.0"/> <attribute type="float" name="valid_min" value="-5.0"/> <attribute type="float" name="valid_max" value="50.0"/> <attribute type="short" name="_FillValue" value="-1"/>
</variable>
Please note that all types match as float._FillValue’s type matches the data type.
www.hdfgroup.org2014 Summer ESIP Federation Meeting 17
Finally, OBPG met the CF conventions.
www.hdfgroup.org2014 Summer ESIP Federation Meeting 18
But tools behave differently.
The same NcML doesn’t work for GoDiva2. NC4 “Group” is the reason. GoDiva2: lat/lon outside groupIDV/Panopy: lat/lon inside group
Is CF/NcML ready for “Group”?
www.hdfgroup.org2014 Summer ESIP Federation Meeting 19
GoDiva2
www.hdfgroup.org2014 Summer ESIP Federation Meeting 20
Panoply
www.hdfgroup.org2014 Summer ESIP Federation Meeting 21
Acknowledgement
This work was supported by Subcontract number 114820 under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration.