noaa’s national weather service accessing the grib2 encoded national digital forecast database

NOAA’sNOAA’sNational Weather ServiceNational Weather ServiceAccessing the GRIB2 encoded Accessing the GRIB2 encoded National Digital Forecast DatabaseNational Digital Forecast Database

Arthur TaylorArthur TaylorNOAA/NWS/Meteorological Development LaboratoryNOAA/NWS/Meteorological Development Laboratory

NOAA WebShop 2004NOAA WebShop 2004

July 28, 2004July 28, 2004

NDFD / DegribNDFD / Degrib

NDFD: What is it?NDFD: What is it?

• The National Digital Forecast Database (NDFD), is a mosaic of The National Digital Forecast Database (NDFD), is a mosaic of the National Weather Service’s (NWS) digital forecaststhe National Weather Service’s (NWS) digital forecasts

• It is available free to all.It is available free to all.

• It is updated hourly with the best forecast available to the NWS.It is updated hourly with the best forecast available to the NWS.

• It contains the following variables at various temporal steps out It contains the following variables at various temporal steps out to 7 days: to 7 days: – Maximum Temperature, Minimum Temperature, Probability of Maximum Temperature, Minimum Temperature, Probability of

Precipitation, Temperature, Wind Direction, Wind Speed, Dew Point Precipitation, Temperature, Wind Direction, Wind Speed, Dew Point Temperature, Sky Coverage, Quantity of Precipitation, Snow Amount, Temperature, Sky Coverage, Quantity of Precipitation, Snow Amount, Wave Height, and Weather (which includes visibility).Wave Height, and Weather (which includes visibility).

Advantage? Advantage? • The NDFD provides a single place to go in order to get the best The NDFD provides a single place to go in order to get the best

forecast available from the NWS each hour.forecast available from the NWS each hour.


NDFD: How do I get it?NDFD: How do I get it?

• The NDFD is available in the World Meteorological The NDFD is available in the World Meteorological Organization’s (WMO) GRIB (version 2) format at:Organization’s (WMO) GRIB (version 2) format at:– http://weather.noaa.gov/pub/SL.us008001/ST.expr/DF.gr2/DC.ndfd/http://weather.noaa.gov/pub/SL.us008001/ST.expr/DF.gr2/DC.ndfd/

– ftp://tgftp.nws.noaa.gov/SL.us008001/ST.expr/DF.gr2/DC.ndfd/ftp://tgftp.nws.noaa.gov/SL.us008001/ST.expr/DF.gr2/DC.ndfd/

• It is broken up into various geographic areas which include:It is broken up into various geographic areas which include:– The Continental U.S. (CONUS), The Continental U.S. (CONUS),

– Various predefined sectors which are: Pacific North West, Pacific South Various predefined sectors which are: Pacific North West, Pacific South West, Northern Rockies, Central Rockies, Southern Rockies, Northern West, Northern Rockies, Central Rockies, Southern Rockies, Northern Plains, Central Plains, Southern Plains, Upper Mississippi Valley, Central Plains, Central Plains, Southern Plains, Upper Mississippi Valley, Central Mississippi Valley, Southern Mississippi Valley, Central Great Lakes, Mississippi Valley, Southern Mississippi Valley, Central Great Lakes, Eastern Great Lakes, North East, South East, and Mid Atlantic sectors.Eastern Great Lakes, North East, South East, and Mid Atlantic sectors.

– Puerto Rico.Puerto Rico.

• Inside a sector it is broken up into the various weather Inside a sector it is broken up into the various weather elements.elements.


NDFD: Which Sector?NDFD: Which Sector?


NDFD: Custom Sector?NDFD: Custom Sector?

• The predefined sectors are created hourly, and are already The predefined sectors are created hourly, and are already waiting to be downloaded. waiting to be downloaded.

Advantage: Speed. Advantage: Speed. • Data is already available and ready to transmit when the request Data is already available and ready to transmit when the request

for the data arrives on the server.for the data arrives on the server.

Disadvantage: Flexibility. Disadvantage: Flexibility. • If a user is interested in a specific area, he/she will either need If a user is interested in a specific area, he/she will either need

to download multiple sectors, or download excess information.to download multiple sectors, or download excess information.

Alternate solution?Alternate solution?• Provide a web service which allows the user to specify the area Provide a web service which allows the user to specify the area

they are interested in. The service encodes just that portion of they are interested in. The service encodes just that portion of the NDFD in GRIB2 and sends it back.the NDFD in GRIB2 and sends it back.


NDFD: Custom Sector.NDFD: Custom Sector.

• The experimental custom sector web service is available at:The experimental custom sector web service is available at:– http://ndfd.weather.gov/cgi-bin/ndfd/gribcuthttp://ndfd.weather.gov/cgi-bin/ndfd/gribcut

• The following call gets the maximum temperature in a subgrid which is The following call gets the maximum temperature in a subgrid which is defined with its lower left corner at lat=30, lon=-100, and its upper right defined with its lower left corner at lat=30, lon=-100, and its upper right corner at lat=40, lon=-90:corner at lat=40, lon=-90:– gribcut?var=maxt&lat1=30&lon1=-100&lat2=40&lon2=-90gribcut?var=maxt&lat1=30&lon1=-100&lat2=40&lon2=-90

• For more information see:For more information see:– http://www.nws.noaa.gov/ndfd/grib.htmhttp://www.nws.noaa.gov/ndfd/grib.htm

Advantage: Flexible. Advantage: Flexible. • The user can choose any latitude longitude box. Note the box is not The user can choose any latitude longitude box. Note the box is not

allowed to be larger than 15 degrees on a side.allowed to be larger than 15 degrees on a side.

Disadvantage: Speed. Disadvantage: Speed. • The data is encoded upon request, which means some work on the The data is encoded upon request, which means some work on the

server side. server side. • In a high demand period this method may slow down.In a high demand period this method may slow down.• If this becomes an issue, various methods can be used to speed it up.If this becomes an issue, various methods can be used to speed it up.


NDFD: How do I use it?NDFD: How do I use it?

Since NDFD uses the WMO’s GRIB2 format, how do I Since NDFD uses the WMO’s GRIB2 format, how do I read a GRIB2 file?read a GRIB2 file?

The NWS has provided NDFD users a program (“degrib”) to The NWS has provided NDFD users a program (“degrib”) to easily download and locally decode GRIB2 files see:easily download and locally decode GRIB2 files see:• http://www.nws.noaa.gov/mdl/NDFD_GRIB2Decoder/http://www.nws.noaa.gov/mdl/NDFD_GRIB2Decoder/

• ““degrib” can be installed on most local operating systems (Windows, degrib” can be installed on most local operating systems (Windows, Linux, UNIX).Linux, UNIX).

• It provides a graphical user interface for specifying products to It provides a graphical user interface for specifying products to download and decode, along with some image generation capability. download and decode, along with some image generation capability.

The software is based on the GRIB2 library available at:The software is based on the GRIB2 library available at:• http://www.nws.noaa.gov/mdl/iwt/grib2/decoder.htmhttp://www.nws.noaa.gov/mdl/iwt/grib2/decoder.htm

• http://www.nws.noaa.gov/mdl/iwt/grib2/encoder.htmhttp://www.nws.noaa.gov/mdl/iwt/grib2/encoder.htm


Degrib: What can it do?Degrib: What can it do?

• Download NDFD DataDownload NDFD Data

• Inventory GRIB and GRIB2 filesInventory GRIB and GRIB2 files

• Convert GRIB and GRIB2 data to other formatsConvert GRIB and GRIB2 data to other formats– ESRI .shp (point, small polygon, large polygon) filesESRI .shp (point, small polygon, large polygon) files

– .flt files (used by GrADS and ESRI Spatial Analyst).flt files (used by GrADS and ESRI Spatial Analyst)

– netCDF filesnetCDF files

– .csv files.csv files

• Probe GRIB and GRIB2 data at a given point.Probe GRIB and GRIB2 data at a given point.

• Interpret the meta-data provided in the GRIB and GRIB2 files.Interpret the meta-data provided in the GRIB and GRIB2 files.

Note: Both a Graphical User Interface (GUI) and command line Note: Both a Graphical User Interface (GUI) and command line interface are provided. See “c:/ndfd/degrib15/docs/degrib.txt” interface are provided. See “c:/ndfd/degrib15/docs/degrib.txt” for information on the command line interface.for information on the command line interface.


Degrib: InstallationDegrib: Installation

•Degrib can be downloaded from:Degrib can be downloaded from:– http://www.nws.noaa.gov/mdl/NDFD_GRIB2Decoder/register.htmhttp://www.nws.noaa.gov/mdl/NDFD_GRIB2Decoder/register.htm

• On the download page, MS-Windows users should get “ndfd-On the download page, MS-Windows users should get “ndfd-demo.exe”, which contains:demo.exe”, which contains:

• An installation wizardAn installation wizard

• Compiled codeCompiled code

• Source code for the libraries, and the “degrib” and “tkdegrib” programs Source code for the libraries, and the “degrib” and “tkdegrib” programs (superImageGen and htmlmaker source code are not provided)(superImageGen and htmlmaker source code are not provided)

• DocumentationDocumentation

• Unix users should get “degrib-unix.tar.gz”, which contains:Unix users should get “degrib-unix.tar.gz”, which contains:• Source code for the libraries and the “degrib” and “tkdegrib” programsSource code for the libraries and the “degrib” and “tkdegrib” programs

• Documentation.Documentation.

• To compile the code you need: To compile the code you need: • FORTRAN (g77 should work) : for the decoder libraryFORTRAN (g77 should work) : for the decoder library

• C (gcc should work) : for the “drivers” and projection libraryC (gcc should work) : for the “drivers” and projection library

• Optional : Tcl/Tk (free from Optional : Tcl/Tk (free from http://scriptics.comhttp://scriptics.com) : for the Graphical User Interface) : for the Graphical User Interface


Degrib: Data downloadDegrib: Data download

Step 1: Download some data.Step 1: Download some data.• The GUI version of degrib, The GUI version of degrib,

“tkdegrib”, provides an “tkdegrib”, provides an interface to get NDFD either interface to get NDFD either via ftp or http.via ftp or http.– Highlight the desired sectorHighlight the desired sector

– Click on either “Download by ftp” Click on either “Download by ftp” or “Download by http”or “Download by http”

• Alternatively, you can get the Alternatively, you can get the data directly from:data directly from:– http://weather.noaa.gov/pub/SL.uhttp://weather.noaa.gov/pub/SL.u

s008001/ST.expr/DF.gr2/DC.ndfs008001/ST.expr/DF.gr2/DC.ndfd/d/

– ftp://tgftp.nws.noaa.gov/SL.us008ftp://tgftp.nws.noaa.gov/SL.us008001/ST.expr/DF.gr2/DC.ndfd/001/ST.expr/DF.gr2/DC.ndfd/


Degrib: Data downloadDegrib: Data download

Step 1 continued:Step 1 continued:• The GUI version of degrib now provides a The GUI version of degrib now provides a

“custom” sector, which you control by:“custom” sector, which you control by:– Use the “file->configure” menu.Use the “file->configure” menu.

– Choose the “Custom” tab. Choose the “Custom” tab.

– Enter in the desired area, press ok.Enter in the desired area, press ok.

• In addition there is a command line method In addition there is a command line method to download various sectors and variables, to download various sectors and variables, using: using: – cd C:\ndfd\degrib15\bin\cd C:\ndfd\degrib15\bin\

– tcldegrib web.tcltcldegrib web.tcl


Degrib: Convert to .shp fileDegrib: Convert to .shp file

To convert to .shp format:To convert to .shp format:– Double click on the GRIB2 file (top Double click on the GRIB2 file (top

pane)pane)– Select message to convert (middle Select message to convert (middle

pane)pane)– Choose an “output file name”Choose an “output file name”– Click on “Generate .shp file”Click on “Generate .shp file”

• ““Small Polygon” creates better Small Polygon” creates better images in ESRI and is easier to images in ESRI and is easier to manipulate, but it is larger than manipulate, but it is larger than the “Point .shp”the “Point .shp”

• ““Large Polygon” merges Large Polygon” merges polygons together to create polygons together to create smaller files, but makes grid to smaller files, but makes grid to grid comparison more difficult.grid comparison more difficult.

• ““Include Missing Values” allows Include Missing Values” allows you to study the entire grid, or you to study the entire grid, or limit the result to where data limit the result to where data existsexists


CONUS MaxT in ArcViewCONUS MaxT in ArcView

After converting the grid, start up ESRI ArcView, then load the layer (.shp After converting the grid, start up ESRI ArcView, then load the layer (.shp file) that you just created. Use the color legend found in:file) that you just created. Use the color legend found in:

C:/ndfd/degrib15/arcview/poly_legend (for polygon shapefiles)C:/ndfd/degrib15/arcview/poly_legend (for polygon shapefiles)

C:/ndfd/degrib15/arcview/point_legend (for point shapefiles)C:/ndfd/degrib15/arcview/point_legend (for point shapefiles)


Polygon vs Point, DC areaPolygon vs Point, DC area


NDFD CONUS Map ProjectionNDFD CONUS Map Projection

For CONUS (continental U.S.), NDFD uses a Lambert Conformal For CONUS (continental U.S.), NDFD uses a Lambert Conformal Conic Map Projection:Conic Map Projection:• Tangent latitutude 25, orientation longitude -95, mesh latitude 25, mesh Tangent latitutude 25, orientation longitude -95, mesh latitude 25, mesh

size 5079.406 m.size 5079.406 m.

• In ESRI ArcMap that would be:In ESRI ArcMap that would be:– 1st Parallel = 21st Parallel = 2ndnd Parallel = 25 degrees north (tangent latitude) Parallel = 25 degrees north (tangent latitude)

– Lon of center of projection = -95 degrees (orientation longitude) Lon of center of projection = -95 degrees (orientation longitude)

– Lat of center of projection = 25 degrees (tangent latitude) Lat of center of projection = 25 degrees (tangent latitude)

– False easting = 0 m, False northing = 0 m, X Shift = 0, Y Shift = 0 False easting = 0 m, False northing = 0 m, X Shift = 0, Y Shift = 0

In addition, NDFD uses a spherical earth with radius 6371.2 kmIn addition, NDFD uses a spherical earth with radius 6371.2 km

To assist ESRI ArcView 3.x, degrib creates a .ave (ArcView Avenue To assist ESRI ArcView 3.x, degrib creates a .ave (ArcView Avenue script) to set the projection and radius correctly.script) to set the projection and radius correctly.

For ArcGIS, please see /degrib/arcview/prj.adf, and ndfd.prjFor ArcGIS, please see /degrib/arcview/prj.adf, and ndfd.prj


Un-projected vs ProjectedUn-projected vs Projected


Degrib: Convert to .flt fileDegrib: Convert to .flt file

To convert to .flt file (for use To convert to .flt file (for use with GrADS or ESRI Spatial with GrADS or ESRI Spatial Analyst):Analyst):

– Double click on the GRIB2 file Double click on the GRIB2 file (top pane)(top pane)

– Select message to convert Select message to convert (middle pane)(middle pane)

– Choose an “output file name”Choose an “output file name”– Click on “Generate .flt file”Click on “Generate .flt file”

• ““Grid” allows you to select Grid” allows you to select what kind of interpolation to what kind of interpolation to do (or keep the original do (or keep the original projection).projection).

• ““GrADS .ctl file”: creates a GrADS .ctl file”: creates a control file which GrADS can control file which GrADS can use to access the .flt fileuse to access the .flt file

• ““M.S.B. first”: creates the .flt M.S.B. first”: creates the .flt files in “Big Endian” format.files in “Big Endian” format.


CONUS MaxT in GrADSCONUS MaxT in GrADS


Degrib: Convert to netCDF fileDegrib: Convert to netCDF file

To convert to netCDF file:To convert to netCDF file:– Double click on the GRIB2 file Double click on the GRIB2 file

(top pane)(top pane)


– Choose an “output file name”Choose an “output file name”

– Click on “Generate NetCDF Click on “Generate NetCDF (.nc) file”(.nc) file”

• This creates a netCDF file This creates a netCDF file according to the “CF” according to the “CF” metadata conventions. metadata conventions.

• This converts 1 GRIB This converts 1 GRIB message to 1 netCDF file.message to 1 netCDF file.

• An improvement would be An improvement would be for multiple GRIB messages for multiple GRIB messages to go to 1 netCDF file. to go to 1 netCDF file.


CONUS MinT in NetCDFCONUS MinT in NetCDF


Degrib: Convert to .csv fileDegrib: Convert to .csv file

To convert to CSV file:To convert to CSV file:– Double click on the GRIB2 file Double click on the GRIB2 file

(top pane)(top pane)


– Choose an “output file name”Choose an “output file name”

– Click on “Generate .csv files”Click on “Generate .csv files”

• This creates a comma This creates a comma separated text file which can separated text file which can be worked with in excel.be worked with in excel.

• Note: Excel has problems Note: Excel has problems loading the CONUS NDFD, as loading the CONUS NDFD, as it appears to be limited to it appears to be limited to 65,536 records, while the 65,536 records, while the NDFD typically has 739,297 NDFD typically has 739,297 recordsrecords


CONUS MinT in .csv in ExcelCONUS MinT in .csv in Excel

X


Degrib: Probe pointDegrib: Probe point

Degrib also provides the ability to probe a GRIB file at a Degrib also provides the ability to probe a GRIB file at a set of latitude longitude locations. It computes the set of latitude longitude locations. It computes the value at that spot, and returns a text message. Note: value at that spot, and returns a text message. Note: this is not available in the GUI. To do so from the this is not available in the GUI. To do so from the command line:command line:

– C:\ndfd\degrib\bin\degrib maxt.bin –P –pnt 38.99,-77.03C:\ndfd\degrib\bin\degrib maxt.bin –P –pnt 38.99,-77.03

– C:\ndfd\degrib\bin\degrib maxt.bin –P –pntFile point.txtC:\ndfd\degrib\bin\degrib maxt.bin –P –pntFile point.txt

• The first example probes just 1 point, while the second reads a The first example probes just 1 point, while the second reads a file which is of the form: StationID, lat, lon per line. The second file which is of the form: StationID, lat, lon per line. The second form allows for numerous stations.form allows for numerous stations.

• By default, probe chooses the “nearest” grid cell, but using the By default, probe chooses the “nearest” grid cell, but using the “-Interp” option, causes it to perform bi-linear interpolation.“-Interp” option, causes it to perform bi-linear interpolation.

See (/degrib/docs/degrib.txt for more help)See (/degrib/docs/degrib.txt for more help)


Probe point (MaxT, MinT, PoP)Probe point (MaxT, MinT, PoP)

element, unit, refTime, validTime, (38.993600,-element, unit, refTime, validTime, (38.993600,-77.022400)77.022400)

MaxT, [F], 200307212300, 200307230000, 86.882MaxT, [F], 200307212300, 200307230000, 86.882

MaxT, [F], 200307212300, 200307240000, 82.375MaxT, [F], 200307212300, 200307240000, 82.375

MaxT, [F], 200307212300, 200307250000, 85.233MaxT, [F], 200307212300, 200307250000, 85.233

MaxT, [F], 200307212300, 200307260000, 87.753MaxT, [F], 200307212300, 200307260000, 87.753

MaxT, [F], 200307212300, 200307270000, 89.395MaxT, [F], 200307212300, 200307270000, 89.395

MaxT, [F], 200307212300, 200307280000, 87.775MaxT, [F], 200307212300, 200307280000, 87.775

MaxT, [F], 200307212300, 200307290000, 86.800MaxT, [F], 200307212300, 200307290000, 86.800

element, unit, refTime, validTime, (38.993600,-element, unit, refTime, validTime, (38.993600,-77.022400)77.022400)

MinT, [F], 200307212300, 200307221200, 72.625MinT, [F], 200307212300, 200307221200, 72.625

MinT, [F], 200307212300, 200307231200, 73.165MinT, [F], 200307212300, 200307231200, 73.165

MinT, [F], 200307212300, 200307241200, 69.025MinT, [F], 200307212300, 200307241200, 69.025

MinT, [F], 200307212300, 200307251200, 65.965MinT, [F], 200307212300, 200307251200, 65.965

MinT, [F], 200307212300, 200307261200, 67.945MinT, [F], 200307212300, 200307261200, 67.945

MinT, [F], 200307212300, 200307271200, 69.745MinT, [F], 200307212300, 200307271200, 69.745

MinT, [F], 200307212300, 200307281200, 69.025MinT, [F], 200307212300, 200307281200, 69.025

element, unit, refTime, validTime, (38.993600,-element, unit, refTime, validTime, (38.993600,-77.022400)77.022400)PoP12, [%], 200307212300, 200307220000, 39.000PoP12, [%], 200307212300, 200307220000, 39.000PoP12, [%], 200307212300, 200307221200, 39.000PoP12, [%], 200307212300, 200307221200, 39.000PoP12, [%], 200307212300, 200307230000, 50.000PoP12, [%], 200307212300, 200307230000, 50.000PoP12, [%], 200307212300, 200307231200, 50.000PoP12, [%], 200307212300, 200307231200, 50.000PoP12, [%], 200307212300, 200307240000, 50.000PoP12, [%], 200307212300, 200307240000, 50.000PoP12, [%], 200307212300, 200307241200, 50.000PoP12, [%], 200307212300, 200307241200, 50.000PoP12, [%], 200307212300, 200307250000, 20.000PoP12, [%], 200307212300, 200307250000, 20.000PoP12, [%], 200307212300, 200307251200, 20.000PoP12, [%], 200307212300, 200307251200, 20.000PoP12, [%], 200307212300, 200307260000, 14.000PoP12, [%], 200307212300, 200307260000, 14.000PoP12, [%], 200307212300, 200307261200, 5.000PoP12, [%], 200307212300, 200307261200, 5.000PoP12, [%], 200307212300, 200307270000, 5.000PoP12, [%], 200307212300, 200307270000, 5.000PoP12, [%], 200307212300, 200307271200, 11.000PoP12, [%], 200307212300, 200307271200, 11.000PoP12, [%], 200307212300, 200307280000, 11.000PoP12, [%], 200307212300, 200307280000, 11.000PoP12, [%], 200307212300, 200307281200, 21.000PoP12, [%], 200307212300, 200307281200, 21.000PoP12, [%], 200307212300, 200307290000, 21.000PoP12, [%], 200307212300, 200307290000, 21.000


NDFD, GRIB2 & WeatherNDFD, GRIB2 & Weather

To encode Weather (Wx), NDFD disseminates “weather To encode Weather (Wx), NDFD disseminates “weather strings”.strings”.• Advantage: A flexible format that can encode a description of Advantage: A flexible format that can encode a description of

weather such as “Chance of thunderstorms and chance of weather such as “Chance of thunderstorms and chance of heavy rain showers”heavy rain showers”

Difficulty: A “weather string” is not a number, so it is Difficulty: A “weather string” is not a number, so it is challenging to store in GRIB2.challenging to store in GRIB2.• Solution: Use GRIB2 section 2 to provide an ASCII look up table, Solution: Use GRIB2 section 2 to provide an ASCII look up table,

and store the numeric values in the regular GRIB2and store the numeric values in the regular GRIB2– Note: each weather grid, for each forecast projection, has a different Note: each weather grid, for each forecast projection, has a different

ASCII look up table, so without section 2 it has no meaning.ASCII look up table, so without section 2 it has no meaning.

• Some Questions:Some Questions:– What does “Chc:T:<NoInten>:<NoVis>:^Chc:RW:+:<NoVis>:” mean, and What does “Chc:T:<NoInten>:<NoVis>:^Chc:RW:+:<NoVis>:” mean, and

how, for example, is an Emergency Manager supposed to know that?how, for example, is an Emergency Manager supposed to know that?– What can be done for the .flt file? What can be done for the .flt file?


Degrib: Future?Degrib: Future?

Maintain the program by keeping up with NDFD as more Maintain the program by keeping up with NDFD as more variables / sectors are introduced.variables / sectors are introduced.

– In order to inform users of updates, degrib has an “announcement mailing In order to inform users of updates, degrib has an “announcement mailing list”, which at the last release had 434 people on it.list”, which at the last release had 434 people on it.

Improvements: Based on user feedback, current Improvements: Based on user feedback, current thoughts are:thoughts are:

– Create some form of graphical probe point capabilityCreate some form of graphical probe point capability

– Add some way for UNIX users (without GrADS) to plot the imagesAdd some way for UNIX users (without GrADS) to plot the images

– Add some way for users to zoom in on a mapAdd some way for users to zoom in on a map

– Improve the NetCDF and GrADS file outputs to contain multiple GRIB Improve the NetCDF and GrADS file outputs to contain multiple GRIB messages.messages.

noaa’s national weather service accessing the grib2 encoded national digital forecast database

Documents

ndfd degribndfd

ndfd users

best forecast available

various weather elements

various predefined sectors

central rockies

central plains

maximum temperature