tigge, grib to netcdf converter doug schuster (ncar/ecmwf)

TIGGE, GRIB to NetCDF TIGGE, GRIB to NetCDF converterconverter

Doug SchusterDoug Schuster

(NCAR/ECMWF)(NCAR/ECMWF)

Topic OutlineTopic Outline

IntroductionIntroduction Ensemble NetCDF File StructureEnsemble NetCDF File Structure Converter HighlightsConverter Highlights ChallengesChallenges Future PlansFuture Plans

GRIB to NetCDF ConversionGRIB to NetCDF Conversion

t, EGRR, 1

t (1,2,3,4)

d (1,2,3,4)

Metadata

t, ECMF, 2

t, EGRR, 2

t, ECMF, 1

d, EGRR, 1

d, EGRR, 2

d, ECMF, 1

d, ECMF, 2 Gather metadata and message locationsCreate NetCDF file structure

Populate NetCDF parameter arrays(1,2,3,4) represents ensemble member id (Realization)

GRIB File NetCDF File

Ensemble NetCDF File StructureEnsemble NetCDF File Structure NetCDF File formatNetCDF File format

– Based on available CF conventionsBased on available CF conventions

– File organization built according to Doblas-Reyes File organization built according to Doblas-Reyes (ENSEMBLES project) proposed NetCDF file structure(ENSEMBLES project) proposed NetCDF file structure

– Provides grid/ensemble specific metadata for each Provides grid/ensemble specific metadata for each member member

Data ProviderData Provider Forecast type (perturbed, control, Forecast type (perturbed, control, deterministic)deterministic)

– Allows for multiple combinations of initialization Allows for multiple combinations of initialization times and forecast periods within one file.times and forecast periods within one file.

Pairs of initialization and forecast stepPairs of initialization and forecast step

Ensemble NetCDF File StructureEnsemble NetCDF File Structure

NetCDF Parameter structure (5 dimensions):NetCDF Parameter structure (5 dimensions):– Reftime Reftime – Realization (Ensemble member id)Realization (Ensemble member id)– LevelLevel– LatitudeLatitude– LongitudeLongitude

““Coordinate” variables are use to describe:Coordinate” variables are use to describe:– RealizationRealization

Provides metadata associated with each Provides metadata associated with each ensemble grid.ensemble grid.

– ReftimeReftime Allows for multiple initialization times and Allows for multiple initialization times and forecast periods to be contained within one forecast periods to be contained within one filefile


Coordinate variables for Coordinate variables for RealizationRealization– Provide ensemble member metadataProvide ensemble member metadata

InstitutionInstitution Forecast TypeForecast TypeRealization 1 2 3 4

Param(Realization)

Institution(Realization) ECMF ECMFEGRR EGRR

ForecastType(Realization) cf pf cf pf

Grid 1Grid 2Grid 3Grid 4

-At fixed Reftime and Level


Coordinate variable for ReftimeCoordinate variable for Reftime– Provide Forecast Times for each Provide Forecast Times for each InitializationInitialization

LeadtimeLeadtimeReftime 20071012,12z 20071013,00z

20071013,00z

Param(Reftime)

Leadtime(Reftime) 24 12 18

Grid 1 Grid 2 Grid 3

-At fixed Realization and Level


CDL Examples (reftime and coordinate CDL Examples (reftime and coordinate variable)variable)

int reftime(reftime) ;int reftime(reftime) ;

reftime:data_type = "long" ;reftime:data_type = "long" ;

reftime:units = "hours since 1950-01-01 00:00:00" ;reftime:units = "hours since 1950-01-01 00:00:00" ;

reftime:standard_name = "forecast_reference_time" ;reftime:standard_name = "forecast_reference_time" ;

reftime:long_name = "Time of model initialization" ;reftime:long_name = "Time of model initialization" ;

int leadtime(reftime) ;int leadtime(reftime) ;

leadtime:data_type = "int" ;leadtime:data_type = "int" ;

leadtime:units = "hours" ;leadtime:units = "hours" ;

leadtime:standard_name = "forecast_period" ;leadtime:standard_name = "forecast_period" ;

leadtime:long_name = "hours since leadtime:long_name = "hours since forecast_reference_time" ;forecast_reference_time" ;

Ensemble NetCDF File StructureEnsemble NetCDF File Structure CDL Examples (realization and coordinate variables)CDL Examples (realization and coordinate variables)int realization(realization) ;int realization(realization) ; realization:data_type="int"realization:data_type="int" realization:units = "1" ;realization:units = "1" ; realization:standard_name = "realization" ;realization:standard_name = "realization" ; realization:long_name = "Number of the simulation in the realization:long_name = "Number of the simulation in the ensemble" ;ensemble" ;char forecast_type(realization, string2) ;char forecast_type(realization, string2) ; forecast_type:data_type = "char" ;forecast_type:data_type = "char" ; forecast_type:standard_name = "forecast_type" ;forecast_type:standard_name = "forecast_type" ; forecast_type:long_name = ”Forecast type" ;forecast_type:long_name = ”Forecast type" ; char institution(realization, string4) ;char institution(realization, string4) ; institution:data_type = "char" ;institution:data_type = "char" ; institution:standard_name = "institution" ;institution:standard_name = "institution" ; institution:long_name = "Institution responsible for the institution:long_name = "Institution responsible for the forecast system" ;forecast system" ;


CDL Examples (Ensemble parameter example)CDL Examples (Ensemble parameter example)

short t(reftime, realization, level, latitude, short t(reftime, realization, level, latitude, longitude) ;longitude) ;

t:data_type = "short" ;t:data_type = "short" ;

t:units = "K" ;t:units = "K" ;

t:standard_name = "air_temperature" ;t:standard_name = "air_temperature" ;

t:coordinates = "leadtime institution t:coordinates = "leadtime institution forecast_type" ;forecast_type" ;

t:scale_factor = 0.00190138198339349 ;t:scale_factor = 0.00190138198339349 ;

t:add_offset = 253.616560226903 ;t:add_offset = 253.616560226903 ;

t:_FillValue = -32767s ;t:_FillValue = -32767s ;

Conversion Tool HighlightsConversion Tool Highlights

Tool User optionsTool User options– Pack to 16 bit shorts.Pack to 16 bit shorts.– Select grids based on forecast typeSelect grids based on forecast type

PerturbedPerturbed ControlControl DeterministicDeterministic Control and DeterministicControl and Deterministic All 3 forecast typesAll 3 forecast types

Large file support with NetCDF 3.6.0 Large file support with NetCDF 3.6.0 or later.or later.


Tool LimitationsTool Limitations– First version First version – TIGGE dataTIGGE data– Regular Lat/Lon gridsRegular Lat/Lon grids– Requires Multiple Outside LibrariesRequires Multiple Outside Libraries

NetCDFNetCDF GRIB APIGRIB API Jasper (JPEG 2000)Jasper (JPEG 2000)

– Hard Coded Cell MethodsHard Coded Cell Methods– SlowSlow


Tool PerformanceTool Performance– GRIB-2 Simple Packing to NetCDF 32 BITGRIB-2 Simple Packing to NetCDF 32 BIT

GRIB-2 size x ~2GRIB-2 size x ~2– GRIB-2 Simple Packing to NetCDF 16 BITGRIB-2 Simple Packing to NetCDF 16 BIT

Similar sizeSimilar size– GRIB-2 JPEG 2000 to NetCDF 32 BITGRIB-2 JPEG 2000 to NetCDF 32 BIT

GRIB-2 size x ~8GRIB-2 size x ~8– GRIB-2 JPEG 2000 to NetCDF 16 BITGRIB-2 JPEG 2000 to NetCDF 16 BIT

GRIB-2 size x ~4GRIB-2 size x ~4


Conversion Time vs Number of Grids

0

20

40

60

80

100

120

140

160

0 2000 4000 6000 8000 10000

Number of Grids

Convers

ion T

ime (

Sec)

ECMWF sl params, 2x2 Degree Global Grid, simple packing, converted on 3.4 GHz processor Linux Box

ChallengesChallenges

Parameter namesParameter names– Map available GRIB-2 names to CF names.Map available GRIB-2 names to CF names.

Product Discipline, Parameter Category, Parameter NumberProduct Discipline, Parameter Category, Parameter Number

– Map dynamically generated names and GRIB-2 encoding to Map dynamically generated names and GRIB-2 encoding to CF parameter namesCF parameter names

Abstract DefinitionsAbstract Definitions Not a simple Discipline/Category/Number combination.Not a simple Discipline/Category/Number combination.

– Require statistical processing and level information.Require statistical processing and level information. Will generally be CF name + Cell methodWill generally be CF name + Cell method

– Get all centers to use this style of parameter Get all centers to use this style of parameter encoding.encoding.

NCEP only uses the “Discipline/Category/Number” NCEP only uses the “Discipline/Category/Number” parameter encoding scheme.parameter encoding scheme.– GRIB API doesn’t recognize hard coded parameters GRIB API doesn’t recognize hard coded parameters that include statistical processing (e.g. that include statistical processing (e.g. maximum temperature). maximum temperature).

ChallengesChallenges

User ToolsUser Tools– Compatibility with NetCDF ensemble Compatibility with NetCDF ensemble file structure.file structure.

Deal with the ensemble dimension.Deal with the ensemble dimension. Properly ingest, compute valid Properly ingest, compute valid times.times.

Read Coordinate variables to Read Coordinate variables to provide ensemble member metadata.provide ensemble member metadata.

NetCDF 3.6.0 or later required for NetCDF 3.6.0 or later required for large file support.large file support.

Future PlansFuture Plans

Implement on TIGGE portals to provide Implement on TIGGE portals to provide option for NetCDF format data download.option for NetCDF format data download.

Generalize converter tool to encompass Generalize converter tool to encompass all types of GRIB data.all types of GRIB data.

Add GRIB API “Definitions” files for CF Add GRIB API “Definitions” files for CF parameter names, and cell methods.parameter names, and cell methods.

Support Additional GRID typesSupport Additional GRID types Explore NetCDF 4 Explore NetCDF 4

– API with greater functionality and API with greater functionality and flexibility.flexibility.

Add user options.Add user options.

tigge, grib to netcdf converter doug schuster (ncar/ecmwf)

Documents