wmo bufr &crex gil ross, uk met office 2009-09-21

WMO BUFR &CREX

Gil Ross, UK Met Office

2009-09-21

WMO BUFR Standard• BUFR

– Binary Universal Form for the Representation of meteorological data– Operational 1988

• CREX – Character form for the Representation and EXchange of data– Semi-operational 1994

• Difference between BUFR and GRIB (generally)– GRIB is for uniform (multidimensional) array coverages– BUFR is for keyword pairs (hashes in perl, dictionaries in python)

• Codes (keyword pairs) are "self-descriptive“– the description and content of the data are both contained within the BUFR or CREX message

itself– Unlike XML elements (or hashes and dictionaries), keyword pairs are split into descriptor

groups and data groups allowing independent compaction techniques

• Same philosophy and the same era as SGML (1986) • predates XML (late 1990s)• Major difference

– BUFR and CREX formats have brevity as a major design requirement.– XML philosophy Terseness in Markup is of minimal importance”⁜

What is WMO standard: BUFR• Domain Specific Language and Schema

– Specific application is Meteorology in general, also Oceanography– Domain is any data type fully definable in formal table structure– Everything is predefined in references

• Even operators which redefine the references

• Data model and data instance– Metadata – standard and locally defined metadata– Local metadata allows foreign key structures

• enabling BUFR to be used as RDBMS objects.– Data definition structure

• Every instance contains its own “DTD”– Composed of full or partial specification – or reference(s) to predefined components of the “DTD”

• Dynamic data specification operations – effectively allows a dynamic “feature catalogue”, – defining new features in an instance– Dynamic specification not achievable in ISO 191xx

• Controlled vocabulary, reference and code specifications• Decoded/expanded BUFR is equivalent to a full markup specification

BUFR Data model

• Descriptor definition structure– Simple descriptor classes and elements– Operators– Coding operators – of course - but also– Definition operators (coverage operators, generalised

coordinates)• Grouping-hierarchy operator• Descriptor (Feature) attribute operator• Descriptor (Feature) context modifier

• Data specification– Highly defined and modifiable data precision specification

• Registry table structure– WMO standard versions– Local versions– Complete independent registries (oceanography)

BUFR Standard

• Terseness of BUFR and CREX – Predefine all characteristics – Refer to published tables for all descriptors, operations and data

codes. – BUFR is formed from main tables and subsidiary code tables

• However BUFR standard mixes a number of ISO 19100 concepts together– One difficulty in converting from BUFR to ISO/GML like XML is

deciding which ISO concept to use

• When the concepts are separated – BUFR realises most of the basic ISO modelling. – BUFR uses generalised coordinates to do all three ISO/XML

things

BUFR in operation

• Exchange container – BUFR– millions of BUFR files exchanged each day– BUFR can even encode its own registry tables for

exchange.– Several compression mechanisms

• The design decisions allow conflicting requirements of “full specification” and high “compactness”.– Everything is a reference.

BUFR – ISO Concepts• Main classification table A

– hierarchy of 20 feature catalogues. • Table D sequences (around 300 sequences with 4000 elements)

– collections of features or collections of collections of features. • Table B elements (around 1200)

– 800 simple features in 30 classes, – 8 classes (with 400 members using 1500 code values) define generalised coverage grids– others define operations on the data values.– refers to around 370 code tables, flag tables and enumerations for classifications of discrete

data features.• Table C defines data descriptors and operations on the data descriptors

– compression, sequencing and repetition.– operators which redefine the nature of the features

• mutable features.– also defines operations on the data which include data compression.– ISO has feature operations BUT they change the VALUE of the feature, – BUFR generalised Coordinates - particularly the significance coordinates in group 8, change

the MEANING and CONTEXT of the feature. • BUFR message

– feature instance– Millions of new BUFR messages are exchanged each day between and within WMO

Member States

class WMO BUFR Standard

Classifica tionTables

BUFR_M essage

TableElements

(from ClassificationTables)

CodingM ethods

PackingMethods

(from CodingMethods)

CodeTables

Table A – 20 Feature Catalogues

Table D – Feature collections 300 collections 4000 elements in sequenceTable B – 8 Coverage GRID Definition Classes 1200 simple elementsTable B references 370 Code Tables code tables – flag tables - enumerations

Table C – operators defining coding Changing table B attributes Changing descriptors

Table C also defines further compression

BUFR Message – Feature Instance

BUFRFeature Catalogue

class BUFR_featureCatalogue

«feature»BUFR_featureCatalogue

«feature»BUFR_specialisedFeatureCatalogue

«feature»BUFR_table_D_ sequenceClass

«code table»table_D_reference

- table_D_comment: char- table_D_indicator: int = 3- table_D_sequenceClass: int- table_D_sequenceClassName: char- table_D_sequenceCodeValue: int- table_D_sequenceName: char

«feature»TableElements::

BUFR_tableB_s impleFeature

- CREX_dataValueCharLength: int- CREX_scalingMultiplier: int- CREX_unitsOfMeasureCode: int- dataValueBitLength: int- referenceValue: int- scalingMultiplier: int- Table_B_codeValue: int- Table_B_featureName: char- unitsOfMeasureCode: int

«code table»Table_A_reference

- Table_A_codeValue: int- Table_A_dataCategory: char

«code table»Common_code Table_C-13

- Table_C-13_codeValue: int- Table_C-13_dataCategory: int

«code table»TableElements::Table_B_reference

- Table_B_comment: char- Table_B_elementClass: int- Table_B_elementClassComment: char- Table_B_elementClassName: char- Table_B_indica tor: int = 0

«feature»BUFR_coordinates::

BUFR_CoordinateGrid


BUFR_Table B_cov erage

+ cancel() : void+ redefine() : void+ set() : void+ setRange() : void

0..*

1

1. .*

1

0..*

1. .

elementClass<=9

Table A - Feature Catalogue

20 sub catalogues

class BUFR_dataCategory

BUFR_featureCatalogue

- Table_A_versionNumber: int = 13

Table_A_dataCategoryCode

*PK Table_A_codeValue* Table_A_category

+ PK_Table_A_dataCategoryCode()

+ UQ_Table_A_dataCategoryCode_Table_A_codeValue()

surfaceData_land

- Table_A_category: char = Surface data - land- Table_A_codeVa lue: int = 0

surfaceData_sea

- Table_A_category: cha r = Surface data -sea- Table_A_codeValue: int = 1

v erticalSoundings_notSatellite

- Table_A_category: cha r = Vertical soundi...- Table_A_codeVa lue: int = 2

v erticalSoundings_satellite

- Table_A_category: cha r = Vertical soundi...- Table_A_codeVa lue: int = 3

singleLev elUpperAirData_notSatellite

- Table_A_category: cha r = Single level up...- Table_A_codeValue: int = 4

singleLev elUpperAirData_satellite

- Table_A_category: cha r = Single level up...- Table_A_codeVa lue: int = 5

radarData

- Table_A_category: char = Radar data- Table_A_codeVa lue: int = 6

synopticFeatures

- Table_A_category: char = Synoptic features- Table_A_codeVa lue: int = 7

physicalChemicalData

- Table_A_category: cha r = Physical/chemic...- Table_A_codeVa lue: int = 8

dispersalTransportData

- Table_A_category: cha r = Dispersal and t...- Table_A_codeVa lue: int = 9

radiologicalData

- Table_A_category: cha r = Radiological data- Table_A_codeVa lue: int = 10

BUFR_tableUpdate

- Table_A_category: cha r = BUFR tables, co...- Table_A_co deValue: int = 11

surfaceData_satellite

- Table_A_category: cha r = Surface data (s...- Table_A_codeVa lue: int = 12

statusInformation

- Table_A_category: cha r = Status information- Table_A_codeVal ues: int = 20

forecastData

- Table_A_category: char = Forecasts- Table_A_codeVa lue: int = 13

warningData

- Table_A_category: char = Warnings- Table_A_codeValue: int = 14

radiancesData_satellite

- Table_A_category: cha r = Radiances (sate...- Table_A_co deValue: int = 21

oceanographicData

- Table_A_category: cha r = Oceanographic data- Table_A_co deValue: int = 31

imageData

- Table_A_category: char = Images- Table_A_co deValue: int = 101

otherData

- Table_A_category: cha r = Other data - e....- Table_A_codeVal ue: int = 255

BUFR Table A is the list of data categories,and together with a public declared l ist of local data sub-categories l isted in Common Table C-13, these distinguish thespecialisations which are thematic feature catalogues which together form the WMO BUFR feature catalogue for non-gridded features and coverages.

class BUFR_simpleFeature

«feature»BUFR_tableB_s impleFeature


«code table»Table_B_reference


vertical-parametricFeatures

- Table_B_elementClass: int = 10 windFeatures

- Table_B_elementClass: int = 11

«feature»temperatureFeatures


hygrographicFeatures


radianceFeatures


physical-chemicalFeatures


synopticFeatures


radarFeatures


«feature»observedPhenomena


oceanographicFeatures


dispersalFeatures


radiologicalFeatures


processingFeatures


nonCoordinateLocations

timeFeatures


1stDimHorizontalFeatures


mapProj ections


imageFeatures


dataDescriptorOperatorQualifiers


qualityInformation


dataMonitoringInformation


satelliteFeatures


2ndDimHorizontalFeatures


Table B simple Feature Classes

class BUFR_tableB_cov erage

BUFR_coordinates::1s tHorizontalLocation


BUFR_coordinates ::identification

- Table_B_el ementClass: int = 1

BUFR_coordinates::instrumentation


BUFR_coordinates::2ndHorizontalLocation


BUFR_coordinates::significanceQualifier


BUFR_coordinate s::timeLocation


+ setDuration() : void+ setIncrement() : void+ setLocalDisplacement() : void+ setPeriodOrDisplacement() : void

BUFR_coordinates::v ertical-parametricLocation



BUFR_Table B_cov erage

+ cancel() : void+ redefine() : void+ set() : void+ setRange() : void

«feature»BUFR_tableB_s impleFeature


«code table»Table_B_reference


BUFR_coordinates::BUFR_TableEntries


elementClass<=9

Coordinates- Coverage Grids

Code Tablesdiscrete Coverages

class Observed phenomena code table examples


BUFR_tableB_s impleFeature


«code table»TableElements::Table_B_reference



observedPhenomena


Table_B_indicator = 0Table_B_elementClass = 20Table_B_elementClassName = observedPhenomenaTable_B_codeValue = 21Table_B_featureName = Type of Precipitation

0 20 021Type of precipitation

Data Datavalue meaningBit No.1 Precipitation-unknown type2 Liquid precipitation not freezing3 Liquid freezing precipitation4 Drizzle5 Rain6 Solid precipitation7 Snow8 Snow grains9 Snow pellets 10 Ice pellets 11 Ice crystals12 Diamond dust13 Small hail14 Hail15 Glaze16 Rime17 Soft rime18 Hard rime19 Clear ice20 Wet snow21 Hoar frost22 Dew23 White dew24-29 ReservedAll 30 Missing value

Note: Mixed precipitation is indicated by setting to one the bits of all the observed single types of precipitation

«code table»typeOfPrecipitation

- Table_B_co deValue: int = 21- Table_B_elementClass: int = 20- Table_B_elementClassName: char = observedPhenomena- Table_B_featureName: char = typeOfPrecipitation- Table_B_indica tor: int = 0- unitsOfMeasure: char = code table

Table_B_indicator = 0Table_B_elementClass = 20Table_B_elementClassName = observedPhenomenaTable_B_codeValue = 37Table_B_featureName = Ice development0 20 037Ice development

Data DataValue meaning0 New ice only (frazil ice, grease ice, slush, shuga)1 Nilas or ice rind, less than 10 cm thick2 Young ice (grey ice, grey white ice), 10-30 cm thick3 Predominantly new and/or young ice with some first year ice4 Predominantly thin first year ice with some new and/or young ice5 All thin first year ice (30 70 cm thick)6 Predominantly medium first year ice (70 120 cm thick) and thick first year ice (>120 cm thick) with some thinner (younger) first year ice7 All medium and thick first year ice8 Predominantly medium and thick first year ice with some old ice (usually more than 2 metres thick)9 Predominantly old ice11 29 Reserved30 Unable to report, because of darkness, lack of visibil ity or because only ice of land origin is visible or because ship is more than 0.5 nautical mile away from ice edge31 Missing value

«code table»iceDevelopment

- Table_B_codeVa lue: int = 37- Table_B_elementClass: int = 20- Table_B_elementClassName: char = observedPhenomena- Table_B_featureName: char = iceDevelopment- Table_B_indica tor: int = 0

There are 370 code tables, flag tables and enumerations for BUFR and CREX.Two examples here are code tables for two discrete features of the observedPhenomena Feature Type.

typeOfPrecipitation is observedPhenomena type 21 and the values it can take are not exclusive, there can be mixed types of precipitation, e.g. rain and snow (sleet). The subtleties of identification means that this codetable would be mainly used by a human observer.

iceDevelopment is observedPhenomena type 37, and describes the development of sea ice observed by a (ship-borne) human observer.

• Continuous

Coverage

class Temperature feature code table example

Table_B_elementClass = 12 Table_B_elementClassName = temperatureFeatures

F=Table_B_indicator Table_B_featureName UNIT = unitsOfMeasureCode UNIT = CREX_unitsOfMeasureCode X=Table_B_elementClass SCALE = scalingMultiplier SCALE = CREX_scalingMultiplier Y=Table_B_codeValue REFERENCE VALUE = referenceValue REFERENCE VALUE =

CREX_referenceValue DATA WIDTH (Bits) = dataValueBitLength DATA WIDTH (Characters) =

CREX_dataValueCharLength BUFR CREX TABLE

REFERENCE F X Y

TABLE

ELEMENT NAME

UNIT

SCALE

REFERENCE VALUE

DATA WIDTH

(Bits)

UNIT

SCALE

DATA WIDTH

(Characters) 0 12 001 Temperature/dry-bulb temperature K 1 0 12 °C 1 3 0 12 002 Wet-bulb temperature K 1 0 12 °C 1 3 0 12 003 Dew-point temperature K 1 0 12 °C 1 3 0 12 004 Dry-bulb temperature at 2 m K 1 0 12 °C 1 3 0 12 005 Wet-bulb temperature at 2 m K 1 0 12 °C 1 3 0 12 006 Dew-point temperature at 2 m K 1 0 12 °C 1 3 0 12 007 Virtual temperature K 1 0 12 °C 1 3 0 12 011 Maximum temperature, at height and over period

specified K 1 0 12 °C 1 3

0 12 012 Minimum temperature, at height and over period specified

K 1 0 12 °C 1 3

0 12 013 Ground minimum temperature, past 12 hours K 1 0 12 °C 1 3 0 12 014 Maximum temperature at 2 m, past 12 hours K 1 0 12 °C 1 3 0 12 015 Minimum temperature at 2 m, past 12 hours K 1 0 12 °C 1 3 0 12 016 Maximum temperature at 2 m, past 24 hours K 1 0 12 °C 1 3 0 12 017 Minimum temperature at 2 m, past 24 hours K 1 0 12 °C 1 3 0 12 021 Maximum temperature at 2m K 2 0 16 °C 2 4 0 12 022 Minimum temperature at 2m K 2 0 16 °C 2 4 0 12 030 Soil temperature K 1 0 12 °C 1 3 0 12 049 Temperature change over specified period K 0 -30 6 °C 0 2 0 12 051 Standard deviation temperature K 1 0 10 °C 1 3 0 12 052 Highest daily mean temperature K 1 0 12 °C 1 3 0 12 053 Lowest daily mean temperature K 1 0 12 °C 1 3 0 12 061 Skin temperature K 1 0 12 °C 1 3 0 12 062 Equivalent black body temperature K 1 0 12 °C 1 3 0 12 063 Brightness temperature K 1 0 12 °C 1 3 0 12 064 Instrument temperature K 1 0 12 K 1 4 0 12 065 Standard deviation brightness temperature K 1 0 12 K 1 4

Table CDescriptor coding operators

class BUFR_descriptorCodingOperators

«implementationClass»BUFR_descriptorOperators

+ set() : void+ unset() : void

coordinateOperator

identificationCov erageOperator

instrumentationCov erageOperator

timeCov erageOperator

positionCov erageOperator

parameterCoverageOperator

coordinateRangeOperator

coordinateIncrementOperator

significanceQualifiierOperator

replicationOperator

delayedRepetitionOperatordelayedReplicationOperator

dataMissingOperator

dataCompressionOperator

dataAttributeOperator

associatedFieldOperator

freeTextOperator

qualityAssessmentOperator

localInformationOperator

unitChangeOperator

scaleChangeOperator

referenceChangeOperatordataWidthOperator

class BUFR_message

BUFR_Metadata

BUFR_containerMetadata

BUFR_indica torSection0

- BUFR_editionNumber: byte- BUFR_messageLength: byte- BUFR_name: char = "BUFR"

BUFR_endSection5

- BUFR_endFlag: char = "7777"

BUFR_discov eryMetadata

BUFR_dataDescriptionSection3

- dataDescriptionSe ctionSize: byte- extendedDescription: byte

BUFR_data Section4

- dataSectionSize: byte- extendedDataValues: byte

BUFR_identificationSection1

- dataCategoryCode: byte = Table A- dataSubCategoryCode: b yte = Local code table- identificationSectionSize: byte- localTableVersi onNumber: byte- masterTableCode: byte = 0 for WMO- masterTableVersi onNumber: byte- optionalSectionFlag : boolean = False- originatingCentreCode: b yte = Table C-1 0 01 033- originatingSubCentreCode: byte = Table C-1 0 01 034- referenceDateTime: byte- updateSeequenceNumber: byte

BUFR_optionalSection2

- optionalMeta data: byte- optionalSecti onSize: byte

BUFR_struc tureValues

- dataDescriptionSe ctionSize: byte- dataSectionSize: byte- indicatorSecti onSize: byte- optionalSecti onSize: byte

Section 3 - the description section - in a BUFR message is the coded version of the feature collection declaration in the message - the BUFR instance.BUFR simple features are continuous coverages, discrete coverages (code flags) and operation declarations on the simple features (data types) which modify those feature types.

Section 3 iis a set of table references to BUFR Table D (for generic feature collections defined for each Table A BUFR specialisation) Table C for operations (such as replications representing meteorological observations at multiple locations and times) on the section 3 references, and references to Table B (simple features and further operations).

When the Table D references are expanded to sets of Table B references, and the Table C operations are performed on the section 3 elements, the result is an expanded set of simple Table B references which give a one-to-one mapping to the values implicit in Section 4.

The elemental Table B references contain codes for units of measure, numerical offsets and multipliers, and the number of bits in which each value is stored in Section 4. There can also further information on any extra compression used on Section 4 values. When this compression is expanded, the bit sections can be separated and the numerical multipliers and offsets applied to the number represented by the bit string to return fixed-point values for features whichare continuous coverages. For example this might be a temperature specified to 0, 1 or 2 decimal places in degrees Kelvin. For discrete coverages, the Table B entries refer to BUFR code tables describing distinct weather elements, such as present weather codes refering to mist, fog, thunder etc..

The BUFR tables define in full detail every non gridded feature known to WMO. The Tables also allow individual instances to be coded extremely concisely (the codes are used for very expensive and very efficient communications from Met Satelites, for example. MSG2 sends out 26GB of new data each day).

Although this seems very different from ISO using GML, the parallels are clear. BUFR Tables are feature catalogues. They are complete descriptions of WMO data and the table mechanism - like ISO feature catalogues - can be extended to many other domains.

«feature instance»BUFR_M essage

0. .1

• BUFR Message– Feature Instance

wmo bufr &crex gil ross, uk met office 2009-09-21

Documents