SDMX implementation: Status and next steps Item 3.2 – Standards Working Group 2-3 July 2018 Luca Gramaglia, Unit B5

SDMX and its benefits

Technical standards (Information Model and formats)

Statistical guidelines (common concepts and code lists)

Automation benefits (notably for validation)

Semantic harmonisation benefits

Current status of SDMX implementation

History

• In 2009, Eurostat started implement SDMX in all new or considerably changed datasets and reference metadata sets.

• Virtually all SDMX implementation projects carried out in Eurostat have sought to achieve compliance to both the SDMX technical standards and to the SDMX statistical guidelines.

A look at EDAMIS

• To assess the status of SDMX implementation, the EDAMIS transmission logs from 2015 to 2017 were analysed.

• Focus was on official data transmissions (no metadata transmissions or transmissions for validation or testing purposes).

• Data transmissions from Member States and to/from international organisations were analysed

Focusing on data transmissions…

National Authorities

Eurostat

International organisations

74515

11

11146

2398 2083

413

Data provider categories

% of SDMX transmissions by data provider category

Preferred SDMX formats

% of EDAMIS datasets that predominantly use SDMX formats

International data exchange

2015 2016 2017

EWF 0.0 0.0 0.0

GESMES 58.8 42.9 34.3

SDMX-ML 35.3 42.9 54.3

TOTAL

SDMX 94.1 85.7 88.6

% of EDAMIS datasets that predominantly use SDMX formats

National data collection

2015 2016 2017

EWF 18.8 17.1 14.5

GESMES 6.3 6.1 5.6

SDMX-ML 12.2 14.0 14.1

TOTAL

SDMX 37.3 37.2 34.2

Main conclusions

• For data exchange with international organisations, SDMX is the de facto standard

• For data collection from national data providers, sow or no growth over past three years

• Certain categories of national data providers (ONAs in particular) experience difficulties. SDMX implementation is successful with these data providers when the complexity of SDMX is hidden (e.g. when Webforms are used)

Moving forward: Benefit-based implementation

packages

• The difficulties in implementing SDMX can be summarised by the fact that the approach to implementation requires a lot of changes from business stakeholders, both in Eurostat and in Member States.

• Changes mostly come from: Changes are required to the structure of the data

Changes are required to the codes used

Changes are required in the format used

• The solution are different implementation packages that are adapted to specific business needs

Issues

SDMX compliance – technical side

Definition Business benefits

Criterion 1

A DSD which describes the structure of the incoming data is available. The data is not necessarily in a standard SDMX format, but can be mapped to the DSD.

Structural validation can be performed based on the DSD. Content validation rules can be implemented based on the DSD.

Criterion 2

The data is in a standard SDMX format.

Once the DSD is available, no additional specifications are required to describe the format.

SDMX compliance – statistical side

Definition Business benefits

Criterion 1

Concept names are aligned to the ones recommended in the SDMX Glossary.

Basic level of semantic interoperability between different datasets

Criterion 2

The guidelines for SDMX code lists are applied and cross-domain SDMX code lists are used.

Reduced need for transcoding when integrating or joining data across different domains.

Criterion 3

Compliance to the SDMX modelling guidelines (e.g. standard recommended concepts are used)

Data integration across different domains is easier, with little or no mapping between structures and codes necessary.

Definition of the implementation packages

SDMX-based validation for everyone The light implementation

• The light implementation focuses on making available SDMX-based validation tools to all statistical domains

• Does not foresee changes in structure/codes and does not prescribe switching to an SDMX standard format. Only seeks to describe the current structure of the data via a DSD to enable automated SDMX-based validation

• Effort is foreseen to be small for domains that currently receive data in CSV or FLR, larger for domains using Excel.

• Applicable to both aggregate data and micro-data.

SDMX-based validation for everyone The light implementation

• Light implementation gives access to the validation architecture developed by the ESS Vision 2020 Validation project.

• Should represent the minimum SDMX implementation level for all domains

SDMX-based content harmonisation The medium and full implementation

• Full implementation of the SDMX standard still available when content harmonisation benefits are a key requirement. A current example is the foreseen implementation of SDMX in the context of FRIBS.

• To be truly effective, a more comprehensive strategy for content harmonisation that goes beyond data collection and exchange is needed.

Next steps

Next steps

• Q4/2018: Eurostat will draft a plan for introduction of the light implementation package in all the domains that have not yet implemented SDMX.

• Item 3.3 of the agenda of the Standards Working Group discusses some proposals and ideas to move forward on the management of SDMX code lists.

Thank you for your attention!

Any questions