SDMX implementation in Istat: progress report

Meeting of the Expert Group on Statistical Data and Metadata Exchange

27-30 October 2014, Korea

Francesco Rizzo

Department for Integration, Quality, Research and Production Networks Development (DIQR)

Aim of presentation

• Stat2015 – Istat modernization multiannual program

• Enterprise Architecture in Istat

• SDMX strategy in Istat

• Istat SDMX Architecture

• Funding

• Collaboration with other organizations

• Streamlining the internal workflow

• Capacity building actions

• SDMX in the National Statistical System

• SDMX in production – state of art

Stat2015 – Istat modernization multiannual program

• Industrialization and standardization of the statistical processes in order to overcome the stovepipes statistical production model

• Harmonization of statistical information

• Foster the efficiency and reuse of processes, and the reuse of collected data

• Raising the quality of statistics and reduce the statistical production costs

Stat2015: Enterprise Architecture in Istat

Business Architecture

Information Architecture

Application Architecture

Technical Architecture

Conceptual layer

Logical Layer

Application

Layer

Physical Layer

GSBPM

GSIM

DDI Other

SOA, WS, Tools

Web Service and Databases on load balancing/business

continuity

Defines what the industry does and how it is done: • Business Function • Business Process • Business Service

Describes the information, its flows and uses across the industry, and how that information is managed

Describes the set of practices used to select, define or design software components and their relationships

Describes the infrastructure technology underlying (supporting) the other perspectives

SDMX

CS

PA

Stat2015: SDMX Istat Strategy

Implementing the SDMX strategy (2013 -2015)

1. From a tactical to a strategic approach

2. Design a suitable architecture that could satisfy the whole SDMX strategy;

3. Looking for the necessary funds to support the implementation;

4. Collaborating with other organizations;

5. Streamlining the internal workflow;

6. Capacity building actions;

From a tactical to a strategic approach

OECD SIS - CC

ESSnets

SDMX

for standardization

and industrialization

(Stat2015)

beyond the data

exchange

SDMX Istat Strategy

SODI

Demography

EGR

Census Hub

getting experience from ESS

pilot exercises

collaborative sharing of knowledge

and software

The overall architecture

Structural Metadata Web GUI

RSS feed

Tablet Smartphone

Excel Plug-in

SDMX-RI

Reference Metadata

Structural Metadata

I.Stat

Data warehouse

Eurostat Census Hub

Eurostat ICT survey

Eurostat National Accounts

OECD STES

Bank of

Italy

Private enterprises

Researchers

Other Italian

Institutions

RDF channel

Google DSPL

channel

JSON channel

. . .

Subject matter domain production units

SEP

Push

Pull

SUM

Financing the implementation: Eurostat – ISTAT grant

• Total estimated cost: 464.983,87 €

• Duration: 24 months (Extended to August 2015)

• 24 deliverables split among 4 work packages:

WP1 – Coordination

WP2 – Enhancement of the SDMX-RI, and its integration into the Istat information system

WP3 – Enhancement of the Istat Metadata Management System

WP4 – Contribution to the development of SDMX standard and implementation of capacity building actions

• Involved: 2 Directorates, 3 Services and 7 Units (19 internal people involved for a total of 778 man/days)

• > 1000 man/days subcontracting

Collaboration with other organizations

ISTAT

OECD

SIS-CC

Statistical

Network

SDMX

TWG/SWG

SDMX-RI

User group

INEGI

(MEX)

Streamlining the internal workflow: Centralized Vs Decentralized

Centralized actions

Subject Matter Domain

actions

I.STAT

(macro)

SDMX WS

DSDs

Reporting Mapping (MA)

OECD (STES)

IMF (SDDS+)

DSDs

Dissemination Data modelling (DSW) Mapping (MA)

Metadata

and

Mapping

Repositories

Eurostat

Reporting Mapping (MA)

Production DBs

(macro)

Reporting (Push)

Reporting (Pull)

Dissemination (Pull)

TCI

Capacity building actions (2014)

Internal training

Introduction to SDMX 0.5 day 2 sessions

SDMX Data modeling 2 days 2 sessions

How to use the SDMX-RI 3 days 3 sessions

Programming SDMX 5 days 1 session

Other actions

Workshop on SDMX Source.NET 3 days 10--12 March 2014

How to use the SDMX-RI for reporting

to International Organizations (OECD

SDMX Expert Group)

2 days 29-30 October 2014

ABS-ISTAT-SNZ 15 days 24 Nov – 5 Dec 2014

SDMX in the National Statistical System – SISTAN Hub Objectives and phases

• Disseminate from a single web application data stored in a distributed environment (data are not stored centrally but grabbed from the peripherals nodes upon request of the end users)

• End users could access and compare harmonized datasets (DSDs will be defined centrally by Istat), or navigate data that refers to a specific peripheral node (e.g. a Region or another SISTAN Institution)

• Proof of concept already started: • Lombardi and Sicily Region involved • Reused/adapted the SDMX-RI Web Client to simulate the Hub (it

acts as a Dispatcher • Istat Regional Offices involved for supporting • Training already started

• Prototype in 2014-2015 • Central Hub Application under design (the development should start

in Autumn)

SDMX in the National Statistical System – SISTAN Hub

SDMX-RI

.STAT Existing

Data

warehouse

SDMX-RI SDMX-RI

DB

DL *

Ista

t S

IS

TA

N I

nstitu

tion

s

File CSV

e/o

SDMX

Web

DL

DB DB

DB

**

HUB

I.Stat

**

**

WEB

GUI

SDMX-RI

SDMX-RI

SDMX in production – state of art

• Metadata repositories • Data flows: 318 I.Stat + 1071 Agriculture Census + 481 Industry

Census) • Code lists: 412 I.Stat + 102 Agriculture Census+ 69 Industry Census • Concept schemes: 56 I.Stat + 23 Agriculture census + 5 Industry

Census) • Data dissemination (machine to machine)

• All cubes of I.Stat + censuses databases available from the SEP • Data reporting

• Census Hub • ICT VIP pilot project • National Account pilot project • Job Vacancy Survey (in testing) • Labour Cost Indicator (in testing) • STS VELA (in testing) • OECD STES • IMF SDDS plus (prototype)

Thanks