essnet on sdmx - statistiska centralbyrån (scb) · essnet on sdmx –technical issues objectives...

«

Workshop on ESSnets

31/5 – 1/6 2010, Stockholm

«

ESSnet on SDMX«Technical issues

«

Technical coordination

Work packages

Dissemination (wiki, web site)

«

«

ESSnet on SDMX – Technical issues

«

«


Objectives

The main contents of the terms of reference:

• Implementation of SDMX at international and

national level

• The further development of the SDMX standards

used for data and metadata

•The further development of the SDMX Content-

Oriented Guidelines

• The further development of the SDMX IT architecture

and IT tools

• The increase of SDMX capacity and knowledge

«

«


Objectives

Broadening and strengthening a NSI standard SDMX framework

(WP8);

Making standard tools available to a more easy implementation of

the standard SDMX in the processing and exchange of data and

metadata files between national and international organizations (WP3,

WP4, WP7);

Improving the semantic harmonization of concepts and terminology

supporting this standard (WP2);

Disseminating related tools and documentation widely (WP9);

«

«


Objectives

• Results have to be broadly reusable within the ESS

• Software tools developed on an Open Source basis and registered

under European Union Public licence (EUPL);

Ensures the following rights:

Obtain the source code of the software from a free access repository;

Use the software in any circumstances and for all usage

Reproduce (copy, duplicate) the software

Modify the original software

The development of software should take care about:

A modular design

The use of a version control system

Clear documentation

• Detailed information at http://ec.europa.eu/idabc/eupl

http://ec.europa.eu/idabc/eupl

«

«


Success factors

Communication: all the members of the project shall communicate

anything that might affect or contribute to the project;

Good inter co-ordination between Work Packages, namely

between WP’s responsibles , in order to minimize the overlapping of

work and to assure interoperability as always as needed;

Strong involvement of all the participants and stakeholders;

Allocation of the right people for each task ;

Stick to the approved working plan, reporting as soon as possible

any problem that could compromise it;

«

«


Main achivements

• Involvement and increase of knowledge on NSI’s about SDMX;

• Better inter co-ordination between Work Packages;

• Improvement of ROI ( Return On Information);

• A common framework to support integrated and interoperable

solutions;

• Dramatic reduction of developments costs on each NSI, due to

common tools and solutions;

«

«



WP1 – Management of the Action (PT)

Work:

• Monitoring the work in the different WP’s, the coordination between WP’s and the quality of the deliverables;

• To Identify problems and make proposals to overcome them;

• To Produce progress reports;

• Organizing coordination meetings;

Meetings:

• Kick-off meeting – 16/10/2009 - Eurostat and partner countries:Start the project, make a project review, establish theproject governance, define roles and responsibilities, clarifythe expectations of all parties;

• 2 nd coordination meeting – 26/03/2010 - progress evaluation

Reports:

Quarterly progress reports (January, April)


Difficulties:

Sound knowledge of the state-of-the-art is needed

In SDMX field there are too many software tools not yet

broadly known and commonly accepted

There are other standards for the same purpose

Oportunities:

Increase knowledge on SDMX

An oportunity to implement SDMX in the Organization

The possibility of influencing the way forward on the short and

medium run

«

«


Proposals

A group of experts to assist the coordinator on

technical evaluation of WP’s would be desirable;

A repository of specialized tech skills existing on

each NSI, could resolve eventual gaps on a particular

task, i.e. sharing knowledge not just information;

Improvement on the current practise of reusing

existing pieces of software already developed by

someone in any project.

«

«


Work Packages

WP2 – MCV Ontology (PT)

Responsible: Statistics Portugal

Goal: To create an ontology of statistical metadata based

on Metadata Common Vocabulary (MCV) concepts

Deliverables: 1st version of the ontology - July 2010

2nd version of the ontology – November 2010

Final version of the ontology – March 2011

Report describing the methodology used – March 2011

«

«


Work Packages


«

«


Work Packages


Terminological point of view

Improve the quality of the definitions:

More complete and accurate

More clear

More coherent among them

«

«


Work Packages


Technological point of view

Semantic Web Technologies(*)

Web services – software tools designed to support interoperable machine-to-machine

interaction over a network

Ontology – Set of concepts, the relations among them and software

for computing them

Knowledge representation languages – RDF, OWL (Web Ontology

Language)

Turn an ontology from a model of concepts into something that is

computational

Make inferences with the concepts in an ontology

(*) The ideas and definitions in this slide were extracted from “Open issues on the semantic web”, Dan Gillman,

Work session on statistical metadata (METIS), March 2010

«

«


Work Packages

WP3 – SDMX Data Explorer (PT, IT, ES)

Responsible : Statistics Portugal

Participants: ISTAT, INE-SP

Goal: To deliver a tool (using a generic SDMX class library) to

visualize SDMX data files under different perspectives

(html, CSV, Charts, SDMX format conversion) in a more

user friendly way.

Progress: Functional and technical specifications (03.D-1 finished);

Analysis and design documentation (03.D-2 finished);

User Interface prototype description(03.D-3 finished);

The development of SDMX-ML Parser has been started;

Source code available for partners in repository

«

«


Work Packages


Deliverables: Application package intended to explore statistical data

using an SDMX data file and related DSD;

Documentation associated with the tool:

Functional and technical specifications

Analysis and Design documentation

User interface prototype

User Manual

Implementation report

Application package: Web application, class

library and web services

«

«


Work Packages


«

«


Work Packages

WP4 – Generalized SDMX Framework (IT, PT, SK)

Responsible: ISTAT

Participants: Statistics Portugal, INFOSTAT (Slovak Republic)

Goal: To complete the “Generalized SDMX Framework” already

implemented by ISTAT, with more functionalities, in order

to make it available as a complete product;

«

«


Work Packages

WP7 – PC-Axis with SDMX-ML (SE, NO, IT)

Responsible: SCB (Sweden)

Participants: ISTAT, Statistics Norway

Goal: To create a PCAxis <–> SDMX converter, based on

PCAxis files and on an SQL database structured

according to the Nordic Metadata Model

«

«


Work Packages

WP8 – SDMX in Survey Design and Data Collection (SK, PT)

Responsible: INFOSTAT (Slovak Republic)

Participants: Statistics Portugal

Goal: To implement SDMX in the survey design and data

collection

Pilot implementation on a selected statistical

domain (tourism and census);

Progress: Design of system architecture and system functionality

(May 2010) – 1st deliverable

Deliverables: Pilot implementation results

System documentation

Report on the methodological results achieved

«ESSnet on SDMX – Technical issues

Work Packages

Implementation of SDMX in survey design and data collection

processes at NSI

METADATA

MDSD

FORMS

SURVEYSDATA

STRUCTURE

DEFINITIONS

SDMX DB

RESPONDENTS

REGISTRY

SOURCE

DB

eFORMS

REPOSITORY

e FORMS

GENERATOR

DOCUMENTS

GENERATOR

RESPONDENTS

RESPONSES

MANAGER

LOADER

METADATA

MANAGER

EXTERNAL

METADATA

SOURCE DATA DATA

EDITING

IMPUTATION

REGISTER

DATA

LOADER

System

architecture

«WP8 – SDMX in survey design and data collection

«Proposal for e-form (e-questionnaire) standard

Preamble section

Identification section (zero modul)

Information section

Interpretative note section

Data section (modules)

Declaration section


«Structure of preamble section

LOGO Official name of the statistical office

Statistical survey nameForm ID

Surveyed period

Declaration of data protection

Form registration No.Address of form submitting

Date of submitting

Subbmitt to:

Until:

Contact::

Contact address


«

Matrix Data Strucure

Heading of columnsHeading of rows

Numbering

of rows

Numbering

of columns

Variable

Matrix


«Report structure for describing e-form

The structure of report on concepts in Concept Scheme for filling the tables describing statistical e-form - Heading level and Modules level (descriptive metadata for database tables) Relationships Concepts Sub-concepts The new semantic role of the

concept, which is used from another concept

1. .1 ORGANISATION_RESPOSIBLE The organization responsible for survey

CONTACT_ORGANISATION 12.1 Responsible organization

Abbreviation

LOGO

1. .1 FORM_IDENTIFICATION Identification Form

CONTACT_NAME 12.3 Statement founded

SURVEY_TITLE

SURVEY_CODE

IKF_CODE

SURVEY_YEAR

SURVEY_PERIOD

REGISTRATION_PLACE

REGISTRATION_NUMBER

REGISTRATION_DATE

1.. N IDENTIFICATION_KEY Identification Items

ROW_IDE

KEY_ORDER

KEY_NAME

KEY_SIZE

INSTRUCTFILL

1.. N INDICATION_DATA Indicator data

ROW_IND

KEY_ORDER

KEY_NAME

KEY_SIZE

INSTRUCTFILL

1.. N KCOMPLEMENTARY Descriptive information on the identifier and indicator

COMPLEMENTARY

COMPL_ORDER

1. .1 CONFIDENTIALITY Confidentiality

CONF_POLICY

CONF_DATA_TR

1. .1 INST_MANDATE Institutional mandate

Addressing

INST_MAN_LA_OA

INST_MAN_SHAR

1. .1 DELIVER_CONTACT Data for service form

DELIVER Deliver - description

CONTACT_NAME 12.3 Deliver - the person's name

CONTACT_ORGANISATION 12.1 Deliver - Organization (Regional Report)

CONTACT_MAIL 12.5 Deliver - postal address

1:1 METHODOLOGICAL_ EXPLANATION

Methodological notes on title

EXPLANATION_HEAD

EXPLANATION

1.. N SURVEY_MODULES

MODULE_ORDER

MODULE_SIGN

MODULE_NAME

The structure of report on concepts in Concept Scheme for filling the tables describing statistical e-form - Heading level and Modules level (descriptive metadata for database tables) Relationships Concepts Sub-concepts The new semantic role of the

concept, which is used from another concept

1. .1 ORGANISATION_RESPOSIBLE The organization responsible for survey

CONTACT_ORGANISATION 12.1 Responsible organization

Abbreviation

LOGO

1. .1 FORM_IDENTIFICATION Identification Form

CONTACT_NAME 12.3 Statement founded

SURVEY_TITLE

SURVEY_CODE

IKF_CODE

SURVEY_YEAR

SURVEY_PERIOD

REGISTRATION_PLACE

REGISTRATION_NUMBER

REGISTRATION_DATE

1.. N IDENTIFICATION_KEY Identification Items

ROW_IDE

KEY_ORDER

KEY_NAME

KEY_SIZE

INSTRUCTFILL

1.. N INDICATION_DATA Indicator data

ROW_IND

KEY_ORDER

KEY_NAME

KEY_SIZE

INSTRUCTFILL

1.. N KCOMPLEMENTARY Descriptive information on the identifier and indicator

COMPLEMENTARY

COMPL_ORDER

1. .1 CONFIDENTIALITY Confidentiality

CONF_POLICY

CONF_DATA_TR

1. .1 INST_MANDATE Institutional mandate

Addressing

INST_MAN_LA_OA

INST_MAN_SHAR

1. .1 DELIVER_CONTACT Data for service form

DELIVER Deliver - description

CONTACT_NAME 12.3 Deliver - the person's name

CONTACT_ORGANISATION 12.1 Deliver - Organization (Regional Report)

CONTACT_MAIL 12.5 Deliver - postal address

1:1 METHODOLOGICAL_ EXPLANATION

Methodological notes on title

EXPLANATION_HEAD

EXPLANATION

1.. N SURVEY_MODULES

MODULE_ORDER

MODULE_SIGN

MODULE_NAME

…


«Concept Scheme


«DSD for data collected

«

«


Work Packages

WP9 – Dissemination (PT, all co-partners)

Responsible: Statistics Portugal

Participants: All partner countries

Goal: Dissemination of the results of this Action within

and outside the European Statistical System

Progress: ESSnet on SDMX web site (available April 2010)

http://sdmxessnet.ine.pt

Deliverables: Training Plan on the tools produced

Presentations about the project (ITDG, STNE,

MWG)

Final workshop (March 2011, Lisbon)

«

«


Dissemination

WIKI and WEB SITE

WIKI

Populated by the participants on the ESSnet

Authentication needed

Documents can be uploaded on draft status

Collaboration

WEB SITE

Populated by the co-ordination

Public access

Documents available with on final status

No collaboration

Structure of the wiki content at 19/03/2010

ESSnet

Main page

ESSnet on SDMX

Coordinator

Project details

Project description

Main objectives

Main milestones

Work Packages

Participants

«

«

WP

Main page

WP – name of the WP

(what is expected):

Participants

Main objectives

Description of the work

Deliverables

(In progress):

Structure of the wiki content at 19/03/2010

«

Web site

http://sdmxessnet.ine.pt

«

Web site