Training Interactive Demonstrations Mapping Assistant ISTAT Loader module

7th Meeting of the Expert Group on SDMX 27-30 October 2014, Seoul, Korea

Daniel SURANYI, Eurostat Daniel.Suranyi@ec.europa.eu

Click to add text

http://1drv.ms/ZIHPSX

What we have done before this demo

• Installed Microsoft SQL Server Express 2014

• Created an empty database MASTORE_NA

• Created a demo database NA with some sample data

• For part 1: Installed SDMX-RI tools (xcopy versions) • Mapping Assistant .net 3.2.1 (intermediate version)

• TestClient .net 2.9.2 (intermediate version)

• NSI Web Service .net 2.7.1 (intermediate version)

• NSI Web Client .net 3.1.1 (ultimate version)

• Configured Web Service and Web Client in IIS

• Downloaded SDMX-NA package (DSDs and Dataflows)

• For part 2: Installed ISTAT Loader application

Interactive Demonstrations

Part 1: you already have a dissemination DB

• SDMX-RI Mapping Assistant, basic features

• SDMX-RI TestClient (for PUSH mode)

• SDMX-RI Web Services

• SDMX-RI Web Client

• Part 2: you do not have a dissemination DB yet

• ISTAT Loader & Mapping Assistant

• SDMX-RI TestClient

• SDMX-RI Web Client

SDMX Reference Infrastructure https://webgate.ec.europa.eu/fpfis/mwikis/sdmx/index.php/SDMX_Reference_Infrastructure_SDMX-RI

Data Provider Data Collector

Non-SDMX local database

SDMX data set

SDMX Registry

DSD

SDMX-RI

Mapping Assistant

Web Svc

Test Client

Web Client

What we will do in this demo…

• Launch Mapping Assistant for the first time

• Configure mapping store database (MASTORE_DB)

• Setup a dissemination tree for National Accounts

• Create a mapping between a NA DSD and the sample disseminiation database

• Download a file to PUSH data

• Show a web interface PULLing data

Mapping Assistant first start

Create Category Scheme

Create Categories

Note: The Category Scheme and the Catgories below represents the "navigation tree" of the dissemination system. It can be customised as you wish. It will be displayed like you set it up here in the web client later on. In that example we only create a single Category.

Add NA_MAIN DSD

Note: Browse to • Desktop

• Korea • SDMX-NA

• DSDs-2.0 And choose • NA_MAIN+ESTAT+1.2

Then wait a few seconds…

Add Dataflow defintions

Note: The download contains all National Accounts dataflows, but we only imported the NA_MAIN DSD. Thus an error may appear for non-NA_MAIN dataflows, which you can ignore.

Browse for: Desktop \ Korea \ SDMX-NA SDMX-NA_AllDataflows_SDMX20.xml

Move selected Dataflow to prepared tree (NAMAIN_T0103_A)

2 possibilites: Drag & drop Or Right-click Attach

Mapping Assistant DB configuration

Settings: NA-Connection SqlServer localhost\SQLEXPRESS NA √ Use Windows authentication Test connection Save

Setup database query

Select table and click OK

Add mapping between DB and DSD

Dataset DSD Comment

FREQ FREQ TIME_FORMAT

Transcode: A A, P1Y

ADJUSTMENT ADJUSTMENT Transcode: NOT_ADJ N

Others 1:1

REF_AREA Constant: AT

COUNTERPART_AREA Constant: W2

INSTR_ASSET Constant: _Z

REF_PERIOD_DETAIL Constant: C

DECIMALS Constant: 2

TABLE_IDENTIFIER Constant: T0103

UNIT_MULT Constant: 6

COMPILING_ORG Constant: AT1

Create transcoding

NSI Web Client

Web Client pivot table

Interactive Demonstrations

Part 1: you already have a dissemination DB

• SDMX-RI Mapping Assistant, basic features

• SDMX-RI TestClient (for PUSH mode)

• SDMX-RI Web Services

• SDMX-RI Web Client

Part 2: you do not have a dissemination DB yet

• ISTAT Loader & Mapping Assistant

• SDMX-RI TestClient

• SDMX-RI Web Client

Data Provider Data Collector

No database SDMX data set

SDMX Registry

DSD

SDMX-RI

Mapping Assistant

Web Svc

Test Client

ISTAT Loader & SDMX-RI http://bms.istat.it/sodidownload/Download.aspx

Web Client

ISTAT Loader

SDMX compliant local database

SQL Server: create new empty DB

Mapping Assistant: add new dataflow

Settings: Login to Mapping Assistant Create a new category (T0101) Move dataflow NAMAIN_T0101_Q into the new category

Loader – initial configuration

Loader db: refers to disseminiation database to be used (we created a blank one)

Loader – initial configuration

Mapping Store db: refers to same database Mapping Assistant is using we use

the same as in teh first example

Loader – initial configuration

Directories: refers to default directories to be used by Loader

Loader – first start

Dataflows: Select dataflows from mapping store to be used by Loader

Data Tables: Create tables in the database for each dataflow

Note: MSSQL supports only 16 keys unselect mandatory

attributes from key ("mandatory fields") after error message

Check database

Fact Tables: Loader created a table per code list Data Tables: Loader created a relational table for the data Note: mapping store is automatically configured

Map to input files (e.g. CSV)

Browse to • Desktop

• Korea • SDMX-NA

• Excel-Templates And choose • SDMX-NA_MAIN_T0101Q_Sample.csv

Map columns of CSV to DSD

• Match header

• Save

• Done

Put dataflow in production

Can load: • SDMX-EDI (Gesmes) • SDMX-ML 2.0

• Generic • Compact • Cross-Sectional

• Fixed length record • Mapping needed

• CSV • Mapping needed

Load data file

Check database

• Laoder fills data table

Open Web Client to browse data