ai - circabc.europa.eu gr… · web viewas a deliverable the npms structure in bulgarian and...

REPUBLIC OF BULGARIANATIONAL STATISTICAL INSTITUT

FINAL TECHNICAL IMPLEMENTATION REPORT

ACTION: “IMPLEMENTATION OF SDMX TECHNICAL AND STATISTICAL STANDARDS IN THE NATIONAL STATISTICAL INSTITUTE OF BULGARIA”

GRANT AGREEMENT NUMBER 11152.2015.001-2015.245

Prepared by:Galya STATEVA

November 2017

CONTENTS

I. OBJECTIVES.......................................................................................................................................... 3

II. IMPLEMENTING THE ACTION ACTIVITIES........................................................................4

II.1. INITIAL WORKSHOP......................................................................................................................... 4II.2. COMPONENT 1................................................................................................................................... 4II.3. COMPONENT 2................................................................................................................................... 9II.4. COMPONENT 3................................................................................................................................. 23

III. FINAL WORKSOP....................................................................................................................... 32

IV. OUTCOMES AND BENEFITS OF THE ACTION................................................................33

V. SUBMISSION OF THE REPORT AND OTHER DOCUMENTS TO EUROSTAT.....................34


INTRODUCTION

The National Statistical System of the Republic of Bulgaria is part of the European Statistical System and implements the collection, processing, analysis and storage/archiving of statistical data, as well as the activities on provision and dissemination of official statistics.

The Vision for development of National Statistical System in the period 2013 - 2017 is the follows: "NSS to produce and provide to all user groups high quality statistical products and services that are produced by harmonized European standards and methodologies".

The main objective for the development of the NSS is to improve the development, production and dissemination of quality statistical information for all user groups, according to their information needs.

Mission and vision for the development of the NSS will be implemented in accordance with the Fundamental Principles of Official Statistics of the United Nations and the European Statistics Code of Practice principles.

The action “Implementation of SDMX technical and statistical standards in the National Statistical Institute of Bulgaria” was launched in September 2015 in the framework of Grant agreement Number 11152.2015.001-2015.245. Duration of the project was 24 months. The action was directed towards implementation of SDMX technical and statistical standards in the National Statistical System of Bulgaria.

This Final technical report describes activities that were undertaken by BNSI during the implementation of the project within 24 months period (September 2015 - September 2017).

I. OBJECTIVES

In the beginning of September 2015, the Grant agreement for the action “Implementation of SDMX technical and statistical standards in the National Statistical Institute of Bulgaria” was signed. The following general objectives were outlined in the framework of the action:

Objective A: Implementation of the ESS metadata standards at national level.

Objective B: Implementation of the SDMX technical and statistical standards.

Objective C: BNSI IT infrastructure applications for SDMX.

Objective D: Popularization of SDMX technical and statistical standards and results in the NSS.


II. IMPLEMENTING THE ACTION ACTIVITIES

During September 2015 –September 2017, as a project implementation period, the NSI of Bulgaria had implemented all planned activities that were specified in the Annex I of the Grant Agreement № 11152.2015.001-2015.245.

The implementation of action “Implementation of SDMX technical and statistical standards in the National Statistical Institute of Bulgaria” was organized in three main components:

1. Component 1: Implementation of the ESS metadata standards at national level.

2. Component 2: Implementation of the SDMX technical and statistical standards.

3. Component 3: BNSI IT infrastructure applications for SDMX.

The current report’s structure is organized by main components and respective tasks include in each component (as defined in the Grant agreement documents).

II.1. INITIAL WORKSHOP

An Initial Workshop was organized in October 2015. The workshop was attended by 26 NSI participants from different subject-matter domains.

At the workshop were made several presentations. The project leader had outlined the general objectives of the action, current situation in the BNSI and the expected outcomes of the project. The component leaders had presented the main activities for each component and explained in details to BNSI experts the planned activities. Directions for the future joint work were outlined. In addition, the project leader presented the detailed action plan and distribution of tasks by metadata experts, IT experts and statisticians from subject-matter domains involved in the project.

The overall evaluation of the Initial Workshop showed that the timing and the objective of the training were reached. The participants were satisfied in biggest extent of the success, contents, usefulness, duration as well as of the workshop organization and atmosphere. The participants were also satisfied of the NSI experts’ knowledge on the presented subject, coachers’ communication and responsiveness to participants as well as of the provided workshop materials.

All initial workshop documents – Agenda and Presentations - were provided to Eurostat in the Annex 1.The original signed List of participants were archived in the documentation of the project according to national legislation rules.

II.2. COMPONENT 1

Task: Update currently used Single Integrated Metadata Structure (SIMS) in the National reference metadata store in order to describe ESS Standard for Quality Reports Structure


(release 2, December 2014) and EURO-SDMX Metadata Structure (release 4, December 2014)

In the beginning of the project, updating activities on currently used Single Integrated Metadata Structure (SIMS) in the IT tool “National reference metadata store” have been done in order to describe ESS Standard for Quality Reports Structure (release 2, December 2014) and EURO-SDMX Metadata Structure (release 4, December 2014). Both metadata structures were reviewed and relevant changes has been made by statisticians directly in the IT tool. Then the final validation by metadata experts was done. The updating structures have been published on the Internet site of the BNSI.

The updating of the ESMS has been done with accordance with SIMS 2.0 by metadata experts using the IT tool “National reference metadata store”. At the end of the June 2016, all ESMS 2.0 files are published on the NSI’s website for all statistical surveys. This task hasn’t been included under the initial application form. Therefore it was done as extra-work within the project.

Task: Review of availability of the quality reports in line with ESQRS (ESS Standard Quality Report Structure (release 2, December 2014)) in the BNSI

Review of availability of the quality reports in line with ESQRS (ESS Standard Quality Report Structure (release 2, December 2014)) was done for “Demographic and social statistics”, “Business statistics” and “Macroeconomic statistics” directorates. The availability of the ESQRS has been checked for the total 65 statistical surveys in November 2015 for the “Business statistics” directorate. 17% of them have and transmit ESQRS to Eurostat in English and 83% of them do not produce quality reports in line with ESQRS format. The same check has been done for the 40 statistical surveys in December 2015 for the “Demographic and social statistics” directorate. 35% of them have positive answer, but 65% don’t have ESQRS at all. The availability of the ESQRS has been checked for the total 42 statistical surveys in January 2016 for the “Macroeconomic statistics” directorate. 10% of them have and transmit ESQRS to Eurostat in English and 90% of them do not produce quality reports in line with ESQRS format.

Task: Development of quality reports following the ESQRS (ESS Standard for Quality Reports Structure (release 2, December 2014) format (in Bulgarian and English) for all statistical domains applying “step – by - step” approach

The work on quality reports in line with ESQRS ver.2 has been started first for statistical surveys in the “Demographic and Social Statistics” directorate. The work on developing quality reports continued in the other directorates, namely “Business statistics” and “Macroeconomic statistics”. Each survey manager has the possibility to fill the quality reports on-line using the IT tool “National reference metadata store”. The metadata experts validated all ESQRS files before their publishing on the NSI’s website.


The development on quality report in line with ESQRS ver.2 for statistical surveys is finished.

Task: Upload the ESQRS data in the “National reference metadata store”

The uploading in the “National reference metadata store” on the available quality reports in line with ESQRS ver.2 for statistical surveys in the “Demographic and Social Statistics”, “Business statistics” and “Macroeconomic statistics” directorates has been done though “step-by-step” approach.

Task: Dissemination of metadata information in the ESQRS format on the BNSI website

The BNSI website was changed in order to show ESQRS metadata. A new classification for metadata type with two codes ESMS and ESQRS was added to the content type for metadata pages on the website. The existing metadata pages were codes as ESMS. The existing content views were filtered with the ESMS parameter. New content views were created with ESQRS filter parameters. New dictionary page called “Quality reports” with ESQRS content was created. Each page of those new heading on the website automatically includes one export content page from “National reference metadata store” (http://www.nsi.bg/en/content/15006/basic-page/quality-reports ).

.


http://www.nsi.bg/en/content/15006/basic-page/quality-reports

Task: Creation of the National Process Metadata Structure extending Euro Process Metadata Structure (EPMS) on the base of GSBPM

For implementation the task of creation of the National Process Metadata Structure extending Euro Process Metadata Structure (EPMS) on the base of GSBPM, it was first necessary to develop a national version of the GSBPM. At the end of 2015 started the development of a national statistical description of the business process in accordance with GSBPM Version 5.0. In its development, both the team of project experts and a wide range of NSI experts were involved. The national version of Version 5.0 was broadly discussed through all statistical subject-matter domains. The document was published on the Internet site of BNSI at the beginning of the July 2016.

Meantime work on developing of the National Process Metadata Structure (extending EPMS) has already started. The first draft of the NPMS has been sent to the statistical subject-matter statisticians for broad discussions and additional comments. Then the final version of the National Process Metadata Structure (extending EPMS) were finished and adopted. As a deliverable the NPMS structure in Bulgarian and English is available in Word, SDMX 2.0 and SDMX 2.1 formats. (see Annex 2, Annex 3, Annex 41 and Annex 42).

Task: Pilot description of the National Process Metadata Structure for Structural Business Statistics (industry and services), Agricultural statistics and Labour Force Survey

The work on pilot description of the NPMS for Structural Business Statistics (SBS - Annual data of non-financial enterprises), Agricultural statistics (Economic accounts for agriculture – quarterly data) and Labour Force Survey have been done by subject-matter statisticians and validated by metadata experts. The above-mentioned three NPMS structures are bi-lingual


(Bulgarian and English versions) and are available in Word, SDMX 2.0 and SDMX 2.1 formats. (see Annexes: 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15).

In mean time as an additional test, the pilot descriptions for the HBS (see Annexes: 16, 17, 18, 19) and EU-SILC (see Annexes: 20, 21, 22, 23) was performed in Bulgarian and English. This task wasn’t included initially in the Grant agreement but has been done as extra-work. By applying the "step-by-step" approach, BNSI intend to develop and upload into “National reference metadata store” the NPMS for all statistical surveys in order to comply with Peer-review 2014 recommendations.

The special guidelines as a support tool for statisticians has been developed in Bulgarian (see Annex 24).

Task: Upload the National Process Metadata Structure for Structural Business Statistics (industry and services), Agricultural statistics and Labour Force Survey in the “National reference metadata store”

All produced NPMS within the project were uploaded in the “National reference metadata store” in SDMX format (see Annexes: 6, 7, 10, 11, 14, 15, 18, 19, 22, 23).Directory structure of the “National reference metadata store” file export of ESMS, ESQRS and NPMS in SDMX 2.0 and 2.1 versions with available DSD and MSD.

The “National reference metadata store” provides export function of the defined subsets into SDMX metadata message files. The SDMX files contain both the English and Bulgarian reference metadata information. The National reference metadata store is compatible with ESS Metadata Handler and allows metadata export in SDMX v2.0 and loading into other systems.This functionality is very convenient and important for statisticians.


All tasks foreseen in the framework of the project were successfully completed.

Deliverables for Component 1:

Annex 2: NPMS_BG.docx Annex 3: NPMS_EN.docx Annex 4_NPMS_LFS_BG.docx Annex 5_NPMS_LFS_EN.docx Annex 6_NPMS_LFS_MSD__2.0.xml Annex 7_NPMS_LFS_MSD_2_1.xml Annex 8_NPMS_Agriculture_BG.docx Annex 9_NPMS_Agriculture_EN.docx Annex 10_NPMS_Agriculture_MSD_2.0.xml Annex 11_NPMS_Agriculture_MSD_2_1.xml Annex 12_NPMS_SBS_BG.docx Annex 13_NPMS_SBS_EN.docx Annex 14_NPMS_SBS_MSD_ 2.0.xml Annex 15_NPMS_SBS_MSD _2_1.xml Annex 16_NPMS_HBS_BG.docx Annex 17_NPMS_HBS_EN.docx Annex 18_NPMS_HBS_MSD__2.0.xml Annex 19_NPMS_HBS_MSD__2.1.xml Annex 20_ NPMS_SILC_BG.docx Annex 21_ NPMS_SILC_EN.docx Annex 22_NPMS_SILC_MSD__2.0.xml Annex 23_NPMS_SILC_MSD__2.1.xml Annex 24_NPMS_guidelines.xlsx

II.3. COMPONENT 2

Task: Re-design of the national DSD of Short-term buisness statistics from SDMX 2.0 to SDMX 2.1 (based on SDMX COG).

Unlike the previous project, there is already suitable DSD for Short-term buisness statistics (STSALL) on the Euro SDMX Registry, which is used for data exchange. Because of that, we naturaly decided to use it. The comparison between the DSD of the fist project and STSALL was carried on in order to determine the differences and the work required.

The Bulgarian translations of Concepts and Code list was appended to the original DSD for Short-term buisness statistics (STSALL) (see Annex 25). The Data Structure Wizard software was used for the task. The task is completed.


Task: Re-design of the national DSD of Tourism statistics from SDMX 2.0 to SDMX 2.1 (based on SDMX COG)

The Tourism statistics DSD in SDMX 2.0 verion was re-designed to SDMX 2.1 version (see Annex 26). The translations of concepts and code lists were updated where needed. The Code list for time format from the 2.0 version was omited, since it is not relevant any more in the new SDMX version. The Data Structure Wizard software was used for the task. The task is completed.

Task: Re-design of the national DSD of Business statistics on non-financial enterprises from SDMX 2.0 to SDMX 2.1 (based on SDMX COG)


The action is postponed in the begining due to allocation of the efforts for implementation of the SIMS 2.0, ESMS 2.0 and ESQRS 2.0 in the “National reference metadata store”.

The national DSD of Business statistics on non-financial enterprises in SDMX 2.0 version was re-designed to SDMX 2.1 version (see Annex 27). The translations of concepts and code lists were updated where needed. The Code list for time format from the 2.0 version was omited, since it is not relevant any more in the new SDMX version. The Data Structure Wizard software was used for the task.

Task: Update the BNSI’s SDMX-RI information with the SDMX 2.1 compliant DSDs for Short-term buisness statistics, Tourism statistics and Business statistics on non-financial enterprises

The SDMX 2.1 compliant DSDs for Short-term buisness statistics (STSALL) and Toursim statistics are uploaded to the BNSI SDMX-RI. Some of the artefacts in STSALL were changed as “isfinal=true”, since the original DSD is not final yet in the Euro SDMX Registry. The SDMX 2.1 compliant DSD for Business statistics on non-financial enterprises was uploaded to the SDMX-RI. The mapping between the DSDs concepts and database tables fields were created with the help of Mapping Assistant tool of the SDMX-RI. The confidentiality filters were added to the mapped data according to the relevant concepts and code lists. The task was performed several times, each time after updating the SDMX-RI version. We used FireFox add-on HttpRequester for testing the results from SDMX-RI.

List of available DSDs form the SDMX-RI web service.


The DSD for Tourism statistics form the SDMX-RI web service.

The data for STS form the SDMX-RI web service.


The data for Tourism statistics form the SDMX-RI web service.

The DSD for Business statistics on non-financial enterprises form the SDMX-RI web service.


The data for Business statistics on non-financial enterprises form the SDMX-RI web service.

Task: Implementation of the National accounts (annual data) DSD in BNSI’s SDMX-RI

The task was postponed initially, due to focusing the efforts over the Agricultural statistics DSD. The National Accounts DSD’s (The last available version from Eurostat NA_MAIN+ESTAT+1.7.xml) code lists and concepts were translated in Bulgarian. The Data Structure Wizard software was used for the execution of task. The new DSD was uploaded


into BNSI SDMX-RI. The mapping between the DSD concepts and database table fields were created with the help of Mapping Assistant tool of the SDMX-RI. The confidentiality filters were added to the mapped data according to the relevant concepts and code lists. The uploading, mapping and confidentiality filtering were performed each time after updating the SDMX-RI version.

List of available Dataflows form the SDMX-RI web service.

The DSD for National Accounts form the SDMX-RI web service.


The data for National Accounts form the SDMX-RI web service.

Task: Creation (based on SDMX COG) and implementation of the DSD for Agriculture statistics in BNSI’s SDMX-RI

The national DSD of Agricultural statistics (Economic accounts in Agriculture) was created in SDMX 2.1 version (see Annex 29). The Data Structure Wizard software was used for the execution of the task. The cross domain concepts and code lists from Euro SDMX Registry were used in the creation proccess. Some specific concepts and coresponding code lists were developed for the DSD needs. The new DSD was uploaded into BNSI SDMX-RI. The mapping between the DSD concepts and database table fields were created with the help of Mapping Assistant tool of the SDMX-RI. The confidentiality filters were added to the mapped data according to the relevant concepts and code lists. The uploading, mapping and confidentiality filtering were performed each time after updating the SDMX-RI version.



The DSD for Agricultural statistics form the SDMX-RI web service.

The data for Agricultural statistics form the SDMX-RI web service.


Task: Creation (based on SDMX COG) and implementation of the DSD for Structural Business Statistics (industry and services) in BNSI’s SDMX-RI

The national DSD of Structural Business Statistics (industry and services) was created in SDMX 2.1 version (see Annex 30). The Data Structure Wizard software was used for the task. The cross domain concepts and code lists from Euro SDMX Registry were used in the creation proccess. Some specific concepts and coresponding code lists were developed for the DSD needs according to the COMMISSION REGULATION (EC) No 250/2009 of 11 March 2009. The new DSD was uploaded into SDMX-RI. The mapping between the DSD concepts and database table fields were created with the help of Mapping Assistant tool of the SDMX-RI. The confidentiality filters were added to the mapped data according to the relevant concepts and code lists. The uploading, mapping and confidentiality filtering were performed each time after updating the SDMX-RI version.



The DSD for Structural Business Statistics form the SDMX-RI web service.

The data for Structural Business Statistics form the SDMX-RI web service.


Task: Preparation of the adapted concepts and standard code lists for updating DSDs in order to be implemented in SDMX 2.1 (STS, Tourism statistics and Business statistics on non-financial enterprises) and for new DSDs (Agriculture statistics, National accounts and SBS – industry and services)

An analysis was performed on the existing concepts in the datasets for Business statistics on non-financial enterprise, Agriculture statistics, Tourism statistics and Business statistics on non-financial enterprises. Then the nessesery concepts were defined. An analysis was performed on the existing codes in the datasets relevance to the concepts. Codel lists were created with the identifieid codes. When available, the cross domain concepts and code lists from Euro SDMX Registry were used in the creation proccess. The bilateral translations in Bulgarian and English was done for the concepts and code lists of the STS, Tourism statistics and Business statistics on non-financial enterprises, Agriculture statistics, National accounts and SBS – industry and services. The information was used as a source for descibing the concepts and codelists in SDMX format with Data Structure Wizard tool.

Task: Visualization and user access of non-confidential data about Tourism statistics, Agriculture statistics, National accounts and Structural Business Statistics (industry and services) on the BNSI website through the SDMX-RI web service

The visualization through the SDMX-RI web service of Tourism statistics was upgraded (http://www.nsi.bg/ddb/sdmx_compact_view_tourism_mysql.php?l=en), which increase the overall performance almost two times. Some links to SDMX sources was also added. A new script was added to allow the users to download the whole dataset in SDMX compact message (http://www.nsi.bg/ddb/sdmx_compact_view_tourism_mysql_xml.php?dataflow=BNSI_TOURISM_TABLES&sender=bnsi&version=1.0). The script was also


http://www.nsi.bg/ddb/sdmx_compact_view_tourism_mysql_xml.php?dataflow=BNSI_TOURISM_TABLES&sender=bnsi&version=1.0

http://www.nsi.bg/ddb/sdmx_compact_view_tourism_mysql_xml.php?dataflow=BNSI_TOURISM_TABLES&sender=bnsi&version=1.0

http://www.nsi.bg/ddb/sdmx_compact_view_tourism_mysql.php?l=en

linked on the Bulgarian Government Open Data Portal (http://opendata.government.bg/dataset/business-statistics/resource/8157d3df-acb9-4b85-8237-7080592aca17).

A new visualization tool was developed to use the SDMX-RI web service compliant with SDMX 2.1. The tool is writen in PHP programing language. The tool lists available data flows from the SDMX-RI as a hierarchical menu. The data flows are defined over the existing DSDs in the SDMX-RI by filtering the available data on some of the concepts. When the user choose a data flow, the common codes for the whole data flow are listed in front of the vizualized table. The data for the choosen dataflow then is parsed as HTML table. The HTML table then is visualized with JQuery Library called DataTables. The data and the coresponding DSD can be downloaded in SDMX 2.1 format.

A very simple visual analysis was added to the tool. The chart is generated when the user filters the table data. Google Charts is used.

The non-confidential data flows for STS and Business statistics on non-financial enterprises were also added to the visualization scope.

http://www.nsi.bg/ddb2.1/



http://opendata.government.bg/dataset/business-statistics/resource/8157d3df-acb9-4b85-8237-7080592aca17

http://opendata.government.bg/dataset/business-statistics/resource/8157d3df-acb9-4b85-8237-7080592aca17

Deliverables for Component 2:

Annex 25_STSALL+ESTAT+2.0.xml Annex 26_ BNSI_TOURISM_TABLES+BNSI+1.0.xml Annex 27_ BNSI_NFE_BNB+BNSI+1.1.xml Annex 28_ NA_MAIN+ESTAT+1.7.xml Annex 29_ EAA+BNSI+1.0.xml Annex 30_ SBS+BNSI+1.1.xml Annex 31_ Concepts_and_Codelists_in_SDMX_2.1.zip Annex 32_ ddb.zip Annex 33_ ddb2.1.zip Annex 34_ CATEGORY_SCHEMES+BNSI+1.0.xml

II.4. COMPONENT 3

Task: Reinstall the latest versions of software components of SDMX-RI compatible with SDMX 2.1

Before the beginning of the work on SDMX the computer, on which we wanted to install SDMX-RI, was virtualized and its capabilities were enhanced. The operating system and the database server were reinstalled.

Тhe following components of SDMX-RI were downloaded:

MappingAssistant_.NET_v5.3.1_2014.11.21-ultimate-capi-sdmx21

NSIWebService_.NET_v4.2.1_2014.11.21-ultimate-capi-sdmx21

TestClient_.NET_v4.3.1_2014.11.21-ultimate-capi-sdmx21

They were successfully installed on the computer designed for SDMX-RI. Installation of the NSIWebService needed ASP.Net v.4.0 Application pool instead of Default Application pool which was new for us. The web service is accessible through browser at: http://nsi-sdmx-01/sdmx21ws/.

In the beginning of 2017 a new release of SDMX-RI components was published. Then we downloaded and installed the latest version of SDMX-RI.The modules installed are:

MappingAssistant_.NET_v5.13.2_2017.01.04

NSIWebService_.NET_v5.2.3_2017.01.27

TestClient_.NET_v4.13.2_2017.01.04

The web service is accessible through browser at: http://nsi-sdmx-01/sdmx21ws_new/.


http://nsi-sdmx-01/sdmx21ws_new/

http://nsi-sdmx-01/sdmx21ws/

Task: Re-design of the Dissemination Data Base (DDB) for the subject-matter domains within project’s framework – changes concern the new version of SDMX-RI and the newly chosen statistical domains – Agriculture statistics, National accounts and Structural business statistics

It was decided to define new dissemination data base and new Mapping store instead of upgrading the old databases. New Mapping store was defined with a name MappingStore_SDMX2.1. New dissemination database was defined with a name DDB_BNSI_SDMX2.1. The data table for short term statistics was created with a new structure, according the new DSD – STSALL, which was downloaded from SDMX Registry. The old data for short term statistics was reloaded into the new data table. With this data we tested the installation of the SDMX-RI 2.1.

Task: Transform and re-upload the data for Short term statistics, Tourism statistics and Non-financial enterprises in compliance with SDMX 2.1 into DDB

The data for short term statistics had to be modified, because the concept TIME_PERIOD in SDMX 2.1 has changed as opposed to SDMX 2.0.

The modified DSD for short term statistics (with code lists translated in Bulgarian) was reloaded into the Mapping Store and the mappings were re-built.

The data table for tourism statistics was created into Dissemination database with a new structure, according to the new DSD. The old data for tourism statistics was reloaded into the new data table. The mappings between the DSD and the data table for tourism statistics were performed.


For tourism statistics – the data with TIME_PERIOD from 2012-10 to 2016-04 was reloaded into dissemination database.

The data table for Non-financial enterprises was created into Dissemination database. There were no changes with the structure from SDMX 2.0

The data for Non-financial enterprises was loaded into Dissemination database. The data covers 2009, 2010, 2011, 2012, 2013, 2014 and 2015 years. The mappings between data and relevant DSD were made using Mapping Assistant.

Task: Upgrade of the software for transformation of national data concerning selected DSDs – redesign the software to make it compatible with SDMX 2.1 and to include the new statistical domains - Agriculture statistics, National accounts and Structural Business Statistics (industry and services)

The application for transformation of national data, manipulating data from the dissemination database, defining new tables and loading data into the dissemination database was redesigned according to the changes in SDMX 2.1 and the new areas included in the project. For example the concept TIME_PEROID for short term statistics has changed. The application was modified to be able to deal with data of Agriculture statistics, National accounts and Structural business statistics. The application is included in Annex_43.rar


Task: Transform and upload the data for new statistical domains - Agriculture statistics, National accounts and Structural Business Statistics (industry and services) in the DDB

The structure of the data, the requirements and the DSD for Agriculture statistics were analyzed. New data table for Agriculture statistics was created into the dissemination data base. The new data was loaded into the dissemination data base. The mappings between data and relevant DSD were made using Mapping Assistant.

The structure of the data, the requirements and the DSD for National accounts were analyzed. New data table for National accounts was created into the dissemination database. The new data was loaded into the dissemination database. The mappings between data for National accounts and relevant DSD were made using Mapping Assistant.

The structure of the data, the requirements and the DSD for Structural business statistics were analyzed. New data table for Structural business statistics was created into the dissemination database. The new data was loaded into the dissemination database. The mappings between data for Structural business statistics and relevant DSD were made using Mapping Assistant.


TestClient was used for the testing after adding each new statistical area. All the tests were successful.

The sources for Agriculture statistics were explored and analyzed. The data for 2015 was prepared according to the DSD for Agriculture statistics. The data files were produced using SPSS.

The sources for National accounts were explored and analyzed. The data for 2000 to 2015 year was prepared according to the DSD for National accounts. The data files were produced using SPSS.

The sources for Structural business statistics were explored and analyzed. The data for 2011 to 2014 year was loaded into Dissemination database. The structure of the files was compliant with the DSD for Structural business statistics so there was no need to prepare new file format for loading data into the DDB.

The SPSS scripts for transformation of data of National accounts and Agriculture statistics were prepared and tested. The data was transformed from working Excel files into CSV files suitable for uploading into dissemination database using the application for manipulating the dissemination database. The scripts are included in Annex_44.

Task: Performance and load testing of the newly installed and redesigned applications and databases

We preformed three test scenarios over the data sets for STS, Tourism statistics and Business statistics on non-financial enterprises, Agriculture statistics, National accounts and SBS – industry and services:

1. Get data from the WebService at http://nsi-sdmx-01/sdmx21ws_new/SdmxService

2. Get data from the application at http://www.nsi.bg/ddb2.1/ redirected to http://nsi-

sdmx-01:82/ddb2.1/

3. View data at http://www.nsi.bg/ddb2.1/ redirected to http://nsi-sdmx-01:82/ddb2.1/

The smallest data set has 866 records and the biggest data set has 44267 records. Our server environment performed as follows (see Annex 35):

1. It takes 2 seconds average to download the biggest data set with the WeService and less than a second for the smallest one

2. It takes 4 seconds average to download the biggest data set with the application and less than a second for the smallest one

3. It takes 143 seconds average to view the biggest data set as HTML page with DataTables JQuery Library and Google Charts and a second for the smallest one. It takes one to three seconds for each thousand records to view as HTML page.


http://nsi-sdmx-01:82/ddb2.1/





http://nsi-sdmx-01/sdmx21ws_new/SdmxService

Task: Upgrade of the IT tool for creating and storing national reference metadata to be compatible with SDMX 2.1 and update the “National reference metadata store” with the latest versions of the core and modules of the base system Drupal CMS

The IT tool for creating and storing national reference metadata was updated to the latest versions of the Drupal CMS core (7.56) and modules. We updated the Drupal CMS core and modules several times to the latest available versions during the project schedule.

The “National reference metadata store” was updated to be compliant with SIMS 2.0, thus the old 21 main concepts with sub concepts were transformed into new 19 concepts with subconcepts acording to SIMS 2.0. Statistical data and metadata module of the IT tool (Annex statistical_metadata2.0.zip) was developed to install SIMS 2.0 structure and to update the old EIMS structure into SIMS 2.0 structure. The ESMS 2.0 and ESQRS 2.0 export definitions were created. The export definitions are used from the IT tool to export the metadata from SIMS 2.0 structure into ESMS 2.0 and ESQRS 2.0 structures in SDMX format. The exported ESMS 2.0 and ESQRS 2.0 files were only in SDMX 2.0 version at first. The ESMS v2.0 metadata export in SDMX v2.1 was created for the “National reference metadata store” to be added to the existing SDMX v2.0 export. Similar export was prepared for the ESQRS v2.0. Two MSDs were developed for ESMS 2.0 and ESQRS 2.0 in SDMX 2.0 format and two more in SDMX 2.1 format (see Annexies: 36, 37, 38 and 39).

Part of the SIMS 2.0 concepts listed in the administrative module of the IT tool for creating and storing national reference metadata.

Part of the SIMS 2.0 concepts listed in the database of the IT tool for creating and storing national reference metadata.


Screen for editing SIMS 2.0 metadata concepts , in particulare Unit of measure, of IT tool for creating and storing national reference metadata.

The “National reference metadata store” was updated to the latest versions of the base system core and modules. The store is now fully compliant with SDMX 2.1.


Task: Extend “National reference metadata store” and IT tool functionalities with National Process Metadata Structure

The work started to integrate National Process Metadata Structure into National Reference Metadata Store. The National Process Metadata Structure should use the existing SIMS 2.0 concepts in the store. Suitable MSDs are going to be prepared in order to export the process structures into SDMX v2.0 and v2.1.


The National Process Metadata Structure concepts were added into “National Reference Metadata Store”. The existing SIMS 2.0 concepts in the store were used when they corresponds to the concepts of National Process Metadata Structure. Suitable MSDs were prepared in order to export the process structures into SDMX v2.0 and v2.1. The Data Structure Wizard software was used for the task (see Annex 41 and Annex 42).

National Process Metadata Structure concepts listed in the administrative module of the IT tool for creating and storing national reference metadata.



Deliverables for Component 3 :

Annex 35_ Prerformance and load testing.zip Annex 36_ ESMS_MSD+BNSI+2.0+SDMX.2.0.xml Annex 37_ ESQRS_MSD+BNSI+2.0+SDMX.2.0.xml Annex 38_ ESMS_MSD+BNSI+2.0+SDMX.2.1.xml Annex 39_ ESQRS_MSD+BNSI+2.0+SDMX.2.1.xml Annex 40_ Statistical Metadata Module 2.1.zip Annex 41_ NPMS_MSD+BNSI+1.0+SDMX.2.0.xml Annex 42_ NPMS_MSD+BNSI+1.0+SDMX.2.1.xml Annex_43_VBNetApplication.7z Annex_44_SPSSscripts.7z

III. FINAL WORKSOP

A Final Workshop was organized from 11 to 13 in September 2017. The workshop was attended by 38 participants from BNSI and ONAs within Bulgarian National Statistical System. The workshop was organized as the sessions by components for reporting final deliverables of the project.

At the workshop were made several presentations. The project leader had outlined the general objectives of the action, current situation in the BNSI, SDMX philosophy and the achieved outcomes of the project as a whole. The component leaders had presented the main activities for each component and explained in details the final results. In addition, responsible statisticians from subject-matter departments were presented the content of the NPMS structure (LFS, SBS, HBS, EU-SILC and Agricultural statistics).


There were extensive discussions on the possibilities for horizontal and vertical integration in the National Statistical System. In the discussions that ensued, the project leader highlighted the importance of good coordination at national level. Conclusions were made and possibilities for the future applicability of the achieved results were identified, particularly in ONAs.

All final workshop documents – Agenda and all presentations are provided to Eurostat (See Annex 45).

IV. OUTCOMES AND BENEFITS OF THE ACTION

The Grant action outcomes are supported the National Statistical System on different aspects:

ESQRS metadata information loaded in the “National reference metadata store”;

ESQRS metadata information published on the BNSI website;

MSD for the National Process Metadata Structure;

National Process Metadata Structure loaded in the “National reference metadata store” for Structural Business Statistics (industry and services), Agricultural statistics and Labour Force Survey;

SDMX COG implemented at national level for the newly created DSDs;

Code lists and concepts adapted at national level compliant with SDMX 2.1;

Existing national DSDs for Short-term buisness statistics, Business statistics on non-financial enterprises and Tourism statistics upgraded from SDMX 2.0 to SDMX 2.1;

SDMX techical standart 2.1 implemented for Agriculture statistics, National accounts and Structural Business Statistics (industry and services);

National DSDs created for Agriculture statistics and Structural Business Statistics (industry and services);

Implemented procedures, scripts and configurations for automated transform and upload of data for the subject-matter domains within project’s framework in the dissemination database to be used by SDMX-RI;

Better pull access of statistical data and better visualization of non-confidential data;

Dissemination Database compliant with SDMX 2.1 standard;

Nationl SDMX-RI compliant with SDMX 2.1 standard;

Agriculture statistics, National accounts and Structural Business Statistics (industry and services) ready for pull mode via SDMX-RI;

National reference metadata tool capable of storing of the process-related metadata.


Most important outcome is better understanding of SDMX 2.1 standard by users at natonal level (statisticians, other national authorities of statistics, BNB).

V. SUBMISSION OF THE REPORT AND OTHER DOCUMENTS TO EUROSTAT

The Final technical and financial reports, including all Annexes will be provided to Eurostat only by e-mail.

ANNEXES

List of all deliverables for the period September 2015 – September 2017Deliverable name

Initial workshop.zip Annex_1 NPMS_BG.docx Annex_2 NPMS_EN.docx Annex_3 NPMS_LFS_BG.docx Annex_4 NPMS_LFS_EN.docx Annex_5 NPMS_LFS_MSD__2.0.xml Annex_6 NPMS_LFS_MSD_2_1.xml Annex_7 NPMS_Agriculture_BG.docx Annex_8 NPMS_Agriculture_EN.docx Annex_9 NPMS_Agriculture_MSD_2.0.xml Annex_10 NPMS_Agriculture_MSD_2_1.xml Annex_11 NPMS_SBS_BG.docx Annex_12 NPMS_SBS_EN.docx Annex_13 NPMS_SBS_MSD_ 2.0.xml Annex_14 NPMS_SBS_MSD _2_1.xml Annex_15 NPMS_HBS_BG.docx Annex_16 NPMS_HBS_EN.docx Annex_17 NPMS_HBS_MSD__2.0.xml Annex_18 NPMS_HBS_MSD__2.1.xml Annex_19NPMS_SILC_BG.docx Annex_20NPMS_SILC_EN.docx Annex_21

NPMS_SILC_MSD__2.0.xml Annex_22

NPMS_SILC_MSD__2.1.xml Annex_23

NPMS_guidelines.xlsx Annex_24STSALL+ESTAT+2.0.xml Annex_25


Deliverable nameBNSI_TOURISM_TABLES+BNSI+1.0.xml Annex_26BNSI_NFE_BNB+BNSI+1.1.xml Annex_27NA_MAIN+ESTAT+1.7.xml Annex_28EAA+BNSI+1.0.xml Annex_29SBS+BNSI+1.1.xml Annex_30Concepts_and_Codelists_in_SDMX_2.1.zip Annex_31ddb.zip Annex_32ddb2.1.zip Annex_33CATEGORY_SCHEMES+BNSI+1.0.xml Annex_34Prerformance and load testing.zip Annex_ 35ESMS_MSD+BNSI+2.0+SDMX.2.0.xml Annex_36ESQRS_MSD+BNSI+2.0+SDMX.2.0.xml Annex_37ESMS_MSD+BNSI+2.0+SDMX.2.1.xml Annex_38ESQRS_MSD+BNSI+2.0+SDMX.2.1.xml Annex_39Statistical Metadata Module 2.1.zip Annex_40NPMS_MSD+BNSI+1.0+SDMX.2.0.xml Annex_41NPMS_MSD+BNSI+1.0+SDMX.2.1.xml Annex_42VBNetApplication.7z Annex_43SPSSscripts.7z Annex_44Final Workshop.zip Annex_45

Galya STATEVAProject leader

Done in Sofia,November 2017

In duplicate in English


ai - circabc.europa.eu gr… · web viewas a deliverable the npms structure in bulgarian and...

Documents