coe assessment 2015 - europa

GOPA Consultants

Hindenburgring 18, 61348 Bad Homburg, Germany

Phone +49 6172 9818812 Fax: +49 6172 930-500 Email: [email protected]

Deliverable 2

Task 2 (Subtask 2a): Assessment of the three existing CoE

Project: ESSnet Assessment Quality, methodology and research Lot 1: Methodological support

Specific Contract N° ° 000044 ESTAT N°11111.2013.00 1-2015.647 under the Framework Contract ESTAT 11111.2013.001-2013.251 Lot 1

February 2016 Final

Document Service Data

Assessment of the three existing CoE i



Type of document Deliverable 2 – Task 2(Subtask 2.a)

Reference: Assessment of the three existing CoE, including a global assessment and improvement of the evaluation framework.

Issue: 18.01.2016 Revision: - Status: Final

Created by: GOPA Consultants Date: 05.02.2016

Distribution European Commission – Eurostat B1 GOPA Consultants

Contract full title ESSnet Assessment

Service contract number

Specific Contract N° ° 000044 ESTAT N°11111.2013.001-2015.647

under the Framework Contract ESTAT 11111.2013.001-2013.251 Lot 1

Document Change Record

Version Date Change

1. 20.01.2016 Eurostat comments received on 28.01.2016

2. 05.02.2016 -

Contents

Assessment of the three existing CoE ii

Contents

1. Introduction ...................................................................................................................... 1

1.1 Operational model for CoE.................................................................................................2

1.2 Criteria for assessment of CoE ...........................................................................................5

2. Overview ........................................................................................................................... 8

3. Summary and Conclusions ............................................................................................ 10

3.1 General Findings .............................................................................................................. 10

3.2 Conclusions for future actions .......................................................................................... 15

4. Current experience with the Centres of Excellence ..................................................... 17

4.1 Centre of Excellence on Data Warehousing (CoE-DWH) ................................................ 17

4.1.1 CoE description ................................................................................................................ 17

4.1.2 Assessment ....................................................................................................................... 20

4.1.3 Scoreboard ........................................................................................................................ 26

4.2 Seasonal Adjustment Centre of Excellence (SACE) ........................................................ 27

4.2.1 CoE description ................................................................................................................ 27

4.2.2 Assessment ....................................................................................................................... 29

4.2.3 Scoreboard ........................................................................................................................ 35

4.3 Centre of Excellence on Statistical Disclosure Control (CoE on SDC) ............................ 36

4.3.1 CoE description ................................................................................................................ 36

4.3.2 Assessment ....................................................................................................................... 38

4.3.3 Scoreboard ........................................................................................................................ 43

Figures

Assessment of the three existing CoE iii

Figures

Figure 1: Life cycle model for CoE ................................................................................................. 3

Figure 2: Communication between stakeholders involved in a CoE .............................................. 4

Figure 3: Active and evaluated CoE ................................................................................................ 8

Figure 4: Classification of the three CoE according to user groups ................................................. 8

Figure 5: Participation in the three CoE by countries ...................................................................... 9

Figure 6: Overview of the scoreboards for CoE ............................................................................ 14

Figure 7: CoE-DWH scoreboard ................................................................................................... 26

Figure 8: SACE scoreboard ........................................................................................................... 35

Figure 9: SDC scoreboard ............................................................................................................. 43

Acronyms

Assessment of the three existing CoE iv

Acronyms

CASC Computational Aspects of Statistical Confidentiality

CoE-DWH Centre of Excellence on Data Warehousing

CoE Centre(s) of Excellence

ESCB European System of Central Banks

ESS European Statistical System

ESSnet Collaborative ESS network

ESTP European Statistics Training Programme

EU European Union

FPA Framework Partnership Agreement

GSBPM Generic Statistical Business Process Model

ICT Information and Communication Technologies

KPI Key performance indicator

MBGA Multi-Beneficiary Grant Agreement

NSI National Statistical Institute

P-SDC Partners in SDC (Centre of Excellence on Statistical Disclosure Control)

SACC Seasonal Adjustment Centre of Competence (Seasonal Adjustment Centre of Excellence)

SACE Seasonal Adjustment Centre of Excellence

S-DWH Statistical Data Warehouse

SASG Seasonal Adjustment Steering Group

SAUG Seasonal Adjustment User Group

SGA Specific Grant Agreement

SDC Statistical Disclosure Control

Section 1. Introduction

Assessment of the three existing CoE 1

SECTION 1. Introduction

In the communication from the Commission to the European Parliament and the Council on the production method of EU statistics in 20091, a Vision for the next decade is outlined. Starting from the actual production practices, which are described as “augmented stovepipe model”, the actual challenges in the European Statistical System (ESS) business environment are analysed. Three challenges were identified as the most important ones:

• New requirements for statistics in terms of quantity and quality;

• Improvement of the regulatory environment for businesses and citizens in the area of statistics, by reducing response burden and priority setting in the field of community statistics;

• Improvements in efficiency and enhancement of statistical quality by using new Information and Communication Technologies (ICT) tools.

In response to these challenges, proposals for a future ESS business architecture were made. The following proposals are of special relevance:

• Development of a more integrated production system for statistics;

• Reuse of administrative data for statistical purposes;

• A new European systems method to statistics which supports horizontal and vertical integration.

Besides community legislation, the realisation of the proposal requires harmonisation by the use of common tools and methods within the ESS and by strategies for knowledge sharing within the ESS. Instruments for achieving these two goals are projects (ESSnets) and Centres of Excellence (CoE).

ESSnet projects should support the development of the infrastructure necessary for implementing the ESS Vision 2020. Projects can aim at common reference architecture, or at assisting Eurostat in the preparation of regulations, standards, and manuals, or target at the development of statistical methods. The formal structure of such projects is usually a grant agreement for an action with multiple beneficiaries, a Multi-Beneficiary Grant Agreement (MBGA).

1 Communication from the Commission to the European Parliament and the Council on the production method

of EU statistics: a vision for the next decade COM(2009) 404 final. (http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2009:0404:FIN:En:PDF )



The role of CoE is rather complementary to the ESSnet projects. This is emphasised in the explicit definition of the CoE in the ESS ESS Vision 20202 as follows:

Centres of Excellence will provide the institutional framework through which the results (e.g. methods, tools and good practices) are distributed and maintained across the ESS. Centres of Excellence as a form of collaboration refers to a team of persons or an entity that provides methodological expertise, support on IT solutions, best practices and/ or training in a focused area. The guiding principle should be to establish centres based on networks of National Statistical Institutes (NSIs) collaborating with each other to pursue excellence, in a particular area for the benefit of the whole ESS.

The emphasis on distribution and maintaining of statistical knowledge across the ESS assigns the CoE a specific role within the ESS: CoE are of utmost strategic importance for the development of the ESS and all partners within the ESS should benefit from the activities of the CoE. This implies that Eurostat is not only in the role of the financing authority of CoE but as part of the ESS, similar to NSIs, also a beneficiary from the activities and results of the CoE.

The fulfilment of this role needs some additional considerations about an operational model for a CoE. In the following sub-section 1.1 such an operational model is outlined. The model is a minor modification of the model discussed in the previous evaluation3. Based on this model the basic principles of the assessment are explained in sub-section 1.2.

1.1 Operational model for CoE

From the definition of the CoE in the ESS Vision 2020 it is evident that the responsibility of a CoE is not so much the production of knowledge (e.g. development of new methods and tools) but the transfer of already existing knowledge to the whole ESS. Examples of transferred knowledge are:

• A software product for a certain production step of the GSBPM, together with methodology and guidelines how to use it;

• A software repository for a certain production step of the GSBPM, together with methodology and guidelines how to use it;

• A conceptual model together with best practice examples which allows ESS members more cost-effective and standardized statistical production.

Using terms rather popular in business intelligence and business management one could call the main tasks of a CoE as “activities in deployment and dissemination”, i.e. moving a product from the development state to a desired state and make it available in the whole ESS.

2 http://epp.eurostat.ec.europa.eu/portal/page/portal/pgp_ess/0_DOCS/estat/ESS_Vision_2020.pdf 3 Proposal for a full set of evaluation criteria, concerning both ex-ante and ex-post evaluations, on Essnets and

Centres of Excellence, June 2015



A CoE is an appropriate structure for such activities if substantial efforts over a longer time period are necessary which are beyond the knowledge transfer of an ESSnet.

Usually deployment and dissemination requires a number of actions which have to be seen in the context of a life cycle model for CoE as sketched in Figure 1.

Figure 1: Life cycle model for CoE

Usually, a CoE will follow an ESSnet and will be responsible for making the results of former ESSnets available to the whole ESS. However, it should be noted that CoE are not limited to results of ESSnet projects. CoE can also promote the use of any new methodological and technical developments for statistics which are defined by the entire statistical lifecycle defined by the Generic Statistical Business Process Model (GSBPM). In the following the boxes of the life cycle model shown in Figure 1 are briefly explained.

Strategic plan

As shown in Figure 1 the strategic plan is a prerequisite for establishing a CoE. It includes three decisions about:

(1) Stakeholders;

(2) Readiness of the existing methodology for knowledge transfer in the whole ESS;

(3) Cost-benefit analysis.

Stakeholders:

Based on the structure of the Seasonal Adjustment Centre of Excellence (SACE) four groups of stakeholders were identified in the previous evaluation:

• Administration group: The CoE is administered by Eurostat and a responsible project officer is nominated.



• Steering group: Members of the steering group are experts in the subject area. The project officer is also member of the steering group.

• Developer group: Experts in the topic, which are either affiliated to the NSIs which cooperate in the CoE or external methodologists.

• Independent Testers: Practitioners at NSIs or other institutions, who use the results of the CoE in their work.

The communication between these groups is shown in Figure 2. The full arrows show the information flow between the partners, the dashed arrows show the feedback information. For example, the CoE starts a negotiation with the Developers for a methodological solution and the Developers respond to the requests.

Figure 2: Communication between stakeholders involved in a CoE

In ideal case the steering group is constituted in advance as part of the strategic plan.

Readiness

Regards to readiness different scenarios are possible. In some cases the envisaged solution is at hand, in other cases the strategic decision is to develop the existing knowledge further within the CoE.

Cost-benefit analysis

Cost-benefit analysis attempts to estimate the strengths and weaknesses of different options and puts the costs of a project in relation to the benefits.

Deployment and Dissemination

The main activities of the CoE during its life cycle are the following five steps:

(1) Application setup: undertake the necessary procedures for implementation of the existing methods within the whole ESS;

(2) Testing: checking the readiness by a selected user group (early adopters);



(3) Training: development of a training material and organisation of courses;

(4) Implementation: making the results available within the ESS, but – in some cases – also other national and international agencies interested in the development, for example, National Banks or the European Central Bank.

(5) Maintenance setup: Development and implementation of structures necessary for keeping the system running and providing user support, for example, a help desk. Other aspects of maintenance are necessary updates of the solutions.

The application setup is usually the first activity of the CoE. Depending on the readiness of the knowledge for implementation in the whole ESS this application setup can cover a broad range. In some cases it may be mainly an organisational issue, in other cases further developments are needed for improving the maturity of the solution.

The sequence of the other activities has to be defined according to the goals of a CoE. This sequence has to be defined in advance in the project description with milestones and deliverables. During the lifetime a redefinition of the work programme of the CoE is possible. The steering group in close cooperation with Eurostat are the stakeholders who are responsible for such a redefinition.

Maintenance setup

As shown in Figure 1 the CoE is responsible for the setup of a structure for maintenance. Typical examples are setting up a help desk or a user forum. The support over the whole lifetime of the solution in the ESS goes beyond the activities of the CoE and needs a special structure, for example a CoE-light which administers the help desk and, if necessary, adapts the solution.

Monitor, evaluate, redefine

Dissemination of a solution in the whole ESS is a many a time a rather complex process. Monitoring and evaluation of this process is of utmost importance. Such a monitoring has to be done at time points defined by the milestones for the CoE. If necessary, also a redefinition of the tasks is possible. In Figure 1 this project dependent monitoring of a CoE is shown by the different spacing between the arrows.

1.2 Criteria for assessment of CoE

The evaluation criteria are defined in a similar way as for ESSnets. As in the previous evaluation 2015, the evaluation criteria were defined according to the European Statistical Programme 2013-174 and the ESS Vision 2020. Based on these documents common

4 Regulation (EU) No 99/2013 of the European Parliament and the Council of 15 January 2013 on the

European Statistical Programme 2013-17.



evaluation criteria and indicators for ESSnet and CoE were developed and grouped into the following categories:

• Policy-related criteria;

• Compliance with the ESSnet principles;

• Technical characteristics of the project;

• Specific criteria for CoE;

• Additional aspects of the ex-post evaluation.

The additional aspects of the ex-post evaluation include three criteria which are of special relevance for CoE:

Involvement of NSIs not participating in the CoE

Provision of documentation for different user groups

Performance

For the first three topics, one can use the same criteria as in the case of the ESSnets with minor modifications for the sets of indicators. However, it should be noted that in the case of CoE the policy related criteria refer more to the implementation and not so much to the development of knowledge. A CoE will be established only in cases where knowledge is already available and the strategic decision is transfer to the whole ESS by a CoE. The evaluation of the activities of a CoE itself is based mainly on the criteria for compliance with ESSnet principles, the CoE specific criteria, and the ex-post evaluation criteria.

The CoE specific criteria reflect the following goals of CoE:

• Provision of methodological expertise for ESS members;

• Keeping methodological knowledge within the ESS up to date;

• Provision of common IT solutions for statistical production systems;

• Best practices and training courses for ESS members;

• Community building within the ESS;

• Appropriate institutional environment.

Because (at least usually) CoE have a longer life cycle than an ESSnet, a repeated monitoring and an interim evaluation is necessary which can lead to a redefinition of the activities of the CoE. This is shown in Figure 1 by the boxes at the bottom of the figure and the arrows to the time line. The evaluation criteria for these interim evaluations and the ex-post evaluation are defined similar to the criteria for the ex-post evaluation of ESSnets. For the interim evaluation of CoE the statement: up to the time of evaluation in the brackets apply.



• Did the project deliver the expected results (up to the time of evaluation)?

• Quality of reporting (up to the time of evaluation);

• Dissemination activities (up to the time of evaluation);

• Cost-effectiveness (up to the time of evaluation);

• Involvement of NSIs not participating in the CoE (up to the time of evaluation);

• Provision of documentation for different user groups (up to the time of evaluation);

• Performance (up to the time of evaluation).

All evaluations are based on the work accomplished up to now by the CoE and the information available at the CROS portal.

Section 2. Overview


SECTION 2. Overview

As shown in Figure 3 three Centres of Excellence (CoE) are active at the moment:

Figure 3: Active and evaluated CoE

Acronym Centres of Excellence

CoE-DWH Centre of Excellence on Data Warehousing

SACE Seasonal Adjustment Centre of Excellence

CoE on SDC Centre of Excellence on Statistical Disclosure Control

The work in the CoE on Data Warehousing was financed up to now by two MBGAs (10.2013 – 10. 2015) and aims at the promotion of a new business architecture. The Centre on Seasonal Adjustment is based on an MBGA for 24 month (4.2014 - 4.2016) and focuses on the deployment of a new software product as standard solution for seasonal adjustment. The Centre of Statistical Disclosure Control deals with an important aspect of data dissemination and is based on a FPA running for four years (2014 – 2018). A first grant agreement for public use files for Eurostat microdata was financed as SGA for 11 month in the year 2015.

The titles of the CoE used in the present report follow the ones displayed on the CROS portal (as of December 15, 2015), although sometimes CoE appear under different titles. This evaluation uses the acronyms of the assessed CoE from Figure 3.

These CoE are again – as in the previous round - classified according user groups as shown in Figure 4. Some CoE can be allocated to more than one user group.

Figure 4: Classification of the three CoE according to user groups

Practitioners

engaged in the

production process

Developers

engaged in the

implementation of

the Vision 2020

Decision makers

who need the CoE

results as a

strategic resource

1 CoE-DWH X X X

2 SACE X x

3 CoE on SDC X x

X Primary attribution

x Secondary attribution

Section 3. Summary and Conclusions


The CoE-DWH serves the needs of all user groups whereas the other two CoE are mainly oriented towards the needs of practitioners. The SACE project is in the last phase and has almost accomplished the envisaged goals. The CoE-DWH is in the middle of the deployment and dissemination process and needs further activities. In the case of the CoE on SDC a first promising step was done. The advantage of the SACE is that it had from the beginning a well-defined objective and an administrative structure (see Figure 2) which supported the activities of the CoE. In the other two cases the administrative structures are not so well defined. The goals are formulated more general and the realisation of selected goals is more dynamic.

An overview of the involved countries is shown in Figure 4. From this figure it can be derived that altogether 12 countries participated and five countries in two CoE. This shows that knowledge transfer in these three areas to all ESS partners is an important task.

Figure 5: Participation in the three CoE by countries

Country CoE -DWH SACE CoE on SDC

Austria X

Estonia X

Finland X X

France X**) X

Germany X

Hungary X

Italy X X

Luxembourg X

Netherlands X**) X**)

Portugal X

Slovenia X

United Kingdom X*) X

X partner

X**) coordinator

X*) associate partner



SECTION 3. Summary and Conclusions

3.1 General Findings

The assessment of the three Centres of Excellence (CoE) operating at the end of 2015 was based on the evaluation framework for ESSnets and CoE developed last year5. The main goal of the evaluation was to shed light on the generic role of CoE for the ESS, not so much on the contributions of the various projects themselves.

Contrary to ESSnets the main tasks of Centres of Excellence (CoE) are the knowledge transfer to the whole ESS. The existing CoE show that the terms ‘knowledge’ and ‘transfer’ can be understood in different ways:

• In the case of the Centre of Excellence on Data Warehousing (CoE-DWH) the ‘knowledge’ is a conceptual model for a statistical data warehouse (S-DWH), and the ‘ transfer’ is done by guidelines for S-DWH design, examples for best practice how to realize such a S-DWH in NSIs, and individual consulting services for NSIs.

• In the case of the Seasonal Adjustment Centre of Excellence (SACE) the ‘knowledge’ is a software product for seasonal adjustment, together with methodology and guidelines, which can be used by all NSIs, central banks and research institutes for accomplishing an important statistical production step. The term ‘transfer’ encompasses deployment of the software in the whole ESS and support for users of this software.

• In the case of the Centre of Excellence on Statistical Disclosure Control (CoE on SDC) the ‘knowledge’ is a set of standardized methods for statistical disclosure control and the ‘transfer’ is accomplished by a repository of algorithms for performing this production step.

Compliance with the policy related evaluation criteria for CoE

All three types of ‘knowledge transfer’ are of importance for achieving the policy goals formulated in the European Statistical Programme 2013-176. These goals are reflected in the policy-related criteria of the evaluation framework. All three CoE contribute to the goals “More flexibility in responding to user needs”, “ Efficiency gains in production”, and “Knowledge sharing in the ESS”.

All three CoE comply with the policy goal “Innovation”. Dissemination of a conceptual model for a statistical data warehouse by the CoE-DWH is innovative for the whole ESS. The

5 See the “Proposal for a full set of evaluation criteria, concerning both ex-ante and ex-post evaluations, on

ESSnets and Centres of Excellence”, June 2015 6 Regulation (EU) No 99/2013 of the European Parliament and the Council of 15 January 2013 on the

European Statistical Programme 2013-17.



most important innovations of the approach are the emphasis put on registers and on metadata. In the case of the SACE the most important innovative aspect is the standardization of a statistical production step by a new software product. The CoE on SDC works in close cooperation with the scientific community in this area and develops new and innovative algorithmic solutions for disclosure control. Future activities will make the solutions more accessible to all data producers.

All three CoE contribute to the three criteria “Cost-effectiveness”, “ Coherence and comparability”, and “Accessibility and clarity” defined in the Code of Practice7. It is quite obvious that the dissemination of a standardized solution for certain steps of the statistical production process is more cost effective than the development of solutions in all NSIs independently. In a similar way the cooperative approach of CoE supports “Coherence and comparability” and improves “Accessibility and clarity”.

With respect to the policy goal “Securing statistical confidentiality” there is naturally a strong contribution by the activities of the CoE on SDC, but for the CoE on DWH and the SACE this topic is outside their scope. Similarly, the contribution to the goal “Non-excessive burden on respondents” is marginal by the CoE-DWH and not applicable to the SACE and the CoE in SDC.

Compliance with the ESS Vision 2020

Looking at the performance of the CoE from the perspective of the ESS Vision 2020 one can find a high degree of compliance with a number of the objectives formulated in the five key areas. However, one must be aware that the objectives in the ESS Vision 2020 are formulated at a rather general level and the activities of the CoE deal with different aspects how these objectives can be met. For example, all three CoE contribute to the objectives: “Identify and implement standards for statistical production”, and “Benefits from our experts working together”. The CoE-DWH deals with an important aspect of the objective: “Adoption of enterprise architecture as a common reference framework”, the SACE contributes in a specific area to the objective: “Advance in sharing IT services and infrastructure”, and the activities of the CoE on SDC has made an important contribution to the objective: “Benefit from the exchange of (micro)data, while fully respecting statistical confidentiality”. These examples show that the activities of a CoE can be more cross-cutting issues for the different objectives of the ESS Vision 2020.

Compliance with ESSnet principles

As regards compliance with the ESSnet principles, the chosen methodological approaches of the CoE guarantee “Knowledge sharing within the whole ESS”. The criterion of “Sustainability” is met because one can expect in all three cases improvements for the ESS in the medium (or even longer) term. The general structure of a CoE seems to be appropriate for achieving the envisaged goals of knowledge transfer in a cost-effective way.

7 http://ec.europa.eu/eurostat/web/quality/european-statistics-code-of-practice



Evaluation of the performance of the CoE in 2015

The performance of the CoE can be evaluated from two different perspectives. The first one is the completion of the work defined by the financial agreements. Comparing the performance with the description of the actions in the financial agreements one can state that the performance of all three CoE is fully compliant with these descriptions.

More interesting is the evaluation of the performance of the CoE up to now, in particular in the period since the last evaluation round, with respect to the activities necessary for achieving the overall goal knowledge transfer, i.e. the desired impact for the whole ESS. These activities can be categorized in the following five steps:

• Application setup: undertake the necessary procedures for implementation of the existing methods within the whole ESS;

• Testing: checking the readiness by a selected user group (early adopters);

• Training: development of a training material and organisation of courses;

• Implementation: making the results available within the ESS, but – in some cases – also other national and international agencies interested in the development, for example, National Banks or the European Central Bank;

• Maintenance setup: Development and implementation of structures necessary for keeping the system running and providing user support, for example, a help desk. Other aspects of maintenance are necessary updates of the solutions.

As regards to these steps the three CoE are at different levels of progress:

Seasonal Adjustment Centre of Excellence The SACE is in a mature phase of knowledge transfer. Application setup, Testing, and Training, were almost completed and in the last period the main activities were implementation and setup of a maintenance structure. These activities have been done and it can be foreseen that in the last month of this CoE (until April 2016) the work will be completed. One advantage of the SACE is the fact that an administrative framework existed from the beginning. This framework proved to be useful for community building. Summing up one can state that the SACE has already achieved the overall goal of knowledge transfer and has created high impact for the whole ESS as well as for other institutions interested in seasonal adjustment.

Centre of Excellence on Data Warehousing (CoE-DWH) The CoE-DWH is in the middle of the envisaged activities. In the last period the application setup was completed and the conceptual model seems to be fit for dissemination in the whole ESS. Testing of the conceptual model for a S-DWH has been done by members of the CoE, and a number of NSIs outside the CoE started with independent testing. For training purpose, work on a S-DWH Design Manual as a “living document” has started in the last period. Successful community building was also an activity in the last period and a help desk for maintenance was installed. Regards to the institutional framework it seems to be a drawback that there is only an informal steering group and the community building in this area is rather new for the ESS. Summing up one can say that there is a high degree of fulfilment of the



evaluation criteria and it can be envisaged that appropriate planned further work will increase the already existing impact on knowledge transfer in the whole ESS. Topics for future activities could be the organisation of courses in S-DWH within the European statistical training programme (ESTP), enlargements of the metadata knowledge repository in the direction of a semantic web for official statistics, or tighter coupling of more production steps of the GSBPM with the S-DWH.

Centre of Excellence on Statistical Disclosure Control (CoE on SDC) The CoE on SDC is in the initial phase and there was up to now only one 11 month SGA for specific work. The expertise of the CoE builds on a long lasting cooperation of experts in the field of SDC. Due to the many facets of disclosure control it is quite obvious that this CoE is in the application setup as the first step of a CoE. In the last period for this application setup new algorithms for the production of public use files (PUF) were developed and tested, and comparison with already existing methods was done. Interesting new points are the systematic assessment of the disclosure risk and assessment of the utility of public use files. Also in this case there is only an informal administrative framework for the CoE; a more formal steering group could be useful. In summary one can say that the work of the CoE on SDC is up to now in line with the specific evaluation criteria for CoE. For further work of this CoE on SDC the setting of priorities seems necessary. The framework partnership agreement mentions for example the development of training material and the setup of a software repository for disclosure control. Such activities could have high impact on knowledge transfer in the area of statistical disclosure control.

Scoreboard Figure 1 provides an overview of the evaluation of the three CoE according to the evaluation criteria for CoE. Remarks made in the individual scoreboards for each CoE are not included. A scoreboard will always be based on simplifications and generalisations and cannot replace a sound assessment in detail. This statement is in particular valid for the following Figure, because, in this overview, all the additional remarks given in the individual scoreboards had to be omitted. Furthermore, note that the evaluation with respect to the aspects of the ex-post evaluation considers only the state of the art of the activities up to now.



Figure 6: Overview of the scoreboards for CoE

CoE-DWH SACE CoE on SDC

Criteria Score Score Score

1. Policy-related criteria

More flexibility in responding to new user needs

Efficiency gains in production

Knowledge sharing in the ESS

Innovation

Securing statistical confidentiality

Non-excessive burden on respondents

Cost-effectiveness

Coherence and comparability

Accessibility and clarity

2. Compliance with ESSnet principles

Involvement of several ESSnet partners and knowledge sharing

Sustainability

Cost-effectiveness, high impact

3. Technical characteristics of the project

Quality of the project proposal

4. Cluster-specific criteria

Specific criteria for CoEs

Provision of methodological expertise

Keeping methodological knowledge within the ESS up-to-date

Provision of common IT solutions for statistical production systems

Best practice and training courses

Community building within the ESS

Approriate institutional environment

5. Additional aspect of the ex-post evaluation

Did the projects deliver the expected results?

Quality of reporting

Dissemination activities

Cost-effectiveness



Performance

Score:

Policy and cluster related criteria

Major contribution

Contribution

Compliance with ESSnet principles and ex-post evaluation criteria

Not met

Partly met

Fully met



3.2 Conclusions for future actions

CoE are a useful structure for knowledge transfer in the whole ESS. Setting up of a CoE is advisable in cases where for this transfer substantial efforts over a longer time are necessary which are beyond the knowledge transfer of an ESSnet. Usually such a CoE will contribute to more than one objective of the ESS Vision 2020 and can lead in this way to more efficient implementation of the ESS Vision 2020.

A requirement for successful operation is a strategic plan (overall goal, roadmap). This plan has to be defined in more detail for a time horizon of about 2 – 4 years and encompass the governance of the CoE. The existence and efficient operation of a steering group which monitors the performance of the CoE is of utmost importance for the success of a CoE. This instrument would support more adaptive long range planning. More active community building in the form of user groups is desirable. The roadmap has to include the five main activities necessary for successful knowledge transfer: application setup, testing, training, implementation, and maintenance setup.

The application setup depends on the readiness of the solution and can include further methodological development. Regards to testing independent testers (early adopters) have to be defined. For training a coordination of the activities of the CoE and the ESTP is desirable. (One way could be that the CoE is responsible for the training material and the training itself is organised within the ESTP.) For implementation the required form of output (for example, standardized software, repositories of algorithm for specific methods, or documents for different user groups) has to be specified. An appropriate maintenance structure (e.g., help desk, consultation forum) must be defined.

The technical characteristics used for the ex-ante evaluation of a CoE have to refer to the specific strategic plan. The strategic plan has to include precise targets and milestone that have to be accomplished within certain periods.

The evaluation of the performance of the CoE has to take the five main activities into account. The dates for interim evaluations should already be laid down in the strategic plan.

With respect to maintenance, a CoE can do only the first step. For achieving sustainability in the longer run, a new institutional framework (low budget “CoE - light”) for user support and keeping important results up-to-date should be considered.

Pros and cons of different financial agreements for CoE

Because CoE have to provide the institutional framework through which the results (e.g. methods, tools and good practices) are distributed and maintained across the ESS, different types of financial agreements can be useful.

Long term financial agreements

Long term agreements up to four years are useful in the case of dissemination of a complex statistical objective, for example, a data warehouse solution for the ESS. Success depends on



a well-defined strategic plan, an appropriate administrative infrastructure (e.g. steering group, independent testers), and a well-defined monitoring regime for the achievements of the CoE. This strategic plan must define milestones which can be evaluated in interim evaluations. Besides qualitative performance criteria also quantitative measureable criteria formulated as key performance indicators (KPIs) are necessary. Examples of KPIs are the number of implementations in different NSIs, the number of support activities, or the number of downloads from a software repository.

The risks of long term agreements are the non-optimal use of scarce resources, the loss in flexibility for dealing with new emerging issues, and legal and administrative problems in ‘downgrading’ a CoE if the expected results cannot be achieved.

Medium term financial agreements

Medium term financial agreements of up to two years are in particular recommendable for the dissemination of already well-developed products, and (very relevant) if a number of NSIs are already using this method or tool (such as DEMETRA + in the case of seasonal adjustment). The main strategic goal is the implementation of the solution in all or almost all Member States. Also in this case the setup of an administrative structure is necessary and a monitoring strategy with respect to measureable performance criteria has to be defined.

The pros of such an approach are the lower risk of binding resources for a long time.

The danger of medium term financial agreements is that that an ad-hoc solution which is not long-lasting might be chosen. Hence, the criterion sustainability is not achieved. run.

Short term financial agreements

Short term agreements are useful if implementable solutions for the objectives are in a less mature stage and the precise formulation of a comprehensive strategic plan for dissemination is at the moment infeasible. In such cases a FPA is a possible solution.

The pros of such an observant approach are high flexibility and low financial risks.

The disadvantage of short term financial agreements is that projects with a minor contribution to the overall strategic goal are carried out.

CoE –light short term financial agreements

Such agreements are useful after the successful dissemination phase for maintenance of a solution. It offers at a low budget the possibility for user support and for minor improvements of the solutions for the whole ESS.

To establish a CoE light is highly recommendable in any case for long term support of implemented solutions.

Section 4. Current experience with the Centres of Excellence


SECTION 4. Current experience with the

Centres of Excellence

The following assessments of the three CoE are all structured in the same way. The short project description is followed by the assessment itself. The CoE description is based on the information available on the CROS portal as of December 2015.

The costs mentioned in each of the project descriptions are based on the information provided on the CROS portal; these are the costs as foreseen in the grant agreement. Only orders of magnitude are provided also because in the case of many projects the real costs were considerably lower.

As mentioned before, the assessment is based on an extended version of the framework for assessing ESSnets and CoE8 and always makes use of the sets of indicators selected for each of the criteria in the framework.

4.1 Centre of Excellence on Data Warehousing (CoE-DWH)

4.1.1 CoE description

General

Acronym: COE-DWH9

Project: Centre of Excellence on Data Warehousing

Contract type: MBGA, 2 contracts

Project type: Centre of Excellence

Project start date: 04.10.2013 Project end date: 4.10.2015

Duration: 24 months

Project status: Part 1 finished, Part 2 finished

Total eligible costs: 100 000 € + 100 000 €

Total funding: 100 000 € + 100 000 €

Website: http://www.cros-portal.eu/content/centre-excellence-data-warehousing

8 See the “Proposal for a full set of evaluation criteria, concerning both ex-ante and ex-post evaluations, on

ESSnets and Centres of Excellence”, June 2015. 9 We use the new acronym “Centre of Excellence on DWH” as on the new CROS portal. Documents of the

first phase are still under the name “Centre of Competence on Data Warehouse” with the acronym COC-DWH.



Objectives

Main goal of the Centre of Excellence on Data Warehousing (CoE-DWH) is to continue the work of the ESSnets on Data Warehousing in order to ensure sustainability of the work, the results, and the acquired expertise of the project after its completion.

The CoE-DWH started in October 2013 as virtual body that enables to manage and actualize the expertise and knowledge acquired during the project in an active & dynamic manner. The CoE relies on the group of NSI’s, which were already partners in the ESSnets on DWH and one NSI that was closely and actively working with the ESSnet.

Based upon the needs of NSIs as inventoried by the ESSnet, the main tasks of the Centre are:

• Support and facilitate projects of member states implementing ESSnet results; • Ad-hoc support, consultancy and/or expert reports, on request of ESS member states; • Maintain the knowledge and expertise repository on the CROS portal, including the

overview of best practices; • Keeping the handbook on S-DWH up-to-date and include it in the knowledge

repository; • Active dissemination of results, knowledge and new developments.

In the first phase, the following activities were carried out:

• Provision of an updated handbook guiding users to the various aspects of creating a S-DWH and incorporating the practical experiences from supporting and advising ESS members;

• An inventory of tools and solutions developed for a S-DWH which could be used for exchange and sharing knowledge in this area;

• An integrated glossary as starting point for a knowledge base.

The second phase of the CoE-DWH started in October 2014. The following activities were envisaged:

• Implementation of a helpdesk structure: ESS members can express their needs and ask questions. Furthermore, the helpdesk should be combined with a communication channel on the CROS portal where the interested parties can consult and comment the development of the action.

• Further elaboration and maintenance of the ESS-wide network of specialists in the field of S-DWH, as a pool of experts that can be consulted by the CoE-DWH.

• Identification and prioritisation of relevant projects within the ESS and, in addition, actively identification of the needs for support expressed by ESS members.

• Keeping the Handbook on setting up an S-DWH up to date. The Handbook can be considered as knowledge repository. These updates include:

− New results of theoretical research and studies;

− Incorporation of all practical experiences from supporting and advising ESS members.



During both phases the following actions and activities regarding dissemination towards the ESS were carried out:

• Organisation of two workshops to report on the activities of the CoE-DWH with two main goals:

− Dissemination and exchange of results, knowledge and best practices from recent developments in data warehouse projects in different NSIs,

− Sharing experiences on the general aspects of running a CoE-DWH in order to further evolve the CoE-DWH concept;

• Promotion of the results of the CoE–DWH at other fora;

• Visiting of NSIs on site.

Participants

Name Country Function

Centraal Bureau voor de Statistiek (CBS)

Netherlands Coordinator

Statistics Estonia (SE) Estonia Co-partner

Statistics Finland (STATFI) Finland Co-partner

Istituto Nazionale di Statistica (ISTAT)

Italy Co-partner

Statistics Lithuania (SL) Lithuania Co-partner

Instituto National de Estatistica (INE)

Portugal Co-partner

Office for National Statistics (ONS) United Kingdom Associate

Reports available via CROS portal:

The main documents are:

• S-DWH - Handbook: http://www.cros-portal.eu/content/s-dwh-handbook;

• S-DWH Design Manual: http://ec.europa.eu/eurostat/cros/content/s-dwh-design-manual_en ;

• S-DWH Glossary: http://www.cros-portal.eu/content/s-dwh-glossary;

• S-DWH Best Practice Cases: http://www.cros-portal.eu/content/best-practice-cases-0;

• Data Warehousing Help Desk: http://ec.europa.eu/eurostat/cros/content/centre-excellence-help-desk_en ;

• Operational activities:

o http://ec.europa.eu/eurostat/cros/content/operational-activities-deliverables-2013-2014_en ;

o http://ec.europa.eu/eurostat/cros/content/deliverables-2014-2015-1_en .



4.1.2 Assessment

As stated in the introduction of this document fulfilment of at least one of the policy related criteria are a basic prerequisite for establishment of a CoE. For CoE knowledge transfer with respect to the policy goals is the main task. In the introduction of the evaluation five important activities were identified for such a knowledge transfer: application setup (procedures for implementation), testing, training, implementation, and maintenance setup. The evaluation of the policy-related criteria refers mainly to the contribution to these activities. Evaluation of the activities is included in the parts dealing with the criteria for compliance with ESSnet principles, the CoE specific criteria and the ex-post evaluation criteria. Because the CoE are not finished the ex-post evaluation always should be understood as an interim evaluation of the work up to now.

Policy-related criteria

More flexibility in responding to new user needs:

As pointed out in the ESS Vision 2020 the implementation of S-DWH is crucial for arriving at higher flexibility and for becoming more responsive to users’ needs.

One of the goals of this CoE is dissemination of the conceptual framework for a statistical data warehouse (S-DWH) in the ESS. The proposed IT infrastructure promoted by the CoE improves interoperability by sharing data and metadata, and a common data model. Hence this CoE is fully compliant with this policy goal.

Efficiency gains in production:

One central idea of the proposed S-DWH is putting business registers in the centre of the warehouse. This model supports the use of administrative records for the statistical production process. The overall architecture takes a holistic perspective at the statistical business process and is in agreement with the GSBPM and the statistical information model. Hence, success of the CoE in transferring this knowledge to other ESS members will lead to significant efficiency gains in the medium and long term perspective.

Knowledge sharing in the ESS:

This CoE can contribute significantly to knowledge sharing. The project uses as input the state of the art in S-DWH in member states and the results will facilitate the implementation of common solutions in the ESS. Various perspectives of metadata like functionality, quality and governance of metadata are part of the envisaged solution and this knowledge will help in the process of standardisation of metadata for the ESS.

Innovation:

The concept of a statistical data warehouse is still new to many NSIs and the dissemination of the S-DWH will lead to innovation in the ESS in a number of ways:

• Consequent application of GSBPM in the statistical production process;



• More focus on the quality of metadata and application of criteria for the assessment of metadata quality;

• Promotion of the importance of registers as a backbone for all kind of statistical activities.

Securing statistical confidentiality:

This is not an important goal within this CoE.


The S-DWH supports data sharing for a wide range of statistical activities and helps in this way to reduce response burden. Hence, knowledge transfer in the area of S-DWH will help in achieving progress in this policy goal.

Cost-effectiveness

Investment in the S-DWH improves the statistical production in many ways and the promotion of this structure will contribute to reduced costs in data production in NSIs.

Coherence and Comparability:

The S-DWH promotes coherence and comparability in data production. In particular, it guarantees that standard concepts, definitions and classifications are consistently applied throughout the system. The dissemination of the S-DWH structure will lead to more coherence and comparability in the NSIs.

Accessibility and clarity:

The accessibility of data will be improved by the implementation of the S-DWH. Available data can be accessed in a more flexible way and the detailed treatment of metadata supports proper interpretation and meaningful comparisons. Consequently the proposed structure, which is along the standard of nowadays IT-infrastructure for data dissemination, will help NSIs to meet this policy goal.


Involvement of several ESSnet partners and knowledge sharing:

Compliance with this central criterion is given: There is a high involvement of the partners in the CoE. Besides the involvement of the partners, the project has also contacts to other NSIs and other national statistical agencies.

Sustainability:

The underlying structure of the S-DWH is at the conceptual level. It can be adapted to the needs in different NSIs and offers in that sense a sustainable solution which can be used for a longer period. Overall the CoE has made good progress but follow-up activities and further resources are

necessary. These further activities are necessary for broadening of the scope to other statistical



domains, for meeting new data challenges (e.g. big data), and for possible adaptation to new technical challenges (e.g. S-DWH and Statistics knowledge web).

Cost-effectiveness, high impact:

A S-DWH helps in the avoidance of duplication of activities within the ESS. It helps to overcome the stove-pipe structure in statistical production and improves many production steps of the GSBPM. The CoE is a very good operational approach to achieve the planned goals in the area of

data warehousing. Due to the importance of this topic this CoE has high impact for the ESS.

CoE-specific criteria

Provision of methodological expertise:

A helpdesk has been set up for the support of users and a glossary as useful reference for all activities in the S-DWH has been published. The help desk is organized quite well according to the main topics of the S-DWH and the structure of the information available at the CROS portal. Besides the helpdesk the topic data warehouse requires probably additional structures for communication. For example, a blog could be useful for discussion of certain aspects. Another important topic is providing more consultation. The planned site visits are an excellent idea and more resources should be devoted to such activities.

Keeping methodological knowledge within the ESS up-to-date:

The handbook for the S-DWH developed in the projects on data warehousing has been updated and enlarged. The S-DWH design manual as a “living document” is very appropriate for keeping the knowledge up-to date. However, as the name “living document” indicates these activities are by no means finished. For example, the last chapter 4: Methodology shows a high potential for further improvement. (http://ec.europa.eu/eurostat/cros/content/s-dwh-design-manual_en)


Establishing a S-DWH is an ongoing effort for all NSIs. It should be based on the common principles which support a more unified method of statistical production according to the principles of the GSBPM. The S-DWH is more a conceptual approach towards data warehousing, which can be adapted by the NSIs according to their infrastructure and resources. Hence, the CoE fulfils exactly the indicator: Conceptual frameworks for statistical production of this criterion.

Best practice and training courses:

The examples of best practice available from the CROS portal show how the conceptual model can be realized. The templates designed for description of best practice seem to be the right approach for promotion of the ideas of the S-DWH which can be used by other NSIs. Because of limited resources a number of planned training materials were not realized up to



now. For example, the visualization of the ideas which was postponed could be an interesting new approach for communication of the ideas. A further activity of the CoE could be the development of information material for different groups. For example, the living documents could be enlarged by examples from different statistical domains. Probably one should consider the opportunity of training courses within the ESTP.

Community building within the ESS:

For community building in this area, it is important that the project partners have a good overview about the activities in data warehousing in the different NSIs. A first step in this direction done by the CoE was accomplished by a survey. Up to now two workshops were organized which helped in community building. Such activities should continue.

Appropriate institutional framework:

The CoE was successful in encouraging other NSIs to cooperate with the CoE and to realize the ideas. The work of the CoE has also supported partnerships within the ESS.

The steering group for this CoE seems to be more an informal group. Due to the importance of data warehousing for statistical production, Eurostat should think about a more structured approach for defining priorities for the activities of the CoE and appropriate resources should be provided. This idea was already expressed in a document of the CoE after the first year (http://ec.europa.eu/eurostat/cros/content/coc-dwh-wp1-17-report-effective-organisation-coc_en).

Additional aspects of the ex-post evaluation

As mentioned already at the beginning this is in fact an interim evaluation of the work up to now.

Did the project deliver the expected results?

Up to now the project seems to be in line with the expected and promised results. Due to lack of resources and also organizational problems it was not possible to complete a number of planned activities such as site visits, or provision of additional training material.

Quality of reporting:

The reports of the first year of the CoE are of high quality. The reports of the second year are not completely finished, in particular the S-DWH design manual as a living document needs some further editing. A nice thing is that the connection between the S-DWH and the GSBPM is now made explicit. Most of the reports are written in a non-technical manner and should be useful for a broad audience. Examples from other statistical domains would be useful.



Dissemination activities:

The reports on the CROS portal give a good overview of the activities of the CoE. The organization of the documents on the CROS portal is rather transparent. The helpdesk is also well structured but needs probably additional features (see criterion: Provision of methodological expertise). In part 4 of the design manual more links to other methodological projects would be nice. For example reference to the Memobust handbook.

Cost-effectiveness:

Comparing the costs for the project with the results of the promotion of a S-DWH one can say that the project is cost-effective. The structure proposed for the S-DWH can be implemented in other NSIs. It helps to reduce the development costs for a S-DWH.

Involvement of NSIs not participating in the CoE:

Other NSIs are involved in many ways with this CoE: A survey about tools and methods used in the NSIs was carried out, workshops were organized, and a helpdesk was established. The CoE was successful in establishing a more active cooperation with the German statistical office (Destatis), Deutsche Bundesbank, Statistics Poland, CSO Ireland, and Montenegro (MONTSTAT). These contacts should be continued.

Provision of documentation for different user groups:

Establishing a S-DWH is an activity which involves different user groups inside NSIs, for example persons responsible for registers, persons responsible for the production and dissemination of statistics, and persons responsible for IT infrastructure. Hence, it is of utmost importance to build networks of experts from different areas which cooperate. This networking is just at the beginning. The promotion of an appropriate information network should be supported more directly by an institutional framework. Within such networks information needs should be defined by the steering group and the idea of documentation for different user groups should be discussed. But this is definitely a challenge beyond the CoE.

Performance:

Coaching activities for other NSIs have been started and should continue. Compared to the available resources it is quite impressive that contacts with 5 institutions were established. There were not so many questions at the helpdesk (altogether 3) but this has to be seen in context of the nature of the problem. The help desk is more a first contact point for a NSI and further cooperation must be done by other ways of communication. Probably Eurostat could advertise the activities of the CoE more prominently on the CROS portal and encourage discussions about several important topics in statistical data warehousing. But one has to be aware that such an activity needs additional resources.

With respect to performance in the five main activities of a CoE one can conclude that the activities for application setup are completed. Testing was done by several NSIs and the



involvement of other NSIs will produce more testing results. Training is at the beginning and should be increased. With respect to implementation, the CoE is on a good track but a more strategic approach from Eurostat is desirable. Strategies for maintenance and possible further development should be discussed in a steering group.



4.1.3 Scoreboard

Figure 7: CoE-DWH scoreboard

Criteria Score Remark


More flexibility in responding to new user needs

Efficiency gains in production

Knowledge sharing in the ESS

Innovation

Securing statistical confidentiality not applicable


Cost-effectiveness

Coherence and comparability

Accessibility and clarity



Sustainability



Quality of the project proposal not part of this evaluation











Quality of reporting The reports of the last year need some revision

Dissemination activities

Cost-effectiveness


Provision of documentation for different user groups Strategic plan for information dissemination necessary

Performance Resources for more active promotion necessary

Score:


Major contribution

Contribution


Not met

Partly met

Fully met



4.2 Seasonal Adjustment Centre of Excellence (SACE)10


General

Acronym: SACE

Project: Seasonal Adjustment Centre of Excellence

Contract type: MBGA/



Duration: 24 months

Project status: Ongoing

Total eligible costs: 500 000 €

Total funding: 450 000 €

Website: http://ec.europa.eu/eurostat/cros/content/seasonal-adjustment-centre-excellence-0_en

Objectives

Seasonal adjustment (SA) is an important process step of the official statistics business architecture. Since the 1990s Eurostat has been playing a leading role in the harmonisation of practices in the European Statistical System (ESS).

The general purpose of this seasonal adjustment centre of excellence (SACE) is to provide assistance and services to the ESS and to the European System of Central Banks (ESCB) in seasonal adjustment activities. More specifically the CoE aims to support and assist ESS and ESCB members in implementing JDemetra+ and thus to ensure a successful release of the tool as a reliable and user friendly production instrument for official statistics. The software JDemetra+ was developed as an open source tool for seasonal adjustment.

The contract for the CoE was signed in April 2014 under the name Centre of Competence on Seasonal Adjustment (SACC) and runs for twenty-four months. According to the project contract the following goals are pursued:

• Support and assisting ESS and ESCB members in implementing JDemetra+, in order to ensure a successful release of the tool, as a reliable and user-friendly production instrument for official statistics;

10 In agreement with the new CROS portal the name Seasonal adjustment centre of excellence (SACE) is used

in this evaluation. In a number of documents the former name Seasonal adjustment centre of competence (SACC) occurs.



• Facilitating the migration to the new software by ensuring that JDemetra+ is fit to enter into production, through testing, and provision of usability features such as high quality documentation;

• Addressing missing functionalities, and elaborating a sustainable model in the maintenance and further development of JDemetra+ over the next decade;

• Improving the documentation;

• Providing help and training on the use of JDemetra+, and on the implementation of the ESS guidelines for seasonal adjustment;

• Sharing the technical knowledge on JDemetra+ to facilitate the development of new modules or plug-ins and testing these new developments.

Participants


Institut National de la Statistique et des Etudes Economiques (INSEE)

France Coordinator

ISTAT Italy Partner

ONS United Kingdom Partner

lnstitut National de la Statistique et des Etudes Economiques (STATEC)

Luxembourg Partner

The CoE has to be seen in context with the activities in the area of seasonal adjustment where the following groups of actors are involved:

• The Seasonal Adjustment Steering Group (SASG);

• The Seasonal Adjustment User Group (SAUG);

• The JDemetra+ Developers.


There is a lot of material available on the new CROS portal. The entry point for non- registered users is: http://ec.europa.eu/eurostat/cros/content/documentation_en .From this Link one can access a number of documents. Two important sources are:

• Guidelines for Seasonal Adjustment 2015 (Direct Link: http://ec.europa.eu/eurostat/documents/3859598/6830795/KS-GQ-15-001-EN-N.pdf/d8f1e5f5-251b-4a69-93e3-079031b74bd3) ;

• JDemetra+ seasonal adjustment software (download and documentation): http://ec.europa.eu/eurostat/cros/content/jdemetra-seasonal-adjustment-software_en .

The help desk: http://ec.europa.eu/eurostat/cros/content/ess-seasonal-adjustment-helpdesk_en is accessible only for registered user. As a registered user one can obtain detailed information



about the CoE and the associated groups in a well-organized form. For almost all topics zip-files are available:

• Coordination meetings: http://ec.europa.eu/eurostat/cros/content/coordination-meetings-0_en ;

• JDemetra+ testing: http://ec.europa.eu/eurostat/cros/content/jdemetra-testing_en ;

• Monitoring reports: http://ec.europa.eu/eurostat/cros/content/reports-1_en ;

• Seasonal adjustment user group: http://ec.europa.eu/eurostat/cros/content/seasonal-adjustment-user-group_en ;

• Training for users: http://ec.europa.eu/eurostat/cros/content/training-users_en .

4.2.2 Assessment




The SACE promotes a business architecture which allows more integrated production of seasonal adjustment. The JDemetra+ software improves accuracy and reliability of seasonal adjustment calculations.


The solution for seasonal adjustment promoted by the SACE allows more integrated production of European statistics. It implements a flexible IT reference infrastructure and defines technical standards for improving interoperability. The usage of JDemetra+ is one important step towards standard IT tools across statistical business processes.


This criterion is met to a high degree. The SACE aims for the development of instruments for facilitated exchange of best-practices and implementation of common solutions in the ESS.



Furthermore, a training programme in this area is provided. The installed help desk is another indicator for successful knowledge sharing. The associated user groups support the collaboration of experts.

Innovation:

With respect to innovation a number of important indicators are applicable for this CoE. The work of the SACE contributes to the standardization of certain production steps of the GSBPM. The close cooperation with the scientific community in time series analysis helps improving the methodology and the SACE is effective in the promotion of better tools.


This policy goal is not applicable for the topics treated within the SACE.


This policy goal is not applicable for the topics treated within the SACE.

Cost-effectiveness

The promotion and implementation of a standardised solution in seasonal adjustment increases effectiveness and efficiency.


The work of this CoE is in agreement with a number of indicators of this policy goal: The promotion of the JDemetra+ software is an important step in vertical consistency in seasonal adjustment across ESS partners; The SACE implements methodological standards in order to increase the use of harmonized methodologies; It promotes the production of comparable statistical outputs for seasonal adjustment. Furthermore, a business architecture which guarantees the application of standard concepts is supported.


The promotion of the JDemetra+ software is an important step for making the tool accessible within the ESS. The outputs of this software help in the dissemination and visualization of seasonal adjusted data.


A main innovative approach of this CoE is the promotion of a well-tested software product for all ESS members and for the ESCB members. The ideas behind establishing the CoE are in full

agreement with the criteria for compliance with the ESSnet principles. In particular, the chosen structure seems to be the best operational approach for achieving the results.




The contribution to this goal is obviously met to a high degree. The internal structure of the SACE uses the expertise of certain Member States for the benefit of the whole ESS. It responds to the needs of ESS partners in the area of seasonal adjustment. The training material offered by the SACE supports excellence among the partners involved in the CoE. The cooperation with the central banks promotes alliances and partnerships with data owners.

Sustainability:

There is a high probability that the activities of this CoE are useful for a longer time period. After the end of the work of the SACE a strategic decision is necessary to what extent follow up

activities are necessary. In particular decisions about the maintenance of the help desk, improvements of the JDemetra+ software, and organisation of training courses will become necessary. Training courses could be organized within the framework of the European Statistics

Training Programme (ESTP).


The SACE helps to avoid duplication of activities within the ESS. It improves the production

steps in the GSBPM relevant for seasonal adjustment and the CoE is the best operational approach to achieve the planned goals in this area. Due to the importance of this topic this CoE has high impact for the ESS.



The CoE has set up a help desk and operates this help successfully. Seminars and workshops are organized and training material is made available. Hence this criterion is met to a high degree.

Keeping methodological knowledge within the ESS up-to-data:

The CoE operates in close connection with the developers in the area of seasonal adjustment which guarantees that the knowledge is up-to-date. Documents and knowledge is updated according to the needs of the seasonal adjustment user group (SAUG).


The set-up of the CoE followed an implementation plan for the JDemetra+ software which is in agreement with the general practice in the field of software deployment. New developments are tested within the community. An excellent manual for JDemetra+ is available and the start for beginners is facilitated.




Inside the seasonal adjustment community, a variety of training material is available, for example, an online tutorial. The CoE uses this material and improves it continuously.


As already stated in the description of the CoE, a community in the area of seasonal adjustment exists. This community has a well-organised structure. There are regularly meetings of the SAUG which provide feedback about utilization of the software. All members of the group are notified about the activities of the CoE.


The advantage of this CoE is that an appropriate institutional framework existed from the beginning. There is a steering group, a user group and a developer group which supports the activities of the SACE. There are regularly meetings of these groups which define objectives for new development activities. The SACE supports by its activities the partnership in seasonal adjustment within the ESS.

Aspects of the ex-post evaluation



The work of the SACE is fully compliant with the objectives of this CoE. All work is based on the standard terminology in the area of seasonal adjustment.


The reports are of high quality. All manuals and the training material are well organized. Implementation of JDemetra+ can be done easily from the CROS portal.


There have been a lot of dissemination activities by the CoE. These activities are well documented by a number of reports on the CROS portal:

• Coordination meetings

• JDemetra+ documentation: a very detailed and reader-friendly handbook which explains the use of JDemetra+ step by step;

• JDemetra+ testing: a number of benchmark data sets have been selected and a detailed testing plan has been specified which compares JDemetra+ with other software (SEAT, TRAMO and X13);



• Monitoring reports: the monitoring reports document quite well the results obtained up to now;

• Seasonal Adjustment User Group: the minutes of the user group meetings are available.

A weak point is probably that the actual version of the CROS portal splits the information between group members and other visitors of the pages of the SACE. For example, the material available on http://ec.europa.eu/eurostat/cros/content/jdemetra-reference-manual_en (downloading, manuals, …) should be incorporated in the member section. Conversely the help desk should be available for all visitors of the CROS portal. But these are more organisational issues.

Cost-effectiveness:

Development of one standardized solution for an important step in data production is definitely more cost-effective than to develop separate solutions in the NSIs and central banks. Using the documentation the solution can be implemented with low costs in different places.


The administrative infrastructure guarantees that not participating NSIs and other institutions are involved in the activities of the CoE. These meetings of the SAUG were attended by representatives from different countries, the ECB, the Organisation for Economic Co-operation and Development, and Eurostat.


The users of the activities of the SACE are persons interested in seasonal adjustment. For those who are beginners, the guidelines for seasonal adjustment are probably the right document. For those who want to use JDemetra+ the Quick start is a good approach. For advanced users the manual is an excellent reference.

Performance:

The performance of the SACE is quite excellent. There are regular training courses and the helpdesk is operated successfully. The following table shows the activities on the help desk:

# of Posts

Other Cases

# of Questions

# of Real Questions

Institutes % Solved Average Duration

101 3 104 80 38 100.00 5.37

Most questions were answered quite quickly, the same day or the day after. In most cases, the requester did not provide feedback on the given answer and the case was closed after a few days. On average, it takes about 5 days to solve a case.



With respect to the five main activities identified for a CoE one can summarize the performance as follows: Application setup (procedures for implementation) has been done completely; there was intensive testing of the software; training material and courses were offered and this work continues; dissemination works quite well due to the SAUG; maintenance activities are done. After the end of the SACE one has to discuss the responsibility for future maintenance.



4.2.3 Scoreboard

Figure 8: SACE scoreboard



More flexibility in responding to new user needs in the case of seasonal adjustment

Efficiency gains in production in the case of seasonal adjustment

Knowledge sharing in the ESS in the case of seasonal adjustment

Innovation in the case of seasonal adjustment

Securing statistical confidentiality not applicable

Non-excessive burden on respondents not applicable

Cost-effectiveness in the case of seasonal adjustment

Coherence and comparability in the case of seasonal adjustment

Accessibility and clarity in the case of seasonal adjustment



Sustainability














Quality of reporting

Dissemination activities structure on the CROS portal should be improved

Cost-effectiveness



Performance

Score:


Major contribution

Contribution


Not met

Partly met

Fully met



4.3 Centre of Excellence on Statistical Disclosure Control (CoE on SDC)11


General

Acronym: CoE on SDC

Project: Centre of Excellence in Statistical Disclosure Control

Contract type: FPA


Project start date: 9.10.2014 (signed) Project end date: 9.10.2018

Duration: 48 months

Project status: Ongoing

Total estimated costs:

Requested grant:

Website: http://ec.europa.eu/eurostat/cros/content/centre-excellence-statistical-disclosure-control-0_en

Objectives

The international cooperation in statistical disclosure control started many years ago. In 2009, an Expert Group on SDC was created with the aim of solving specific SDC issues and discussing developments in this field.

The contract for this CoE was signed in October 2014 and is formally a Framework Partnership Agreement (FPA) in Statistical Disclosure Control. It aims at bringing together experts in the area of statistical disclosure control which is understood as the first step for further activities in this area. The partnership should help to continue the excellent co-operation in the area of SDC between several NSIs and to add expertise from other NSIs which showed interest in this topic recently. In more detail, the contract mentions the following objectives:

• Development of new methodologies, for example in view of Big Data or Open Data;

• Extending already-existing SDC tools like improving already existing software, dealing with singleton problems in large hierarchical tables, and improving protection of weighted tables;

• Development of guidelines for SDC applications at NSIs, including ways on how to document applied rules and methods;

11 In agreement with the new CROS portal the name Centre of excellence on Statistical Disclosure Control

(CoE on SDC) is used in this evaluation. In a number of documents the former name Partners in SDC is used.



• Support of NSIs in the use of existing software by on-the-job training, the organisation of specific workshops, or training courses;

• Improvement of the co-operation by using remote-access facilities which allow testing of methods on sensitive data from other partners;

• Development of training courses for specific areas including e-learning material for SDC;

• Development of public use files and synthetic training data;

• Testing of new or improved approaches.

The first Special Grant Agreement (FPA-SGA1) with the title: “Public Use Files for Eurostat Microdata” started in January 2015.

Acronym: SGA on PUF

Project: Public Use Files for Eurostat Microdata

Contract type: FPA-SGA1

Project type: Part of CoE on SDC


Duration: 11 months

Project status: Finished

Remark: First Grant of the CoE on SDC

Total eligible costs: 100 000 €

Requested grant: 100 000 €

Website: http://ec.europa.eu/eurostat/cros/content/centre-excellence-statistical-disclosure-control-0_en

Participants of CoE on SDC


CBS Netherlands Coordinator

Statistisches Bundesamt (DESTATIS)

Germany Co-partner

Institut National de la Statistique et des Etudes Economiques ( INSEE)

France Co-partner

Statistics Finland Finland Co-partner

Statistics Austria (STAT) Austria Co-partner

Hungarian Central Statistical Office (HCSO)

Hungary Co-partner

Statistical Office of the Republic of Slovenia (SURS)

Slovenia Co-partner

GROUPE DES ECOLES NATIONALES D’ECONOMIE ET STATISTIQUE (GENES)

France Co-partner (not active in the first grant)



The action of the first grant was split into four phases: a preparation phase, an implementation phase, an evaluation phase, and a dissemination/documentation phase. In the preparation phase procedures and methodology was discussed. Synthetic data sets and more traditional methods for disclosure control were explored. In the implementation phase the different methods were implemented and applied to EU-SILC and the LFS results. The evaluation phase assessed the disclosure risk of the produced public use files and assessed the utility of the produced public use files. Finally, the dissemination/documentation phase drafted guidelines for the production of harmonised public use files for EU-SILC and the LFS.


At the moment the documents on the CROS portal are available only for members of the group under: http://ec.europa.eu/eurostat/cros/group-content/8960/cros_document_en. One can find on this page public use files from the participating partners for EU-SILC and the LFS. Furthermore the work of the group is documented by a number of presentations.

4.3.2 Assessment


This evaluation considers only the work of the CoE on SDC up to now, i.e. the first SGA within the FPA.



The indicator of this criterion which is applicable for the activities of this CoE is: Development of models of granting access to confidential data which only allow for indirect identification of the statistical units to researchers carrying out statistical analyses for scientific purposes.


Development of harmonised tools for the production of confidential microdata is an efficient solution for the demand of users for such data.




The CoE on SDC, in particular the first grant, supports the development of instruments for the facilitated exchange of best practice. The activities of the CoE support also different forms of collaboration of experts. Hence, the CoE on SDC meets this policy goal.

Innovation:

With respect to innovation the first two indicators for innovation are applicable: Development and implementation of new processes for certain steps in the statistical production are supported; Development of new statistical products.


The agenda of the CoE is fully compliant with this policy goal: standardised methods for disclosure control are developed and promoted within the ESS; Technical and organisational solutions for granting access to confidential data for research purposes under the provisions of Article 23 of the Regulation on European statistics are developed; Technical and organisation measures to protect statistical confidentiality in the case of exchange of confidential micro data are developed and propagated within the ESS.


This policy goal is not applicable for the topics treated within the CoE on SDC.

Cost-effectiveness

The promotion and implementation of a standardised solution in statistical disclosure control by the CoE supports the goal of cost-effectiveness.


The activities of the CoE on SDC in the development of harmonised methodological standards for statistical disclosure control improve the comparability of the statistical outputs.


Public use files for scientific purposes improve the accessibility of data for research purposes. Hence the activities of the CoE on SDC support this policy goal.



The contribution to this goal is obviously met to a high degree. The activities will support the intention to make methods for SDC available for non-participating partners. The first grant shows quite well that the specialization of certain member states in the production of public use files (synthetic data, traditional SDC) may contribute to the benefits for the ESS as a



whole. Furthermore, there is an increasing demand for confidential micro data by the scientific community. The activities of the CoE on SDC can contribute to the promotion of excellence beyond the partners directly involved.

Sustainability:

There is a high probability that the methods developed under the first grant agreement will be useful over a longer time period. Due to the fact that the activities of the CoE are at an early stage

further efforts in the area of securing confidentiality are necessary. From the intended plans of the CoE stated in the FPA and the work up to now it can be expected that the future work will produce sustainable solutions.


The CoE is complementing but not duplicating ongoing or planned actions in the ESS. The standardisation and harmonisation of methods in SDC will lead to cost-effective solutions with

high impact for the ESS. The organization of the activities as a CoE is a very good operational approach to achieve the planned results.



With respect to this criterion the following indicators can be applied: further development of methods according to new needs within the ESS; participation in research projects on the academic level. The provision of public use files for scientific purposes and the development of methods according to new needs have definitely a scientific character.

Whether a helpdesk should be established needs to be discussed. If a software repository is set up the help desk could be useful for users of this software. Probably support by blogs or consultation is more effective in this case.

Keeping methodological knowledge within the ESS up-to-data:

The CoE is a partnership of many of the key players in statistical disclosure control inside the ESS which guarantees that the knowledge is up-to-date.


It can be envisaged that the activities of this CoE will result in a repository for common software tools for statistical disclosure control.


Up to now there were no activities in this direction by the CoE on SDC. According to the general goals of this CoE it can be anticipated that training courses will be developed.




An expert group on statistical disclosure control already exists and it can be expected that community building in this area will be improved by the activities of the CoE


The steering group for this CoE seems to be more an informal group. Due to the importance of the topic for ‘data dissemination’ Eurostat should think about a more structured approach for defining priorities for the activities of the CoE on SDC; appropriate resources should be provided. Obviously, the CoE was up to now successful in bringing together interested parties but a more active promotion of the activities of the CoE on SDC should be envisaged.

Additional aspects of the ex-post evaluation



Regarding the description of the first grant agreement the CoE on SDC seems to be in line with the expected and promised results.


Only drafts of the results of the first grant agreement were available to the evaluators. These papers are written in an understandable way and the results are well presented.


At the moment only few documents are available on the CROS portal. It is also not clear whether the CASC website (http://neon.vb.cbs.nl/casc/) is used for further dissemination activities. However, this CoE is rather in an early stage; in the future dissemination activities should play a bigger role.

Cost-effectiveness:

Comparing the costs for the first grant with the results one can say that the project is cost-effective. Common solutions in the area of disclosure control will help to reduce the development costs for such activities.


Networking activities in the area of statistical disclosure control were mentioned in the description of the FPA but have not been started yet.




Similar to the other criteria of the ex-post evaluation it is too early for this CoE to make an evaluation. A typology of information requirements for different user groups should be developed. For example, requirements for users of public use files or requirements for statistical data producers.

Performance:

Looking at the five main activities for a CoE, one can state that main activities of the first grant falls in the activities application setup and testing. The development of common solutions for public use files could be understood as starting point for the development of a common software repository. Detailed test for the solution were made. A general plan for further activities should be formulated explicitly.



4.3.3 Scoreboard

Figure 9 SDC scoreboard



More flexibility in responding to new user needs met in the context of SDC

Efficiency gains in production met in the context of SDC

Knowledge sharing in the ESS met in the context of SDC

Innovation met in the context of SDC

Securing statistical confidentiality

Non-excessive burden on respondents not applicable

Cost-effectiveness met in the context of SDC

Coherence and comparability met in the context of SDC

Accessibility and clarity met in the context of SDC



Sustainability









Best practice and training courses not applicable up to now


Approriate institutional environment general structure for CoE should be implemented


Did the projects deliver the expected results? this refers to the first grant

Quality of reporting not applicable at the moment

Dissemination activities preliminary evaluation

Cost-effectiveness this refers to the first grant

Involvement of NSIs not participating in the CoE not applicable at the moment

Provision of documentation for different user groups not applicable at the moment

Performance preliminary evaluation

Score:


Major contribution

Contribution


Not met

Partly met

Fully met