draft methodology for impact analysis of ess.vip projects resources... · eurostat a.tfip/b1 2 rdg...

EUROPEAN COMMISSION EUROSTAT Directorate A: Cooperation in the European Statistical System; International cooperation; Resources

RDG 2014/ 5/ EN

1st Meeting of the

ESS Resource Directors Group

Luxembourg, 13th

June 2014

Draft Methodology for impact analysis of ESS.VIP Projects

Item 5 of the Agenda

Eurostat A.TFIP/B1 2 RDG 2014/5/ EN

Draft Methodology for impact analysis of ESS.VIP Projects

1. Recommendation for action by the RDG

Following the discussion held in the ESSC on 13 February 2014, and in the Joint

meeting of DIME and ITDG on 26 March 2014, the Resource Directors Group is

invited:

- To give feedback on the revised draft methodology for the impact analysis of the

projects enabling the implementation of the ESS Vision 2020 that builds on the

work of both the Task Force Impact Analysis and the ESSnet on Standardisation.

2. Background and brief history

The need for a methodology for the impact analysis of ESS.VIPs, and more in general,

for the assessment of projects with relevance for the development of the ESS, has been

underlined on several occasions.

2.1. Two initiatives assessed the possible approaches

Sponsorship on Standardisation

The Recommendations of the Sponsorship on Standardisation, adopted by the

ESSC in September 2013, proposed to develop a standardised qualitative

assessment tool to facilitate informed decisions in the field of standardisation.

This task was followed up by the ESSnet on Standardisation. The ESSnet on

Standardisation, which started its work at end 2012, was asked to further develop

and test the impact assessment tool developed by the Sponsorship on

Standardisation on the basis of a SWOT methodology. First results were

presented in the Forward Looking Feedback Workshop in Den Haag (May 2013)

where the Sponsorship discussed its draft recommendations with a broad group

of interested experts from inside and outside the ESS.

The recommendations, adopted by the ESSC in September 2013, state the

following:

For important decisions in the field of standardisation, it is imperative to make a

systematic assessment of the costs and benefits implied by the decision proposed.

For strategic decisions the assessment often needs to rely on qualitative

information only. As a minimum, this assessment has to reflect the impact on the

Member States, Eurostat and on the ESS as a whole.

Task Force "Impact analysis of ESS Vision Infrastructure Projects"

On 15 November 2012, the ESSC noted the importance of estimating

implementation costs of ESS Vision Infrastructure Projects (ESS.VIPs), bearing

in mind that substantial upfront investment would be necessary under budget

constraints. In particular, the ESSC decided that the ESS.VIPs, besides the three

that were already endorsed in 2012 (SIMSTAT, Common Data Validation Policy


and ESBR) needed to be evaluated on the basis of implementation costs and

future benefits before being rolled-out. On 14 December 2012, the Partnership

Group proposed to set up a Eurostat Task Force of Resource Directors to discuss

those issues, with the aim, in its first function, to assist Eurostat in the conduct of

impact assessments of different projects including ESS.VIPs.

The objective of the “Task Force Impact Analysis” was to support the work of

development projects such as the ESS.VIPs by facilitating an analysis of their

impact for the ESS, in particular from a cost/benefit perspective. Specific

objectives were to make a proposal for a common methodology and to test it in a

pilot in the given time frame. The Task Force was established for the period

April-December 2013, chaired by Eurostat and included representatives of NSIs

responsible in the field of resources and cost assessments from 7 Member States.

After an analysis of national practices for assessing projects, the Task Force

explored qualitative and quantitative approaches. The qualitative part takes into

account the experience of the sponsorship group/ESSnet on standardisation, with

specific adaptations when necessary. The methodology was tested in parallel on

a pilot project: PRIX (PRIce statistics neXt generation).

The draft methodology was discussed by the ESSC on 13 February 2014, and it

was decided that the methodology would be further discussed in relevant

Directors group meetings, in particular the DIME and the Resource Directors’

Group and tested on new pilots.

2.2 Key elements of the two proposals

Both the Task Force Impact Analysis and the ESSnet on Standardisation analysed

potential impact assessment methodology and elaborated the proposals for possible

approaches to the assessment. The key elements of the two approaches are summarised

as follows (for more details see the final reports1)

The proposed methodology of Sponsorship on Standardisation is based on:

- Structured approach with a fixed set of aspects;

- Assessment by all stakeholders in the ESS from their own perspective; this is

essential to establish consensus;

- All types of initiatives in the ESS use the same assessment approach; this

allows comparisons over initiatives;

- Decisions around a new initiative follow a life cycle; through time the

process tends to start with decisions of a strategic nature, then a tactical

nature and finally an operational nature. At the start of the process only

qualitative information is available, and this is usually sufficient for the type

of decisions required. Over the cycle of the initiative the assessment should

be updated and new more detailed information could be added. More

quantitative information should fit in.

- To build up trust, a transparent process is necessary;

1 https://circabc.europa.eu/sd/a/15e27166-60d1-4441-950b-8e498b5b9a29/ESSC_2014_20_EN_8_ESS.VIP_Projects_Impact_Analysis.pdf

https://circabc.europa.eu/sd/a/041f4624-9018-41d6-bb06-0791017e07e7/DIME-ITDG%202014%20Item07b%20Annex%201%20Draft%20deliverable%20on%20Impact%20Assessment.pdf

https://circabc.europa.eu/sd/a/15e27166-60d1-4441-950b-8e498b5b9a29/ESSC_2014_20_EN_8_ESS.VIP_Projects_Impact_Analysis.pdf




- The approach is supplemented by a scoring mechanism. The scoring

mechanism is a powerful tool to analyse, summarise and present the results.

It is not meant to derive from the stakeholder information automatically the

European business case. The open text explanations to the scores are

probably even more important for the preparation of a useful discussion.

- The scores and comments provide the basis for discussing the European

business case. It shows not only the positive and negative aspects, but also

the variation among the ESS stakeholders.

- The approach is a proportional way to involve all the partners in the

European Statistical system and supply the basis for a European business

case without excessive burden on the respondents. The process will be run

over a short period and produce sufficient information for the appropriate

body to make an informed decision on launching a project. It will function as

a kind of written consultation, but in this case also involving Eurostat.

Key elements of the draft methodology of TF "Impact analysis of ESS.VIPs"

- While information is collected and the assessment is done at NSI level, the

purpose is to assess the impact of a development for the European Statistical

System as a whole.

- It requires contributions and cooperation from both statistical and financial

(resource) experts and the involvement of NSI’s senior management.

- It combines quantitative and qualitative assessments. The quantitative

assessment relies on an estimation of current costs, development costs and

future costs of production. The qualitative assessment is based on a SWOT

analysis (Strengths, Weaknesses, Opportunities and Threats) largely based on

the methodology of Sponsorship on Standardisation. The conclusions of the

quantitative analysis feed into the qualitative analysis.

- The ESSC formulated its request in terms of “cost-benefit”. In the

quantitative assessment “benefits” were understood as the potential cost

savings, and in the qualitative assessment “benefits” are reflected in the

strengths and opportunities.

2.3 Need for a common approach in the context of the ESS Vision 2020

On 26 March 2014, the DIME/ITDG appreciated the work done by both groups and

stressed the need to reconcile the approaches and fine tune the methodologies. A

suggestion was made to apply the different types of assessment in different phases of the

project: an initial qualitative approach for providing the key elements for taking an

informed strategic decision establishing the need for the project and a more in depth

assessment, based on a refined qualitative analysis combined with a quantitative

approach to decide on the implementation aspects of the project. The DIME/ITDG

suggested also assessing the type of SWOT analysis to be carried out, notably with or

without fixed items. Eurostat was mandated to elaborate a proposal to be further

discussed in the Resource Directors Group meeting taking place on 13 June and in the

DIME/ITDG steering group taking place on 18-19 June 2014.

Following the work of the Task Force Vision 2.0, the ESSC agreed on the ESS Vision

2020 on 14 May 2014. The Vision foresees that, the concrete project proposals for

implementing the Vision will be evaluated and selected on the basis of several general


criteria. A first list of criteria is attached to the Vision document and will be fine-tuned

by the temporary high level task force, mandated to support the PG in the preparation of

a first draft implementation strategy for the ESS Vision 2020.

3. Policy context

The work of the Eurostat Task Force "Impact analysis of ESS.VIPs" was launched in the

context of the ESS.VIP programme. The work of the ESSnet on Standardisation was

launched in the context of the Sponsorship on Standardisation with the aim to establish

the business case of European standardisation initiatives at the strategic level. It should

be noted that the implementation of the ESS.VIP Programme will be aligned with the

conclusions of the Task Force on Vision 2.0 (ESS Vision 2020).

4. Consequences for the NSIs

Once the final methodology adopted, all NSIs will be requested to contribute to the

impact assessments of ESS Vision 2020 implementation projects.

5. Outstanding problems

The testing of the draft methodology developed by the TF impact assessment on

ESS.VIPs on the pilot project PRIX (price statistics) has shown that currently no

systematic evaluation/ impact analysis of the modernisation process is available at NSI

level. Some NSIs are working on the elaboration of a methodology for impact

assessment, but for the time being only a limited number of NSIs for few parts of the

project are capable of estimating costs and/or savings.

Both the Sponsorship on Standardisation and the TF Impact analysis of ESS.VIPs

stressed the importance of testing the methodology on a number of projects and to

review it in the light of the experience gained with those cases.

6. Risk assessment

If no progress is made on the elaboration of a common methodology, there will be no

common basis for assessing the proposals for the ESS Vision 2020 projects and taking

decisions at the ESS level.

The rating of the SWOT by each NSI includes a degree of subjectivity. In order to

mitigate this risk, it should be justified by facts (or if available by the quantitative

analysis) and represent the position of the management (directors responsible for the

subject areas), taking into account the input from statistical and financial experts.

7. Next steps

The draft methodology will be discussed in the Resource Directors Group on 13

June and submitted to DIME/ITDG steering group on 18-19 June;

The output from the discussions of the RDG and DIME/ITDG will be included in

to the work of the high level TF attached to the PG with the view of preparing the

discussion on the ESS Vision 2020 Implementation strategy at the DGINs in

September 2014.


The methodology will be applied for the assessment of projects to be proposed to

the ESSC for the implementation of the ESS Vision 2020;

The methodology will be revised on the basis of the experience cumulated.

Annexes

- Annex 1 to the methodology: Template to be used for the qualitative analysis

- Annex 2 to the methodology: Template to be used for the quantitative analysis


DRAFT METHODOLOGY FOR IMPACT ANALYSIS OF ESS.VIP PROJECTS

The methodology described in this document aims at assessing the impact of the projects that will

enable the implementation of the ESS Vision 2020. It is based on the work of the ESSnet on

Standardisation and of the Eurostat Task Force Impact assessment of ESS.VIPs.

The methodology requires the contributions of different actors at different stages (statistical and

financial– experts), and the involvement of NSI’s senior management. The methodology relies on two

types of assessments:

The qualitative assessment based on a SWOT analysis (Strengths, Weaknesses, Opportunities

and Threats).

The quantitative assessment based on an estimation of current costs, development costs and

future costs of production.

It is important to note that, while information is collected and the assessment is done at NSI level, the

purpose is to assess the impact of a development for the European Statistical System as a whole.

1. Overall description

Decisions around a new initiative follow a life cycle (see Fig. 1). The impact assessment methodology

is split in two phases that correspond to the decision points “establish the need” and “adopt”.

Fig. 1: Approval of life cycle phases (extract from the Final Report of the ESSnet on

Standardisation)

Phase 1 of

methodology:

Business case

including SWOT

and Vision

general criteria

Phase 2 of

methodology

including detailed

quantitative analysis


Phase 1: Business case including SWOT and Vision general criteria

At the start of the process, the strategic nature of the decision to be taken aims at deciding a

"go/non-go" of a project that, because of its characteristics, represents an important initiative

at ESS level. At this stage, it can be assumed that the available information is qualitative. This

information is usually sufficient for the type of decisions required (strategic). Pragmatically,

the strategic ESSC decision on an initiative will rely on a business case based on a standard

project management methodology PM² including the list of general criteria included in the

ESS Vision 2020 and an impact analysis based on a qualitative assessment (SWOT). The

ESSC decides on launching the project and could request moving to phase 2 for the whole

project or specific Work Packages of the project.

In order to provide the necessary elements for an informed decision of the ESS on a specific

initiative, the SWOT analysis should encompass the information necessary to contextualise

the initiative in the ESS strategic orientations. It includes a certain level of harmonization

(based on 3 fixed items) that could allow comparisons among competitive initiatives although

the comparison would have significant limitations.

Phase 2: Quantitative and qualitative analysis

Following the decision made at the end of phase 1, a more in depth impact assessment might

be needed. This second phase is based on a detailed quantitative analysis and a refined

qualitative analysis (SWOT) including detailed information derived from a more dedicated

and rich set of qualitative nature.

Phase 3 (not covered by this document)

Implementation of the initiative.

2. Roles of statistical experts and financial experts in the implementation of the methodology

Statistical experts (from the specific domain) and financial experts (on the level of the NSI) have to

work hand-in-hand to implement the methodology. The results should be validated by senior

managers in the NSIs.

In particular, statistical experts are involved in the preparation of business case, quantitative and

qualitative analysis and analysis of the results of the process. An early involvement of the statistical

experts is extremely important to agree on the main goals/achievements and cost drivers of the

project. Financial experts are involved mainly in phase 2 of the process (quantitative and qualitative

analysis) but their early involvement in phase 1 would also be beneficial.

3. Tools

The two main tools supporting the methodology are available in annex. They include:

1) The template to be used for the qualitative analysis (SWOT questionnaire),

2) The template to be used for the quantitative analysis


4. Main steps of the impact assessment methodology

Table 1: Impact assessment steps

Phases and Steps Actors

Phase 1. Business case including SWOT and Vision general criteria

(3 months)

Step 1.1. Preparation of the business case, including project scope and description, expected

achievements, Vision general criteria

Project manager in consultation of statistical

experts (national and Eurostat)

Step 1.2. The business case includes a SWOT

analysis based on fixed and flexible items

1.2.1. Identification of the main SWOT items

Focus group of statistical and financial experts

from MS, project leader, support from Eurostat

Project Management Office (PMO)

1.2.2. SWOT rating: questionnaire filled in by

each NSI

+ possible questions on some of the Vision

general criteria

Senior Managers responsible for the statistical

area with the support of statistical and financial

experts from all MS

Step 1.3. Synthesis report including the national

and European perspectives feeds into the business

case, Vision general criteria are completed.

Project leader, support from Eurostat PMO

Step 1.4 Discussion by the relevant WG and

Director’s group.

Project leader, Working Group, Directors Group

Step 1.5. Decision by the appropriate body For ESS.VIPs, decision made by ESSC

Phase 2. Quantitative and qualitative analysis if required by decision taken under 1.5 (4-5 months)

Step 2.1. Quantitative analysis Statistical and financial experts from all MS,

project leader, support from Eurostat financial

unit

Step 2.2. Qualitative analysis

2.2.1. The initial SWOT analysis is refined based

on the evolution of the project and the outcome of

the quantitative analysis

Focus group of statistical and financial experts

from MS, project leader, support from Eurostat

Project Management Office (PMO)

2.2.2. SWOT rating by each NSI Senior Managers responsible for the statistical

area with the support of statistical and financial

experts from all MS

Step 2.3. Synthesis report including the national

and European perspectives

Project leader, support from ESTAT financial unit

and PMO

Step 2.4. Finalised business case and

Discussion by the relevant WG and Director’s

group.

Project leader, Working Group, Directors Group

Step 2.5. Decision by the appropriate body For ESS.VIPs, decision made by ESSC


Phase 1. Business case including SWOT

b. Identify the different work packages that deserve a specific SWOT analysis

a. Create a focus group of statistical and financial experts (6 to 8 countries)

c. Elaborate a list of SWOT items relevant for the project

a. SWOT questionnaire filled in by all NSIs

1.4.2 Discussion of business case by WG and Directors’ group

1.2. Qualitative analysis – SWOT items

1.2.2 SWOT rating

1.5. Decision to launch the project or go to

phase 2 by appropriate body (e.g. ESSC)

1.1 Preparation of the business case, including Vision general criteria

1.4.1 Finalised business case

1.3 Synthesis report including EU perspective


Phase 2. Quantitative and qualitative analysis

b. Grouping of the countries

c. Identification of cost drivers

2.2.1 Refined SWOT

d. Estimate development costs related to the projector countries in

each group

e. To be state: Estimate future costs for countries in each group

f. To-be state: Estimate savings for countries in each group

2.4.2 Discussion by WG and Directors’ group

a. As-is situation: Identify current costs for areas affected by change

2.2.2 SWOT rating by each NSI

2.1 Quantitative analysis 2.2 Qualitative analysis (refined

SWOT)

2.4.1 Finalised business case.

2.5. Decision by appropriate body

(e.g. ESSC)

2.3 Synthesis report including EU perspective


5. Detailed description on the methodology

PHASE 1. Business case including SWOT and general Vision criteria

STEP 1.1. Preparation of the business case

The business case is developed with the standard project management methodology PM². The busi-

ness case will include the scope, approach, objectives, resource implications and impact on

stakeholders of the project.

a. Determine the scope and expected achievements of the project:

The scope and expected achievements of the project can be determined on the basis of classical

project management methodology, they are a pre-condition to a good impact assessment.

b. Determine the timeline of the impact analysis:

The timeline of the impact analysis has to be determined, i.e. the period which should be covered by

the analysis:

Expected duration of the project + N years of operation

The timeline depends on the project to be assessed and should be in line with expectations as regards

the results of the project2, including quantifiable savings. If savings are expected within 5-7 years, the

impact analysis should cover at least this timeframe.

c. Identify the main areas of the GSBPM3 model affected by change:

The next step is to identify the main areas affected by change. To this end and when appropriate4, use

is made of the statistical business processes according to the GSBPM model:

2 N is a hypothesis that may be modified after a first assessment of the project.

3 Generic Statistical Business Process Model The reader can check the website www.unece.org/stats/gsbpm to be sure of having the latest version.


Main business processes affected by a development project would typically be “Collect, Process,

Analyse and Disseminate”.

d. Determine the Vision general criteria for which specific information from MS is needed (e.g.

national legal issues)

General criteria for Vision implementation projects

A. Does the proposal have a positive business case at both ESS and national levels, in the sense that

benefits outweigh costs?

B. Does the proposal contribute to the needs and drivers identified in the ESS Vision 2020 such as

efficiency, quality and promoting public trust?

C. Can the proposal be clearly positioned according to our ambitions for collaboration set out in the

ESS Vision 2020?

D. Does the scope in that sense look manageable?

E. Is the proposal consistent with the full set of other proposals?

F. Does the proposal take into account generic European principles like proportionality and

subsidiarity?

G. Is the proposal in line with the European Statistics Code of Practice?

H. Does the proposal observe existing legislation, both at national and EU level?

I. Is the proposal in line with the ESS reference architectural decision and design principles?

J. Does the proposal sustain a simultaneous production of European and national statistics?

STEP 1.2. Qualitative analysis- SWOT

1.2.1: SWOT items

The qualitative analysis is based on a SWOT tool including 3 fixed items and 3 items which can be

partially adapted to the specific needs of the project. It builds on the work on Standardisation in the

European Statistical System 5. The exercise can be piloted by a focus group but the SWOT tool is to

be sent to all NSIs and filled in by as many NSIs as possible.

a. Create a focus group of statistical and financial experts (6 to 8 NSIs)

The first steps of the qualitative analysis can be more efficiently managed by a small group. Therefore

a focus group needs to be created, representing as far as possible NSIs with different levels of

modernization. It will gather both statistical experts and financial experts from the same NSI.

b. Identify the different work packages that deserve a specific SWOT analysis

Depending of the complexity of the project and the interdependencies between the different Work

Packages, it may be useful to develop one SWOT analysis for the whole project or different SWOTs

for the different Work Packages.

4 The GSBPM model might for instance not be appropriate when processes / activities are strongly integrated costs are difficult to allocate to specific statistical domains. 5 http://www.cros-portal.eu/sites/default/files//NTTS2013fullPaper_202.pdf


For example, for the PRIX project, it was initially considered that the Work package 2 (Modernisation

and harmonisation of basic data collection) and Work package 3 (IT tools in particular system of data

warehouses) could be subject of different SWOT analyses.

c. Elaborate a list of SWOT items relevant for the project

Once the scope of the analysis is clearly defined, the focus group (in a joint meeting of the statistical

experts and the finance experts that may last at least half day) will reflect, on the main strengths,

weaknesses, opportunities and threats of the project or selected Work Package.

The list of SWOT items that is provided in the template is a combination of items identified by the

ESSnet on standardization and the testing by the Task Force when assessing the PRIX project Work

Package 2. SWOT tool includes 3 fixed items and 3 flexible items that have to be modified or

adapted to be as relevant as possible to the project;

List of SWOT items (fixed items in bold)

1. STRENGTHS

a. Saving of resources

b. Improved process and systems quality

c. Enhanced flexibility

d. Increased harmonisation

e. Increased level of detail

f. Better availability of metadata

2. WEAKNESSES

a. Costs (of development, transition, support and maintenance)

b. Lack of agility/flexibility of the new system

c. Heterogeneity of the starting point between countries and partners

d. Complexity

e. Impact of changed process (risk of errors, discontinuity of the data)

f. Lack of appropriate skills in the NSIs

3. OPPORTUNITIES

a. Reduced burden on respondents

b. Enhance standardisation (also outside the ESS)

c. Better communication with users and stakeholders d. Opportunity to change to cope with resource constraints

e. Shared technical developments

f. Possibility to provide tailored feedback to data providers

4. THREATS

a. Lack of consistency with national policies (including legal framework)

b. Lack of support from stakeholders/financers (including budget cuts)

c. Lack of suitable skill profiles

d. Threat to data security

e. Competition from other providers

f. Dependency from external actors/providers


When defining the SWOT items, the support of staff experienced in the subject area is crucial. The

group should assess the project and not the current situation. Strengths/Weaknesses are mainly related

to what is internal the ESS and directly related to the project. Opportunities/Threats are elements that

cannot be influenced by the ESS. The result should be six items for each category (S, W, O, T). The

items should be as far as possible one-dimensional, exclusive and exhaustive. They can be detailed

with information specific to the project.

Example: Generic item: d. Increased level of detail

Specific item for PRIX project: d. Increased level of detail (higher frequency of data collection,

territorial coverage) and potential dissemination.

1.2.2. SWOT - rating

One questionnaire per NSI should be filled in by management (directors responsible for the subject

areas), taking into account the input from statistical experts and financial experts.

The contribution is made from the perspective of the NSI, In particular they will be asked to:

- Give a score to each item in case of relevance and impact.

Relevance: Scores are from 1 to 4, where 1 is minimum relevance and 4 is maximum

relevance.

Impact: Scores are from 0 to 3, where 0 is no impact and 3 is maximum impact.

- Provide comments on the justification of the score. This information will be used to draft a

text supporting the analysis of the scores.

Additional questions may be sent to NSIs at the same time to gather information on some vision

general criteria as outlined in Step 1.1.d (e.g. national legal issues).

Step 1.3 Synthesis report including the national and European perspective

The project leader will combine in the report the synthesis of the national perspectives and the

European perspective (e.g. savings at the level of the ESS, increased comparability of the data may

not be of high relevance at national level but very relevant at European level).

The project leader will identify patterns of the “fingerprints”, in particular in comparison with the

grouping of NSIs and will summarize the main comments provided by NSIs for each category (S, W,

O, T). The graphical presentation in “fingerprints” can help visualising the SWOT of the same project

from the perspective of each NSI. Example:


The synthesis feeds into the business case that also addresses the Vision general criteria outlined

in Step 1.1 d.

The analysis may result in:

- Keeping the initial project, the proposal will indicate how to exploit the strengths, mitigate

the weaknesses, develop the opportunities and avoid the threats,

- Modifying the project,

- Proposing to launch first some Work Packages of the project for which the SWOT

analysis is more positive,

- Proposing to launch the project with a limited group of NSIs,

- Proposing to go to phase 2 for the project or some Work Packages, for a more in depth

quantitative and qualitative analysis.

- Proposing to abandon the project.

STEP 1.4. Discussion by the relevant Working Group and Director’s Group

The impact analysis will feed into the business case that will be discussed in the relevant Working

Group and Director’s Group.

STEP 1.5. Decision by the appropriate body

In the case of the ESS.VIPs, the decision will be taken by the ESSC and will include the possibility of

moving to phase 2 for the whole project or parts of it.

PHASE 2: Quantitative and qualitative analysis (if required by decision taken in 1.5)

STEP 2.1. Quantitative analysis

The quantitative analysis shall focus only on the business processes affected by change.

The quantitative analysis should be carried out by as many NSIs as possible in order to have a

complete and sufficiently reliable picture as extrapolating costs from one NSI to another might cause

biased results. However, if certain NSIs are not in a position to assess their costs and quantifiable

savings, at least one NSI per group (see Grouping of NSIs) should provide input.

When carrying out a quantitative analysis, the following issues have to be kept in mind:

An accurate quantification of costs is not always possible for various reasons (no cost accounting,

integrated system not allowing identifying costs for a specific module …). Quantification of potential

savings is in general even more difficult. In many cases the quantitative analysis will therefore have to

be carried out based on (best) estimates. The methodology might therefore have to be adapted

according to the information at hand.

A quantitative assessment of the costs and potential savings is nonetheless a very useful exercise as it

obliges to think in much more concrete terms about the project than would be the case under a purely

qualitative approach. It completes the information derived from the qualitative SWOT analysis. In


case only limited quantifiable information is available, the overall assessment will have to rely to a

greater extent on the qualitative SWOT analysis. This is why it is important to document the extent to

which the quantitative assessment of costs and potential savings has relied on estimates.

Whenever possible, costs should be determined for the following categories:

Staff (including IT staff and interviewers in case surveys are not sub-contracted)

IT solutions needed to implement the project successfully

Overhead (either 30% of staff costs or based on real costs)

Other costs

The distinction of costs according to the different categories allows a better understanding of the costs

and also facilitates the identification of cost drivers (see also c. Cost drivers). Metadata should be

associated with the budget items to improve the comparability of the data.

a. As-is situation: Identify current costs for areas affected by change

Current operating/production costs

All current operating/ production costs for the statistical business processes affected by change have

to be determined. Costs have to be identified for the last available full calendar year. In order to

reduce the complexity of the methodology, operating costs for this last calendar year will be

extrapolated for the entire timeline of the cost-benefit analysis; in case operating costs over one year

are not representative, average operating costs over a reference period to be determined may be used.

In order to ensure comparability of results across NSIs, total costs per NSI have to be PPP-adjusted.

Evolutive maintenance costs

For development projects, evolutive maintenance costs foreseen until the end of the timeline of the

cost-benefit analysis have to be identified in addition to current operating/production costs. These

evolutive maintenance costs would have occurred even in case the development project would not

have been carried out.

b. Grouping of NSIs (option)

For various reasons not all NSIs might be in a position to quantify or even estimate their costs and

potential savings (no cost accounting, integrated system not allowing to identify costs for a specific

module). In this case a possible option would be to identify 3-4 groups of NSIs with similar

production models. The underlying assumption is that NSIs with similar models have comparable cost

structures (after exclusion of price level differences) and expectations as to the value-added of the

project (i.e. in relation to their current production processes). It would thus be possible to arrive at

some conclusions for a group of NSIs, even if not all of them were in a position to quantify their

costs. It has nonetheless to be kept in mind that extrapolation of results from one NSI to the other has

to be handled with great care.

c. Identification of cost drivers

Given that the assessment of future costs is extremely difficult, an intermediate step consists in

identifying the impact of change on the costs (the so called “cost drivers”):

For which areas and/or cost categories do we expect higher costs? These could be either one-

off development costs (for instance HR costs for training and requalification of staff) or

permanently higher costs (for instance for infrastructure).


For which areas and/or cost categories do we expect a reduction in costs (for instance lower

dissemination costs)?

The cost drivers should be identified by the statistical experts together with the financial experts.

d. Estimate development costs and adaptation costs related to the project

A distinction has to be made between development costs of the project at EU-level (i.e. not to be

borne by each NSI), and costs for adaptation of the local system in each NSI. Given the purpose of the

ESS.VIP projects, main developments may be carried out by Eurostat, one NSI or a small group of

NSIs and are assessed by Eurostat in cooperation with the relevant NSIs. As to the adaptation costs,

these have to be estimated for NSIs in each group based on the identification of main cost drivers.

Here again, the principles outlined before apply:

Focus on adaptation costs for the statistical business processes affected by change.

If possible, estimation of costs for the different categories (Staff, IT, overhead, other).

At least one NSI per group should provide an estimation of adaptation costs, as appropriate,

(if not all are in a position to do so).

In case the scope of the project is not yet completely defined, the estimation of development costs and

adaptation costs has to be based on possible options and/or the most probable scenario.

e. To-be-state: Estimate future costs for NSIs in each group.

Based on the cost drivers, the next step consists of estimating future costs for areas affected by

change. However, the pilot project has shown that even the most advanced NSIs in terms of cost

accounting that can provide reliable information on the “as is” situation, are currently not fully in a

position to quantify the to-be state. An alternative is to calculate the (large) economic and quality

savings instead (see also f Estimation of potential savings).

f. To-be-state: Estimate potential savings for NSIs in each group.

First of all it has to be pointed out that only a small part of potential benefits are quantifiable in the

form of potential savings. There are two ways of quantifying the potential savings:

The first way is to calculate the net potential savings by comparing the total costs in case of a

standstill situation to the total development costs/adaptation costs of the project plus future costs:

Total costs in case of standstill

- (Total of development costs/adaptation costs + future costs)

_______________________________________

= Net potential savings

When it is not possible, an alternative is to build on concrete experiences of NSIs which have already

modernized certain parts of their process. Ideally, these experiences have been subject to ex-ante

and/or ex-post evaluation of the modernization process. But even if not the case, they can provide a

useful input for the estimation of savings by illustrating for instance the potential for savings in

certain areas or the impact of the change on the cost structure.

Based on the results of the analysis carried out so far (current costs, development and adaptation

costs, future costs), a last step is be to bring the different elements together in order to make a

simulation: Taking into account the expected savings, how long would it take to absorb the


development and adaptation costs of the project? In order to do so, the yearly percentage of savings

has to be estimated as a percentage of current overall costs. In a next step, the most realistic potential

savings (which could also be a percentage range) have to be compared to the estimated development

costs in order to have an idea on how long it might take to absorb the development costs of the

project. In the absence of easily quantifiable information on savings, the advantage of such a scenario

is to think in more concrete terms about the impact of the assessed project on the cost structure. Given

the limited quantifiable information on future costs currently available, such a simulation has to be

carried out with care. The idea is to identify the most realistic scenario(s).

The conclusions of the quantitative analysis can feed into the refined qualitative SWOT analysis: In

particular, reductions in costs and potential savings can feed into the assessment of item a of the

strengths (saving of resources), and estimated costs can feed into item a of the weaknesses (Costs of

development, transition, support and maintenance). The availability of information and the resulting

margin of error/uncertainty have of course to be taken into account.

STEP 2.2. Qualitative analysis- refined SWOT

The purpose is to refine the SWOT analysis, on the basis of the new elements of the project. It may

also be decided to focus the SWOT items on specific work packages.

2.2.1. SWOT items (same as 1.2.1)- the flexible SWOT items may be changed

2.2.2. SWOT rating (same as 1.2.2)- the SWOT rating may be conducted again to take into account

the new elements of the project.

STEP 2.3. Synthesis report including the quantitative and qualitative assessments

Eurostat project leader will combine in the report the synthesis of the national perspectives and the

European perspective (e.g. savings at the level of the ESS, increased comparability of the data may

not be of high relevance at national level but very relevant at European level). The project leader will

identify patterns of the “fingerprints”, in particular in comparison with the grouping of NSIs and will

summarize the main comments provided by NSIs for each category (S, W, O, T).

The analysis may result in:

- Implementing the project, the proposal will indicate how to exploit the strengths, mitigate

the weaknesses, develop the opportunities and avoid the threats,

- Modifying the project,

- Proposing to launch implementation first in some Work Packages of the project for

which the SWOT analysis is more positive,

- Proposing to launch the implementation with a limited group of NSIs,

- Proposing to abandon the project.

STEP 2.4. Discussion by the relevant Working Group and Director’s Group

The impact analysis will feed into the business case that will be discussed in the relevant Working

Group and Director’s Group.

STEP 2.5. Decision by the appropriate body

In the case of the ESS.VIPs, the decision will be taken by the ESSC.

draft methodology for impact analysis of ess.vip projects resources... · eurostat a.tfip/b1 2 rdg...

Documents