the european evaluation consortium (teec)elibrary.lt/resursai/eu integracija/eu knowledge... · the...

Framework Contract No: BUDG-02-01 L2 Specific Budget No: BUDG/01/EVAL/2002/04

Interim evaluation of the eEurope Standardisation Action Plan 2002

(eSAP)

FINAL REPORT: PART I: MAIN TEXT

The European Commission

The Enterprise Directorate-General (DG ENTR)

Submitted by:

The European Evaluation Consortium (TEEC)

The Evaluation Partnership Limited (UK) Economisti Associati (Italy) Particip GmbH (Germany)

navreme knowledge development (Austria)

Authorised Representative The Evaluation Partnership Limited (TEP)

6 Cole Park Road Twickenham, Middlesex TW1 1HW

United Kingdom

22nd November 2003

Interim evaluation of ESAP: FINAL REPORT

THE INTERIM EVALUATION OF ESAP

TABLE OF CONTENTS

0 SHORT EXECUTIVE SUMMARY .............................................................................. I

1 THE POLITICAL CONTEXT....................................................................................... 1 1.1 THE E-EUROPE INITIATIVE .................................................................................. 1 1.2 E-EUROPE AND THE ROLLING STANDARDISATION ACTION PLAN........... 2 1.3 THE E-EUROPE STANDARDISATION ACTION PLAN...................................... 4

2 THE STANDARDISATION CONTEXT ...................................................................... 6

3 THE EVALUATION APPROACH ............................................................................... 6 3.1 OBJECTIVES AND SCOPE ..................................................................................... 7 3.2 THE EVALUATION METHODOLODY ................................................................. 8

3.2.1 The scope of the evaluation ............................................................................................................ 8 3.2.2 The approach to the evaluation ...................................................................................................... 8

4 PRESENTATION OF THE EVALUATION RESULTS........................................... 10

4.1 RESULTS OF THE DESK BASED ANALYSES................................................... 10 4.1.1 Review of the implementation documentation .............................................................................. 10 4.1.2 Analysis of the eSAP documentation............................................................................................. 10 4.1.3 Financial data on eSAP implementation ...................................................................................... 10

4.2 EVIDENCE FROM THE FIELD............................................................................. 11 4.2.1 Views of all actors on eSAP progress against policy objectives................................................... 11 4.2.2 The differences: actors involved/not involved in action lines ....................................................... 12 4.2.3 Views of eSAP progress against policy objectives by stakeholder................................................ 12 4.2.4 The views of the European level standards institutions ................................................................ 14 4.2.5 The views of those within the European Commission................................................................... 14 4.2.6 The case studies ............................................................................................................................ 16

4.3 EVALUATION AGAINST THE MAIN QUESTIONS .......................................... 19

5 ANALYSIS AND INTERPRETATION ...................................................................... 20

5.1 A CONCEPTUAL FRAMEWORK FOR ESAP...................................................... 22 5.1.1 A framework for the measuring policy impacts ............................................................................ 22 5.1.2 A framework for measuring process efficacy................................................................................ 22

5.2 RELEVANT INDICATORS AND BENCHMARKS ............................................. 24 5.2.1 Selection of areas for impact indicators: the current eSAP.......................................................... 24 5.2.2 Selection of areas for process indicators: ongoing eSAP............................................................. 24 5.2.3 Potential ex-ante indicators: a future eSAP ................................................................................. 25

6 CONCLUSIONS AND RECOMMENDATIONS....................................................... 26 6.1 CONCLUSIONS OF THE EVALUATION ............................................................ 26

6.1.1 Conclusions from the analysis of stakeholders views ................................................................... 26 6.1.2 Conclusions relating to eSAP procedure ...................................................................................... 28 6.1.3 Conclusions from the analysis of case study results ..................................................................... 29 6.1.4 Possible approaches to evaluating eSAP...................................................................................... 30

6.2 RECOMMENDATIONS OF THE EVALUATION................................................ 30

The European Evaluation Consortium (TEEC) i


0 SHORT EXECUTIVE SUMMARY

The e-Europe Standardisation Action Plan 2002 (eSAP) has a €12.5 million budget, is based on a Rolling Action plan which supports the aims of e-Europe, is described in detail through a scoreboard of action lines, is financed by the European Commission and delivered through European Standardisation (ESO) contracts. It includes circa 140 separate action lines covering the 12 domains of e-Europe and supports its 3 policy aims. The basic approach to the evaluation was as follows: • To undertake a descriptive analysis of eSAP on the basis of three main

documents: the Rolling Action Plan, the eSAP scoreboard and the GEFINOR financial administration system, used by DG Enterprise for contract management and to highlight eSAP progress against deliverables.

• To elicit the views of a representative sample of eSAP stakeholders

about the relevance and effectiveness of eSAP policy against its objectives and the contribution of a selection of action lines through the use of a special questionnaire and some face to face interviews.

• To analyse, interpret and report the findings with a view to drawing out

useful recommendations to enhance the process supporting eSAP and to improve the content.

The conclusions of the interim evaluation represent the considered judgements of the evaluation team on the basis their analysis and interpretation of the evidence and are briefly summarised as follows: Conclusions from the analysis of stakeholder views • Stakeholders give a positive assessment of eSAP’s contribution • eSAP may be better on standardisation design than on implementation • eSAP sometimes sets unrealistic targets and then cannot keep to them • eSAP has participation deficits, particularly end users and consumers • eSAP needs greater visibility and a more proactive dissemination effort • Suitable indicators for eSAP monitoring have yet to be developed • Member States could help to strengthen eSAP involvement and impacts Conclusions relating to eSAP procedure • eSAP logic and terminology could be clearer, especially for end users • ESO contracts could embody the eSAP strategic logic more robustly • eSAP programme & financial monitoring needs more customisation • eSAP administration needs to be integrated with related processes Conclusions from the analysis of case study results • Relevant projects are undermined by low involvement and awareness

The European Evaluation Consortium (TEEC) I


• A more flexible and proactive approach could give better results • The representation of users and consumers needs to be strengthened Possible approaches to evaluating eSAP • An ESAP strategy framework is needed to develop indicators • The supporting management process needs its own separate measures It is recommended for the future that: 1. A strategic framework should be agreed for any successor programme

that shows the how policy intervention through stakeholder engagement lends itself to results dissemination and user application, bearing in mind the voluntary nature of standardisation.

2. A set of clear, transparent and relevant indicators and benchmarks are

developed with the objective of assessing the financial support given to the e-Europe Standards Action Plan in an objective manner.

3. A set of 3 ex-ante indicators should be developed to reflect the intrinsic

intervention logic of the eSAP intervention:

Indicator Ex-ante definition The relevance of the action lines to the policy

The relevance of each action line at the beginning of the new eSAP period would be considered to be on a scale from 0 to 100.

The expected results through implementation of the action lines

Introduce targeted expected results into new contracts to give a reference point for future evaluations.

The resulting impacts from delivering these results

Select indicators that express the contribution of eSAP to the Information Society for All.

4. At least 5 good value for money indicators are developed in more detail

for use with the current and future eSAP:

eSAP Policy Objectives Suggested indicators Policy intervention 1. Progress in delivering actual standards;. Stakeholder engagement 2. Record of participation in activities

3. Balance of representation at eSAP action line initiatives; Results dissemination 4. Summary table of dissemination actions taken against targets User application 5. Perceptions of contribution of eSAP action lines.

5. A differentiated approach is taken in the choice and negotiation of

suitable ex-ante and value for money (VFM) indicators eSAP for the 3 European Standardisation Organisations (ESOs) and that these are embedded into any future eSAP contracts with the ESOs.

6. A methodology is developed to measure stakeholder satisfaction about

the responsiveness of the ICT standardisation process to their needs and requirements with a view to looking at the costs and benefits of developing this as a “soft” indicator.

7. Management performance indicators are added to show the speed,

contract quality and cost of delivery in achieving eSAP deliverables.

The European Evaluation Consortium (TEEC) II


Indicator Proposed definition Delivery speed

Performance against agreed Commission milestones for delivery of key domains

Contract quality

Performance against contract quality criteria (clarity of purpose, ability to monitor, presentation of logic, etc.)

Cost of delivery

Performance against targeted manhours to deliver key domain components

8. Statistical and financial monitoring systems are more closely aligned so

that indicators are built into contract monitoring and actual achievements can be compared with budgeted achievements.

9. A common terminology is used on which to base action line descriptors

that are grouped under understood domain themes, used for all stakeholders and are “user (and consumer) friendly”.

10. Scoreboard nomenclature aligns with, but is not different to, the e-Europe

policy nomenclature to avoid another element of confusion. 11. Member States are encouraged to take a more proactive approach:

• to stimulate more user group involvement • to bring in more national consumer groups • to take a common position with dominant market players • to engender wider ownership of eSAP deliverables • to work towards better eSAP accessibility at national level.

12. The www.e-europestandards.org website should be developed and

promoted in a more proactive way by creating linkages to those web sites used more frequently by targeted end user and consumer groups and through using the right search engines.

13. There is more follow through on progress within action lines by the

Commission, especially in showing interest in the outcomes and helping to promote the benefits of these.

The contract team would like to acknowledge the time, effort and helpful contributions made to the interim evaluation work by the European Standardisation Organisations and by those involved in the cases studied.

The European Evaluation Consortium (TEEC) III


1 THE POLITICAL CONTEXT

In this chapter of the report, the e-Europe Standardisation Action Plan (eSAP) is placed in its policy context of e-Europe and the Rolling Standardisation Action Plan and the balance of its content is briefly described.

1.1 THE e-EUROPE INITIATIVE

The e-Europe initiative (e-Europe 2002 - An Information Society for All), was adopted by the European Council of Ministers at their June 2000 meeting in Portugal with the aim of accelerating Europe’s transition towards a knowledge based economy and to realise the potential benefits of higher growth, more jobs and better access for all citizens to the new services of the information age. The initiative intends to accelerate positive change in the EU and is a key element in the strategy for modernising the European economy. It aims to secure equal access by all of Europe's citizens, to promote computer literacy and to create a partnership environment between the users and providers of systems, based on trust and enterprise. The ultimate goal is to bring everyone in Europe on-line as quickly as possible. The first phase of eEurope was the eEurope 2002 Action Plan1 which comprised a total of 64 targets under 11 action catogories in support of 3 main policy aims.

e-Europe Action Plan: Policy aims

Action category

e-Europe Action Line e-Europe description of the actions

Cheaper, faster, more secure internet

1a 1b 1c

Cheaper Internet e-Research e-Security

Cheaper & faster internet access Faster internet for researchers & students Secure networks & smartcards

Investing in people, skills

2a 2b 2c

e-Learning e-Working e-Accessibility

European youth into the digital age Working in the knowledge-based economy Participation for all in the knowledge-based economy

Stimulate the use of the internet

3a 3b

3c 3d 3e

e-Commerce e-Government e-Health e-Content e-Transport

Accelerating e-commerce Government on-line: electronic access to public services Health on-line European digital content for global networks Intelligent transport systems

For each action category, there have been initiatives to benchmark the progress being made, usually through Flash Eurobarometer surveys:

e-Europe Action Lines Benchmarking results listed for 2001 and 2002 Cheaper internet Internet penetration and access

1 http://europa.eu.int/information_society/eeurope/2002/action_plan/pdf/actionplan_en.pdf

The European Evaluation Consortium (TEEC) 1


e-Europe Action Lines Benchmarking results listed for 2001 and 2002 Internet access costs

e-research Core speeds for researchers and students e-security Secure servers

% of internet users experiencing problems: - security, viruses, credit cards, spamming

e-education (or learning)

No of internet PCs in schools % of schools connected

e-working Study on computer training and work e-accessibility e-commerce % of internet and companies buying and selling on line e-government On line availability of government services

% of users using Govt web-sites

e-health Transfer of patient identifiable data Types of patient data transferred

e-content e-transport

Source: http://europa.eu.int/information_society/eeurope/2002/benchmarking/list/2001/index_en.htm The majority of the targets were reported to have been successfully completed and in June 2002 the European Council launched a second phase, eEurope 2005, which focuses on exploiting broadband technologies to deliver online services in both the public and private sector. e-Europe is not a public expenditure programme and does not make new funds available. It rather aims to provide a policy framework within which existing expenditure can be better focused and to accelerate the adoption of relevant legislation. The eEurope 2005 Action Plan proposes a mid-term review to ensure that its actions are appropriate to an enlarged EU. The intention of this review is not to limit the mid-term review to enlargement issues but to open a debate on eEurope goals and implementation to give all stakeholders the opportunity to contribute. The mid-term review, foreseen for the Spring European Council in March 2004, should be a stocktaking exercise making proposals for possible adjustments in the eEurope Action Plan and will be based on results from a wide consultation process.

1.2 e-EUROPE AND THE ROLLING STANDARDISATION ACTION PLAN

In response to an invitation by Commissioner Liikanen, also in June 2000, the European Standardisation Organisations (CEN and ETSI) presented a Rolling Standardisation Action plan in support of the eEurope initiative, and Enterprise DG with other Commission services accepted to support.

The objective of this market-driven initiative is to better streamline the European ICT standardisation work towards the targets of the eEurope initiative, to accelerate the implementation of the standardisation work programmes and to ensure the involvement of all relevant stakeholders in the standardisation process. DG Enterprise decided to financially contribute to the initiative for the years 2000-2002, with an overall budget of € 12.5 million.



The main rationale for the financial contribution is to promote standardisation work in support of public interest in areas such as: Design for all and assistive technologies, e-Learning, secure and rapid access to Internet, smart cards, electronic signatures and road transport applications. The expected deliverables in these areas are European standards, ETSI standards and specifications, CEN workshop agreements and other forms of consensus documents. These standards shall be used, as widely as possible, in support of the implementation of European Directives or specific European policy initiatives, such as eLearning, the European Smart Card Initiative and other well defined policy objectives. To fully reap the potential benefits of voluntary standards in support of public interest, a close liaison between standardisation and European policy initiatives needs to be established. DG ENTR will ensure that the results of the Rolling Standardisation Action Plan will be fully taken into account by other European policies and initiatives, either through legal recognition or informally through recommendations by the Commission and Member States. In this respect, the Rolling Standardisation Action plan also contributes to the practical implementation of the concept of co-regulation.

Another priority is to better involve the users into European standardisation, from the definition of the standardisation work programmes and their implementation up to a wider dissemination of the results. Under the contracts signed, strong emphasis has been given to open hearings in preparation of future standardisation work, the dissemination of standards and, as a new element, interoperability test beds. European standardisation in support of eEurope should be based as widely as possible on open, transparent and inclusive processes. In conclusion the Commission’s support to the Rolling Standardisation Action Plan contributes to a further modernisation of European standardisation. As a result, the ICT standardisation activities have been streamlined towards the aims of the eEurope initiative and more emphasis will be given at all stages to the involvement of users. The Rolling Action plan was produced in July 2001 and outlines the commitment of the European Standardisation Organisations (ESOs) to respond to the challenge faced, providing details of activities performed or underway. The reader is recommended to make use of the Action Line information to obtain details on specific standardization activity. In July 2002, the ESOs together with partners set up a revised common Rolling Action Plan, which provides a 3-year work plan in support of the new priorities, i.e. "the widespread availability and use of broadband networks throughout the Union by 2005 and the development of Internet Protocol Ipv6 …. and the security of networks and information, eGovernment, eLearning, eHealth and eBusiness". It is important to note that the Rolling Action Plan



contains actions that are relevant to e-Europe but beyond the scope of this evaluation.

1.3 THE E-EUROPE STANDARDISATION ACTION PLAN

Standardisation is a major tool to achieve e-Europe policy aims for a number of reasons, such as the following: • It helps create a clear and predictable framework for e-Business. • It supports targeted activities to further foster the take up of ICT • It supports the integration of converging technologies • It promotes openness and involvement of representatives of society • It requires the co-operation of the stakeholders to find global solutions • It harnesses the expertise to required to address interoperability issues. The overall objectives of the Commission’s financial support to standardisation in general are as follows: 1. to streamline the European ICT standardisation work towards the targets

of the e-Europe initiative and to accelerate the implementation of the standardisation work programmes;

2. to promote the participation of all relevant stakeholders in the e-Europe

standardisation process, ensuring full transparency and inclusiveness; 3. to disseminate the results of e-Europe standardisation as widely as

possible through public events, information brochures and free of charge on-line access to the deliverables;

4. to encourage the use of e-Europe standards, thus helping to build an

Information Society for All. The e-Europe Standardisation Action Plan 2002 (eSAP), which is the subject of this interim evaluation, is a selection of actions from the Rolling Action Plan 2002 which have been funded by DG Enterprise through a series of contracts with the European Standardisation Organisations. The actions included in these contracts are listed in the eSAP “scoreboard”, which is presented in Annex 4.6 (Part III of Final Report). eSAP supports the implementation of e-Europe policy aims, more specifically through the implementation of the targets identified for the e-Europe action plan.

e-Europe policy aims

Action categories

Action domains

Budget (€)

Action lines (no.)

Cheaper, faster, more secure internet

1a, 1b, 1c

Cheaper internet, e-Research e-Security

2,970,600 19

Investing in people 2a, 2b, 2c e-Learning 2,882,314 34



e-Europe policy aims

Action categories

Action domains

Budget (€)

Action lines (no.)

and skills e-Working e-Accessibility

Stimulate the use of the internet

3a, 3b, 3c, 3d, 3e

e-Commerce e-Government e-Health e-Content e-Transport

6,827,954 72

Total 12,680,8682 125 Source: eSAP scoreboard and Annexes 3.5, 3.6

The higher figure of €12.7 million includes both Commission and EFTA contributions, the Commission contribution amounting to 95% and EFTA 5%. The Commission has earmarked an overall commitment of 12.5 M Euro to eSAP and the overall real cost is clearly lower than this commitment. There are differences in the way in which action lines are reported. In the eSAP score board, it is understood that all dissemination action lines are merged into one, hence the lower figure (125) than the 140 separate actions reported in GEFINOR elsewhere. As the eSAP scoreboard figures are for internal use only and include both European Commission and EFTA funding contributions, as well as the other differences, it is more accurate to describe the eSAP budget commitment as €12.5 million and the number of action lines as 140. Many types of actions have been funded, including sponsorship of ESO activity such as strategy development; pre-standardization activity such as events, open meetings and the preparation of reports; elaboration of relevant standards through CEN workshops, technical committees and ETSI specialist task forces. Further to actions in these domains, a range of promotion and dissemination actions have also been funded as part of eSAP. Some actions are in support of legal frameworks and some in support of policies. The three ESOs (CEN, CENELEC and ETSI) are the EU level institutional vehicles through which eSAP deliverables are elaborated and realised. The ESOs also assist the EC in giving advice and in reaching consensus with other stakeholders in relation the issues and problems that need to be tackled in the process. More about the role of the ESOs is found in Annex 3.1. The e-Europe Standardisation Action Plan 2002 (eSAP) has a €12.5 budget commitment from the Commission, is based on a Rolling Action plan which supports the policy aims of e-Europe, is described in detail through a scoreboard of action lines, is financed by the European Commission and delivered through ESO contracts. It includes around 140 separate action lines covering the 12 domains of e-Europe and supports its 3 policy aims.

2 eSAP scoreboard figures include both Commission and EFTA funding



2 THE STANDARDISATION CONTEXT

The e-Europe standardisation action plan has been implemented on the basis of the requirements, rules and processes as defined by the legal basis for European standardisation policy Directive 98/34, complemented by Council Decision 87/95 and by the Framework contract between the Commission and the ESOs and the annual infrastructure contract providing a financial contribution to the ESO's Infrastructure on a yearly basis. The eEurope standardisation, pre-standardisation and promotion actions are described by Order Vouchers (contracts), compliant with the approach described above and identifying additional generic principles such as: • Use of open consensus -building tools, non- discriminatory , inclusive,

open and transparent processes; • Information should be available on the e-Europe standardisation web site

and results of consensus decisions should be available for free download by interested parties.

• Each activity should be documented by a final report containing detailed

information on the execution of the work; • Due account shall be taken of related standardisation work at regional and

international level. An implementation strategy for each task described in the contract is agreed on the basis of a detailed technical annex proposed by the contractor and agreed by the Commission. A more detailed presentation of the wider legislative context for eSAP is given in Annex 3.3.

3 THE EVALUATION APPROACH

This chapter describes the context for the evaluation, its main objectives and the methodology that has been used. The terms of reference for the interim evaluation are in Annex 1. The e-Europe Standards Action Plan (eSAP) 2002, comes to an end in 2004 and the Commission wishes to undertake an interim evaluation before giving full and final approval to a successor plan, due to be effective from 2005. A fuller explanation of the context is included in Annex 1.1.



3.1 OBJECTIVES AND SCOPE

The evaluation takes place in the context of the new Financial Regulation3 as well as the need to demonstrate reasonable progress of the existing eSAP implementation before approving any new funding. The objective of this eSAP evaluation is to assess whether the initiative has helped to implement the e-Europe policy aims and, in particular, to contribute to the overall policy objectives of the Commission’s financial support to standardisation in general, and to eSAP in particular, which are as follows: • to streamline the European ICT standardisation work towards the targets

of the e-Europe initiative and to accelerate the implementation of the standardisation work programmes;

• to promote the participation of all relevant stakeholders in the e-Europe

standardisation process, ensuring full transparency and inclusiveness; • to disseminate the results of e-Europe standardisation as widely as

possible through public events, information brochures and free of charge on-line access to the deliverables;

• to encourage the use of e-Europe standards, thus helping to build an

Information Society for All. The final results of eSAP will need to be assessed against these objectives, however, full implementation has not yet been achieved and this present evaluation of the results is necessarily of an interim nature. Certain policy objectives (referred to as as benchmarking criteria in the tems of reference) have been agreed and these are summarised as follows for eSAP:

i. Promotion of public interest ii. More user involvement iii. Open and inclusive platforms iv. Free availability and use of e-Europe standardisation deliverables. Financial support can only be considered as successful if it can be demonstrated that the expectations included in these four benchmarking criteria are likely to be met. The interim evaluation of eSAP’s effectiveness was contracted to carry out the following tasks following validation of the relevance of the questions raised (see Annex 1.3): • Identification of a set of clear, transparent and relevant indicators and

benchmarks with the objective of assessing the financial support given to the e-Europe Standards Action Plan in an objective manner.

3 Commission Regulation No 2342/2002 of 23 December 2002 laying down the detailed rules for the implementation of Council Regulation (EC, Euratom) No 1605/2002 on the Financial Regulation applicable to the general budget of the European Communities



• Development of a clear concept to gather the necessary information on

the relevant indicators and benchmarks, taking into account the voluntary nature of standardisation and the general conditions of the contracts with the European Standards Organisations.

• Proposal of a methodology to measure stakeholder satisfaction about the responsiveness of the ICT standardisation process to their needs and requirements.

. • Proposal of general provisions to be included into future contracts with the

European Standardisation Organisations to facilitate the future evaluation of the results.

The final report of the interim evaluation addresses these tasks.

3.2 THE EVALUATION METHODOLODY

The methodology used in the evaluation is given in full in Annex 2 and the questionnaire used is given in Annex 5. The methodology is summarised as follows.

3.2.1 The scope of the evaluation The scope of the evaluation was defined as those projects that had been financed by the 2002 e-Europe Standardisation Action Plan. A full listing can be found in Annex 4.6.

3.2.2 The approach to the evaluation The three stages of the methodology are described in full in Annex 2 and summarised as follows: • The preparation stage began with a briefing meeting and familiarisation

by the evaluation team. Descriptive and evaluative analyses of the action lines making up the ESAP scoreboard were undertaken along with a validation of the relevance of the questions included in the terms of reference. The targets for questionnaire distribution were determined and, on the basis of agreed selection criteria, a total of 40 interviews were planned. Interviews with 15 stakeholder representatives and 25 action line participants were planned, the latter being grouped around a selection of 11 contracts around key action line themes and representing 20% of the ESAP budget. The contracts used for the case studies are presented in the table below.

Ref Dom Domain and description Status Contract Budget Year

Cheaper, faster, more secure Internet: Cheaper Internet (2 action lines in 1 case)

B5.1 1 Harmonization EP/Tiphon with UMTS/3GPP Ongoing ETSI 2001/02/3.1.1 350,000 2001

B 5.1 1 Architectural approach TIPHON Ongoing ETSI 2001/02/3.1.2 290,000 2001

Investing in people and skills: e-Accessibility

B 5.2 6 Work programme assistive tech Ongoing CEN 2000/39/1.2.4 32,500 2000



Ref Dom Domain and description Status Contract Budget Year

B 5.3 6

Speech recognition voice user interface : generic user command control and editing vocabulary for telecommunication products and services-main EU languages

Completed ETSI 2000 /02/3.4 245,000 2000

Stimulate the use of the Internet: e-Commerce

B 5.4 7a Signature environment and classes for electronic signatures Completed CEN 2000/42/1.2 166,042 2000

B 5.5 7a Standards for signature policies Planned ETSI 2002/496/3.3.3 76,000 2002

B 5.6 7c Tex-spin Ongoing CEN 2001/22/3.2.1 400,000 2001

B 5.7 7c EFNET 3 Ongoing CEN 2001/22/3.2.2 480,000 2001

Promotion, awareness, and pre-standardization

B 5.8 12 Digital TV Planned CLC 2002/497/4.1 25,000 2002

B 5.9 12 Organisation of Interoperability events Ongoing ETSI 2001/02/4.1 400,000 2001

B5.10 12 Promotion of ETSI e-Europe related activities (portfolio of existing deliverables) Ongoing ETSI 2001/02/4.5 100,000 2001

Case studies value 2,564,542 21%

Total contracts value 12,248,112 100%

Notes: The “Ref” numbers are the case study chapter headings in Appendix B. The “Total contracts value” is the commitment figures in GEFINOR as at end Sept 2003

The first two contracts in the list were studied as one case study, bringing the number of case studies to 10.

• During the field work stage, a questionnaire was used to glean the views

of all stakeholders about the progress of ESAP 2002. It included policy level questions (relating to the 4 main policy objectives of ESAP) and questions targeted at those who had (and had not) specific knowledge and involvement in the action lines. There were also a series of questions to those involved in the action lines, asking them to clarify their involvement in specific action lines and asking a separate list of questions for each of the 6 types of action lines:

Internal ESO actions Pre-standardisation: Events and meetings Pre-standardisation: Reports Standards elaboration Promotion The use and support of standards

A meeting took place with ESO participants to feed back and discuss some of the preliminary findings emerging from the questionnaires responses. Evidence collected during the field work also included the views of 15 stakeholder representatives and the analysis of the 10 case studies.

• The reporting during the project consisted of an inception report, after

initial familiarisation, followed by a progress report when the field work approach had been determined. Following the field work, an interim report was submitted, which included preliminary findings. The draft final report addresses the comments made by the Commission on the interim report, adjusts the findings to reflect additional evidence and includes preliminary conclusions and recommendations.

The experience of implementing the methodology chosen for the evaluation highlighted some of the problems relating to eSAP. It was




found that the descriptions used to identify different eSAP action lines were often not fully understood (particularly in the case of ETSI), even by those involved in the actions. No common terminology for categorising and referencing the action lines had been developed, presenting a potential difficulty for users in trying to understand what some action lines refer to. The final report presents the evaluation results (Chapter 4), analysis and interpretation (Chapter 5) and conclusions and recommendations (Chapter 6).

4 PRESENTATION OF THE EVALUATION RESULTS

The presentation of the evaluation results includes the summary results of both desk and field work, as well as the analysis of the case studies. The results are supported by the more detailed analysis of Appendix A.

4.1 RESULTS OF THE DESK BASED ANALYSES

4.1.1 Review of the implementation documentation The legal directives that underpin eSAP form a necessary background for the authentication, visibility and monitoring of eSAP. The fact that the eSAP action lines are embedded in the wider work programmes of the ESOs makes it difficult to identify and extract what impacts are due to eSAP itself and what are the results of other policy interventions.

4.1.2 Analysis of the eSAP documentation The information used to describe the content of eSAP has been extracted from data supplied by the Commission. The content of eSAP in complex and not easily accessible, but GEFINOR (the contract management system) gives a comprehensive financial view. The allocation of action lines to objectives is based on a judgement by the TEP team. The identification of the action lines covered by eSAP is difficult for those outside the European Commission and ESOs. The system of contract identification, the titles used to describe contracts and the way the contents of one contract link with the contents in another can be confusing for external parties.

4.1.3 Financial data on eSAP implementation The contract administration is complex and not very transparent as regards the total funding position for the ESOs as a whole. In practice, the ESOs pursue their standardisation priorities and different strands of EC funding are brought together under strategic themes, making it extremely difficult (and often impossible) to allocate different deliverables to different ESO contracts. Even within the eSAP contracts, action lines are spread across different projects. The method of showing the progress against invoicing deliverables in GEFINOR does not hold true for eSAP actions, other than the normal



standardisation process, so the timing of the invoicing at different stages, identified by GEFINOR, gives an unclear picture of the progress of other deliverables and of eSAP as a whole. Furthermore, the system of confirming the quality of deliverables is informal at best.

4.2 EVIDENCE FROM THE FIELD

The evidence from the field is based on 105 questionnaire returns. This is not a large number and the findings need to be seen as only one source of evidence for the evaluation. The analysis on which the results are based is presented in Appendix A3 (Part II: Final Report).

4.2.1 Views of all actors on eSAP progress against policy objectives The views of questionnaire respondents, differentiated by those who participate in action lines and those that do not is summarised as follows: • Overall the assessment of the contribution of eSAP to the 1st policy

objective (promotion of public interest) is relatively positive, but action line participants are more positive than non-participants and there are clearly some doubters. It is in the areas of stakeholder support, the realism of the timetables and the general progress of eSAP work where views are the least positive and the differences of viewpoint most marked.

• Respondents are neutral in their assessment of the contribution to the 2nd

policy objective (more user involvement), with action line participants holding by far the most positive viewpoint on this aspect. In fact those whose perceptions are at policy level are negative regarding this objective, suggesting that eSAP’s contribution to the 2nd policy objective is weak.

• Respondents are also neutral in their assessment of the contribution of

eSAP to the 3rd policy objective (open and inclusive platforms). The higher the rating given by action line participants, the greater the difference of viewpoint between them and non-participants; the latter having a more negative overall. All consider think that consumers and social partners do not see themselves as adequately represented in the standardisation process.

• The overall assessment of respondents to the 4th policy objective (free

availability and use of eSAP deliverables) is towards the positive, but with some contrasts that demonstrate that action-line participants are particularly convinced that deliverables are freely and widely available and non-participants not convinced that the action lines have been well implemented and communicated.

The assessment by questionnaire respondents is positive with regard to the contribution of eSAP to promotion of the public interest and towards the positive with regard to the free availability and use of eSAP deliverables. The evidence suggests that more action is needed to attain more user involvement and more open and inclusive platforms.



4.2.2 The differences: actors involved/not involved in action lines The fact that the perceptions of those not involved in the action lines is less positive than those who are which may just reflect that they are likely to give a more upbeat assessment (because they are involved). It could also mean that good eSAP work is not getting the visibility it deserves. The action line participants are very supportive of the eSAP process and appreciate the contribution it is making to the attainment of overall EU policy objectives and outcomes. However, they would like to see a more open, transparent and inclusive process and a greater dissemination effort.

4.2.3 Views of eSAP progress against policy objectives by stakeholder The views of questionnaire respondents, differentiated by four key groups of stakeholders (advisers, Govt & standards, ICT industry and user representatives), is summarised as follows: • In their assessment of the contribution or eSAP to the 1st policy objective,

all stakeholders are in agreement about the relevance of eSAP, that it includes additional standards work of public interest and has contributed to an acceleration of this work. The advisors are particularly positive about the relevance of eSAP and tend to be the most optimistic stakeholder group overall. Stakeholders are less convinced that the work is supported by the market players, that the timetables are realistic and the progress satisfactory.

• The analysis of stakeholder comments shows that there are concerns by

some individuals about the fact that it is the US that exploits the standards drawn up in the EU, that the current weakness of the ICT industry and the narrow motivations of some players is weakening the contribution of eSAP to the public interest and that the ISO forum may be a more appropriate forum for users to get engaged with. Whilst these are all minority views, they do serve to illustrate that there are pressures working against attainment of the 1st policy objective.

• In their neutral assessment of the contribution of eSAP to the 2nd policy

objective, advisors tend to be the more positive stakeholders, particularly with regard to the involvement of users in open public events, where other stakeholders hold a more negative viewpoint. It is only the advisors who think that open public events have managed to involve stakeholders; others do not agree fully with this statement.

• The interview comments show that there are factors working against more

user involvement, even though they seem to stay involved when they engage with the process. The lack of end user involvement is most marked, involvement is biased towards the larger players, complexity of the subject matter is an obstacle, as is the time and effort needed from busy users. Whilst these problems emerge from individual stakeholders, there is a sense in which the impact on the 2nd policy objective is threatened by such factors.



• The overall assessment of the contribution of eSAP to the 3rd policy objective also tends towards the neutral, although the there is a measure of agreement that awareness of the actions has encouraged additional activity and that procedures are seen to be open and transparent. Advisers are neutral regarding the representation of consumers and social partners, but other stakeholders appear to agree that more needs to be done.

• A general problem of low awareness of the broad canvass that eSAP

represents emerges from the individual comments of some stakeholders, even though the action lines are open, transparent and accessible. There appears to be room to be more promotion of the value of a wider engagement with the standardisation process and customisation to the needs of certain groups, particularly consumers, to underpin the 3rd policy objective.

• In their assessment of the contribution of eSAP to the 4th policy objective,

stakeholder respondents score between neutral and positive on statements about the availability of deliverables, the value of the website, the usefulness of the outputs and participant involvement in promoting the use of these outputs. However, they are neutral in their scoring of the statement “The ESAP action lines are perceived to have been well implemented and communicated”, with user groups being the least convinced of all.

• Much of the individual comment focuses on the www.e-

europestandards.org website which is seen as in need of attention by government and standards organisations, as well as other stakeholders. The challenge of poor eSAP visibility in the eyes of users, allied to the centrality of good web site performance in modern communications, suggests that a more proactive approach needs to be taken in managing this site if the 4th policy objective is to be seriously addressed.

The assessment by the different stakeholder respondents brings out the fact that work may not be sufficiently supported by the market players, that timetables may not be realistic and that the progress is not always satisfactory. Advisers tend to be the most positive stakeholder group, often contrasting with the more neutral views of the others who, for instance, appear to suggest that more needs to be done in terms of the representation of consumers and social partners. User groups are the least convinced that eSAP action lines have been well implemented and communicated. The individual comments of different stakeholders suggests that there are different pressures working to undermine the attainment of the different policy objectives. Whilst these can only be attributed to one or two individual players in some cases, they serve to illustrate that more may need to be done to deliver a final impact from existing eSAP deliverables.



4.2.4 The views of the European level standards institutions The views of the ESOs and the one or two national bodies are summarised as follows: • The ESOs are supportive of eSAP and appreciate the strategic direction

and framework this has brought to standardisation related activities within the e-Europe initiative. It is clear that delivering the impacts more rapidly and making the benefits visible to potential beneficiaries will remain a major challenge and there is room to work with the EC in a more “joined up” way to attain these desirable goals.

• Whilst ESOs are clearly experienced in EC procedures and have learnt to

cope with these adequately, the new FR promises to bring new demands to contractors that, if poorly implemented, could adversely affect the willingness of actors to remain inside the process in an increasingly innovative ICT sector.

• The evidence suggests that it is not easy to differentiate the activities

within eSAP from other activities being undertaken by the ESOs, that the operations of the 3 ESOs are quite different and the speed at which they can operate varies significantly. It point to a differentiated approach to the choice of indicators for both the process and the impacts.

• It is likely that the choice of appropriate indicators for capturing the

progress of the eSAP process and its impact will be controversial. Added value offers a potential route for determining the contribution of different types of actions, but it may be necessary to resort to softer options, such as peer reviews and user surveys, to define indicators that show to what extent the eSAP strategy is working.

The ESOs are keen to see a more joined up approach by the EC, with a more speedy and more visible implementation. Efficient implementation of the new FR is crucial for ESOs and the evidence suggests that a differentiated approach to the choice of indicators for the 3 organisations would be welcome. Added value indicators could be a way forward, but it is likely that eSAP will need to resort to “softer” measures to see if the eSAP strategy and process is working.

4.2.5 The views of those within the European Commission The views and perceptions of those within the EC are summarised as follows: • There is a need for a revision of the legal basis to bring it in line with the

new financial regulations and also to (re-)establish an effective Member States consultation and feedback mechanism, such as SOGITS (Senior Officials Group for Standardisation in the field of Information Technology). There is also a need to update the legal basis to bring it in line with the evolving standardisation context. In particular new deliverables, such as CWAs, are not mentioned in the current legal basis; also interoperability test beds, as offered by ETSI, are understood to be in a "grey zone".



With regard to the contribution to eSAP policy objectives: • Objective 1: The promotion of public interest

There are clearly some very good examples of meeting the public interest objective in eSAP, but progress can be slow to underpin inclusiveness and the involvement of some big market players can dilute the attainment of this objective in certain areas. Commission action is still needed to steer the process and to stimulate Member State support.

• Objective 2: More user involvement

Business and the public authorities clearly emerge as the most involved user groups. However, SMEs are not often aware of the importance of standardisation and consumer groups are struggling to remain engaged with the ICT standardisation world. There may be room for a more two-pronged approach to effective consumer involvement in eSAP, through both ANEC and the national consumer organisations at the same time.

• Objective 3: Open and inclusive platforms

Whilst there is a real effort to get all types of actors to engage with the different platforms created, the complexity of some issues and the dominance of market players in others create practical barriers to the achievement of this policy objective which will be difficult to overcome.

• Objective 4: Free availability and use of e-standards

The situation is moving slowly towards the free availability of eSAP deliverables and the main website is seen to be a good tool for those who can engage with eSAP through this means. The EC needs to strengthen its internal communications to reduce the danger of duplication and other parties (e.g. Member States) may need encouragement to assist with accessibility to standards at local levels.

Regarding management organisation, processes and procedures:

• Inter-service collaboration is one of the more important challenges in the management of eSAP which, whilst being driven by legislation, may be being hampered by the agendas and priorities of the different DGs, which do not necessarily have standardisation as a top priority.

• The contractual process appears to be soundly based in terms of the

linkage being made between funding and deliverables, even if it is slow in its implementation. However, the new FR presents a challenge in that is might lead to a much higher level of frustration amongst industry actors.

• There is clearly an interest by many DGs to be fully engaged with the

standardisation process and an expectation that the number of challenges ahead will increase and be interesting and important to address. This could be an opportunity for a more proactive top-down approach by the EC.



• DG Enterprise already has experience of a number of performance indicators, ranging from embedded management controls to a wider range of measures, proxies and ideas for monitoring the process and the product. These provide a useful source of ideas when considering appropriate indicators for the future.

Looking at eSAP from an EC viewpoint, it may be necessary to revisit the legal base for some projects and certain actions could help to strengthen impacts, such as negotiations with Member States to stimulate more local support, to bring in more national consumer groups, to take a common position with dominant market players and to work towards better eSAP accessibility at national level. Within the EC, inter-service collaboration needs constant attention as officials move jobs, new initiatives are taken and future challenges identified. A “joined-up” top-down approach in certain fields would be helpful. The EC is a useful source of ideas when considering appropriate indicators for the future.

4.2.6 The case studies The full description and analysis of the case studies is found in Appendix B (Part II: Final Report). The case study reports are based on the considered views of the evaluation team, having familiarised themselves with the content of the particular cases chosen and having met a small selection of actors involved in the actions. Sometimes the evidence is anecdotal or limited to the evidence of a small number of participants. The reports therefore include interpretation by those undertaking the analysis and interviews on which the evidence is based. It has not been part of the methodology to check the statements of participants and other interested parties nor to audit the data and information presented. Where possible, views are compared across different components of the overall eSAP initiative to come to a considered view. Individual comments should therefore not be given a significance beyond the context of the case study itself. The sources of information used are described in each of the case studies. As regards the different domains, the findings are summarised as follows: • The contribution of the TIPHON project to a cheaper internet will be felt

through the monitoring of internet pricing trends, as long as other influencing factors (e.g. demand and supply) can be eliminated. In the meantime, its contribution to this desirable impact is being diminished by the poor representation of consumers and SME users.

• There is a lack of involvement of consumer and social groups in the case

of the e-Accessibility projects studied. However, these projects are considered to be highly relevant and have led to some new strands of networking between key actors and some useful recommendations. This



domain is seen to fit well with e-Europe policy and the wider involvement of final beneficiary representatives is very desirable.

• All 4 e-Commerce projects are seen to be relevant to policy objectives

and have resulted in strong backing from users, who see e-Security as a key issue. There are problems of communication in these projects due to their complexity and there could be a problem in maintaining interest in the interoperability project, which has yet to promise an impact. The uptake of e-Signature standards has been promising, but key industry stakeholders need to be more engaged to have real impact in the WS/E-Sign project.

• Whilst interest in Digital TV pre-standardisation is high, take up has been

slow and the dissemination of results is seen to be weak. The response to interoperability promotion has been encouraging and its impact is growing. User participation in promotional initiatives is not strong and website use is still limited, suggesting that awareness remains low. However, processes are seen to be transparent and impacts need to be judged in the longer term.

A common theme across the domains represented by the case studies is that, although the projects chosen are relevant, inadequate user and/or industry representation and low final beneficiary awareness are undermining the contribution of the projects to the policy objectives. The analysis of comments and suggestions from those interviewed during the case studies is summarised as follows: DOMAIN: CHEAPER INTERNET • ETSI is seen to be doing a good job by TIPHON participants, but

mechanisms to bring in other STF resources could be more flexible. The Commission could be more proactive and committed in following up the project work, such as through an annual open meeting to further project goals.

DOMAIN: E-ACCESSIBILITY • Limited project impact reflects the limited scope and objectives of the e-

Accessibility project and increasing user involvement is not specifically addressed. Although there was no event on this topic, RTD projects on ICT assistive technologies were reviewed by two experts.

• The need for increased communication features highly in the SPEECH

recognition project. More positive and constructive feedback is requested of the EC, more co-ordination between the specific ETSI task force and the Commission, more communication between policy makers and implementers, and increased representation of consumer groups on the ETSI board are considered to be desirable. The need for standards to



be regulated by legislation, and increased funding for the participation of user and consumer groups were also suggested.

DOMAIN: E-COMMERCE • WS/E-Sign must be evaluated together with other related projects. There

has been significant human and financial investment, as well as efforts to ensure international visibility, and some public access.

• E-signature demonstrates how problems with standards are only

apparent at the implementation phase. Therefore, there is a need for increased feedback from the market during the project development. Furthermore, the EC would be better able to evaluate standards by checking the applicability of working group results.

• TEX-SPIN objectives have been met, but stakeholders remain somewhat

dissatisfied with its impact. • Although it is not always possible to gain the consensus of stakeholders in

the EFNET3 project, there is general satisfaction with progress. Increased results’ promotion is required.

DOMAIN: PROMOTION, AWARENESS AND PRE-STANDARISATION • With its potential for helping to achieve the Information Society, Digital TV

fits well with the objectives of the Barcelona Council of 2002. Benefits to standardisation in this sector include more efficient use of spectrum and lower transmission costs, which should enhance competition. Although the project is well received by stakeholders, CENELEC’s dissemination strategy needs to be better defined and more proactively pursued.

• For INTEROPERABILITY it is suggested that more flexibility is required

from the EC in the definition of costs for the organisation of events. One possible solution would be the provision of a generic list of authorised cost headings, within a clearly defined framework. Tests for other protocols are also suggested.

• For promotional activities it is suggested that funding is effective where

there is a concrete goal to be achieved. The Council initiative provides an effective framework for projects, which have demonstrated a high level of success. Slow project administration is problematic and is feared to reduce industry interest.

Common themes in the suggestions emerging from the case studies is more flexibility in project design and implementation, a more proactive approach to stimulating follow up after the production of project results and better consumer representation in standardisation institutional and committee infrastructures.



The general findings emerging from the case studies are summarised as follows: • The response to ESAP policy objectives

The evidence suggests that the cases studied are appropriate and relevant to the eSAP policy objectives and some have been particularly well received in terms of their value to different stakeholders and the openness of the approach. However, user groups could be better represented, particularly consumer groups, and there is evidence that lack of resources have hampered progress in some projects.

• Stakeholder views about attaining policy objectives

There is evidence of a high level of satisfaction in some projects, with networking and learning opportunities for participants. However, attracting a wider range of user groups is sometimes difficult, particularly where the project is complex and difficult to communicate. More promotion and better communications will be needed to overcome deficits in targeted stakeholder engagement.

• Capturing benefits and impacts

A wide diversity of possible indicators emerges through the case studies. One common view is that impacts can only be measured by the uptake of standards in the longer term. At the same time, some useful process indicators have been identified that are relevant to monitoring the degree to which eSAP is delivering against the policy objectives.

The evidence from the case studies serves to reinforce the views that emerge from the analysis of the questionnaires and interviews. The lack of user group representation (particularly consumers) was noted in several cases and the need for better communications seen to be necessary to overcome deficits in targeted stakeholder engagement with eSAP. The case studies highlight some additional possibilities for improving the impact of eSAP: more flexibility in project design and implementation and a more proactive approach to stimulating follow up after the production of project results.

4.3 EVALUATION AGAINST THE MAIN QUESTIONS

This section summarises the examination made of the questions posed in the original terms of reference as a way of checking the overall results emerging from the interim evaluation. The more detailed analysis is found in Appendix C (Part II, Final Report). • In general, the assessment is positive with regard to the contribution of

eSAP to promotion of the public interest. It is relevant to stakeholders and has speeded up the delivery of new standards in the public interest. At the same time a more joined up approach, wider market support and more speedy implementation are needed for impact.



• The evidence suggests that more dissemination action is needed to attain a greater user involvement, particularly in the case of consumers. Indicators are potentially available for the measurement of specific deliverables, and GEFINOR could provide the framework for some tangible targets, but measures are not in a readily accessible form for management purposes.

• The evidence suggests that more action is needed to attain more open

and inclusive platforms. One challenge is to engage a wide range of users and consumers in areas which are hidden and of high complexity, where the delivery of standards could have a substantial impact (e.g. interoperability).

• In general, the assessment is positive with regard to the contribution of

eSAP to the free availability and use of e-Europe standardisation deliverables and web site access and e-availability is making this more of a reality each day. Initiatives to promote the use of standarisatioin deliverables at Member State levels would help the situation.

Attempts to deliver tangible indicators that reflect the achievements of eSAP to date are not easy to find. There is no systematic procedure for gathering the kind of hard information that is needed to measure specific deliverables and there is little that is readily accessible

5 ANALYSIS AND INTERPRETATION

This chapter formulates a conceptual framework for eSAP, differentiating between the external policy impacts it is seeking to attain and the internal processes through which it needs to support and deliver the policy. It introduces some possible indicators to measure performance in these different contexts, thereby setting the stage for the final conclusions and recommendations. In the terms of reference, the evaluation is tasked with making a “proposal of general provisions to be included into future contracts with the European Standardisation Organisations to facilitate the future evaluation of the results.” It is mainly with this in mind that the section below is developed


Table A: The intervention logic of eSAP

INTERVENTION LOGIC

DIFFERENT COMPONENTS MAKING UP THE POLICY OBJECTIVES

EVIDENCE NEEDED TO EVALUATE POLICY IMPACTS

RELEVANT SOURCES OF EVIDENCE AND RELEVANT INDICATORS

1. POLICY INTERVENTION

FOCUS: to streamline the European ICT standardisation work towards the targets of the eEurope initiative and

eSAP funded action lines have contributed to the focusing of ESO activities towards eEurope targets.

Internal: Senior ESO officials: • Change in the direction of ESO work programmes; • Additionality of eSAP action lines to existing work.

SPEED: to accelerate the implementation of the standardisation work programmes;

eSAP funded action lines have speeded up the process of delivering eStandards.

Internal: eSAP action line participants in ESOs: • Progress in delivering actual standards; • Extent to which process is faster.

2. STAKEHOLDER ENGAGEMENT

INVOLVEMENT: to promote the participation of all relevant stakeholders in the eEurope standardisation process,

The relevant stakeholders have got engaged in standards being supported by the eSAP funded action lines.

Internal: ESO members and representatives: • Actual penetration by potential stakeholder groups in eSAP

actions; • Record of participation in activities

TRANSPARENCY: ensuring full transparency and

The process of delivering these standards is visible to all interested parties.

External: Associates: • Awareness of eSAP action line initiatives; • Access to eSAP action line proceedings.

INCLUSIVITY: inclusiveness; The stakeholder parties involved are representative of a wide section of relevant interests

Associates: • Balance of representation at eSAP action line initiatives;

3. RESULTS DISSEMINATION

DISSEMINATE: to disseminate the results of eEurope standardisation as widely as possible through

Stakeholders are committed to the widest possible dissemination of the results from eSAP funded action line work.

Internal: ESO members and representatives: • Progress of dissemination actions within eSAP action lines. • Relevance of target audiences for dissemination activities.

MEANS OF DISSEMINATION: • public events, • information brochures and • free of charge on-line access to

the deliverables

Stakeholders are using these particular means to disseminate results from eSAP funded action line work.

Data from records of eSAP dissemination action lines: • Volume of means of dissemination • Satisfaction indicators from events and periodicals.

3. USER APPLICATION

PROMOTE: to encourage the use of eEurope standards, thus helping

Stakeholders are active in promoting the use of the emerging eEurope standards

External: User groups: • Perceptions of potential applications • Examples of positive experiences • Perceptions of contribution of eSAP action lines.

FINAL GOAL: to build an Information Society for All

The widest possible applications are being sought for the use of the eEurope standards

Not included in this assessment



5.1 A CONCEPTUAL FRAMEWORK FOR eSAP

In considering the current and future performance of eSAP, two elements need to be considered: the impact of the policy and the efficacy of the process. These two aspects are addressed below.

5.1.1 A framework for the measuring policy impacts During the course of the evaluation, the intervention logic of eSAP has been examined and this is presented in Table A above. It takes the different components of the eSAP policy objectives and shows how they express the four main components of the intervention logic: • Policy intervention • Stakeholder engagement • Results dissemination • User application The kind of evidence needed to evaluate policy impacts is then presented and, in the final column of Table A, potentially relevant sources of evidence and indicators are identified. Very limited evidence has been found during the evaluation of the capture of information upon which indicators might be built. Examples of existing indicators are found in Annex 7: • Stakeholder engagement: Workshop creation • Stakeholder engagement: Meeting attendance • Results dissemination: CWA downloads These are focussed on CEN, rather than the other ESOs, but give some idea of the kind of indicators that are currently available.

5.1.2 A framework for measuring process efficacy Supporting the delivery of the impacts is the management process surrounding eSAP and this requires different concepts to be introduced. Assuming that the goal is a high quality of administration efficacy, including effectiveness and efficiency in this concept, a Total Quality Management approach is a good place to start. Based on a TQM approach, the following kinds of indicator choices were presented and tested during the course of the evaluation, both during the interviews and during a meeting with the ESOs:

Indicator Possible description or definition Funding efficiency & relevance score

Rate the relevance of each action line to e-Europe policy objective (0-1) and multiply by the budget allocated. The sum divided by total budget available would be a measure of ongoing policy relevance.

Expected benefits Introduce an estimate of benefits expected (tangible & intangible) into the contracts. This could cover time related



Indicator Possible description or definition impacts, numbers of people/organisations reached etc. and give a reference point for future evaluations.

Value added Estimated benefits less budgeted cost Record of time acceleration expected

Would need to get estimates of time to achieve deliverable without funding to compare with contracted agreements; some actions may only occur with funding, others may be accelerated, some may have occurred just the same anyway.

Number of contract amendments

The target for these indicators would be zero.

Delivery measure Number of actions early, on time or late. Interviewees in case studies generally believed that they could rate the TQM elements in the table above and there was limited positive feedback from the ESOs. However, the way forward is not simple. • The whole issue of measures and indicators of process effectiveness is

a difficult one due to the intangible nature of many outputs (e.g. EN dvB4 ETSI is the most widely accessed or used standard but there is no evidence about how it is used). Question arises as to the acceptability of funding work through trusted organisations that are perceived to be effective overall, versus a system of detailed measures linked to deliverables. There is also the concern the work load created in establishing and tracking such measures.

• Some agreement that measures could be devised that are tailored to

each contractual situation and, in this case, the TQM model could be useful. However, this would require some further study, discussion with the ESO’s and perhaps a few trials. It has been noted that national and European initiatives may be inter-connected and there may be long term macro benefits, so the concept of European level benefit or value-added may be difficult to establish.

• Clearly some concerns exist about the impact of the new Financial

Regulations and the need to take account of the voluntary process time that is incorporated into delivering eSAP outputs. Also, ESO work is occurring in response to e-Europe that is not funded by the Commission.

• Participants already feel that the process of funding is too rigid and often

too slow; greater flexibility would be appreciated and discussions with the ESOs about how this can be achieved may provide room to negotiate the introduction of some process measures. In the absence of these, DG Enterprise can simply establish from its own records if the contracted deliverables have been completed or not. The impact of the policies is then subject to a separate evaluation but would need to focus on measuring perceptions of the various stakeholders.

• The importance of properly defining the constituency was highlighted, if

an action line is to be supported, and any required measures would need to be agreed at the contract stage, although a heavy administration load is to be avoided.



It is concluded that it is possible to formulate a conceptual framework within which to define desirable policy impacts and determine possible indicators and sources of evidence for eSAP. However, this needs to be distinct from the measurement of the performance of the supporting management process, which needs its own framework and distinctive indicators.

5.2 RELEVANT INDICATORS AND BENCHMARKS

The policy impact of the present eSAP and the accompanying management process indicators are addressed separately. In addition, the possibility of building in indicators into the next eSAP at the ex-ante stage is addressed.

5.2.1 Selection of areas for impact indicators: the current eSAP Taking the earlier list of relevant indicators as a starting point, the following table highlights 5 areas in bold that would appear to represent the best value for money as policy impact indicators for the current eSAP.

INTERVENTION LOGIC

RELEVANT SOURCES OF EVIDENCE AND RELEVANT INDICATORS

1. POLICY INTERVENTION

• Change in the direction of ESO work programmes; • Additionality of eSAP action lines to existing work.

• Progress in delivering actual standards; • Extent to which process is faster.

2. STAKEHOLDER ENGAGEMENT

• Actual penetration by potential stakeholder groups in eSAP actions; • Record of participation in activities

• Awareness of eSAP action line initiatives; • Access to eSAP action line proceedings.

• Balance of representation at eSAP action line initiatives; 3. RESULTS DISSEMINATION

• Progress of dissemination actions within eSAP action lines. • Relevance of target audiences for dissemination activities.

• Volume of means of dissemination • Satisfaction indicators from events and periodicals.

3. USER APPLICATION

• Perceptions of potential applications • Examples of positive experiences • Perceptions of contribution of eSAP action lines.

The overall approach is based on securing a limited but balanced cluster of the most tangible evidence available, even where this may not be an ideal solution. This makes it more likely that the capture of the evidence will be inexpensive and that the indicators will be self-evident. The first four indicators should be obtainable through the ESOs; the most difficult will be the 5th and last (perceptions of contribution of eSAP action lines), which will require a regular special survey of user views.

5.2.2 Selection of areas for process indicators: ongoing eSAP It is proposed that the selection of relevant process indicators follows a management approach and is built into the regular management reporting. It is proposed that the main issues for the eSAP management process are: • The speed of delivery of the ESO contracts so that work can start; • The quality of the contracts, expressed by avoiding amendments;



• The cost of delivery of the policy impacts. The original list of process indicators can be divided into those that relate to the “ex-ante” stage (e.g. relevance of the action line) and those which reflect management processes. The table below introduces a set of possible process indicators:

Indicator Proposed definition Proposed source of evidence Delivery speed


Record of flow of documentation between Dir B and Dir G in DG ENTR and between DG ENTR and other DGs involved

Contract quality


Monitor through the progress of service level agreements (SLAs) between different departments in and outside DG ENTR

Cost of delivery


ABB or IRMS within DG ENTR to show actual resources against targeted resources in delivery of key domains

The pre-conditions for the proposed approach are gradually being introduced in the Commission (TQM, ABB, IRMS, etc.), permitting more accountability to be built into DG ENTR management systems.

5.2.3 Potential ex-ante indicators: a future eSAP In addition to the indicators already suggested, it is proposed that the Commission build in some indicators “ex-ante” against which a future impact eSAP can be evaluated. This will help to develop a more sophisticated performance measurement system and emerge from the thinking on TQM already introduced above. It is proposed that the focus for ex-ante indicators should be: • The relevance of the action lines to the policy; • The expected results through implementation of the action lines; • The resulting impacts from delivering these results. The following table goes into greater detail:

Indicator Ex-ante definition Evaluation task Policy relevance


Rate the relevance of each action line to e-Europe policy aims (0-10) multiplied by a rating of the extent of the expected impact (0-10). The combined rating (0-100) would be a measure of ongoing policy relevance.

Expected results


Capture the evidence related to actual results and compare these with those targeted. The results need to capture the different stages of the intervention logic of the new eSAP.

Desired impacts


Carry out a survey amongst targeted user stakeholders to ascertain the degree to which there has been a contribution.



The new eSAP presents an opportunity to build in all these three sets of indicators at the ex-ante stage and be better prepared to undertaken evaluation tasks as the implementation progresses. It is concluded that a set of policy indicators and benchmarks that are relevant to the current eSAP programme needs to be representative of the different stages of the intervention logic, be based on tangible and accessible evidence and reflect what is likely to be available. Management process indicators need to be structured around key issues such as the speed, quality and cost of delivery of the policy. The developing management context within the Commission permits some new approaches to be harnessed, such as Total Quality Management (TQM), Activity Based Budgeting (ABB) and Integrated Resource Management Systems (IRMS) in support of process indicator development. The envisaged successor to the present eSAP programme presents an opportunity to build in ex-ante indicators that cover the relevance of the action lines, the expected results and the resulting impacts. This will make the evaluation of the impact of a future eSAP more meaningful. It would also be possible to test their feasibility retrospectively on the current eSAP programme.

6 CONCLUSIONS AND RECOMMENDATIONS

The conclusions summarise the paragraphs in bold from the preceding text. The recommendations draw on these to propose some approaches to enhancing the performance of eSAP in the future.

6.1 CONCLUSIONS OF THE EVALUATION

The conclusions cover the eSAP process, findings from the questionnaires and interviews, the evidence of the case studies and possible approaches to evaluation in the future. These represent the considered judgements of the evaluation team on the basis the analysis and interpretation of the evidence by the evaluation team.

6.1.1 Conclusions from the analysis of stakeholders views STAKEHOLDERS GIVE A POSITIVE ASSESSMENT OF ESAP’S CONTRIBUTION • Respondents as a whole give a positive assessment regarding the

contribution of eSAP to promotion of the public interest and with regard to the free availability and use of eSAP deliverables.



• Action line participants are very supportive of the eSAP process and appreciate the contribution it is making to the attainment of overall EU policy objectives and outcomes.

• Generally, the evaluation has highlighted the success of open events and

meetings, both pre- and post-standards elaboration. Interoperability events have generated a very positive response from participants and these can be seen to accelerate the standardisation cycle in ICT.

ESAP MAY BE BETTER ON STANDARDISATION DESIGN THAN ON IMPLEMENTATION • ESOs appear to believe that Europe is ahead of the USA in ICT

standardisation, helped by the eSAP funding initiative, which is relatively more speedy and pragmatic.

• At the same time, a few users stated that the ISO was more relevant to

them, working at a global level, and a few advisers think that, whilst the EU builds standards, the US is ahead on implementation.

• Some eSAP action lines need better implementation and communication. ESAP SOMETIMES SETS UNREALISTIC TARGETS AND THEN CANNOT KEEP TO THEM • Timetables are not always realistic and progress not always satisfactory. • The ESOs are keen to see a more joined up approach by the EC, with a

more speedy and more visible implementation. • Efficient implementation of the new FR is crucial for ESOs ESAP HAS PARTICIPATION DEFICITS, PARTICULARLY END USERS AND CONSUMERS • More action is needed to attain more user and social partner involvement

and more open and inclusive platforms. • eSAP work is not always sufficiently supported by key market players. • The www.e-europestandards.org website needs to be developed and

promoted and in such as way that it is more accessible ESAP NEEDS GREATER VISIBILITY AND A MORE PROACTIVE DISSEMINATION EFFORT • The perceptions of those not involved in action lines is less positive than

those who are. This may be just because they are involved, but could also mean that good eSAP work is not getting the visibility it deserves.

• A more open, transparent and inclusive process is called for my all

stakeholders, along with a greater dissemination effort. SUITABLE INDICATORS FOR ESAP MONITORING HAVE YET TO BE DEVELOPED • A differentiated approach to the choice of indicators for the 3 ESOs would

be welcome.



• eSAP will need to resort to “softer” indicators to see if the eSAP strategy and process is working.

• The Commission is a useful source of ideas when considering appropriate

indicators for the future. MEMBER STATES COULD HELP TO STRENGTHEN ESAP INVOLVEMENT AND IMPACTS • Member States are in a position to stimulate more local support. • National consumer groups could be encouraged to advertise the eSAP

process. • Member States are able to take a common position with dominant market

players and to work towards better eSAP accessibility at national level.

6.1.2 Conclusions relating to eSAP procedure ESAP LOGIC AND TERMINOLOGY COULD BE CLEARER, ESPECIALLY FOR END USERS • Descriptions used to identify different eSAP action lines are often not

understood, even by those involved in the actions; • No common terminology for categorising and referencing the action lines

has been developed or is in use. • The strategic significance of any particular action line being funded is not

clear, nor the contribution and interrelationship of projects to a final desired impact.

• The fact that eSAP action lines are embedded in the wider work

programmes of the ESOs makes it difficult to identify and extract what impacts are due to eSAP itself and what are the results of other policy interventions.

ESO CONTRACTS COULD EMBODY THE ESAP STRATEGIC LOGIC MORE ROBUSTLY • The allocation of action lines to objectives is not often clear in the contract

documentation. • The identification of the action lines covered by eSAP is difficult in terms of

contract identification, the titles used to describe contracts and the way the contents of one contract overlap with another.

• It is extremely difficult to allocate policy deliverables to different ESO

contracts. Even within the eSAP contracts, action lines are spread across different projects.

ESAP PROGRAMME & FINANCIAL MONITORING NEEDS MORE CUSTOMISATION



• GEFINOR data gives a useful and clear financial overview of the content of eSAP, but shows that only 64% of commitments have been invoiced to date.

• The contract administration is complex and not very transparent as

regards the total funding position for the ESOs as a whole. • The method of showing the progress against invoicing deliverables in

GEFINOR does not hold true for eSAP actions, other than the normal standardisation process.

• The timing of the invoicing at different stages, identified by GEFINOR,

gives an unclear picture of the progress of other deliverables and of eSAP as a whole.

ESAP ADMINISTRATION NEEDS TO BE INTEGRATED WITH RELATED PROCESSES • The system of confirming the quality of deliverables needs to be more

systematic. • Inter-service collaboration needs constant attention as officials move jobs,

new initiatives are taken and future challenges identified. Inter-service collaboration needs constant attention as officials move jobs, new initiatives are taken and future challenges identified.

• A “joined-up” top-down approach in certain fields would be helpful.

6.1.3 Conclusions from the analysis of case study results The evidence from the case studies serves to reinforce the views that emerge from the analysis of the questionnaires and interviews. RELEVANT PROJECTS ARE UNDERMINED BY LOW INVOLVEMENT AND AWARENESS • Inadequate user and/or industry representation is undermining eSAP. • Low final beneficiary awareness is also undermining eSAP’s contribution. • Better communications are needed to overcome deficits in targeted

stakeholder engagement with eSAP. A MORE FLEXIBLE AND PROACTIVE APPROACH COULD GIVE BETTER RESULTS • More flexibility in project design and implementation is requested. • Follow up after the production of project results needs to be more

proactive. THE REPRESENTATION OF USERS AND CONSUMERS NEEDS TO BE STRENGTHENED • There needs to be better user/consumer representation in standardisation

institutional and committee infrastructures.



• The www.e-europestandards.org website is not sufficiently oriented to targeted end user and consumer group needs.

6.1.4 Possible approaches to evaluating eSAP AN ESAP STRATEGY FRAMEWORK IS NEEDED TO DEVELOP INDICATORS • It is possible to formulate a conceptual framework within which to define

desirable policy impacts and determine possible indicators and sources of evidence for eSAP.

• A set of policy indicators and benchmarks that are relevant to the current

eSAP programme needs to be representative of the different stages of the intervention logic, be based on tangible and accessible evidence and reflect what is likely to be available.

• The envisaged successor to the present eSAP programme presents an

opportunity to build in ex-ante indicators that cover the relevance of the action lines, the expected results and the resulting impacts.

• Attempts to deliver tangible indicators that reflect the achievements of

eSAP to date are not easy to find. • There is no systematic procedure for gathering the kind of hard information

that is needed to measure specific deliverables and there is little that is readily accessible.

THE SUPPORTING MANAGEMENT PROCESS NEEDS ITS OWN SEPARATE MEASURES • The measurement of the performance of the supporting management

process needs its own framework and distinctive indicators. • Management process indicators need to be structured around key issues

such as the speed, quality and cost of delivery of the policy. • The developing management context within the Commission permits some

new approaches to be harnessed, such as score boards, TQM, ABB and IRMS in support of process indicator development.

6.2 RECOMMENDATIONS OF THE EVALUATION

It is recommended that 1. A strategic framework should be agreed for any successor programme

that shows the how policy intervention through stakeholder engagement lends itself to results dissemination and user application, bearing in mind the voluntary nature of standardisation.



2. A set of clear, transparent and relevant indicators and benchmarks are developed with the objective of assessing the financial support given to the e-Europe Standards Action Plan in an objective manner.

3. A set of 3 ex-ante indicators should be developed to reflect the intrinsic

intervention logic of the eSAP intervention:

Indicator Ex-ante definition The relevance of the action lines to the policy


The expected results through implementation of the action lines


The resulting impacts from delivering these results


4. At least 5 good value for money indicators are developed in more detail

for use with the current and future eSAP:

eSAP Policy Objectives Suggested indicators Policy intervention 1. Progress in delivering actual standards;. Stakeholder engagement 2. Record of participation in activities

3. Balance of representation at eSAP action line initiatives; Results dissemination 4. Summary table of dissemination actions taken against targets User application 5. Perceptions of contribution of eSAP action lines.

5. A differentiated approach is taken in the choice and negotiation of

suitable ex-ante and value for money (VFM) indicators eSAP for the 3 European Standardisation Organisations (ESOs) and that these are embedded into any future eSAP contracts with the ESOs.

6. A methodology is developed to measure stakeholder satisfaction about

the responsiveness of the ICT standardisation process to their needs and requirements with a view to looking at the costs and benefits of developing this as a “soft” indicator.

7. Management performance indicators are added to show the speed,

contract quality and cost of delivery in achieving eSAP deliverables.

Indicator Proposed definition Delivery speed


Contract quality


Cost of delivery


8. Statistical and financial monitoring systems are more closely aligned so

that indicators are built into contract monitoring and actual achievements can be compared with budgeted achievements.

9. A common terminology is used on which to base action line descriptors

that are grouped under understood domain themes, used for all stakeholders and are “user (and consumer) friendly”.



10. Scoreboard nomenclature aligns with, but is not different to, the e-Europe policy nomenclature to avoid another element of confusion.

11. Member States are encouraged to take a more proactive approach:

• to stimulate more user group involvement • to bring in more national consumer groups • to take a common position with dominant market players • to engender wider ownership of eSAP deliverables • to work towards better eSAP accessibility at national level.

12. The www.e-europestandards.org website should be developed and

promoted in a more proactive way by creating linkages to those web sites used more frequently by targeted end user and consumer groups and through using the right search engines.

13. There is more follow through on progress within action lines by the

Commission, especially in showing interest in the outcomes and helping to promote the benefits of these.


the european evaluation consortium (teec)elibrary.lt/resursai/eu integracija/eu knowledge... · the...

Documents