IEE Project Performance Indicators Final report – Executive summary

Report for EACI Ricardo-AEA/R/ED57321 Issue Number 3 Date 06/12/2012

IEE Project Performance Indicators

i Ref: Ricardo-AEA/R/ED57321/Issue Number 3

Customer: Contact:

Executive Agency for Competitiveness and Innovation

Lisa Beardmore Ricardo-AEA Ltd Gemini Building, Harwell, Didcot, OX11 0QR t: 0870 190 8242 e: lisa.beardmore@ricardo-aea.com Ricardo-AEA is certificated to ISO9001 and ISO14001

Customer reference:

EACI/IEE/2011/001

Author:

Lisa Beardmore, Heather Haydock

Approved By:

Heather Haydock

Date:

06 December 2012

Signed:

Ricardo-AEA reference:

Ref: ED57321- Issue Number 3

1

Executive Summary Introduction The EU has set itself ambitious targets for reducing greenhouse gas emissions, improving energy efficiency and increasing the use of renewables. The Intelligent Energy – Europe Programme (IEE), now in its second phase (IEE II, 2007-2013), has a key role to play in this by funding specific actions within the market place to overcome (non-technological) barriers to both the efficient use of energy and the greater use of new and renewable energy sources, with the overall aim of contributing to the provision of secure, sustainable and competitively priced energy for Europe. In addressing non-technical barriers, the programme complements work being done under the EC Framework programmes and elsewhere to improve energy efficient and renewable energy technologies.

The IEE programme is managed, on behalf of the European Commission, by the Executive Agency for Competitiveness and Innovation (EACI).

IEE projects typically address one of four areas for funding:

• Energy efficiency and the rational use of energy (SAVE) Funding in this area mainly targets the improvement of energy efficiency and the rational use of resources in the industry, products and building sectors.

• New and renewable resources (ALTENER) In this area, funding is provided to increase the share of renewables in the production of electricity, heat and cooling, and to integrate them in the local energy systems.

• Energy in transport (STEER) This strand covers initiatives targeting energy savings and energy efficiency in the transport sector, including stimulation of demand for alternative fuels and clean and energy-efficient vehicles.

• Integrated initiatives

These initiatives cover several economic sectors or several of the main areas of energy efficiency, new and renewable resources and energy in transport at the same time.

Different projects have a different focus but each addresses at least one of five defined fields of delivery, which are:

• Enabling policy Activities supporting development and implementation of policies and legislation at EU, national and local level, e.g.: strategic projects providing technical inputs and market feedback to policy makers; actions supporting the implementation of European directives and policies; actions assisting regional and local policy makers and civil servants to develop and implement local strategies and actions plans.

• Market transformation Actions transforming markets making them more efficient, reliable and transparent are expected under this field of delivery, e.g.: accelerating uptake of certified products and services; helping industries and ESCOs to promote trading of competitive products across the EU, tackle common market barriers, and improve transparency in the market through improved standards, labelling and quality assurance schemes.

• Preparing the ground for Investment Activities triggering investments and facilitating access to finance e.g.: projects facilitating an extended use of structural funds and other investment funds; actions providing the necessary

2

information and tools to public and private decision makers to mobilise investments; actions supporting project development assistance and making energy efficiency and renewable energy projects bankable; preparation of tendering procedures and contractual arrangements.

• Building capacities and skills Activities which improve the capacity and skills of European market actors e.g.: actions aimed at creating appropriate skills, competencies and abilities of all market players in order to strengthen their role in a sustainable energy market.

• Changing behaviour Activities aimed at informing stakeholders with a view to changing behaviour and fostering commitment e.g.: actions promoting large scale information and raising awareness activities on energy issues; actions encouraging market players to change behaviour and motivate them to take action on sustainable energy issue.

In total, over the period of IEE II more than 370 projects have been funded to date.

Under the Programme, the requirements for projects to adopt an impact-focused approach to performance indicators were strengthened; to this end, the application form and the evaluation criteria were adapted to make SMART indicators an integral part of the project. Moreover, since the 2009 call for proposals, the EACI requires proposers - in addition to their project specific indicators - to deliver Common Performance Indicators (CPIs) covering the renewable energy production triggered, primary energy savings, investments mobilised, and GHG reduction.

In this effort to ensure an appropriate monitoring and evaluation of the impacts of the programme, the EACI launched (in August 2011) an invitation to tender to establish a set of typical outputs and performance indicators of projects supported in the framework of IEE, as well as an application methodology and guide for their use. This aim of the tender was:

o Help proposers to define appropriate performance indicators (and in turn formulate good quality proposals);

o Provide support to IEE project consortia to monitor their performance indicators; o Provide support to IEE project consortia on how to turn their measured impact into

communication messages; o Encourage projects with higher and more visible impacts; o Support the EACI in extracting results and aggregating them at both sector (by field of delivery)

and programme level; o Support the EACI to identify projects with important impacts, to be used as best practices and to

be taken into account for the prioritisation and definition of future work. The main activities of the work carried out within the tender consisted of:

o Review of a number of finished and/or ongoing IEE II projects, in order to carry out a collection, analysis and reality checking of the actual outputs and outcomes and impacts and of the performance indicators;

o Definition of a set of specific performance indicators to measure the outputs and outcomes and impacts of the IEE II projects;

o Preparation of an application methodology and guide for their use; o Analysis of the Common Performance Indicators of the IEE projects approved in the framework

of the IEE Call 2011.

Following the evaluation phase of the received offers, the EACI concluded a service contract with AEA Technology plc (Ricardo-AEA Ltd since November 2012). The contract ran from March 2012 to December 2012.

3

Challenges and approach As IEE projects address non-technological barriers, there are specific issues to be faced in estimating and measuring the impacts of projects on the IEE targets. The development of performance indicators for IEE projects and the IEE programme is particularly challenging because:

• IEE projects address non-technical barriers, which are influenced by many different factors and makes it difficult to link outputs to impacts (the attribution gap)

• Projects vary widely – aims, topics, scope, scale, location • Within the typical 2-3 year duration of the contracts, many projects have relatively limited impact;

longer term impacts can be significant but are difficult to forecast and there is no post-project monitoring in place

• IEE programme participants find it difficult to define project impacts in terms of the existing Common Performance Indicators (CPIs). They take different approaches to the problem; this lack of consistency is a problem for project officers and proposal evaluators

• Diverse impacts are difficult to aggregate at the programme level • All of the above make programme impacts hard to measure or prove

The main purpose of this service contract was to develop a set of typical project outputs and performance indicators, plus guidance, to assist proposers in selecting meaningful and effective ways to monitor, measure and communicate project performance and impact, in a manner which is consistent across the programme.

Insights from Project Coordinators, literature, and expert workshops Within the current system project impact indicators vary depending on:

• Sector & key action • Types of activity undertaken • Level of implementation • Target groups, e.g. policy makers or industry stakeholders • Field of delivery & objective, e.g. technology awareness or market development

Many projects define their performance indicators in terms of involvement of the stakeholder groups that the project is targeting. Most project-specific indicators are output indicators rather than impact indicators. In some projects there is also a tendency to underestimate short term (within project duration) impacts while overestimating likely long term (2020) impacts.

Between May 2012 and August 2012, interviews were carried out with 31 project coordinators. The key feedback that can be taken from the interviews is that an almost equal number of coordinators had some experience in setting IEE indicators before, as the number that had no experience. Most coordinators found indicators useful for shaping and defining their projects, and used them throughout the project to monitor progress. A large number of coordinators found that gaining access to data in order to calculate quantitative impacts was very challenging. Coordinators, mainly those in the ALTENER strand, indicated that a Green Jobs indicator would be a useful and politically relevant indicator for IEE. However, relatively few project coordinators felt that they would be able to calculate this indicator with a reasonable degree of confidence.

In most cases, project coordinators had suggestions to make around Common Performance Indicator use and guidance. Most coordinators wanted better guidance, and felt that the EACI should identify

4

methodologies that they should use when calculating their impacts, so that projects become comparable.

In reviewing the existing literature, and other programmes delivering similar outputs, it was found that very few programmes have performance indicators for “softer” measures (e.g. LIFE+, GEF), and many acknowledge this is very difficult to do. A major difference noted between IEE and other well-established programmes, is that these other programmes tend to use highly prescriptive sets of output indicators, and provide very clear guidance on each stage of the results chain. A key finding from a number of reviews of performance indicator systems is that guidance on how to establish plausible linkages between their outputs and their long term impacts has to be clear and well written. Indicators should be appropriate to the project aims and objectives and not be governed by how measureable they are.

An internal workshop of evaluation, innovation, programme management, renewable energy, energy efficiency, and transport experts brought a number of perspectives to bear on methods to improve the way IEE projects can link their outputs to their impacts, and make the impacts that are measured more reliable and relevant. In this workshop, the idea of using the AIDA framework (awareness, interest, desire, action) was proposed as a way of linking outputs and impacts. This approach is explained in the guidance document, and could be a very useful tool, especially to those projects who’s activities are focussed at the ‘awareness’ end of the chain.

Reviewing the existing performance indicator system based on CPIs The current system of performance indicators based on the four CPIs asks applicants to define impact indicators based on a set of outputs that are linked to the project’s objectives. The CPIs are:

- Investments made by European stakeholders in sustainable energy, triggered by IEE ( EUR) - Renewable energy production triggered by actions supported by IEE (toe/ year) - Energy savings triggered by actions supported by IEE (toe/ year) - Reductions of greenhouse gas emissions triggered by actions supported by IEE (tCO

2e/ year).

The CPIs are estimated for both short-term (by the end of the project) and long-term (by 2020). A review of the 56 Call 2011 projects’ CPI data and methodologies used to estimate the data was undertaken as part of this project. The results of the review fed into the development of the new indicator system and guidance document.

The aim of the review was to separate the methods used by projects to calculate their CPIs into three categories:

• Reliable; • Acceptable; • Uncertain

To do this each of the project's calculations was reviewed from the perspective of the 4 steps outlined in our guidance document.

• Step 1: Define impacts and scope • Step 2: Set baseline • Step 3: Estimate short-term impacts • Step 4: Project forward to long-term (2020) impacts

Awareness Interest

Desire Action

5

Which method each project had used was also determined, in relation to the suggested methodologies in the guidance document. Method 1 involves aggregating the savings attributed to each individual action, e.g. the savings per action plan or per trained person; this being fundamentally a bottom-up approach. Method 2 involves estimating overall savings against a baseline, e.g. using the percentage savings achieved by a pilot project: this being fundamentally an empirical or observational approach.

Each step was given a subjective score of between 0 and 5 where:

• Score of 0: No information is provided on a data source or basis of assumption, or an assumption seems completely illogical

• Score of 5: Data and assumptions are fully documented; assumptions seem very logical; seems likely to lead to a highly accurate estimate of impacts.

This was done for the first three CPIs together (RE deployed, energy saved, GHG) and separately for the fourth CPI (investment). The reasoning being that the same basic method is typically used to estimate either RE deployed or energy savings and then the GHG impacts are calculated from there.

The scores were then weighted for each step as follows:

Step 1 Step 2 Step 3 Step 4

Short term CPIs Method 1 30% N/A 70% Short term CPIs Method 2 30% 30% 40% Long term CPIs Method 1 20% N/A 50% 30% Long term CPIs Method 2 20% 20% 30% 30%

A project achieving a weighted score of >3.5 was deemed Reliable; a score between of 2 and 3.5 Acceptable and a score of <2 Uncertain. Any projects that scored 0 or 1 on step 1, 2 or 3 were automatically moved to the Uncertain category even if they scored highly elsewhere. Any projects that scored 0 or 1 on step 4 only were deemed Uncertain for the long-term (2020) CPIs only.

All of the projects in the 2011 call were found to have made a good attempt at developing their CPIs and the methodologies were well described in the CPI documents. This was not always the case for 2009 and 2010 call projects, which suggests that the project coordinators have benefited from the training they have received from the EACI, the support of project officers and a growing familiarity with the concept of CPIs and their estimation. However, there remains an inconsistency in approach, particularly to the forward projection of 2020 impacts, which confirms the need for further guidance and support. This need was further confirmed by the interviews with project coordinators, most of whom said that the calculation of CPIs was very challenging and they would have liked more detailed guidance and examples.

The review of CPI methodologies helped to inform our thinking on the selection and definition of a new set of Project Performance Indicators (PPIs, see following section), the contents of the guidance document and the way in which PPIs could be aggregated and reported as IEE programme impacts. Specifically:

• Guidance on the attribution gap. Further guidance is needed on how to bridge the attribution gap for those projects choosing to adopt the Method 2 methodology. The attribution of savings was the key uncertainty in several of the project’s CPI calculations. In general the Method 1 approaches were more likely to rate Reliable or Acceptable but even then there were some key assumptions made and not always justified or referenced.

• A 2-stage approach. Many of the projects provided more information on their CPI calculation methodology in the CPI document than the proposal. This suggests that they were short of time when preparing the proposal and dedicated much more time once they were in contract. This supports the recommendation for a two-stage process as regards the calculation of the performance indicators. It may be beneficial to produce a cut-down version of the guidance for use at the bid stage while also allowing applicants to access the larger guidance document should they wish to.

• Assessment of CPIs. Based on the evaluators comments, about half of the 2011 projects were asked to review and clarify one or more of their CPI values. Furthermore, a more in depth analysis of the CPIs in the first months of the project, resulted in further refinement of the values. This suggests that the EACI should not put too much emphasis on the indicator values at the proposal stage or use them as criteria for evaluating proposals.

6

• Top-down methods. The top-down approach should only be used as a method for checking the impacts derived from bottom-up analysis using methods 1 or 2; the results are too unreliable.

• Mandatory CPIs. For some EU-wide policy and market transformation projects, it may not be possible to quantify the impacts except through these unreliable top-down methods and so it is recommended that the existing CPIs are not mandatory for such projects in future. This does not apply to policy projects focused on local or city-level policies or more narrowly defined market transformation projects.

• Decimal places and commas. It may be easier to rule out all use of decimal places and commas to avoid confusion. The problem is highlighted particularly when Excel has been used to calculate a figure that has then been transposed into a Word document.

• Aggregation. Submission of CPI data should be via a spreadsheet to avoid the lengthy exercise of pulling all data from word documents.

Proposed new performance indicator system and guidance The initial guidance document was drafted in month 6 of the project. The document has then been through 4 iterations, based on feedback from the EACI staff, further internal workshops, and the review of 2011 CPI methodologies.

The final guidance document is designed to be the full version of the guidance, which provides information and support for every step of the new performance indicator system. Although a two phase process has been developed with the EACI, the 1st phase guidance will be further developed by the EACI, which will utilise elements of the full guidance document.

The guidance document includes the necessary steps for:

• Selecting typical outputs which are relevant to a specific project, • Selecting performance indicators which are relevant to a specific project, • Adapting them to the objectives of that project.

The guidance document is based around the results chain of objectives – activities - outputs - impacts (short-term) – impacts (long-term), and each stage of the indicator-setting process is clearly explained using examples. A template to be completed by applicants which will capture the chain shown below has also been developed.

7

Applicants are instructed to set no more than 5 objectives, in order to keep them to a manageable and meaningful number, as there will be outputs and impacts related to each objective. Objectives should be a mixture of short and long-term. At the point of setting their objectives, applicants are also asked to identify which of the five fields of delivery their project is associated with:

• Enabling policy • Market transformation • Preparing the ground for Investment • Building capacities and skills • Changing behaviour

It is possible for projects to address more than one field of delivery, but it is recommended that the maximum is three. The reason for this is that projects need to be focussed in order to achieve their aims, and as each field of delivery is quite different, projects may be attempting too much if they try to address more than three fields of delivery.

Outputs are then determined by the objectives and the activities to be delivered through the proposed work packages. An Annex to the guidance document has been provided containing tables of standard outputs in order to guide applicants as to the choices they have in expressing quantity indicators and quality indicators. Examples of basic quantitative output indicators include:

• Number of participants at workshops • Number of training hours provided • Number of technical design solutions

However, total numbers often do not provide the full picture of an output. Examples of indicators that measure interest or effectiveness include:

• % of workshop participants that rated the workshop useful or very useful • % of trainees that rated the training useful or very useful • % of workshop participants that would recommend the workshop to a colleague

The options faced for output indicators were whether to provide a very prescriptive set of indicators for applicants to use vs. a high degree of flexibility supported by detailed guidance. The advantages of providing a standard set of indicators include; an increase in consistency between projects, improved identification of indicators for applicants especially where they have little time/budget to complete the indicator section of the application, and the increased ease of aggregation of outputs to programme level. The disadvantages of this approach include; missing output options due to the wide range of IEE project types, and encouraging a ‘box-ticking’ mentality. It was decided to opt for standard output tables that are to be used as guidance, and not be mandatory. This allows applicants to review their outputs, be guided on the best way to frame those outputs, but still be free to express the innovation in their projects.

The guidance document then focuses on a new proposed set of impact indicators, the new Programme Performance Indicators (PPIs) and methodologies to calculate the impacts of projects more reliably. In looking at prescriptive sets of project impact indicators, it was decided that although this would improve consistency between projects, the very nature of these projects mean that they all have very different impacts. The idea of IEE is to effect a range of impacts, with the goal that these project-specific impacts then go on to be measureable through energy efficiency, renewable energy generation and therefore reductions in GHG emissions.

Feedback from the project coordinators not only indicated that they wanted to see guidance on the methodology they should use for calculating long term impacts, but that they often struggled to find a ‘sensible’ replication factor to use. In reviewing the 35 initial projects, and the 56 Call 2011 projects, many similar projects were found to have vastly different estimates for their 2020 impacts, due to the assumptions made about replication. The new indicator system asks applicants to estimate a minimum and maximum range of impact, which should help improve comparability between direct project impacts by 2020. External experts contracted to evaluate the submitted proposals can also then clearly identify the replication assumptions and review their validity.

8

In reviewing the existing CPIs of the current system, a number of additional indicators were considered that may be relevant to a proportion of projects, and that would allow projects that find the existing CPIs very difficult, to be able to use other metrics to measure their impact at a programme level. A broad range of new PPIs were considered, included green jobs created, SMEs influenced, innovation, increased skills/capability, policy makers influenced, attitudes and behaviour changes. The final set of seven PPIs is presented below, along with the units to be used.

PPI Short term (by end of project) Unit Long term (by 2020) Unit

Additional cumulative investment made by European stakeholders in sustainable energy € Additional cumulative €

Renewable Energy production triggered tonnes of oil equivalent (toe)/year

toe per year AND cumulative to 2020

Primary energy savings compared to baseline projections toe/year toe/year AND cumulative to

2020

Reduction in greenhouse gas emissions compared to baseline projections tCO2e/year tCO2e/year AND cumulative

to 2020

Policy makers influenced total cumulative

Number of people with increased skills/capability/competencies total cumulative

Number of people changing their behaviour total cumulative

In the current system, applicants are asked to estimate cumulative investment impacts and in-year RE, energy efficiency, and GHG reduction impacts. The new system asks for both in-year and cumulative estimates, which will take account of the different impact profiles for different types of project.

We recognised that PPI estimates will always be approximate, particularly for 2020 impacts, and easier to estimate for some projects than others. The purpose of the guidance document is to help applicants to estimate the likely impacts of their projects, during the project duration and by 2020, in a logical and sensible way that is appropriate to their project. To make the PPIs more relevant to each project, a system of mandatory and optional choices has been proposed:

9

Field of delivery Investment

€

RE triggered

Energy saved

GHG reduced

Policy maker

influence (all levels)

No. of people

with increased capability

No. of people

changing their

behaviour

Enabling policy O O O O

Market transformation O O O

Preparing the ground for investments

O

Building capacities and skills

O O O O

Changing behaviour O O O O

Mandatory PPI O Optional PPI

The proposed guide then goes on to describe a 5-stage methodology for estimating PPIs. The methodology was tested during the analysis of the Call 2011 CPI data, where it was found that most projects had used the bottom-up approaches described in the proposed methodology. A key recommendation that has been made is that projects use bottom-up approaches wherever possible, and support their conclusions by using top-down methods to verify the estimations.

STEP 1: DEFINE IMPACTS AND SCOPE

STEP 2: SET BASELINE

STEP 3: SHORT TERM IMPACTS

STEP 4: LONG TERM IMPACTS

STEP 5: CHECK IMPACTS

10

Conclusions and recommendations The key overall conclusions and recommendations that can be made from this project are:

1. IEE is a unique programme There are no other programmes delivering the same breadth and mixture of projects as IEE. This makes it uniquely difficult to assess the impacts of the programme using only a small number of comparable metrics. The key metrics used in the current system are highly important, and are complemented by the new PPIs, which allow measurement of impact for those projects that particularly struggle to measure energy, carbon, or investment impacts in a reliable manner. The PPI data is vital to the programme, for understanding the programmes’ impact, and to justify its existence. It is recommended that the new set of PPIs is released with the next Call for IEE proposals, and supported by the new guidance document.

2. It takes time and effort to calculate reliable impact indicators This means that estimations of the PPIs at the application stage should not be used directly in the evaluation process; although it is reasonable to take into account how well the applicant has thought through their proposed indicators. A new work package/task for indicator development, monitoring, and evaluation (including ex-post evaluation) could be introduced for projects to fully study their impacts. Although the guidance document produced for this project is designed to be a full guide, able to support applicants in completing each step of their performance indicators, it was felt that the effort required to reliably calculate impacts, especially long term impacts, would be a heavy burden at the application phase. The review of Call 2011 CPIs showed that it was necessary for most of the projects to review and clarify their CPI estimations during contract negotiation and in the first months of the action. A two phase approach is sensible, as it allows applicants to focus on determining the most appropriate indicators for their project at the application stage, and then utilises a dedicated work package/task at the start of the project to study the impacts of the project and monitor them in greater depth. It is important to note that many projects have made a very reasonable attempt at estimating the CPIs, given a lack of direction for their methodological approaches. Project Officers at the EACI have also been invaluable in guiding projects to address their proposed CPIs during the negotiation phase. The aim of the guidance document section on methodologies is not only to improve these estimations, but to help projects to fully document their assumptions and data sources so that their estimations can be understood by the EACI, and help more PPIs to be rated as reliable. This is especially important where projects have a multi-national impact.

3. It is important to recognise that some projects will not be able to reliably calculate their energy or investment impact There will be certain projects, especially pan-European policy projects for which it is particularly challenging to reliably estimate any impacts in the form of energy or CO2 savings, RES capacity or investments. The best efforts that can be made will ultimately always result in a rating of ‘uncertain’. However, these policy projects often have the largest predicted impacts if they influence policy in the intended manner. This is an acceptable situation, as indicators should be appropriate to the project aims and objectives and not be governed by how measureable they are. This is why it is proposed to introduce a new indicator on 'policy maker influence'.

4. Project monitoring and evaluation are key to developing our knowledge of impact A number of outputs from this project are designed to aid in the monitoring and evaluation of the outputs and impacts of IEE projects. An Excel template could be used for simpler data-gathering and can be used as an aggregation and tracking tool. The recommended work package/task on indicator development, monitoring and evaluation will also provide project coordinators with a clear message that they are to spend time and effort not only in developing their estimations of their impact at the start of the project, but then to also monitor their progress against their indicators, and to evaluate their success at project completion. If feasible, it is recommended that a database system is used to enter the indicators, so that analysis of aggregated outputs and separation of impacts by field of delivery or key action is relatively simple to achieve.