assessing the cost-effectiveness of environmental policies ... · policy or project appraisal since...

Benjamin Görlach, Anneke von Raggamby1 and Jodi Newcombe2

Assessing the Cost-Effectiveness of EnvironmentalPolicies in EuropeResults of a project produced for the European Environment AgencyPaper presented at the EASY-ECO Conference “Impact Assessment for a NewEurope and Beyond”, 15-17 June 2005, University of Manchester, UK

Abstract

Economic analysis for policy appraisal is generally interested in answering two questions: ‘is a givenpolicy objective worth achieving?’ and ‘If so, what is the most cost-effective way of achieving thisobjective?’. While the first question is addressed in a cost-benefit analysis (CBA), the second questioncan be answered with the help of a cost-effectiveness analysis (CEA). Cost-effectiveness analysis canbe applied both as an ex-ante appraisal and as an ex-post evaluation tool. While an ex-ante CEAdetermines the most cost-effective way of achieving a given target, an ex-post CEA helps to assesswhether a policy measure has been effective in addressing the problem it was designed for, and atwhat cost. It can take the form of an ex-ante / ex-post comparison, assessing whether expected effectswere realised in the projected cost, it can consist of a cross-country comparison (benchmarking), or, ifex-post CEAs are carried out repeatedly, it can determine whether cost-effectiveness has increased.

Although some European countries have moved ahead in this respect in recent years, ex-postevaluation of environmental policy performance remains a relatively recent phenomenon and is notwidely applied. At the European level, there is little experience with carrying out such assessments,and even less with using their results to feed back into policy implementation. With the attentionrecently given to a cost-effective design and implementation of environmental policies – as expressede.g. in the impact assessments carried out for all major policy initiatives – it can be expected that therole of cost-effectiveness assessments is set to increase. The main challenge encountered is toestablish the causality between observed effects and influencing factors. Another main issue relates tothe scale of the analysis – traditionally, CEAs have mainly been applied at the local level, in order toevaluate individual, well-defined measures. Upscaling the analysis to assess the cost-effectiveness ofstrategies or policies at national or European level necessarily increases the uncertainty of relatingobserved impacts to a particular action.

This paper summarises main results of the project “Cost-effectiveness of environmental policies”,carried out by Ecologic, eftec and IVM on behalf of the European Environment Agency. Following thestructure of this project, it first provides an overview of legal requirements for ex-post CEA inEuropean environmental policy, then presents an overview of existing guidance documents andmanuals for carrying out ex-post CEAs, and finally gives a summary of the current practice in EUMember States with regard to ex-post CEA of environmental policies. 1 Both Ecologic, Institute for International & European Environmental Policy, Pfalzburger Straße 43/44,D-10717 Berlin, Germany. Contact email: [email protected] eftec Economics for the Environment Consultancy Ltd, 16 Percy Street, London W1T 1DT, UK

Görlach et al: Assessing the Cost-Effectiveness of Environmental Policies in Europe

2

1 Introduction

1.1 Ex-post Cost-Effectiveness Analysis: What is it and how can it be used?Economic analysis for policy appraisal is generally interested in answering two questions: ‘is a givenpolicy objective worth achieving?’ and ‘If so, has the policy objective been achieved in the most cost-effective way?’. While the first question is addressed in a cost-benefit analysis (CBA), the secondquestion can be answered with the help of a cost-effectiveness analysis (CEA). The two methods arebriefly described in the box below.

Box: Cost-benefit analysis and cost-effectiveness analysis

Cost-benefit analysis (CBA) is carried out in order to compare the economic efficiency implicationsof alternative actions. The benefits from an action are contrasted with the associated costs (includingthe opportunity costs) within a common analytical framework. To allow comparison of these costs andbenefits related to a wide range of scarce productive resources, measured in widely differing units, acommon numeraire is employed: money. This is where most problems usually start for economicpolicy or project appraisal since some resources, especially environmental ones, are difficult to pricein money terms. Many of the goods and services provided by ecosystems – such as amenity, clean air,biodiversity sustenance – are not traded on a market, hence no market price is available which reflectstheir economic value. Such prices need to be estimated instead through the use of valuation studies,e.g. eliciting people’s willingness to pay for a particular environmental good. By comparing costs andbenefits in monetary terms, a CBA provides an assessment of whether a policy option is worthimplementing (i.e. whether the benefits outweigh the costs). The comparison can either be done bydiving benefits by costs (where a benefit-cost ratio larger than one means that the option is worthimplementing), or by subtracting net costs from net benefits (where a positive sum indicates abeneficial option).

A cost-effectiveness analysis (CEA) seeks to find the best alternative activity, process, or interventionthat minimises resource use to achieve a desired result. An ex-ante CEA is performed when theobjectives of the public policy have been identified and an analyst or an agency has to find the leastcost-option of achieving these objectives. An ex-post CEA addresses the question in how farobjectives have been achieved, and at what cost. In either case, the cost-effectiveness of a policyoption is calculated by dividing the annualised costs of the option by a quantified measure of thephysical effect, such as animal or plant species recovered, tons of emissions of a given pollutantreduced, kilometres of river length restored, etc. In this context, the effects of a policy can be bothreduced pressures (e.g. the least-cost option to reduce CO2-emissions), or avoided impacts (e.g. thecheapest way to keep global warming below 2°), where the latter is usually more difficult to assess.Different options that achieve / have achieved the same effect are then compared based on their cost.CEA, therefore, does not ask, nor attempts to answer, the question whether the policy is justified, inthe first place, in the sense that its benefits to society will exceed its costs to society. CEA issometimes used as a second-best option when a full-blown CBA would be desirable, but many effectscannot be captured in monetary form.

Cost-effectiveness analysis can be applied both as an ex-ante appraisal and as an ex-post evaluationtool. If applied ex-ante, a CEA will help to determine the most cost-effective way of achieving a giventarget, assisting policy makers to allocate resources and realise policy objectives in efficiently.


3

The focus of this paper is on ex-post CEAs. Where it is applied ex-post, a CEA may help to assesswhether a policy measure has been effective in addressing the problem it was designed for, and atwhat cost. It can take the form of an ex-ante / ex-post comparison, assessing whether expected effectswere realised in the projected cost; it can consist of a cross-country comparison (benchmarking), or, ifex-post CEAs are carried out repeatedly, it can determine whether efficiency has increased over time.

Although some European countries have moved ahead in this respect in the last years, ex-postevaluation of environmental policy performance remains a relatively recent phenomenon and is notwidely applied. At the European level, there is little experience with carrying out such assessments,and even less with using their results to feed back into policy implementation.

1.2 Critical Issues in ex-post Cost-Effectiveness AnalysisWhether at the European level or at the level of Member States, similar problems are encountered inex-post assessments.

• The main challenge is to establish the causality between observed effects and influencing factors,thereby disentangling the different effects of policies and relating them to individual policymeasures, and separating out the influence of other factors.

• A related problem is that of data gathering: unless specifically tailored monitoring requirementshave been specified up front, it is often difficult to find the data that measures the impact a policyhas had. For this reason, data gathering ex-post can easily become very costly and time-consuming.

• Another main issue relates to the scale of the analysis – traditionally, CEAs were mainly appliedat the local level, in order to evaluate individual, well-defined measures. Upscaling the analysis toassess the cost-effectiveness of strategies or policies at national or European level necessarilyincreases the uncertainty of relating observed impacts to a particular action.

Next to these practical problems, there are also some theoretical issues that merit further discussion,but which are only touched upon in passing in the available literature:

• There is some discussion on which types of costs should be considered in a CEA, ranging fromthe purely financial private costs (i.e. investment and operational costs) of specific measures togeneral equilibrium-estimates of costs to the wider economy, including efficiency losses (foregoneproducer and consumer surplus). At least one guidance document argues that changes in producerand consumer surplus should be included, however none of the case studies identified actuallycalculated these cost components.

• Regarding the treatment of effectiveness, there is an interesting issue of whether measures ofeffectiveness should be discounted even though they are in non-monetary terms. Discounting ofcosts is a standard procedure in most CEAs, and is called for in all guidance documents.Regarding the temporal dimension of effectiveness, there is no guidance on whether some type ofdiscounting should be applied as well. For example, in a comparison of two measures that achievethe same objective at the same cost, but where one takes two years to reach the objective, while forthe other costs are stretched out over five years, the latter would appear more cost-effective.

• Other issues include the distinction between intermediate goals and final goals of a policyintervention, which are often confused. Thus, the effectiveness term in a cost-effectivenessanalysis can either capture a pressure (i.e. tons of emissions reduced) or an impact (avoideddamage or improvements in environmental quality). Which of the two is applicable depends on theoriginal goal of the policy measure. In practice, most assessments tend to focus on pressures, since


4

they are more easy to measure and since the causality between measures and effects is more easyto establish.

1.3 Scope of this PaperThis document summarises and interprets the main results of the project “Cost-effectiveness ofenvironmental policies”, carried out by Ecologic, eftec and IVM on behalf of the EuropeanEnvironment Agency. It is mainly structured along the work packages identified for this project:

I. Overview of legal requirements for ex-post CEA in European Environmental Policy

II. Overview of guidance documents and manuals for carrying out ex-post CEAs

III. Selected case studies of applied ex-post CEA for environmental policy measures.

Due to the prevalence of ex-ante over ex-post CEA, work package III also included selected examplesof ex-ante CEA. For work package II, it emerged that while there are some guidelines that provideinsight on the processes and techniques of ex-post evaluation, and some that would discuss theapplication of CEA to ex-post analysis, there were no guidelines and manuals specifically gearedtowards ex-post CEA.

While this document provides a summary and conclusion of the work package results, the detailedresults (including summaries of legal requirements, guidelines and case studies) are documented in aweb based database, the Pan-European database for applied ex-post Cost-Effectiveness Analyses(PANACEA). The database includes detailed summaries for the most relevant studies and guidelines,as well as bibliographic references and links for the less relevant cases.

The gathering of information and literature involved the following steps:

• Consultation with members of the network of economists in the framework contract;

• Consultation with the EEA project steering group;

• Consultation with EEA national focal points;3

• A detailed web-search, including on-line resources of major research institutes, internationalbodies, relevant national Government departments and European Commission DGs;

• Consultation with in-country contacts in ministries;

• Searches in relevant academic journals.

3 Regarding the consultations with the EEA national focus points and subsequent contacts, the authors would liketo thank the following people for their support: Jan Voet (Belgium), Tapani Saynatkari and Marjukka Porvari(Finland), Denise Juin (France), David Lee (UK), Gerard O’Leary (Ireland), Bernt Rondell and Per Magdalinski(Sweden) and Eric Debrabanter (Luxembourg). The authors would also like to thank Friedrich Hinterberger(SERI), Frans Oosterhuis (IVM), David Pearce (UCL) and Hans Vos (EEA) for reviewing and commenting onthis paper. The summary and overview of legal requirements upon which chapter 2.1 draws was compiled byFrans Oosterhuis and Harro van Asselt (IVM). The evidence on case studies and manuals that is presented inchapter 2.2 and 2.3 was gathered by Jodi Newcombe and Helen Johns of eftec, who have also commented on andcontributed to other parts of this document.


5

2 Summary of the Results

2.1 Legal Requirements for ex-post CEAsIn 2001, the EEA noted that “very few items of EU environmental legislation request information onpolicy effectiveness … even though some EU measures are very costly to implement and should besubject to some kind of cost-effectiveness scrutiny” (EEA, 2001, p. 14). This observation still seems tobe valid, although a (small) number of recent Directives do include a requirement to perform an ex-post cost-effectiveness analysis.

This project identified 18 legislative items that require some type of evaluation, and can be related toenvironmental policy. In the figure below, these items are visualised by grouping them in fourcategories. As the analysis shows, only a small subset (A) indeed meets all three requirements: (i)environmental legislation that (ii) mandates a cost-effectiveness analysis (iii) to be carried out ex-post.If any of these three criteria are relaxed, the scope of relevant items can be expanded. This means thatthe 18 general items covered in this project can be subdivided as follows:

A - Environmental legislation that requires an ex-post evaluation of cost-effectiveness, at least as oneof several factors to be considered in a wider evaluation framework. The current study hasidentified four items in this category: Directives on cogeneration, biofuels, renewable energy andemission ceilings (see below).

B - Environmental legislation that requires an ex-ante evaluation / analysis of cost-effectiveness, or atleast consideration of cost-effectiveness as one of several factors. In this category, six items havebeen identified. The cost-effectiveness requirements in this category may take different forms: e.g.in the case of the Water Framework Directive (WFD), it is not so much the cost-effectiveness ofthe Directive as such that is considered, but rather the cost-effectiveness of combinations ofmeasures mandated by the Directive. Several Directives (e.g. large combustion plants, ozone andbenzene in ambient air) require that experiences with the implementation of the Directive be takeninto account when deciding on the cost-effectiveness of stricter standards, thus connecting ex-postevaluation and ex-ante CEA.

C - Environmental legislation that requires an ex-post evaluation, but not (necessarily) the analysis ofcost-effectiveness. Four items have been identified that fall into this category, including Directiveson marine and air pollution. While none requires explicitly the consideration of cost-effectiveness,

Ex-post evaluationrequired

Environmental legislation

Cost-effectiveness considerations required

CAD

B


6

some items refer to the overall efficiency or the effectiveness of the regulations, implying at least acontributing function for cost-effectiveness.

D - Legislation and regulations requiring ex-post CEA that is not strictly environmental, but has asignificant impact on the environment. This category comprises four items related to fundinginstruments of the Community regional policy (Cohesion Fund, Structural Funds, the Instrumentfor Structural Policies for Pre-Accession (ISPA) and the Financial Instrument for the Environment(LIFE)). It should be noted that this list is not exhaustive. Depending on which policies areregarded as having a significant impact on the environment, more could be included in thiscategory.

This means that while there are several more Directives that involve effectiveness assessments in oneway or another, the set of environmental Directives calling for an ex-post evaluation of cost-effectiveness is limited to four Directives:

• Directive 2001/77 (Electricity from renewable energy sources). Article 4.2 of the Directivedemands that “[t]he Commission shall, not later than 27 October 2005, present a well-documentedreport on experience gained with the application and coexistence of the different mechanisms [...].The report shall assess the success, including cost-effectiveness, of the support systems [...] inpromoting the consumption of electricity produced from renewable energy sources.” The reportingmay include a proposal for a framework for Community activities with regard to support schemesfor Community activities. This framework should “promote the use of renewable energy sourcesin an effective way, and be simple and, at the same time, as efficient as possible, particularly interms of cost”.

• Directive 2001/81 (National Emission Ceilings). Article 9.1 of the Directive demands that “in2004 and 2008, the Commission shall report to the European Parliament and the Council onprogress on the implementation of the national emission ceilings”, and on the extent to which theobjectives of the Directive are likely to be met. The reports shall include “an economicassessment, including cost-effectiveness, benefits, an assessment of marginal costs and benefitsand the socioeconomic impact of the implementation of the national emission ceilings onparticular Member States and sectors.

• Directive 2003/30 (Promotion of biofuels and other renewable fuels). Article 4.2 of theDirective states that “by 31 December 2006 at the latest, and every two years thereafter, theCommission shall draw up an evaluation report [...] on the progress made in the use of biofuelsand other renewable fuels in the Member States.” The report shall assess “the cost-effectiveness ofthe measures taken by Member States in order to promote the use of biofuels and other renewablefuels”, as well as “the economic aspects and the environmental impact of further increasing theshare of biofuels and other renewable fuels”.

• Directive 2004/8 (Cogeneration). Article 7.3 of the Directive demands that the Commissionshould provide “a well-documented analysis on experience gained with the application andcoexistence of the different support mechanisms” in order to “assess the success, including cost-effectiveness, of the support systems in promoting the use of high-efficiency cogeneration.”

All of these Directives have entered into force in 2001 or later. Consequently, they are still in theirfirst reporting cycle. The first assessments of the Directives’ performance was expected for the end of2004 (for the National Emission Ceilings Directive), but has not been published at the time of writing.Most of the assessments will be repeated at intervals of two or four years.

A further question is how the evaluation of cost-effectiveness should be conducted. For the fourDirectives that require an ex-post CEA, neither guidelines nor standards are provided regarding thecontent or the methodology to be applied. For some of the other Directives and regulations, more


7

guidance exists. The guidance is most developed in the case of the Water Framework Directive,Article 11 / Annex III of which requires an ex-ante appraisal of the most cost-effective combination ofmeasures to achieve good ecological status. To support the selection of measures, the Europeanworking group WATECO (established under the WFD Common Implementation Strategy) hasproduced an extensive guidance document. In addition, some Member States have come up withhandbooks and guidance documents for the national implementation (see also the results of WP 2).

Box: National-level requirements in the Member States

While the focus of this study was on requirements for ex-post effectiveness in Europeanenvironmental legislation, it also became evident that the legal requirements for ex-post-evaluation onthe level of the Member States differs markedly. Considerable experience with such assessments existsin the NL and the UK, where requirements are in place to evaluate policies and their impacts,including their (cost-)effectiveness.

• In the Netherlands, Article 20 of the Government Accounts Act (Comptabiliteitswet) states thatMinisters shall be responsible for the effectiveness and efficiency of the policy underlying theirbudgets. This includes conducting regular audits of the effectiveness and efficiency of the policy,and reporting back to the Ministry of Finance. Guidance for this requirement is presented inter aliain the draft “guidance for ex post evaluation research” (Concept wegwijzer evaluatieonderzoek ex-post [G44]).

• For the UK, the Green Book on appraisal and evaluation in central government (G8 below) statesthat “all new policies, programmes and projects, whether revenue, capital or regulatory, should besubject to comprehensive but proportionate assessment, wherever it is practicable, so as best topromote the public interest.” In this context, the Green Book mentions cost-effectiveness analysisas one possible assessment method.

2.2 Guidance DocumentsThere is a large number of textbooks on the use of economic appraisal, most of which focus on cost-benefit analysis but also sometimes cover cost-effectiveness analysis. This abundance in the domain ofacademic publications does not seem to be reflected in the publication of practical guidelines. Inaddition, cost-effectiveness analysis is dealt with to a greater extent in the health sector than in theenvironmental sector. Textbooks on cost-benefit analysis in the environmental sector typically onlymention cost-effectiveness in passing.

For the selection of the guidance documents covered in this study, emphasis was placed on providing arange of the best examples, in order to make an overall assessment of the state of play and thereby toassess the need for a new, specific guidance document to be authored for the EEA’s purposes. Forselecting guidance documents, the criteria for selection were:

• That the guidance was up-to-date (thus only the latest Government guidance from one issuingbody is presented);

• That the guidance is focused on the analysis of environmental policies (or explicitly mentionsthem as one of a number of policies to be assessed);

• That the guidance is issued by or directed at EU Member States (except where other country-levelguidance offers additional insights, as is the case with the USA); and

• Public sector guidance is preferred.


8

Forty-four potential guidance documents, mostly guidelines from various national and internationalpublic sector bodies, but also academic papers and books, were identified during the course of thisproject, of which twenty-four were deemed to be relevant for the purposes of this study. Of these,fifteen are summarised in the PANACEA database. The relevant guidance documents are summarisedin the table below (a grey “(X)” in the ex post or ex ante column indicates that the document isrelevant, but does not explicitly address ex post/ex ante evaluation; a question mark indicates that wewere informed of the document’s existence, but were either unable to obtain a copy or it is in alanguage we were unable to translate):

No. Title Policy Area Country Author/clientType ofanalysis Ex

pos

t

Ex a

nte

Sum

mar

y

G2

CEA and Developing aMethodology for AssessingDisproportionate Costs(2004)

WFD UK

Risk and PolicyAnalysts Ltd /DEFRA and UKEnvironment Agency

CEA &CBA X X

G3

Guidelines for Defining andDocumenting Data onCosts of PossibleEnvironmental ProtectionMeasures (1999)

Environmentgeneral EU

EuropeanEnvironment Agency /no client

Neither X

G8The Green Book: Appraisaland Evaluation in CentralGovernment (2003)

General UK UK Treasury / noclient

CBA &CEA X X X

G9 Guidelines for PreparingEconomic Analyses (2000)

Environmentgeneral USA

US EnvironmentProtection Agency. /no client

CBA &CEA (X) X X

G12

Basic Principles forSelecting the most Cost-Effective Combinations ofMeasures as Described inArticle 11 of the WaterFramework DirectiveHANDBOOK (2004)

WFD GermanyEcologic / GermanFederal EnvironmentAgency

CEA X X

G14

Economics and theEnvironment: theImplementation Challengeof the Water FrameworkDirective, GuidanceDocument (2003)

WFD EU WATECO / no client CEA X X

G18What Constitutes a GoodAgri-Environmental PolicyEvaluation? (2004)

Agriculture OECD Pearce, David / noclient

CEA &CBA (X) (X) X

G20

Guidelines and DiscountRates for Benefit-CostAnalysis of FederalPrograms (1992)

General USAUS Office ofManagement andBudget / no client

CBA &CEA X X

G21Guidelines for theEconomic Analysis ofProjects (1997)

Developmentprojects

Asiancountries

Asian DevelopmentBank / no client

CBA &CEA X X

G22Opportunities EnvelopeGuidelines for Proposals(2004)

ClimateChange Canada Government of

Canada / no client CEA X X

G24PEEM Guidelines 3 -Guidelines for cost-effectiveness analysis of

Vector-bornediseases

Inter-national

Panel of Experts onEnvironmentalManagement for

CEA X X X


9


pos

t

Ex a

nte

Sum

mar

y

vector control. (1993) Vector Control(PEEM), WHO / noclient

G25

Review of TechnicalGuidance onEnvironmental Appraisal(1999)

Environmentgeneral UK eftec / former UK

DETRCBA &CEA X X X

G27Guide to Cost-BenefitAnalysis of InvestmentProjects

General EUEvaluation Unit, DGRegional Policy, EC /no client

CBA &CEA X

G29 Cost-Effectiveness Analysisa Tool for UNESCO General Inter-

nationalSPM consultants /UNESCO CEA X X X

G30 DTLR Multi-CriteriaAnalysis Manual General UK former UK DETR/ no

client

CBA,CEA &MCA

X

G32Making Choices in Health:WHO Guide to CostEffectiveness Analysis

Health Inter-national

World HealthOrganisation / noclient

CEA X

G33

A Handbook for ImpactAssessment in theCommission: How to do anImpact Assessment

General EU

Strategic Planning andProgramming unit,Secretariat-General,EuropeanCommission / noclient

CBA,CEA &MCA

(X) X

G34 Samfundsøkonomiskvurdering af miljøprojekter.

Environmentgeneral Denmark

Danish NationalEnvironmentalResearch Institute / noclient

? ? ?

G35Kosten en baten in hetmilieubeleid, definities enberekeningsmethoden

Environmentgeneral Netherlands

Dutch Ministry forSpatial Planning,Housing and theEnvironment(VROM) / no client

? ? ?

G36 Evaluating EU Activities General EUEuropeanCommission DGBudget / no client

CBA &CEA X X

G37A Framework forEvaluating EnvironmentalPolicy Instruments

Environmentgeneral Finland Mickwitz, Per / no

clientCBA &CEA X X

G38Kosteneffectiviteitnatuurbeleid:Methodiekontwikkeling

Environmentgeneral Netherlands

Rijksinstitut voorVolksgezondheid enMilieu (RIVM) / noclient

CEA X

G39

Evaluating EU expenditureprogrammes: A guide: expost and intermediateevaluation

General EUEuropeanCommission DGBudget / no client

CBA &CEA X

G40

Ex-ante Evaluation: aPractical Guide forPreparing Proposals forExpenditure Programmes

General EUEuropeanCommission DGBudget / no client

CBA &CEA X X

G43YmpäristöpolitiikanTaloudellisten VaikutustenArviointi (Economic

Environmentgeneral Finland

Porvari, M. andHildén, M. (FinnishEnvironment Institute)

? ? ?


10


pos

t

Ex a

nte

Sum

mar

y

Assessment ofEnvironmental Policy)

/ no client

G44 WegwijzerEvaluatieonderzoek ex post General Netherlands

Dutch Ministry forSpatial Planning,Housing and theEnvironment(VROM) / no client

Neither X X

Of the fifteen guidelines summarised, there are:

• Three documents dealing with cost-effectiveness analysis for implementing the Water FrameworkDirective (G2, G12, G14);

• Six government or international body-issued guidance documents for the public sector in general(G8, G20, G24; G 29; G40; G44);

• Two government-issued guidance documents for evaluation of environmental policies (G9, G25);

• Two guidelines on using cost-effectiveness for project appraisal (G21, G22);

• One document primarily concerned with data collection and management as a prerequisite forcost-effectiveness analysis (G3); and

• One academic background paper on cost-effectiveness analysis of agri-environment schemes(G18).

The guidance documents and manuals identified in the study vary substantially in the level of detailthey provide about how to undertake cost-effectiveness analysis, especially with regards to technicalissues such as discounting, distributional impacts, effects on competitiveness, and so on. However, thebasic descriptions of the core stages of the cost-effectiveness analysis differ only a little.

As discussed in greater detail in chapter 3 below, the overall picture that emerges is that many usefulelements are present in the different documents, which together provide good insights on how toconduct an ex-post cost-effectiveness analysis. However, there is not one single document that wouldcombine all of these elements into one volume.

Regarding the distribution and the focus of the documents, the following observations can be made:

• There is a bias in the guidance towards ex-ante analysis. There are, however, some guidelines thatprovide insight in the processes and techniques of ex-post evaluation, including ex-post CEA.These are the Commission’s Guide on Evaluating EU Expenditure Programmes (G39), andEvaluating EU Activities: A practical guide for the Commission Services (G36); the DutchWegwijzer Evaluatieonderzoek ex post (G44), and the HM Treasury Green Book: Appraisal andEvaluation in Central Government (G8). While these documents give insights on ex-postevaluation in general, the treatment of cost-effectiveness analysis in these documents is rathersuperficial. Thus, HM Treasury Green Book (G8) mentions cost-effectiveness on three occasionsonly (p. 4, 37, 38) and defines CEA in one short sentence only (p. 4). Annex E of the EU guide onevaluating EU Activities (G36) lists several evaluation techniques including CBA and MCA,however CEA is not included in this annex (p. 89, 90), but is only briefly defined in the glossary(p. 103). The Commission’s Guide on Evaluating EU Expenditure Programmes (G39) providescomprehensive guidance on how to frame, set up and conduct an ex-post evaluation, e.g. in terms


11

of establishing causality between measures and outcomes, and singling out the effects of aparticular policy measure. Cost-effectiveness analysis is briefly discussed and compared to otherevaluation tools (p. 58), but on a rather abstract level. The Dutch Wegwijzer Evaluatieonderzoekex post (G44) discusses the distinction between efficiency, effectiveness and cost-effectiveness insome detail, but does not provide guidance on which costs to consider or how to measure them.

• In many cases, guidance documents will generally be written with ex-ante analysis in mind,treating ex-post analysis as a special case, and in far less detail (see e.g. HM Treasury Green Book(G8) or the Handbook for Impact Assessment in the European Commission (G33) as well as theDETR Review of Technical Guidance on Environmental Appraisal (G25)). As also documented inthe survey of legal requirements (see chapter 2.1), there are more cases where undertaking an ex-ante analysis is a legal obligation. Where organisations are not legally required to perform ex-postanalysis, the need for guidance will be less pressing and the focus of the guidance less clear-cut.

• In terms of spatial distribution of national-level guidance documents, good examples can be foundin three countries in particular: the UK (G 2, 8, 25), the US (G 9, 20) and the Netherlands (G 35,38, 44).

• The documents relating to the Water Framework Directive (G 2, 12, 13, 14) contain a largeamount of WFD-specific supplementary details and are of limited value as general guidancedocuments. Also, in line with the requirements for CEA established by the WFD, they focus onex-ante analysis only.

• The general guidance documents issued by governments or their agencies or international bodiesare for the most part not specifically related to the analysis of environmental policy, but have amuch broader scope. In this way, e.g. HM Treasury Green Book (G8), the Guide to Cost-BenefitAnalysis of Investment Projects issued by the European Commission, DG Regions (G27) or theDutch guidance on ex-post policy evaluation (G44) do provide general guidelines for assessing thecost-effectiveness of policies, but pay less attention to the specific needs of evaluatingenvironmental policies, such as the valuation of environmental goods and services, or theincorporation of long-term effects and irreversible damages. The HM Treasury Green Bookprovides examples of data sources for a range of impacts, including environmental impacts, withan entire annex devoted to the valuation of non-market goods. However, this type of data is morecommonly used in a CBA than a CEA.

• In G29, a paper on the potential role of CEA within UNESCO, the authors make the observationthat making CEA a permanent feature and an accepted tool within the organisation it will benecessary to modify the culture of the organisation, ‘which is very sceptical of what is seen aslimited "economistic" methods’. The paper makes the case for introducing incentives to ensurethat this evaluation tool is streamlined within the activities of UNESCO.

• At the same time, several guidance documents focus specifically on the evaluation ofenvironmental policy, including e.g. the OECD guide on evaluating economic instruments forenvironmental policy (G7), the US EPA guidelines for preparing economic analyses (G9, ex-anteonly), the eftec / DETR study on Review of Technical Guidance on Environmental Appraisal(G25), the Danish Economic assessment of environmental projects (Samfundsøkonomiskvurdering af miljøprojekter, G34) or the Dutch guidance on costs and benefits in environmentalpolicy (Kosten en baten in het milieubeleid, G35).

• An explicit distinction between financial and economic costs is made in most of the guidancedocuments. The documents use different terms to make this distinction, and sometimes the sameterms are used to mean different things. In some cases, the terms ‘direct’ and ‘indirect’ costs areused instead of ‘financial’ and ‘economic’, in other cases ‘social welfare losses’ are used to mean


12

economic costs. The US EPA guidelines (G9) are the most detailed in this respect, differentiatingbetween compliance costs, government regulatory costs, social welfare losses, transitional costs,and indirect costs. However, environmental costs are not always explicitly mentioned. The focusof some of the guidelines are on the social costs of the options assessed, while others are moreinterested in the costs to industry of proposed environmental regulation.

• At least one of the documents (HM Treasury Green Book (G8)) recommends cost-benefit analysisover cost-effectiveness analysis. A similar tendency to regard CEA as a simpler but inferioralternative to a CBA can also be discerned in the UK Water Framework Directive Guidance (G2).Other documents note that cost-effectiveness analysis should be performed when there aresubstantial doubts about the theoretical basis of the monetisation of benefits, or if environmentaltargets are set politically without a cost-benefit analysis.

• Some of the guidelines (G 2, 8, 44) point out that performing the cost-effectiveness analysis or theevaluation itself can be a significant drain on resources, and the effort put into the analysis shouldbe commensurate with the proposed program or policy.

2.3 Case Studies of applied ex-post Cost-Effectiveness AnalysesThis part of the project has reviewed applications of cost-effectiveness analysis in the evaluation ofenvironmental policies, with a strong focus on European studies and on ex-post analyses. To this end,more than 70 potential case studies were identified, 18 of which passed the selection criteria and werethus summarised and treated in greater detail. The project did not attempt to give a comprehensiveoverview of ex-post CEA in Europe, due to language limitations an emphasis was placed on studiesthat are published in English, French, German or Dutch. To identify a broad scope of potential studies,consultations were carried out with some national authorities as well as with the EEA’s network offocal points, as described in chapter 1. The following table presents a selection of the total casestudies, listing only those that were pre-selected for further analysis.

No. Author Year Title Country Policy area Timing SummCS1 NERA 2002 Fleetwide Emissions and

Cost-Effectiveness of theConsent Decree Pull-AheadRequirements for Heavy-DutyDiesel Engines

USA Air quality ex ante

CS2 Wright et al. 2001 The Cost-Effectiveness ofReductions in DioxinEmissions to Air fromSelected Sources

NewZealand

Air quality ex ante

CS8 Standard &Poor’s DRI

1999 The Auto-Oil II Cost-Effectiveness Study

FI, F, D,EL, IRL, I,NL, E, UK

Air quality ex post

CS11 IVM 2000 Cost-effectiveness of Dutchwater policies

NL Water ex ante X

CS12 RIVM 2000 Cost effectiveness ofenvironmental measures

NL Acidification ex ante X

CS13 RIVM 2004 Environmental costs ofenergy measures 1990-2010

NL Energy,Climate

ex ante /ex post

X

CS15 RIVM 2003 Evaluation of theImplementation memorandumfor emission ceilings,acidification and large-scaleair pollution 2003

NL Air quality ex ante X

CS19 CE Delft 2001 Treatment of plasticpackaging waste from

NL Waste ex-ante


13

No. Author Year Title Country Policy area Timing Summhouseholds

CS20 CE Delft 2000 Accelerated introduction ofcleaner petrol and dieselengines in the Netherlands

NL Air quality ex-ante

CS26 Resources forthe Future

1999 The Enhanced I/M Programin Arizona: Costs,Effectiveness, and aComparison with Pre-regulatory Estimates

USA Air quality ex post X

CS30 HarvardSchool ofPublic Health

2000 Are the Costs of ProposedEnvironmental RegulationsOverestimated? Evidencefrom the CFC phaseout

USA Ozone ex post X

CS31 SwedishUniversity ofAgriculturalSciences

2000 Cost efficient reductions ofstochastic nutrient loads to theBaltic Sea

Baltic Seacountries

Water ex ante

CS47 MacaulayLand UseResearchInstitute

2002 The cost-effectiveness ofbiodiversity management: acomparison of farm types inextensively farmed areas ofScotland

UK Biodiversity ex post X

CS49 Beamount, N.and Tinch, R.

2003 Cost Effective Reduction ofCopper Pollution in theHumber Estuary

UK Water ex post X

CS51 IIASA 1999 Economic Evaluation of aDirective on NationalEmission Ceilings for CertainAtmospheric Pollutants. PartA Cost-EffectivenessAnalysis

EU Air quality ex ante X

CS52 VTT 1999 Integrated cost-effectivenessanalysis of greenhouse gasemission abatement: the caseof Finland

FI Climatechange

ex ante X

CS53 AEATechnology

1998 Options to Reduce NitrousOxide Emissions

EU Climatechange

ex post X

CS54 AEATechnology

1998 Options to Reduce MethaneEmissions

EU Climatechange

ex ante

CS56 WRc Unknown

Examination of ExistingPolicy Options ... toImplement Directive76/464/EEC

EU Water ex post

CS57 eftec 2001 The Potential Cost andEffectiveness of VoluntaryMeasures in Reducing theEnvironmental Impact ofPesticides

UK Agriculture ex ante

CS63 Entec 2004 Review of the LargeCombustion Plant Directive

EU Air quality ex ante

CS69 TyndallCentre

2004 Ex post evaluations of CO2-based taxes: a survey

DK, FI, D,NL, NO, S,UK

Climatechange

ex post X

CS70 DMU 2004 Effectiveness of waste waterpolicies in selected countries– an EEA pilot study

DK, NL, F,E, PL, EE

Water ex post X

CS71 EuropeanTopic Centre

2004 Analysis of effectiveness ofimplementing packaging

AT, DK,IRL, I, UK

Waste ex post X


14

No. Author Year Title Country Policy area Timing Summon Waste andMaterialFlows

waste management systems

CS73 SPRU 2000 The Large Combustion PlantDirective (88/609/EEC): AnEffective Instrument ForPollution Abatement?(IMPOL)

F, D, NL,UK

Air quality ex post X

CS74 CERNA 2000 The Implementation of theMunicipal Waste IncinerationDirectives (IMPOL)

F, D, NL,UK

Air quality ex post X

CS75 SPRU 2000 The Implementation ofEMAS in Europe: a case ofcompetition betweenstandards for environmentalmanagement systems(IMPOL)

F, D, NL,UK

Population &Economy

ex post X

CS86 RIVM 2004 Evaluation of the DutchManure and Fertiliser Policy,1998-2002

NL Agriculture ex post X

It emerged that the practical experience with ex-post cost-effectiveness evaluations is unevenlydistributed in Europe, with much evidence coming from the Netherlands and the UK. The finding thatthese countries have a long tradition for such assessments is in line with the results of a 1998 study forthe European Commission, which surveyed the use of economic evaluation methods for environmentalpolicies in several European countries (Virani 1998).

In general, there is a limited awareness of the precise concept of cost-effectiveness, both byconsultants conducting the analyses and by the officials administrating them. Reports promisingdiscussions of cost-effectiveness sometimes turn out instead to be cost-benefit analyses (e.g., CS75)discussions on whether static or dynamic efficiency are being achieved (especially with respect tomarket-based instruments) (e.g., CS69), or aggregations of cost estimates unrelated to the outcomesachieved (case studies not summarised). Few studies were strict methodical cost-effectivenessanalyses of the type outlined in guidance documents (the most complete example of which was the USEPA guidance G9). Where cost-effectiveness ratios are actually calculated, they are sometimes notclearly defined (e.g. in the IMPOL studies CS73 and CS74).

• As stated in EEA (2001) and by Agnolucci (2004, CS69), environmental effect and environmentaleffectiveness should be treated as distinct concepts. The former is the physical outcome of theintervention, while the latter is a measure of this effect in comparison with what was expected orwith what other interventions have achieved. This distinction is not made in all case studies.

• Many of the aspects of cost-effectiveness analysis recommended by guidance documents are notcarried out in practice in the studies, presumably because of the difficulties of reconcilingtheoretical correctness with time, data, resource and skill constraints.4 For example, none of thestudies reviewed included lost consumer or producer surplus in their costs, as recommended by theUS EPA guidelines (G9). Furthermore, discounting, although recommended in almost all guidancedocuments, was not applied in most studies. This was particularly noticeable in CS52, whichdiscussed greenhouse gas abatement costs in Finland far into the future without the use of

4 Unfortunately, few of the studies are transparent about which aspects were omitted out and why, whichdifficulties and constraints were encountered, and how they were addressed.


15

discounting. As one exception, a study on energy measures in the Netherlands (CS13) not onlyapplied discounting, but also investigated the impact of choosing a social or a private interest rate.

• With regard to the choice of a baseline or reference scenario, business-as-usual baselinesrepresenting “the world without the intervention” are found less often than baselines which use asingle year as a reference point. The latter implies that without the intervention, environmentaloutcomes would have stayed constant at the level of the base year. This can lead to a largeunderestimation of the actual effect that an intervention has had.

• Some of the studies reviewed discussed the marginal abatement costs of emission reductions.However, it should be remembered that marginal abatement cost is only a proxy for cost-effectiveness, and becomes a less accurate proxy the more marginal abatement costs vary fordifferent emission levels. This is because the cost-effectiveness ratio should use the total cost of ameasure,5 whereas the marginal abatement cost is the cost per unit reduction at a particular stageof abatement, and ignores the fact that costs at an earlier stage may very well have been lower.6

Therefore, the marginal abatement cost is only an exact measure of cost-effectiveness if marginalabatement cost is constant across all emission levels, which would be a brave assumption.

• The most widely used sources of information were surveys of regulated business units (CS26,CS47, CS53, CS73, CS75), academic studies (CS12, CS30, CS47, CS51, CS52, CS75), firms’environmental reports (CS12, CS49, CS75), official national statistics (CS51, CS52), datatransmitted to the regulatory agency as part of the regulatory obligation (CS26, CS49, CS70,CS71), including data submitted to international bodies such as the IPCC or CORINAIR database(CS52, CS53). The latter included three studies where data was supposed to be reported toEurostat or other (CS70, CS71, CS73). Strikingly, some of these studies were conducted in thosecases where there least data was available. Other sources were realised using market prices fromtrade journals and newspapers (CS30), consultation with technical experts (CS30, CS75), andgovernment information on subsidy amounts (CS47).

• Some case studies addressed lack of data as a restriction for the analysis. One case study (CS53)noted that commercial sensitivity restricted the availability of data; another (CS69) noted that alack of data on the marginal costs of abating carbon dioxide make attempts to perform CEAproblematic. Other problems with data sourcing were noted in CS70 – i.e. insufficient dataprovided by Eurostat – and CS71, which found that it takes a long time for data to becomepublicly available. However, none of the studies explicitly discussed the cost of conducting theanalysis itself, or of the data gathering in particular

• Methodological considerations, such as the treatment of confounding factors and sensitivitytesting, are variably applied and are sometimes buried in the text rather than explicitly introducedas important parts of the cost-effectiveness analysis. CS12 is a notable exception in this regard,providing a comprehensive set of sensitivity tests that control for variations in the interest rate,depreciation period applied, indirect costs, effect of interactions between measures, timing ofdifferent measures and the impacts of relative price changes. Other case studies reflect uncertaintyby using different weightings for different parts of environmental effectiveness (CS26), differentassumptions about baselines (CS30), different lifetimes for abatement measures (CS49), to widerinfluences like reform of the Common Agricultural Policy (CS51) and economic growth (CS52).

5 Note that total costs here refer only to the additional costs associated with the measure itself, and not the totalcosts of achieving the environmental outcome.6 A more formal mathematical explanation would describe this by showing that the total cost of emissionsreductions is the integral of the marginal abatement cost between two different emissions levels.


16

3 Interpretation of the ResultsThe current project addressed the extent to which cost-effective considerations are taken up in theevaluation of environmental policy in Europe, and where they are, whether the analysis is consistentwith existing guidelines. In other words, is the current practice of ex-post cost-effectiveness analysismaking best use of available advice to quantify the effectiveness of policies and relate it to the costsencountered?

For environmental policy at the Community level, systematic ex-post assessment of cost-effectivenessis a fairly recent phenomenon. Of the total environmental acquis, only four Directives explicitlymandate that an ex-post assessment of cost-effectiveness be carried out. As these Directives all enteredinto force after 2000, no assessment has yet been carried out in response to the reporting obligationsfor these Directives.7

However, several ex-post cost-effectiveness assessments have been carried out to assess theperformance of other earlier Directives and Community programmes, even though the Directives andregulations themselves do not mandate such assessments. This includes assessments of the EU UrbanWaste Water Treatment Directive (CS70), the Directive on packaging and packaging waste (CS71),the Large Combustion Plant Directive (CS74) or the EMAS regulation (CS75). Likewise, there are afew examples where the implementation of European regulations at the Member State level has beenanalysed in a CEA (e.g. CS15 for the National Emissions Ceiling Directive in the Netherlands).

From the analysis of ex-post CEAs surveyed in this study, it has emerged that the scope, level ofdetail and methodological focus of ex-post CEAs differ substantially. As of yet, it is not possible toidentify one “common approach” to ex-post CEA that has been applied in different countries, or todifferent policy questions. On the contrary, a certain tendency of reinventing the wheel can bediscerned, e.g. in the case of the Water Framework Directive, where different Member States havecommissioned guidelines and handbooks in addition to the guidance prepared on the European level.This is not necessarily a negative development, as different approaches to implementing one and thesame Directive may be warranted by different conditions in the Member States (e.g. in terms ofavailable data, complexity of the decision situations, available human resources etc.). Yet it means thatmuch scope remains for policy learning and mutual exchange.

The actual implementation of the CEAs documented in this project differs from the theoretical idealof a CEA, more so in some cases than in others. The real-life practice combines several differentapproaches, all of which include assessments of costs and outcomes of some sort, but which do notalways closely resemble the textbook ideal of a CEA. Such changes are not always due to a lack ofunderstanding, but are often necessitated by data gaps or by time and capacity constraints. To dealwith these, authors will often take methodological shortcuts. For instance,

• A US study on the cost of CFC phaseout (CS30) uses marginal abatement cost as a proxy for cost-effectiveness, an EEA study on packaging waste (CS71) uses budgeted government expenditure asa proxy for costs, and the cross-country study CS74 on the implementation of the municipal wasteincineration directives uses data from two German Länder as representative of the whole ofGermany.

7 For the National Emission Ceilings Directive (2001/81), an extensive ex-ante cost-effectiveness has beencarried out in 1999 in preparation for the Directive (Amann et al. 1999). The first assessment of theimplementation of the Directive is due at the end of 2004, but was not available at the time of writing. Inaddition, a national ex-ante CEA for the implementation of the NEC Directive has been carried out in theNetherlands (Beck et al. 2004, CS15)


17

• Three studies (CS 69, 70 and 71) explicitly note that the lack of data makes analysis difficult, butderive their conclusions on the limited data base available;

• Some studies will omit certain parts of the analysis and certain types of impacts, or treat them in aqualitative way. Thus, many studies do not address impacts to the national economy, such asincreased expenditure, job creation etc. (this omission is explicitly noted in a study on copperpollution in the Humber estuary (CS49), but also applies to other studies). Other studies do notaddress secondary environmental impacts of abatement technologies used, or describe them onlyin qualitative terms (e.g. CS75 for the case of the EMAS scheme).

• Cost estimates are sometimes taken over from previous studies, even though these may not berecent ones (e.g. CS 15, 51).

• Confounding factors and parameters, such as economic growth, technological change, policydevelopments, the interactions and interdependencies between measures, the presence of side-effects, or the difficulty of relating measures to outcomes, are discussed in many studies. Moststudies would either mention them, but not incorporate them into the subsequent analysis (CS 15,30, 70, 71), or are treated in the sensitivity analysis only (CS 12, 51).

• For presenting results, a particular shortcut was applied in the IMPOL study on the largecombustion plant Directive (CS73), which described the cost-effectiveness of the comparedoptions only in qualitative terms as low, medium or high.

The variety of methodological shortcuts employed means that only a minority of case studies hasactually applied the different parts of a CEA that are described in guidance documents. Thus, forexample,

• A third of the summarised case studies do not consider sensitivity testing of any sort. While someothers employ sensitivity testing or at least some type of plausibility check (e.g. by comparingresults with other studies), only two provide an elaborated sensitivity analysis (CS 12, 51). In twostudies, a reduced form of sensitivity analysis is applied by using different baselines (CS 30, 52).

• Only four studies (CS 12, 49, 51, 53) apply discounting and discuss the effect that the choice ofdiscount rate has on the results, while other studies skip this part altogether.

• Only four studies (CS 12, 51, 52, 53) made use of models to estimate the cost-effectiveness ofpolicies.

• While many studies simply applied the status quo (or the situation in a given year) as the baselinefor the analysis, one study (CS70) did not specify a baseline for the analysis, making interpretationof the findings rather difficult.

• None of the studies provided a monetary valuation of environmentally beneficial side-effects, assuggested e.g. by the WFD-related guidance document G2.

The majority of these simplifications, shortcuts and omissions can be related to a lack of data, orrespectively to a lack of resources for gathering the necessary data. While the reviewed case studiesare not very transparent about the cost of conducting the analysis and of gathering the data, some ofthe guidance documents contain insights on this point. The particular difficulties of gathering ex-postdata on costs and effectiveness are discussed e.g. in G44 and G29, both of which note that datagathering ex-post can be more tedious than for ex-ante analysis. For example, the UNESCO guidanceon CEA (G29) notes that “Systematic C-E analysis presumes the existence of clear objectives, costdata and results indicators. Many times, however, organisations request ex-post evaluations of theeffectiveness of interventions that were never designed with any of these aspects in mind.”Consequently, all these steps that should have been taken up front have to be repeated ex-post.


18

When comparing different ex-post CEAs, it has to be considered that not all policy initiatives areequally suited for an ex-post evaluation by means of a CEA. The following conditions wouldappear most relevant for a successful ex-post CEA (see also G44):

• The objectives of the policy intervention have to be clearly identified and defined, ideallyconnected with a quantified target and a clear baseline.

• The policy should be connected to a fixed time period, identifying when policy targets should beachieved.

This diversity in terms of depth and detail also can also be related to the guidelines used. None ofthe guidance reviewed for this study is an “uncluttered”, easily-digestible general guidance documentfor performing CEA with respect to environmental policies. They are either a little too comprehensive,e.g. the US EPA’s Guidelines for Preparing Economic Analyses, or too general for non-economists,e.g. the UK Treasury Green Book, or too specifically-focused on one policy area, e.g. the WaterFramework Directive documents. Also, while most guidelines for ex-post cost-effectiveness analysesstrive to be theoretically comprehensive – which, by itself, is positive – they also need to take intoaccount the likelihood of data gaps and other practical difficulties in conducting analysis, and makepractical recommendations for dealing with these limitations.

On this point, the available guidance documents are mostly confined to a more or less concisetechnical description of cost-effectiveness analysis and its strengths and weaknesses. However, theygive much less guidance at all on how to deal with real-life difficulties, e.g. by specifying whichmethodological shortcuts can be advisable or at least justifiable. The exceptions to this are theguidelines aimed at the WFD (G2, G12 and G14), which are already embedded in a specific regulatorycontext, the section on communicating assumptions and methods in G9, the Dutch guidance on ex-postevaluation G44, and the outlining of issues surrounding the practicalities of data reporting in G3.

• For instance, the WFD-related guidance G2 argues for a tiered approach in determining the levelof detail of the analysis. Thus, it is suggested that the analysis can be limited if there is widespreadagreement among stakeholders on the measures to be implemented, if different alternatives differstrongly in the results that they deliver, or if either of the alternatives delivers significantadditional benefits.

• Likewise, the US EPA guidelines for preparing economic analyses (G9) recognise that someimpacts may escape quantification, and provide brief guidance on which of the markets affectedby a measure can be left out of the analysis.

• The Dutch guidance on ex-post evaluation (G44), by contrast, pays ample attention to theeveryday problems encountered by policy makers, including scarce resources, lack of time,political pressures etc. However, the document only describes evaluation in general and providesno information on how these findings relate to conducting a CEA.

• Practical limitations of CEAs and ways of overcoming them are also sometimes touched upon indiscussions of dealing with risk and uncertainty, but it is not explained how this can be relatedback to carrying out the assessment (see e.g. the US EPA guidelines for preparing economicanalyses (G9) or the European Commission handbook for impact assessments (G33)).

The emerging picture is thus that there is a considerable amount of guidance on Cost-EffectivenessAnalysis, which sheds little light on ex-post CEA, and that there is sufficient guidance on the practicalaspects of ex-post policy evaluation, which does however say little about cost-effectiveness and theway it can be assessed. That is to say: the knowledge of how to conduct an ex-post evaluation ofcost-effectiveness is available, but it needs to be combined from different sources. There is as yetnot one single document which provides all the relevant guidance in a consistent way.


19

As noted above, ex-ante CEAs are relatively more abundant than assessments carried out ex-post, a fact that is also reflected in the focus of most guidance documents. Since this project focusedon ex-post analyses, it was considered (i) whether ex-post assessments would deliver results that aremarkedly different from ex-ante CEAs, and (ii) whether experiences with ex-ante CEAs could beinferred to the practices of ex-post CEAs.

I. Regarding the first point, there are few cases where the results of an ex-post CEA weredirectly compared to an ex-ante analysis previously conducted for the same policy measure.The assessment by Resources for the Future of the enhanced inspection and maintenanceprogramme in Arizona (CS26) is one of the rare examples of such comparisons, concludingthat the ex-ante estimates of the costs of achieving the forecasted emission reductions wereunderestimated. Another assessment by James Hammitt (CS30) of the cost of CFC phase-outfound mixed evidence: while some ex-ante assessments substantially overestimated themarginal costs of limiting CFC consumption, others modestly underestimated this cost.8

II. For the second point, the small amount of studies comparing directly the results of ex-ante andex-post analysis prevents us from inferring specific conclusions regarding the relationshipbetween ex-ante and ex-post CEAs. What can be said, however, is that an ex-post CEA will bemuch easier to perform in cases where an ex-ante assessment has been carried out:

• Certain points that are crucial for a successful ex-post CEA will have been clarified incases where an ex-ante assessment has been carried out. This includes clearly defined andquantified targets for a policy intervention, a baseline scenario, and a timetable forachieving the targets.

• Carrying out an ex-ante assessment presents an opportunity to formulate at an early stagethe questions that should later be addressed in the ex-post CEA. This means thatmonitoring and reporting requirements can be designed accordingly, meeting the dataneeds of an ex-post CEA.

4 Recommendations and possible follow-up ActivitiesOne main objective of this study has been to derive recommendations for potential follow-up activitiesto be implemented by the EEA; these are discussed in chapter 4.3 below. In addition, policyrecommendations were derived regarding the use of project results for the impact assessmentprocedure at the EU Commission (chapter 4.1), as well as the implementation and evaluation ofexisting EU environmental legislation (chapter 4.2).

4.1 Ex-post CEAs and the Commission’s Impact Assessment ProcedureThe results of this project have implications for the trend in European environmental policy towardsmore and better assessment of the impacts of policies, both ex-post and ex-ante. At the same time,policy evaluation is clearly not a goal in and of itself, but has to serve a specific purpose. Consideringthe time and resources that flow into evaluation exercises like an ex-post cost-effectiveness analysis, itis clear that the expenses will be justified only if the results of the analysis have a practical impact onpolicy making. Thus, the evaluation of policies becomes a useful tool once the results feed back into

8 A 1999 study published by the Stockholm Environment Institute, “Costs and Strategies presented by Industryduring the Negotiations of Environmental Regulations” (CS9), was not considered in detail in this project: whilethe study did compare ex-ante and ex-post estimates of costs, it did not relate to these to the effectiveness ormeasures or compare their cost-effectiveness.


20

the policy process: be it for the further implementation of the same policy, or for future policyinitiatives in a related field.

One of the main processes where cost-effectiveness considerations may play a role is the(Sustainability) Impact Assessment - (S)IA. At its Gothenburg summit in 2001, the European Councildecided that an ex-ante sustainability impact assessment should be carried out for all major policyproposals, thereby establishing these assessments as a cornerstone for the coherent implementation ofthe EU Sustainable Development Strategy. With its communication (COM 2002/276). the EuropeanCommission developed a highly comprehensive approach to impact assessment. One motive behindthe current initiatives in the EU is the establishment of more efficient and “leaner” decision-makingprocedures.

The relation between impact assessments (such as (S)IAs) and ex-post assessments (including cost-effectiveness analyses) is ambivalent. Different types of interactions can be conceived of:

• Ex-post assessments can be used to follow up on ex-ante appraisals, and to put their role intoperspective. Ex-post appraisals can be employed to assess whether the predicted costs or theexpected impacts have actually been incurred, or to reveal where they have been clearly under- oroverstated. In the medium term, this information can be used to improve the quality of ex-anteappraisals, by revealing the crucial influence of particular assumptions or methodological choices.

• At the same time, the comparison with ex-post analyses could also help to better define the role ofimpact assessments in the policy making process, showing their usefulness and their limitations.Also, if it is clear from the outset that an (S)IA will be re-evaluated at a later stage, this could givean extra incentive to carry out the assessments more thoroughly, eliminating the likelihood that(S)IAs are drawn up in before a decision is taken, but never re-considered afterwards.

• It is also possible that the existence of ex-post evaluation and monitoring requirements will reducethe burden placed on ex-ante appraisals. Where it is clear from the outset that the performance andthe cost-effectiveness of a policy will be re-evaluated during the implementation, the requirementsfor an ex-ante assessment of all expected impacts may become less strict.

• At the same time, strengthening the link between ex-ante and ex-post assessments can also makeboth more effective: in this sense, the ex-ante impact assessment should comprise a list of issuesthat should later be addressed through an ex-post assessment, including the cost-effectiveness ofmeasures taken.9 This is of particular relevance for the data collection – a recurring problem for anex-post analysis is that necessary data on impacts and expenditure is not available. An ex-anteappraisal would be well suited to identify the data needs that have to be collected during theimplementation phase, as many of the questions later to be answered through the ex-postassessment will also be raised during the ex-ante appraisal. Such an initiative should consider the“Monitoring and Evaluation” requirement of (S)IAs as formulated in the guidance documents forCommission impact assessment (European Commission 2002, European Commission undated(G33)). A review conducted by the Institute for European Environmental Policy (IEEP 2004)concludes that “Almost all IAs make some reference to monitoring procedures [...]. However, fewspecifically address the question of what specific data is required to assess the impact ofmeasures.” Thus, (S)IAs could not only be used to already identify data needs, but ex post CEAscould also reinforce the monitoring and evaluation as required by (S)IAs.

9 In fact, such a requirement exists for impact assessments conducted by the European Commission, but is notalways followed up on in the assessments carried out so far.


21

• Better integration of ex-ante appraisal and ex-post evaluation will also mean that the ex-anteassessment may take on a different form and focus. It would be expected that the assessment couldbecome more action-oriented, identifying weak points and bottlenecks that are crucial for theimplementation, and thereby also setting the focus for an ex-post analysis. In other words, theassessment would be less of a conclusive judgement on which option is or is not worth pursuing,but would rather specify the conditions under which an option is preferable.

However, if a stronger integration between ex-ante appraisal and ex-post evaluation is pursued, twomain caveats should be considered:

• Uncertainty – both ex-ante and ex-post assessment have to deal with uncertainty to a degree.,where the former has the problem of predicting realistic impacts, the latter has the problem ofrelating the observed impacts to individual measures and initiatives. In this sense, both are limited,and it is not necessarily possible to prove the ex-ante appraisal wrong with the benefit ofhindsight.

• Scaling and agency – whereas the sustainability impact assessments are carried out on the EUlevel and by the Commission, ex-post assessments for many Directives would be carried out onMember State level, and by national administration officials. This means that the scale of theanalysis will be different, affecting also the level of detail at which information is obtained; andthis means that the questions initially identified by the Commission may not be equally applicableto all Member States.

4.2 Regarding the Evaluation of existing EU Environmental LegislationThe main findings of this project – a diversity of approaches followed in real-life CEAs, and a lack ofguidance targeted specifically at ex-post CEA – are clearly relevant for the implementation andevaluation of those Directives that require an ex-post evaluation, including cost-effectiveness aspects.For these four Directives identified in Chapter 2.1, the first round of evaluation is either underway orwill be carried out in the coming year, highlighting the need for specific guidance and good-practiceexamples of ex-post CEAs.

At the same time, the findings of this project are also relevant for the implementation of otherDirectives identified in this study, which either provide for an ex-ante cost-effectiveness analysis orwhich require ex-post reporting of effectiveness in a broader sense. In both these cases, findingsrelated to the methodology and practice of ex-post CEAs, including specific guidance, can provideimportant inputs.

I. In policy areas where an ex-ante CEA is required: To support the learning from policyimplementation, it seems advisable to re-consider the results of such an ex-ante analysisduring and after the implementation, in order to see if the ex-ante analysis succeeded inassessing expected impacts, and if the judgement made regarding the most cost-effectivesolution was indeed correct. Such knowledge can be a valuable input for the furtherimplementation process, or for other subsequent policy initiatives in the same field.

The Water Framework Directive provides an example of this. The WFD requires programmesof measures to be drawn up in order to reach good ecological status in all water bodies by2015. The selection and combination of measures shall be guided inter alia by cost-effectiveness considerations. It is foreseen that the programme of measures will be adaptedand revised at six-year intervals, repeating the cost-effectiveness analysis for the selection ofpotential measures. Although there is no formal requirement to do so, it seems highlyadvisable to base the selection of measures after 2015 on an assessment of how far thejudgements made in the first planning cycle regarding the cost-effectiveness of measures were


22

indeed correct. To this end, an ex-post analysis would be necessary to assess the extent towhich the planned objectives have actually been reached, and if not, then why not. Similararguments can be made for other Directives that are implemented over a longer time periodand with more than one implementation and reporting cycle.

II. The second possible application concerns those Directives that mandate an ex-post evaluationof the policies’ performance or effectiveness, but do not explicitly require a cost-effectivenessanalysis. However, even a loose evaluation that does not qualify as a CEA in the proper sensewill often involve a qualitative description of cost-effectiveness, or an unrelated juxtapositionof information on costs and on effects. With some guidance and better data, such assessmentscould be developed further towards a CEA. Here, it needs to be assessed whether theevaluation would benefit from giving a greater weight to cost-effectiveness considerations,e.g. by making the evaluation more stringent and more coherent.

Regarding the actual ex-post assessments that will eventually be carried out – be they fully fledged ex-post CEAs or other types of evaluations involving cost-effectiveness – it should be considered that thisstudy only provides a first scoping of the available evidence. As many of the Directives requiring anex-post evaluation of (cost-)effectiveness are still in their first reporting period, the number of ex-postevaluations carried out both at the EU and the Member State level will increase in the near future.

This raises the question of how the assessments themselves will be assessed: what constitutes asuccessful assessment, and how can the value of an assessment for subsequent policy making beassessed? In this context, it also needs to be established which institutions will be responsible forreviewing assessments, and how the results of assessments will flow back into the policy makingprocess.

As previously mentioned, there are notable differences between individual Member States when itcomes to evaluating the (cost-)effectiveness of environmental and other policies. Judging by thenumber of case studies and guidance documents surveyed in this study, systematic andinstitutionalised procedures for evaluation and appraisal would appear to be furthest developed in theUK and in the Netherlands, supported by cross-cutting requirements to evaluate the performance andcost-effectiveness of major policy initiatives (see also box on p. 7). This observation is also supportedby Virani (1998), who surveyed the use of economic evaluation methods for environmental policies inseveral European countries. For those European Directives that require Member States to report oncost-effectiveness, it can be expected that the capacity and experience built up will also be reflected inthe quality of the assessments (e.g. Directive 2000/60 (WFD), Directive 2002/30 (noise-relatedoperating restrictions at Community airports) and Directive 2004/8 (Cogeneration)).

4.3 Possible Follow-up Activities by the European Environment AgencyBased on the findings of this study we conclude that potential further activities by the EuropeanEnvironment Agency in the area of ex-post CEAs should concentrate on providing appropriateguidance for their application:

• In their thematic scope, most guidance documents reviewed in this study are either too wide or toonarrow – either explaining how to assess the performance of each and every policy initiative, orapplicable to the implementation of one particular Directive only.

• In terms of assessment methods, there is little guidance specifically targeted at cost-effectivenessanalysis. Instead, most guidance documents treat CEA as one (minor) point next to cost-benefit ormulti-criteria analysis, or even present it as an inferior alternative to a CBA.

• Regarding the timing, most guidance documents are geared towards ex-ante assessments, treatingex-post analyses as a special case if at all. There are some guidance documents on ex-post


23

evaluation in a broader sense (e.g. G 8, 36, 39, 44), which also mention cost-effectiveness as onecriterion and explain the concept. However, these documents provide insights, but no hands-onguidance on how to conduct an ex-post CEA, which would lead us to conclude that there is nospecific guidance for ex-post cost-effectiveness analyses.

These findings suggest that there is a need for a specific guidance document on how to conduct ex-post CEAs for environmental policy measures. Based on the outcomes of this project, such a guidancedocument should have the following properties:

Format of the guidance documentA clearly structured guidance document on performing cost-effectiveness analysis forenvironmental policy in the EU, with appendices relating to different policy areas or Directives,and with clear pointers to further information on certain aspects of the process if required, ratherthan attempting to be fully comprehensive.

For this format, a web-based implementation with links to good practice examples, in-depthguidance for particular aspects, downloadable checklists etc. would seem most suited.10 The web-based PANACEA database developed for this project could serve as a starting point for such anapplication.

Focus of the guidance documentThe guidance document should have a clear focus on the application of CEA for the ex-postevaluation of environmental policies. The current and future Directives requiring ex-post assessmentof cost-effectiveness would clearly be a starting point, as would be the follow-up evaluation ofSustainability Impact Assessments carried out by the European Commission. In specifying the level ofdetail for the proposed guidance document, there is an obvious yet unavoidable trade-off: parts of aCEA will necessarily be complex and technical, at the same time the guidance also needs to beunderstandable and useful for non-economists and practitioners in the administration. A practicalguideline will therefore need to skip some of the more complex aspects, or move them to an annex. 11

Approach of the guidance documentAbove all, the guidance document needs to embody a pragmatic approach. There is sufficientguidance to explain the theoretical foundations and to elaborate the requirements for an ideal prototypeCEA, the US EPA Guidelines for Preparing Economic Analyses (GS9) being the most developedguideline in this regard. However, a practice-oriented guideline should not only be theoreticallycomprehensive, but should also consider the likelihood of data gaps and other practical difficulties inconducting analysis, and should make practical recommendations for dealing with these limitations.

10 Some of the manuals and guidelines discussed in this paper have such web-based interfaces, however withmuch less detail and interconnections than suggested here. This includes HM Treasury Green Book (G8) athttp://greenbook.treasury.gov.uk/, the eftec/DETR Review of Technical Guidance on Environmental Appraisal(G25) at http://www.defra.gov.uk/environment/economics/rtgea/, or the DG Regional Policy Guide onEvaluation of Socio-Economic Development (G27) at http://www.evalsed.info/frame_about.asp.11 An alternative would be to develop the guidance in two parts: a summarised guidance for the administrationofficials commissioning the assessment, and a more detailed and technical guidance for those carrying out theactual analysis.


24

Rather than describing a prototype CEA, it should also explain which simplifications and shortcuts arejustifiable under which conditions, and how this will affect the quality of the results.

A guidance document of this type should also help practitioners to determine how much effort to putinto an ex-post evaluation. Although a CEA is generally less of a strain on time and resources thanother appraisal types, a fully fledged CEA can still require substantial inputs of manpower andresources, especially in order to assess the effects of the investigated measures. In order to cope withlimited administrative capacities, a targeted and proportional approach may be called for, whereby thecomplexity of the analysis is adjusted depending on the complexity of the decision situation, e.g.through an initial screening. How this can be achieved in a methodologically sound way would need tobe explained in the guidance document.

Practical impact of the guidance documentNext to providing methodological guidance, a guidance document should also providerecommendations on how the results of an ex-post cost-effectiveness analysis can feed back into thepolicy making process, and how the political impact of results can be enhanced.

• Above all, this concerns the clarity, brevity and the structuring of information for the reader’sbenefit. This is especially appropriate where the output of the analysis is targeted at a non-economic audience, or where it is presented as part of a public participation process. Some of theguidelines surveyed in this project contain such recommendations on how to present results, e.g.HM Treasury Green Book (G8) or the European Commission Handbook for Impact Assessments(G33). However, the majority of case studies summarised in this study are rather targeted at anacademic audience, and are therefore difficult to digest for a non-specialist reader.

• Another point concerns the optimal timing of the assessment in the policy making process. Onthe one hand, the analysis has to take place long enough after the policy intervention to observe aneffect, which may take several years in the case of environmental policy measures. On the otherhand, it also has to take place early enough to have an influence on possible follow-up measures,or the decision on (dis-)continuing the measure. The set of case studies analysed in this study areinconclusive in this regard: while several of them were supposed to feed into a specific policyprocess, there is no indication of the actual impact these studies may have had. Other studies werewritten for informative purposes rather, unrelated to a specific future policy process.

Possible extensions to the guidance documentAn additional feature of the guidance document would be to introduce different types of assessmentmethods, and to support the choice among these. This would take the form of a checklist, indicating inwhich cases cost-effectiveness analysis is the appropriate, necessary or sufficient evaluation method,and in which cases other methods (such as cost-benefit analysis, multi-criteria analysis or semi-quantitative screening methods) are preferable.12 The impression arising from this project is thatcurrently there is no systematic approach regarding the choice of evaluation methods for EUenvironmental policy. The feature of choosing between different evaluation methods would become allthe more relevant if the scope of the guidance were extended to include Directives requiring an ex-postevaluation of effectiveness in general, rather than ex-post CEA only (see 4.2 above).

12 Some information on this can be expected from the ongoing FP6 project “Sustainability A-Test”. However,since one of the objectives of Sustainability A-Test is to support the ex-ante sustainability impact assessmentscarried out by the European Commission, ex-post evaluation does not feature prominently in the project.


25

Another possible extension concerns the further development of the PANACEA database: first of all,PANACEA could be developed further into a portal for the guidance document as described above,linking the guidance elements to data base entries. Secondly, the PANACEA database could beupdated continuously, as more ex-post assessments of European environmental policy becomeavailable. In this way, the database can be developed into a reference inventory for assessments carriedout in response to EU Directives, including assessments carried out on the Member State level.

5 ReferencesAgnolucci, P. (2004): Ex post evaluations of CO2 –based taxes: a survey. Tyndall Working Paper 52,

June 2004. Tyndall Centre for Climate Change Research

Amann, M. et al. (1999): Economic Evaluation of a Directive on National Emission Ceilings forCertain Atmospheric Pollutants. Part A: Cost-Effectiveness Analysis. Final Report, November1999. Laxenburg: IIASA

Beck, J.P., R.J.M. Folkert and W.L.M. Smeets (2004): Beoordeling van de UitvoeringsnotitieEmissieplafonds verzuring en grootschalige luchtverontreiniging 2003 (Evaluation of theImplementation memorandum for emission ceilings, acidification and large-scale air pollution2003). RIVM rapport 500037003/2004. Bilthoven: RIVM

Department of Transport, Environment and the Regions (1999): Review of Technical Guidance onEnvironmental Appraisal. London: DETR

EEA (2001): Reporting on environmental measures - Towards more ‘sound and effective’ EUenvironmental policies. Environmental issue report 25, European Environment Agency,Copenhagen.

European Commission (2002): Communication from the Commission on Impact Assessments(COM(2002)276 final). Brussels, 5.6. 2002

European Commission (undated): A Handbook for Impact Assessment in the Commission: How to doan Impact Assessment.

European Commission, DG Regional Policy (undated): Evaluation of Socio-Economic Developments– THE GUIDE. http://www.evalsed.info/

Institute for European Environmental Policy (2004): “Sustainable Development in the EuropeanCommission’s Integrated Impact Assessments for 2003”. Final Report.

Virani, S. (1998): Economic Evaluation of Environmental Policies and Legislation. Report Preparedfor the European Commission, DG III. Final Report, September 1998. Norfolk: Risk & PolicyAnalysts Ltd.