1 ex post evaluation of cohesion policy interventions 2000-2006 financed by the cohesion fund work...

1

Ex post evaluation of cohesion policy interventions 2000-2006 financed by

the Cohesion Fund Work Package C – cost-benefit analysis of selected

environment projects

Adam AbdulwahabEvaluation network, Brussels, 20th October 2011

#

Background

Study period: January 2010 – September 2011

Consortium:

209 / 2011 Ex post evaluation for Cohesion Fund 2000-2006, WP C

http://www.tecninvest.com/index_e

#

The 3 overall questions the present study set out to address

1. What were the impacts of the examined projects?

2. How can ex post cost-benefit analyses contribute to the practice of ex ante cost-benefit analyses?

3. What are the potentials and limits of carrying out an ex post cost-benefit analysis to identify and/or analyse the impact of projects? Is it an appropriate tool for impact analysis?


Recommendations

#

Methodology


• Initial list of 40 major environmental Cohesion Fund/ISPA projects that were completed (or nearly completed).

• 20 projects were selected to ensure a good mix of sectors (waste management, wastewater and water management) and to ensure wide geographical coverage.

• The short listed 20 projects were briefly examined (brief project descriptions based on available material and initial contacts).

• 10 projects were selected for in-depth case studies using CBA ex post .

• The ten case studies were based on extensive data collection among stakeholders.

The figure presents a brief overview of data collection approaches applied -> -> -> -> ->

#

Methodology used in the case studies

The basis of the study was 10 Cost Benefit Analyses ex post, where the following issues were also addressed:

Utilisation,

Validity of assumptions made during the ex ante analyses,

Risk mitigation measures, including consequences of mitigated and (both foreseen and unforeseen) non-mitigated risks,

Costs and benefits that cannot be expressed in monetary forms,

Meeting environmental requirements,

The contribution of accompanying actions which are outside the project but intended to enhance the project success (qualitative approach only),

Unintended effects, and

The calculation of the margin of error of the ex post cost benefit analysis.


#

Overview of the 10 case studies

609 2011 Ex post evaluation for Cohesion Fund 2000-2006, WP C

Waste management

Water management

Waste water management

01 Bulgaria

03 Pilsen

17 Barcelona

06 Crete

09 Zaragosa

27 Hungary

22 Dublin

13 Madrid

29 Poland

50 Portugal

# 709 / 2011 Ex post evaluation for Cohesion Fund 2000-2006, WP C

Interpretation of the study results – based on the 10 cases

Positive net impacts for some but not all projects

2 projects (solid waste PT, water supply ES) showed net positive impacts

4 projects (water CZ, water management ES, water PL, wastewater IE) showed positive impacts (better services to population, impacts on natural environment) but at high costs resulting in negative ENPV and B/C ratio < 1

4 projects produced a stream of benefits that did not outweigh costs - overcapacity (solid waste ES), demand forecasts (solid waste EL), expensive technical solutions (solid waste BG, wastewater HU)

But caution: relying on ENPV and B/C ratio can be misleading!

ENPV >0 and B/C ratio >1: monetised benefits exceed monetised costs over time

ENPV <0 and B/C ratio <1: monetised cost exceed monetised benefits over time

Benefits are more difficult to monetise (i.e. quantify and attach economic value expressed in money)


Interpretation of the study results – based on the 10 cases

Legal compliance:

Does not ensure positive impacts in economic welfare terms – in most cases this was achieved at costs that exceeded the benefits that could be quantified

Achieving positive ENPV may be impossible on project level; especially where projects were undertaken to meet regulatory requirements

Two cases of non-compliance driven objectives (water supply ES, water management ES): objectives had environmental focus relevant to the needs of the region

Generally, the projects generated positive side effects in terms of environmental awareness and institutional learning

Distributional effects

many projects benefit low-income regions, indirectly paid by higher income regions

CBA as a tool does not capture such distributional effects.

#

01 Bulgaria Waste management Investment 57 million / 228 000 population


Project objectives: to achieve solid waste management in full compliance with EU and Bulgarian Regulations and as a result avoid potential environmental damages from landfills

Comparison of ex ante and ex post CBA results:

Analysis:

FNPV M€

FRR %

ENPV M€

ERR % B/C ratio

Ex ante 4.1 n/a n/a n/a n/a

Ex post -2.2 n/a -4,4 -4% 0.43

Risk assessment: main challenge is affordability because of low income level

Financial sustainability: only operational costs can be financed by the waste tax, reinvestment costs are not included e.g. for sealing of the cells

Direct benefits: Reduction of illegal waste disposal

Other outcomes: A first step towards modern waste handling system

The project could have taken waste management further by introducing much more recycling measures in the project

Legal compliance as driver Yes

Technical solution Ok

Capacity issues Ok

Budget reasonability Ok

Options analysis N/A

#

06 Greece Waste management Investment 28 million / 80 000 population


Project objectives: to collect and treat waste produced by the Municipalities of Chania, Akrotiri, Souda, Keramies, Eleftherios Venizelos, Therisos, Kydonia, Platania and Mousouri


Analysis: Risk assessment: financial sustainability is an issue.

Operations need to focus on improving recycling rate and market uptake of compost.

Financial sustainability: operational costs are much higher than budgeted, and revenues lower than budgeted.

Direct benefits: reduction of illegal waste disposal of 90000 tons/year, recycling of 30000 tons/year



Capacity issues Over capacity


Options analysis Yes

FNPV M€

FRR %

ENPV M€

ERR %

B/C ratio

Ex ante 20 34% na na na

Ex post -3.7 neg -17 0.5% 0.22

Unrealised benefits: use of compost, reduced use of pesticides, refined sorting of water, 100% waste management coverage

Hindrances for benefits: lack of awareness among farmers, lack of education of households to sort waste, lack, lack of reinforcement of illegal waste disposal

Unintended effects: waste management know-how, increased environmental awareness

#

13 Madrid/Spain Waste & energy recoveryInvestment 74 million / 3 million population


Project objectives: to seal off of the 'old' landfill, the construction of an Energy Recovery Plan and landscape restoration on top of the old landfill


Analysis:

Risk assessment: No major risk elements were identified

Financial sustainability: The energy generation component generate a surplus, while the other components require operational funding from Municipality

Direct benefits: Closing off illegal landfill, stop to negative environmental impacts, energy generation

Unrealised benefits: unutilized synergies with other facilities of the entire Valdomingez facility

Hindrance to benefits: Contractual bindings of current contracts with different operators

Other outcomes: the project was important part of the combined Madrid Waste Management facility



Capacity issues Over capacity


Options analysis Part of bidding process

FNPV M€

FRR %

ENPV M€

ERR %

B/C ratio

Ex ante 10.1 6% 36.5 8.9% 1.3

Ex post -9.1 4 -47 n/a 1.08

#

50 Lipor/Portugal Waste management Investment 80 million / 1 400 000 population


Project objectives: to complement and to extend some of the existing activities LIPOR and its city councils shareholders and to remedy some environmental liabilities, aiming at the consolidation of the integrated management system in the region.


Analysis:

Risk assessment: the composting facility is the main reason for poor economic performance of the overall investment

Financial sustainability is ensured by green or 'guaranteed' administrative price and close control of costs

Direct benefits: comprehensive and efficient management of urban solid waste

Hindrances for benefits: implementation delays, acceptance of compost quality by farmers, awareness of waste sorting and recycling among the population



Capacity issues Ok


Options analysis Partly

FNPV €

FRR %

ENPV M€

ERR %

B/C ratio

Ex ante 9.6 7.3 na na na

Ex post -42 <0 na na na


03 Czeck Rep / Pilsen: combined waterInvestment 30 million EUR / 170 000 population

• Project objectives: was to expand and rehabilitate the water and wastewater system in order to comply with the EU environmental acquis

• Comparison of ex ante and ex post CBA results:

Analysis:Legal compliance as driver Yes


Capacity issues Ok


Options analysis uncertain

FNPV M€ FRR % ENPV M€ ERR % B/C ratio

Ex ante -0.19 5.9% -0.3 2.6% 0.74Ex post -0.06 6% -0.45 1.9% 0.66

• Risk assessment: No major risks have been identified, risks were assessed and mitigated

• Financial sustainability: Yes, the affordability of the total tariffs showed no risk for payment problems among the households

• Direct benefits: Improved quality of drinking water, reduction of pollution in the river due to less spill over and connection of 1000 household to sewage system

• Other outcomes: the environment in the river is improving

#

09 Spain /Zaragoza: Water supply Investment 121 million EUR / 821 000 population


Project objectives: to implement a water supply system capable of guaranteeing constant distribution of good quality water in the target area.


Analysis:

• Risk assessment: undertaken and the design is based on the results

• Financial sustainability: The project is sustainable thanks to full cost recovery tariffs

• Direct benefits: improved water quality

• Other outcomes: reduction in water treatment costs – reduced water leakages – awareness raising activities

Legal compliance as driver Not only


Capacity issues Ok



FNPV M€

FRR %

ENPV M€

ERR %

B/C ratio

Ex ante – 71.6 n.a. 180.54 20.0 2.11

Ex post – 67.9 -0.81% 36.65 7.6 1.28

#

17 Spain /Barcelona: Waste water Investment 69 million EUR / 130 000 population


Project objectives: to regulate flooding in certain parts of the city building by the retention tanks and belonging connectors.


Analysis:

• Risk assessment: it was concluded that the project is unlikely to ever generate an economic surplus. Important variable were investment costs, value of beach days, avoided damage costs and the discount rate.

• Financial sustainability: The project is not sustainable. No income is generated. However, the economic assessment shows a sustainable investment.

• Direct benefits: Reduced risk of flooding, improved know-how of water management, access to ground water for irrigation and street cleaning, improved ecosystem in marine water.

• Other outcomes: a lot of knowledge and experience has been gathered. The project has many visits from other cities and countries. High awareness raising among population.

Legal compliance as driver No


Capacity issues Ok



FNPV M€

FRR %

ENPV M€

ERR %

B/C ratio

Ex ante -43 4.87 16.7 14.43 1.47

Recalculated ex ante -52 Na -26 - 0.56

Ex post -58 -7.07 -1.1 5 .98


22 Dublin Ireland: Waste waterInvestment 500 million EUR / 1 145 000 population

• Project objectives: to comply with the environmental legislation, and through this compliance improve the quality of life of the Dubliners


Analysis:


Technical solution Not optimal

Capacity issues Under-capacity

Budget reasonability No


• Risk assessment: The project is very unlikely to generate an economic surplus

• Financial sustainability: need for further identification of investment funds and a revision of user fee structure

• Direct benefits: an increase in housing permissions and recreational benefits, in particular with cleaner Dublin Bay beaches

• Hindrances to realising benefits: the design capacity of the new infrastructure is too low and it does not comply with the requirements of the newly designated sensitive waters, hence, further investments are needed to fully reap the benefits

FNPV M€

FRR %

ENPV M€

ERR %

B/C ratio

Ex ante na na 118 6.7% 1.21

Ex post -675 -11.4% -325 -4.3 0.37


27 Szeged/Hungary: Waster waterInvestment 95 million EUR / 170 000 population

• Project objectives: to comply with the Waste Water Directive, including the expansion of the existing waste-water treatment plant, sewer expansion and household connections

• Comparison of ex ante and ex post CBA:

Analysis:



Capacity issues Ok

Budget reasonability No

Options analysis N/A

• Risk assessment: financial risk was investment costs overrun, economic feasibility of sewage network expansion is uncertain, the plant expansion is not economically feasible.

• Financial sustainability: Yes, but fixed assets are not in the books. Tariffs are sufficient to cover operating and reinvestment costs.

• Direct benefits: Extension of the sewage system to new users

• Other outcomes: increased awareness among population. The project was excellently run.

FNPV M€

FRR %

ENPV M€

ERR %

B/C ratio

Ex ante -25.8 0.9 29 9.9 1.32

Ex post -54.6 -1.5 -70.7 -5.2 0.31


29 Szczecin/Poland: Waste waterInvestment 281 million EUR / 406 000 population

• Project objectives: a complete renovation of an outdated waste water treatment


Analysis:


Technical solution OK

Capacity issues OK

Budget reasonability OK


• Risk assessment: The project is not financially sustainable and likely to lead to an economic return in the future.

• Financial sustainability: the project appears to be financially sustainable as user fees have been allowed to increase in line with the increase in the quality of service provision.

• Direct benefits: improved quality of waste water and water supply services, and improved environmental conditions of the Oder river.

• Other outcomes: Szczecin is no longer a HELCOM hotspot.

FNPV M€

FRR %

ENPV M€

ERR

B/C ratio

Ex ante na 7.4 na na na

Ex post -52 1.9 -57 1.8 0.87

#

2. What have we learned that can improve ex ante CBA?

1910 Mar 2011 Ex post evaluation for Cohesion Fund 2000-2006, WP C

"it is a matter of making the financial analysis look as bad as possible in order to increase the funding need, and to make the economic analysis to look as

positive as possible in order to justify the public funding"

Quote from workshop with MS

The main shortcomings identified:

not integrated in the decision process (merely 'a ticking the box' exercise)

not looking at individual components

missing the bigger pictures (e.g. synergies, risk un-realised benefits)

#

2. What have we learned that can improve ex ante CBA?

2009 / 2011 Ex post eval cohesion fund_DG Regio

Recommendations:

CBA thinking should apply to the selection of alternatives, prior to the final design of the project.

Do the CBAs much earlier in the process. A solid CBA should precede the final technical design of the project.

CBA to be related to the Master Plan context.

Cost efficiency analysis could be considered for 'need to have' projects.

Clearer distinction between the financial and the economic CBA.

CBA training to enhance the CBA culture from the start of project generation

Access to common resources, e.g. best practice examples, catalogue of how to calculate externalities, standard values, etc.

#

3. Is ex post CBA a good methodology to identify and assess the impacts of the projects?

CBA as a methodology is not common practice for ex post evaluation

However this study has applied CBA post-decision

Lessons learnt:

1) Timing matters: ex post analysis is too early right after project completion

2) starting from individual components is the easiest way to identify the benefit elements

3) concentrate on valorisation of the main benefit elements

4) wider benefits are often the important outcomes but difficult to quantify


Considerations

break-even analysis might be more convincing to illustrate economic surplus/deficit

develop a price and benefit catalogue (Excel tool) to support CBA in practice

if the ex ante CBA is done according to the guidelines, the ex post CBA is merely 'a reality check' updating key variables

CBA to be combined with other qualitative impact assessment methodologies to improve the capturing of wider benefits

#

Conclusions

CBA is an efficient tool both for ex ante and ex post analysis

In practice, however, CBA is not used to its potential since practitioners find it too academic, too complicated….

What to do about that?

– encourage CBA thinking early in the project process

– simplify the CBA exercise to focus on the most critical aspects

– CBA training (development of eLearning tools for practitioners)

– access to 'easy to use' CBA resources (best practice, on-line QA, cost/benefit catalogue, benefit transfer studies, etc)

Ex post CBA's?

– yes the CBA methodology works

– Institutional set-up? (MS responsibility or DG Regio?)


# 233 Feb 2011 Ex post eval cohesion fund_DG Regio

Thank you very much for your attention!

1 ex post evaluation of cohesion policy interventions 2000-2006 financed by the cohesion fund work...

Documents

cba ex post

ex post costbenefit

ex post cost benefit

cost benefit analyses

post costbenefit analyses

ex ante analyses

wp c slide

impact of projects