1

Impact measuring, the Quest of the Grail?

Ex post Impact Evaluation of four Bilateral Cooperation projects

Synthesis

KINGDOM OF BELGIUM

Federal Public Service

Foreign Affairs,Foreign Trade andDevelopment Cooperation

Office of the Special Evaluator for Development Cooperation

2

Introduction‘For decades, development agencies have disbursed billions of dollars for programs aimed at improving living conditions and reducing poverty; developing countries have spent hundreds of billions more. Yet the shocking fact is that we have relatively little knowledge about the net impact of most of these programs. In the absence of good evidence about what works, political influences dominate, and decisions about the level and type of spending are hard to challenge. But if evidence about what works were systematically developed and made public, that information could be used for better public policymaking and thus for more effective international aid and domestic spending1.’

Until recently, the evaluations implemented by the Belgian cooperation and by the Special Evaluation Office for International Cooperation mainly analysed the contribution of interventions to achieving results by asking whether the project evaluated was one of the possible causes of the changes observed or not. The results of our cooperation actions have never really been assessed in order to justify whether the changes observed could actually be attributed to the intervention evaluated.

Searching for a needle in a haystack?

Changes over time in a population’s well-being may not automatically be attributed to a development action. Indeed, other events may have contributed to the changes observed in the well-being indicator. Therefore, the analysis of attribution represents a

major challenge. Some believe that it is a task almost as impossible as looking for a needle in a haystack.

For others, assessing the effects truly attributable to an intervention is only possible by comparing the situation of beneficiaries to a

hypothetical situation of what would have happened without the intervention or the ‘counterfactual situation’. It is the comparison of levels

of well-being ‘with’ and ‘without’ the development action which offers an assessment of the effects which really may be attributed to it.

1 From William D. Savedoff and Ruth Levine. Center for Global development. Washington DC May 2006.

3

A pilot evaluationWith a view to learning more about this issue, the Special Evaluation Office for International Cooperation organised an evaluation of the impact of four direct bilateral cooperation projects completed over four years ago. Therefore, it is an ex-post evaluation. For the choice of interventions, we opted for diversity as much in terms of the countries as sectors.

The evaluation objective was both to report on the results which could really be attributed to these projects and to learn from experience with a view to improving Belgian cooperation practices in the field of impact evaluation.

The object of the evaluation was to provide an answer to the following evaluation questions:

Summative evaluation questionsQ1 Alignment To what extent were the projects in line with the

objectives of the partner countries’ policies with a view to maximising the impact of projects?

Q2 Integration of the impact

To what extent did the design and implementation of projects integrate the evaluation of impacts?

Q3 Achievement of outputs

To what extent did implemented projects achieve the expected outputs?

Q4 Achievement of impacts

To what extent did the projects achieve the desired realisations and impacts?

Q5 Sustainability To what extent were the impacts achieved by the projects likely to last in the long-term?

Q6 Cross-cutting themes and undesired effects

To what extent did the projects have effects in terms of gender and environment, as well as undesired effects on beneficiaries?

Formative evaluation questionsQ1 Quantitative Analysis To what extent did the quantitative methods help

to evaluate the impact?

Q2 Qualitative Analysis To what extent did the qualitative methods help to evaluate the impact?

Q3 Combination of the two approaches

To what extent did the combination of the two approaches help to better evaluate the projects’ impact?

Q4 Evaluation process What lessons have been learned from the difficulties encountered and the potential solutions? What were the advantages of the combination of the two quantitative and qualitative approaches?

4

A learning processThe evaluation was carried out by experts from ADE (Aide a la décision économique), a consultancy firm specialising in evaluation using qualitative methods, and researchers from the Centre for Research in Economic Development of the University of Namur (CRED) specializing in the application of quantitative methods. The experts and researchers pooled their observations. The discussions between them as well as with the Special Evaluation Office for International Cooperation and the evaluation’s steering committee led to a process of mutual learning.

The evaluation presented several challenges.

• At first, it was necessary to redefine the notion of impact in the framework of the qualitative and quantitative methods and the following chain of results:

Quantitative methods measure the effects on beneficiaries (‘outcomes’) while attempting to demonstrate a causal relationship between the project and the measured effects.

Quantitative methods generally do not provide indications about the impact, defined as ‘all the effects generated by the project at a general level (without any reference to temporality)’.

Qualitative methods support a rea-soned judgement on the achievement of the ‘outcomes’ and the impact of the projects.

They analyse whether the conditions for achieving the ‘outcomes’ on the beneficiaries and the impact were fulfilled at different stages of the intervention logic summarised in the logical framework.

• The lack of reliable data was another challenge.

In order to be able to quantify and demonstrate the effects of the project, quantitative methods require sufficient reliable data reflecting the situation prior to the intervention as well as the post-intervention situation. Furthermore, this data must be collected from both a sample of the population that has been subject to the project (called the ‘treated group’ or ‘target group’) and a sample of non-beneficiaries, a significant number of which shared similar characteristics with the beneficiaries prior to the intervention, but which did not benefit from the intervention (called the ‘counterfactual group’ or ‘control group’).

OUTCOMESOUTPUTSINPUT

Effets du projet sur les

bénéficiaires directs résultant de l’utilisation des

outputs

Biens, équipements

ou services qui résultent des inputs, et qui produisent les

effets

Moyens mis en œuvre et Activitésentreprise en vue de produire les

effets attendus du projet

Effets induits du projet

à niveau global

IMPACT

Autres termes utilisé qui portent parfois à confusion:

« Résultats intermédiaires » (selon la terminologie du cadre logique)

« Résultats immédiats ou produits » (selon la terminologie du SES)

« Objectif spécifique »(selon la terminologie du cadre logique)

« Résultats à court et moyen terme ou réalisations » (selon la terminologie du SES)

« Objectif global »(selon la terminologie du cadre logique)

« Résultats à long terme ou impact » (selon la terminologie du SES)

5

However, the data about the initial situation before the intervention (the baseline) were not quantitatively exploitable. It was also necessary to find the people with the institutional memory of these interventions.

Eventually, in Rwanda the evaluation of the Public Health Laboratory project could not be completed. The team of consultants did not receive all the permissions required by the Ministry of Health to collect and analyse the data.

Lessons concerning the methodology

Despite these difficulties, the evaluators successfully used some quantitativeinstruments. For two projects, it was possible to select a credible counterfactual group and to reconstruct baseline data from a sample of beneficiaries and a counterfactual group by using data collected through household surveys based on ‘respondents’ memories. The attribution of some ‘outcomes’ to projects could be shown with econometric tools.

By implementing qualitative methods, for the four projects evaluated, the consultants were able to provide a reasoned judgment concerning the achievement of ‘outcomes’ which could not be processed quantitatively and also on the achievement of the impact.

OUTCOMESOUTPUTSINPUT / ACTIVITIES

Impact evaluation based on QUALITATIVE methods of analysis

Impact evaluation based on QUANTITATIVE methods of analysis

IMPACT

Impact Evaluation: ‘Evaluation of causal effects of an intervention on outcomes irrespective of time dimension’ (World Bank and 3ie)

Impact evaluation: Evaluation of the process of achieving the impact, the ultimate, global (and long-term) effect of an intervention.

Measurement and Attribution

Process

6

The evaluation concludes to the relevance of the approach advocated by ‘theory based evaluation’, according to which the combination of qualitative and quantitative methods in the same evaluation process is the right solution for good quality impact evaluations. It must not be denied that the lack of baseline data greatly limited the opportunity of drawing a conclusion about the attribution of projects’ ‘outcomes’. But, despite these limits, the exercise was able to show that qualitative and quantitative methods enrich each other mutually. Indeed, the data collection system set up for the application of quantitative methods also generated a large amount of qualitative data. This was made possible thanks to the information collected during exploratory missions, the implementation of first-hand data collection through household surveys and also, thanks to the relatively long presence in the field. This data was used to set out the arguments for assessing the projects’ effects, thus resulting in a more rigorous qualitative analysis. Furthermore, the qualitative analysis helped to support the quantitative methods’ observations. This enabled the methodological choices to be confirmed and the results of the statistical and econometric tools to be interpreted. Also, thanks to the qualitative analysis, the evaluators were able to provide their conclusions about the achievement of the impact.

Finally, the use of combined methods helped to involve the stakeholders concerned and to better target their role. Impact evaluations are able to provide an adequate response to the objectives of accountability, decision-making and capitalisation on experiences (sharing of the lessons learned).

Disclaimer

The three evaluated projects are case studies. They do not constitute a representative sample. The conclusions and lessons can thus not be generalized for the whole of the governmental co-operation. The evaluation was carried out ex post on projects finished since 2008. Since then, the BTC seeks to solve certain raised issues. Moreover, the fact of reaching or not the expected impact depends on many external factors and involves in particular the responsibility of the partner country and of the political decision makers. Thus it does not concern only the responsibility of the actors in charge of the project implementation. Each evaluated project is the object of an individual evaluation report; those three reports have in turn been used as a basis for the synthesis report. This booklet summarizes the synthesis report and does not reflect all the nuances of the analysis one finds there.  The reader is invited to consult the project reports, the synthesis report as well as the management responses of the BTC and the DGD. All are available on the site mentioned on page 11.

7

Lessons concerning the results and the impact of projects1/ Project supporting Technical and Vocational Education in the Democratic

Republic of Congo (AETP2)

The evaluation concluded that the project has achieved most of its outputs. However the project did not have a significant effect on the quality of the education or on the number of pupils, nor did it improve the schools’ capacity for self-financing or make young graduates more employable.

This failure can be explained by the very partial use of outputs, but, above all, by the extremely unfavourable environment in this country’s technical and vocational education sector. For many decades, the school infrastructures have been in a deplorable condition, exacerbated by the violent conflicts that have ravaged the country. Furthermore, the State has not set up any appropriate policy in technical and vocational training supported by adequate budget resources. For example, the schools are financed mainly by the students’ parents. The absence of appropriate incentives within the institutions hinder any action designed to improve the quality of education in the DRC. This reality was highlighted already in the project design phase. While the expectation was that the situation would improve, this did not materialise.

The AETP2 project was carried out between July 2005 and October 2008 with a budget of €4,100,000. It followed on from the AETP1 project (2001-2004), which supported 13 schools in four provinces, and precedes the AETFP project (2009-2013) currently in progress. AETP1, in addition to supporting 30 new schools, offered minimum support to certain establishments already supported by earlier projects.

The overall objective of the AETP2 project, as de-fined by the intervention logic, was to contribute to the economic development of the DRC.

The specific objective was the improvement of the quality of Technical and Vocational Teaching in selected sectors: mechanics, electricity, construction, cutting and sewing and sales and administration.

The support involved various activities carried out in 20 technical and vocational schools in 8 provinces. The activities were broken down into six sectors identified as being an employment source. The beneficiaries identified were schools, teachers and students.

8

2/ Project for the Improvement and Reinforcement of Water Points in the Groundnut Basin in Senegal (PARPEBA)

The evaluation concluded that the project has contributed significantly to improved living conditions of an important part of the population of the three regions concerned. The use of the water supply from boreholes has resulted in significant time and energy savings, especially for those beneficiaries with access to a private tap (self-financed). Another effect of the new boreholes is an increase in well-being and improved social cohesion (a reduction in conflicts between women, who are primarily responsible for collecting water).

PARPEBA helped to offer improved access to water to at least 15.5% of the population in the region.This percentage is 3 times less, if only boreholes offering water with a good chemical composition are taken into consideration.

Despite those positive effects, the water quality remained a problem. More than two thirds of the sites that were built or rehabilitated through PARPEBA had harmful fluorine and salt levels (WHO standards). Belgium was aware of the problem at the time but decided to align itself with the sectorial policy of Senegal which favoured an increased access to water.

The PARPEBA project followed on from several earlier interventions (since 1983) by Belgian cooperation in the hydraulics field in the Groundnut basin. The PARPEBA project was carried out between April 2003 and December 2008 with a budget of €14,976,444 (contribution of €363,591 by Senegal).

The overall objective was the improvement of the living conditions of rural populations in the Groundnut basin through the provision of satisfactory and sustainable access to a water supply of acceptable quality.

The specific objective was to ensure sustainable access to drinking water for 260,000

beneficiaries in the regions of Diourbel, Fatick and Kaolack.

The project targeted the implementation of 32 new multi-village access to drinking water systems (AEP) in the regions of Diourbel, Fatick and Kaolack, and the improvement of 14 AEP from the previous project: the Rehabilitation, Implementation and Equipment of Borehole Sites in Rural Environments (PRRESFMR) in the regions of Kaolack and Fatic Special attention was paid to the organisational (and federative) aspects of the Association of Borehole Users (ASUFOR).

9

In addition, the risk of bacterial contamination between the place of supply and the place of consumption is high. The population is generally insufficiently aware of these risks. Follow up projects that have been implemented since try to find solutions for improving the water quality.

3/ Project for Small and Medium Hydraulics in the Tiznit region in Morocco (PMH)

The evaluation showed that the operations to rehabilitate the irrigation system, yielded a positive impact on the surface of cultivated land during periods of water stress, although no improvement was observed during wet periods. The farmers who benefited from the project derived most of their income from farming activities during both dry and wet periods. This last conclusion should be treated with caution as it is based on an income assessment by the respondents themselves. The project also encouraged the spontaneous extension of the irrigation networks, especially upstream. Furthermore, it apparently led to a rising of the groundwater level.

However, the project did not appear to have encouraged either crop diversification or changes in farming practices. The strategies of income diversification based on the migration persist. It was unrealistic to expect that the project would have a measurable effect on rural exodus.

The PMH-Tiznit project is the second phase of a project, the first phase of which started in 1996. The second phase for a sum of €960,000 aimed to complete the results not achieved in the first phase.

The project’s overall objective was to ensure income and stability for farmers by reducing the rural exodus.

The specific objective was to ensure that the structures (dams and canals) enabled the proper functioning of the traditional irrigation system.

The Small and Medium Hydraulic project (PMH) was part of the agricultural development policy implemented by Moroccan authorities. The project comprised two main components: the rehabilitation of irrigation infrastructure and the protection of soil from erosion. The project’s two successive phases enabled the improvement and extension of 11 traditional irrigation networks which help to maintain farming through the spreading of flood water, limit their erosion effects and consolidate the good operation of traditional irrigation networks within the perimeters.

10

Lessons concerning project managementThe evaluations highlighted the shortcomings that existed at the time in the design, implementation and monitoring-evaluation (M&E) of the projects.

The projects’ design and their intervention logic were relatively weak. The objectives were not very precise, poorly defined, disconnected from the activities and unrealistic in view of the resources implemented and the local context. The weaknesses identified indicated that the achievement of the ‘outcomes’ and impact was not a priority.

The question of alignment on partners countries policies also raises an issue, especially when national policies do not support the desired ‘outcomes’ and impact. There should be a possibility to stop or redesign an intervention that does not meet the conditions to achieving its expected outcomes.

The project operations were geared towards disbursement and physical achievements. This output based management was detrimental to an approach focused on ‘outcomes’ and impact. It showed a probable absence of incentives within the Belgian bilateral aid system to tackle things the other way round.

The M&E systems used for the four evaluated projects were insufficient and were not appropriate. They did not enable the monitoring or the evaluation of the ‘outcomes’ and impact of the projects. Although the formulation documents defined or referred to indicators, these were never assessed and even when data were available for such assessment, they were not suitable for a quantitative measurement.

RecommendationsPromote the impact evaluations within the wider context of the aid cycle taking account of the future challenges of Belgian cooperation, the multiplicity of stakeholders and the cost-benefit ratio of such evaluations.

Implement these impact evaluations by combining quantitative and qualitative methods within one evaluation process and allowing the necessary time and resources in terms of budget and expertise.

Develop a system for evaluating a project’s ‘outcomes’ and impact during the project design phase. This implies reviewing the design, implementation and M&E of projects so that the pursuit and measurement of the ‘outcomes’ and impact become priorities in the same way as the achievement of ‘outputs’. This also implies using the logical framework as a genuine tool for formulation, results-based management and evaluation. This would also enable a more realistic taking into account of the causal relationships, a more realistic analysis of risks and, ultimately, a more feasible project.

Standardise the project’s implementation with the evaluation system in order to make the use of quantitative methods possible at a lower cost and in an ethical manner.

11

This study was commissioned by the Office of the Special Evaluator for Development Cooperation (OSE). It carries out the strategic evaluation of Belgian federal activities which the OECD’s Development Aid Committee (DAC) recognizes as official development aid (ODA). The objectives are here to give account of the use of public funds as well as to draw lessons to improve the effectiveness of Belgian development cooperation.

The full version of the synthesis evaluation report and the projects reports are available in English, French and Dutch on the CD-ROM attached, on the website http://diplomatie.belgium.be/en/policy/development_cooperation/evaluation/ and at the Office of the Special Evaluator + 32 (0)2 501 38 34 - ses-dbe@diplobel.fed.be

Photographs: © ADE

12

Egmont • rue des Petits Carmes 15, B-1000 Brussels • + 32 (0)2 501 38 34 • www.diplomatie.belgium.be • www.dg-d.be • ses-dbe@diplobel.fed.be

Office of the Special Evaluator for Development Cooperation

Responsible editeur : Dirk AchtenChairman of the Executive Committee

KINGDOM OF BELGIUM

Federal Public Service

Foreign Affairs,Foreign Trade andDevelopment Cooperation