joint evaluation of budget support operations ......oecd-dac working group on budget support...

Methodological Note

on the Application of the

3 Step Methodology to the Evaluation of Budget Support Operations in Mali

Septembre 2011

Evaluation conjointe pour le compte de la Commission de l’Union européenne, de la Belgique et du Canada

comme exercice pilote du Comité d’aide au développement de l’OCDE

JOINT EVALUATION OF BUDGET SUPPORT OPERATIONS

IN MALI 2003 - 2009

COMMISSION EUROPEENNE

Consortium composed by

ECO Consult, AGEG, APRI, Euronet, IRAM, NCG Leader of the Consortium: ECO Consult,

Contact Person: Dietrich BUSACKER [email protected]

Contract N° EVA 2007/geo-acp

This evaluation is mandated by

the Joint Evaluation Unit for :

EuropeAid Cooperation Office (AIDCO) Directorate General for Development and Directorate-General External Relations

the Canadian International Development Agency

the Belgium Technical Cooperation

This evaluation has been accepted by the Mali Government as a pilot exercise of the OECDDevelopment Assistance Committee.

The evaluation has been managed by the Joint Evaluation Unit, who has also chaired the Evaluation Management Committee and the Reference Group in Mali, as well as the OECD-DAC

Working Group on Budget Support evaluation

Evaluation Team : Andrew Lawson (Team Leader & public finance specialist), Josette Habas (Decentralisation), Modibo Keita (Governance), Elisabeth Paul (Education & Health), Bruno

Versailles (Econometric Analysis), Alexandra Murray-Zmijewski (Macroeconomics & Statistics)

The opinions expressed in this document represent the views of the authors, which are not necessarily shared by the Commission of the European Union, nor by the authorities of the

countries or organisations concerned

Joint Evaluation of Budget Support to Mali: Methodological Note EVA 2007/geo-acp Consortium: ECO Consult – AGEG – APRI – Euronet – IRAM – NCG

i

Table of Contents Page

1 Introduction 1 1.1 Scope and Limitations of this Note and Next Steps 1

2 Application of the OECD-DAC methodology in Mali 3 2.1 Overview of the Evaluation Task 3

2.1.1 The Evaluation Questions 5 2.1.2 Overall Phasing of Evaluation Activities 5

2.2 Overview of the Approach to the Three Step Methodology 6 2.2.1 The Approach to Step One 6 2.2.2 The Approach to Step Two 7 2.2.3 The Approach to Step Three 9

3 Successes and Shortcomings of the Methodology 11 3.1 Successful Aspects of the Application of the Methodology 11 3.2 Addressing the Shortcomings in the Evaluation Methodology 14

3.2.1 Reducing the Length and Complexity of the Evaluation Process 14 3.2.2 Maximising the Policy Relevance of the Evaluation 15

4 Two Recommended Approaches for Future Evaluations 16 4.1 Integrating the Evaluation Approach into Domestic Policy Processes 16

4.1.1 An Integrated Country Led Evaluation Strategy 16 4.1.2 A Composite Two Year Evaluation Process 17

4.2 Improving the Conduct of the Evaluation Process: Lessons from Mali 18

Annex One: Reflections on the Statistical Impact Evaluation Methodology Proposed by Gunning et al. 20

List of figures and tables

Figure 1: The Scope of the Evaluation: 5 Interlinked GBS and SBS Arrangements 3 Figure 2: Mali: Aggregate Public Expenditure 1999- 2009 compared to Budget Support and other

Revenue Sources (% GDP) 12 Figure 3: Mali: Public Expenditure in the Education Sector, compared with GBS, SBS, Projects and other

Sources of Revenue 1999 – 2009 (% GDP) 12

Table 1: Budget Support Disbursements by Agency and Type, 2003-2009 (Euro millions) 4 Table 2: Budget Support Disbursements as % GDP, Total Aid & Total Public Spending 4 Table 3: Overview of Evaluation Questions (EQs) and Judgement Criteria (JCs) 5 Table 4 : Overall Phasing: Evaluation Products and corresponding Completion Dates 5 Table 5: Mali - Conclusions of Step Three: Factors determining Key Economic and Social Impacts and

their Relationship to the Budget Support Operations 10


ii

List of Acronyms and Abbreviations

AAP (HIPC) Assessment & Action Plan

BCEAO Banque Centrale des Etats d’Afrique Occidentale

CIDA Canadian International Development Agency

CSCRP Cadre Stratégique de Croissance et Réduction de Pauvreté

(Strategic Framework for Growth & Poverty Reduction – the PRSP)

DP Development Partner

EQ Evaluation Question

EU European Union

GBS General Budget Support

GDP Gross Domestic Product

HIPC Highly Indebted Poor Countries (initiative)

IMF International Monetary Fund

JC Judgement Criterion

NGO Non Governmental Organisation

ODA Official Development Assistance

OECD-DAC Development Assistance Committee of the Organisation for Economic Cooperation & Development

PEFA Public Expenditure & Financial Accountability

PFM Public Finance Management

PRSP Poverty Reduction Strategy Paper

SARPE Stratégie Alternative de Recrutement de Personnel Enseignant

(Alternative Strategy for the Recruitment of Teaching Personnel)

SBS Sector Budget Support

WAEMU West African Economic & Monetary Union (UEMOA in French)


1

1 Introduction

1. An evaluation team managed by ECO Consult, Germany and Fiscus Limited, UK was contracted by the Commission of the European Union to undertake an evaluation of General and Sector Budget Support to Mali over 2003 to 2009. The evaluation was managed by the European Union on behalf of the OECD-DAC working group on budget support evaluation, as one of three pilot applications – together with Tunisia and Zambia - of an updated OECD-DAC methodology for the evaluation of budget support. This methodology is explained in three reports published by the European Union1:

• “Methodology for Evaluations of Budget Support Operations at country level: Issue Paper” (May 2008).

• “Methodology for Evaluations of Budget Support Operations at country level: Methodological Details” (April 2009).

• “Methodology for Evaluations of Budget Support Operations at country level: Tools for Step Two – evaluation of the impact of government strategies” (April 2009).

2. The Mali Budget Support evaluation is now very largely complete with a Synthesis Report having been submitted and presented in Bamako in April 2008 and the comments on that report having now been incorporated into a draft Final Report, which is expected to be finalised and formally approved during May-June 2011. As part of the assignment, a Methodological Note was required to be drafted, presenting the team’s observations on the application of the methodology and its recommendations for future budget support evaluations. This Note addresses this contractual requirement and together with parallel notes drafted for the Tunisia and Zambia evaluations is expected to provide the basis for the refinement of the OECD-DAC methodology.

1.1 Scope and Limitations of this Note and Next Steps

3. This Note has been drafted by Andrew Lawson, Team Leader of the Mali evaluation, drawing on comments provided by the members of the evaluation team, most notably by Elizabeth Paul, Bruno Versailles and Alexandra Murray-Zmijewski. It does not aim to provide full details of the methodology as applied in Mali, but rather an overview, which provides a flavour of the key evaluation tasks undertaken and the process followed, highlighting some of the particular innovations introduced. It identifies the main shortcomings of the evaluation methodology, as observed by the evaluation team, and provides recommendations for the future evaluation of budget support.

4. Ideally, the comments of the Management Committee and the (Mali-based) Reference group for the evaluation would have been sought prior to the drafting of this note. Given that a fundamental objective of evaluations is to help improve development policies and programmes, this is a significant shortcoming and we strongly believe that some framework is needed for the input into the methodology by those designing and managing budget support both within beneficiary governments and development agencies. As a bare minimum, it is hoped that the Management Committee and the Reference group will be in a position to comment on this first draft but we believe that a wider consultation process will also be required.

1 The first two documents draw heavily on a previous report authored by Enzo Caputo, Andrew Lawson and Martin van der Linde and the third on a prior report by Jan Willem Gunning, Chris Elbers, Antonie de Kemp and Phil Compernolle. Each of these was adapted and amended on the basis of discussions and agreements reached within the OECD-DAC working group on budget support evaluation.


2

5. The evaluation team have deliberately avoided having discussions with the Tunisia and Zambia evaluation teams on the methodology, in order to avoid being inadvertently influenced by their perceptions. It should be noted, however, that the methods of application of the methodology varied quite substantially across these countries, reflecting in part the differences in the nature of the budget support operations in each of these countries, in part the differences in data availability within each country, and in part the differences in the financial and human resources available to each team. Thus, while they are comparable they should not be read as identical applications of the same framework.

6. Nevertheless, with the three methodological notes tabled, it would be helpful for the three evaluation teams to confer, share impressions on the application of the methodology and ideally to prepare a joint set of recommendations on future methodology.

7. These three methodological notes are to be the subject of a first discussion by the OECD-DAC working group on evaluation of budget support, scheduled for Tuesday 17th, May 2011. Thereafter, we would recommend that the following steps should be undertaken:

• Circulation of the three Notes for comment by their respective Steering Committees/ Reference Groups.

• Consultation between the three evaluation teams, perhaps culminating in a joint note between the Team Leaders.

• Development of a revised set of guidelines on budget support evaluation, amending the current methodology in the light of the experience of the three pilot exercises and drawing together the reference material currently contained in three documents into a single document of evaluation guidelines.

• Further consultations with OECD-DAC members and with government officials from budget support receiving countries on the draft revised guidelines, leading to an approved and updated methodology.


3

2 Application of the OECD-DAC methodology in Mali

8. In this chapter, we provide a description of the mode of application of the OECD-DAC methodology in Mali. We begin with an overview of the evaluation task, then explain the phasing of the different evaluation activities and reports before describing the approach taken for each of the three “Steps” of the evaluation.

2.1 Overview of the Evaluation Task

9. The Mali evaluation covered a total of 32 General and Sector Budget Support operations (out of a grand total of 332 operations undertaken during the period) provided by 10 Development Agencies over 2003 – 2009. Specifically, these included:

• 17 General Budget Support (GBS) operations, financed by 8 Development Agencies;

• 1 Sector Budget Support (SBS) operation in support of Decentralisation & Reform of the State;

• 9 SBS operations in support of the Education sector;

• 1 SBS operation in support of PFM reform; and

• 4 SBS operations in support of the Health and Social Development sectors.

Figure 1: The Scope of the Evaluation: 5 Interlinked GBS and SBS Arrangements

10. During the initial years of the evaluation period, the GBS operations were managed individually but a common structure for budget support was developed and signed in 2007, with sub-structures for General Budget Support, Education, Health & Social Development and Decentralisation/ Reform of the State. A harmonised programme for the reform of Public Finance Management (PFM) was initiated in 2005 and came to comprise an important part of the assessment framework for GBS. The SBS for PFM was managed as part of this harmonised framework rather than as a separate arrangement. Thus, the evaluation task centred on the evaluation of four linked arrangements for

2 The terms of reference excluded a Sector Budget Support operation for the cotton sector, which had been financed by the EU over 2006 and 2007.


4

GBS, SBS Education, SBS Health & Social Development and SBS Decentralisation/ Reform of the Public State. The funding of each of these was channelled through the national treasury account and managed through the normal budget, with earmarking arrangements being applied for the Education and Health & Social Development SBS over 2007 – 2009. With disbursement conditions on Education, Health and Decentralisation/ Deconcentration being included in the GBS Performance Assessment Framework (PAF) and Decentralisation/ Deconcentration conditions included in the Health and Education SBS performance frameworks, these four arrangements were thus inter-linked within a hierarchical framework as illustrated in Figure 1.

Table 1: Budget Support Disbursements by Agency and Type, 2003-2009 (Euro millions)

2003 2004 2005 2006 2007 2008 2009 Totals Total Budget Support 132.0 50.8 95.4 185.5 141.0 158.7 202.7 966.1 GBS 132.0 50.8 95.4 125.2 74.4 82.3 135.7 695.9 African Development Bank 28.4 0.0 9.9 40.9 0.0 16.6 12.8 108.5 World Bank 37.4 0.0 19.5 20.3 27.4 27.7 51.5 183.9 Canada (GBS) - - - - - - 5.9 5.9 Denmark - - - - - - 3.2 3.2 European Union 36.1 29.7 39.0 32.0 16.9 12.9 26.9 193.6 France 7.0 5.5 9.5 13.0 10.0 6.0 10.4 61.3 Netherlands 17.7 10.1 10.1 9.9 10.1 10.0 10.1 77.8 Sweden 5.5 5.5 7.5 9.1 8.4 7.3 12.4 55.6 Canada - PFM SBS - - - - 1.7 1.8 2.4 5.9 SBS - - - 60.3 66.6 76.4 67.0 270.2 Education - - - 33.0 44.2 49.8 40.7 167.7Belgium - - - - 4.0 4.0 0.0 8.0 Canada - - - - 4.6 7.7 8.7 21.0 France - - - - 3.0 3.0 7.0 13.0 Netherlands - - - 22.0 22.0 25.0 25.0 93.9 Sweden & Norway - - - 11.1 10.7 10.1 0.0 31.8 Health & Social Development - - - 7.3 7.9 15.2 17.0 47.4Canada - - - - 1.7 7.7 8.7 18.1 Spain - - - - 0.0 0.0 1.0 1.0 Netherlands - - - 4.0 4.0 4.0 4.0 15.9 Sweden - - - 3.3 2.3 3.5 3.3 12.5 Decentralisation/ Reform of the State - - - 20.0 14.5 11.4 9.3 55.1European Union - - - 20.0 14.5 11.4 9.3 55.1

Table 2: Budget Support Disbursements as % GDP, Total Aid & Total Public Spending 2003 2004 2005 2006 2007 2008 2009

Total Disbursements (Euro millions) Total Budget Support 132.0 50.8 95.4 185.5 141.0 158.7 202.7 GBS 132.0 50.8 95.4 125.2 74.4 82.3 135.7 SBS - - - 60.3 66.6 76.4 67.0 As % of GDP Total Budget Support 4% 1% 2% 4% 3% 3% 3% GBS 4% 1% 2% 3% 1% 1% 2% SBS - - - 1% 1% 1% 1% As % Total ODA Total Budget Support 41% 18% 27% 35% 33% 44% 42% GBS 41% 18% 27% 23% 18% 23% 28% SBS - - - 11% 16% 21% 14% As % Total Public Spending Total Budget Support 15% 5% 9% 15% 11% 13% 13% GBS 15% 5% 9% 10% 6% 7% 9% SBS - - - 5% 5% 6% 4%


5

11. The two tables above show the values of the disbursements by each Development Agency for each of the operations evaluated and their significance in relation to total aid, GDP and public spending. In summary, the evaluation task comprised a large number of complex operations from a range of Development Agencies, delivered over a 7 year period through inter-linked structures against a range of objectives and through a variety of assessment and disbursement arrangements. In scale, these operations were of considerable importance to the Mali National Budget and, to a lesser degree, to the economy as a whole.

2.1.1 The Evaluation Questions

12. The Terms of Reference required 21 compulsory Evaluation Questions (EQs) to be addressed covering the five levels of the Intervention Logic presented in the OECD-DAC methodology. These were to be addressed respectively in Steps One, Two and Three. During the Inception Phase, 60 Judgement Criteria were identified, to provide the basis of the answers to the EQs. (See Table 3.)

Table 3: Overview of Evaluation Questions (EQs) and Judgement Criteria (JCs)

Steps of the Evaluation Step 1 Step 2 Step 3Number of Evaluation Questions 13 4 4

Corresponding EQs (EQ. 1.1 - 1.3 ; 2.1 – 2.4 ; 3.1 – 3.6) (EQ. 4.1 – 4.4) (EQ. 5.1 – 5.4)3

Number of Judgement Criteria (JC) 42 10 8

2.1.2 Overall Phasing of Evaluation Activities

Table 4 : Overall Phasing: Evaluation Products and corresponding Completion Dates

Evaluation Product Completion Date

Inception Mission November 2009

Draft Inception Report January 2010

Final Inception Report April 2010

Local level Perception Survey May-June 2010

Desk Report August 2010

Draft Perceptions Survey Report September 2010

Main Field Mission October 2010

Final Perceptions Survey Report December 2010

Draft Synthesis Report February 2011

Presentation of findings in Bamako April 2011

Methodological Note May 2011

Final Synthesis Report June 2011 (expected)

3 This table is a simplification. In fact, EQs 5.1 – 5.4 are based on two Judgement Criteria (JCs), of which the first corresponds to Step Two (‘Outcomes and impacts identified can be related to specific determining factors, and in particular to policies put in place by the government) and only the second to Step Three (‘the government policies and interventions induced by GBS/ SBS have influenced outcomes through identifiable processes and mechanisms’).


6

13. Following the requirements of the Terms of Reference, the evaluation was divided into a) An Inception Phase; b) a Desk Phase; c) a Field Phase; and d) a Final Synthesis Phase. In addition, it was decided during the review of the Inception Report, that an additional perceptions survey should be undertaken in a sample of provincial and local governments (‘collectivités territoriales’). The key outputs for each of these phases and the dates on which they were submitted are shown above.

14. A number of observations on the phasing of activities should be made:

• It was required that the Inception Report should be approved by the Management Committee before proceeding with the substantive phases of the evaluation. Its approval was delayed, essentially for two reasons:

o Concerns over the lack of outcome and impact data by which to assess progress,

under Step Two, of the Decentralisation policy and related SBS. This led to the

organisation of the Local level Perceptions Survey under new funding made available

by CIDA;

o Concerns over the need for a methodology, which would allow the respective

effects of the GBS and SBS operations to be separately identified.

• At the time of the Inception Mission, it was not initially envisaged that there might be funding available for primary data collection, which led to initial proposals for Step Two being based exclusively on analysis of existing data ;

• In turn, this led to a relatively rushed approach to the commissioning, design and implementation of the Local level Perceptions survey, which unfortunately had a negative impact on the quality of the final product.

• The main field mission was initially planned for June 2010. With several donor agency staff being unavailable in July and August and the 50th anniversary of Independence being celebrated in September, a delay meant that late October was the next appropriate date.

2.2 Overview of the Approach to the Three Step Methodology

15. Steps One and Two of the methodology were undertaken simultaneously. Initial analysis of available data (and some of the collection process) was completed during the Inception mission, when the key elements of the respective approaches were also defined. With the exception of the information required for analysis of progress in the PFM area under Step One and in the Decentralisation area under Step Two, the bulk of the necessary data was provided before, during and shortly after the Inception mission and, as a consequence, a good part of the analysis was completed during the Desk Phase. The main Field Mission then focused on completing analysis under Steps One and Two (filling remaining data gaps and testing preliminary hypotheses) and on undertaking the analysis necessary for Step Three. The techniques and analytical approaches adopted for each step are presented below.

2.2.1 The Approach to Step One

16. Step One of the methodology involved an analysis of the validity - in relation to the Malian experience - of the “programme theory” of budget support: the hypothesised intervention logic, defined in the OECD-DAC methodology. It comprised 13 Evaluation Questions, across the first three levels of the framework – Inputs, Direct Outputs and Induced Outputs. These questions were addressed through the following techniques:


7

• Analysis of the Financing Agreements for all the Budget Support operations, as well as information on annual progress reviews and annual disbursements by agency and type of Budget Support.

• Construction of a data base on GBS/ SBS disbursements against projections by year and, where possible, by quarter.

• Construction and analysis of a data-base on budgetary and fiscal trends:

o Compilation of data on budgeted revenues and expenditures and actually realised

revenues and expenditures broken down according to the national budget

classifications, covering administrative, economic and functional classifications.

(These data included comprehensive information on sources of funds, broken down

between domestic revenues, aid grants and concessional and other loans.)

o Triangulation of sources of information on revenues and expenditures to ensure use

of the most reliable data - notably, triangulation between Ministry of Economy &

Finance, the Central Bank, the (government) Secretariat for the Harmonisation of Aid, sector ministries and Development Agencies.)

• Construction and analysis of a data-base on output and outcome trends for health, education and social development4.

• Construction and analysis of a data-base on PFM outcomes, based on the results of the IMF/ World Bank HIPC AAP surveys of 2001 and 2004, the 2006/07 PEFA and the draft 2010 PEFA, for which the draft report was tabled in October 2010.

• Review of relevant policy documents, reports and evaluations from government and the Development Partners.

• Interviews with key Resource Persons during Field Mission.

• Focus Group discussions during Field Mission5:

o Ministry of Economy & Finance

o Health stakeholders (sector ministry staff and sector DPs)

o Education stakeholders (sector ministry staff and sector DPs)

o Private Sector

o Civil Society Organisations.

2.2.2 The Approach to Step Two

17. Step Two of the methodology involved an analysis of the determining factors - internal and external – responsible for generating the outcomes and impacts targeted by the national and sectoral strategies supported by budget support. It comprised 4 Evaluation Questions, which aimed to identify potential linkages from levels 4 and 5 of the framework (outcomes and impacts) back to level 3, the induced Outputs. It involved three rather different types of analysis across three different outcome/ impact areas. The choice of techniques was significantly limited by the availability of data – most significantly the absence of a country-wide household budget survey for the post 2006 period, and by the level of

4 The original intention was that this output and outcomes data-base should cover transport as well, which was the subject of GBS disbursement conditions in the earlier years of the period but comprehensive data were not available. 5 The content of the Focus Group discussions was based on pre-prepared questions agreed by the evaluation team. These questions in fact covered elements relevant to Steps One, Two and Three.


8

funding available to the team for analytical activities. Essentially, the level of funding meant that detailed statistical analysis involving econometric techniques could only be envisaged for the health and education sectors, although it is doubtful that the necessary data would have been available for detailed statistical analysis of the macroeconomic and poverty data6.

18. The following types of analyses were undertaken during Step Two :

• In relation to the determinants of outcomes and impacts regarding economic growth and poverty reduction :

o Qualitative analysis of data on the composition of economic growth, on the evolution of poverty and inequality and the evolution of the investment climate;

o Interviews with key resource persons, notably the Central Bank (BCEAO);

o Focus group discussion with private sector representatives.

• In relation to the determinants of outcomes in terms of improved service delivery and improved governance at the level of the collectivités territoriales :

o Qualitative analysis of results from the Local level Perceptions survey;

o Documentary analysis;

o Interviews with key resource persons;

o Focus group discussions with Civil Society Organisations and with Private Sector representatives.

• The most detailed analysis during Step two was reserved for the health and education sectors, where the determinants of outcomes and impacts were examined using different types of econometric and statistical techniques:

o Using the annual school census, two slightly different econometric techniques were used for analysing the education sector. Firstly, a more descriptive analysis was undertaken to assess the relative importance of different outputs of government education spending (numbers of schools, classrooms, and teachers.) on the growth of the number of pupils. This was done through a simple difference-in-difference analysis, very similar to the techniques described in the “Tools for Step Two” methodological note7.

o Second, a fixed effect regression analysis was undertaken to reveal the major

determinants of two key proxy variables for quality: (i) the repeater rate (within

primary) and (ii) the primary-secondary transition rate. This considered the relative significance of different types of teacher training and qualifications, different types of school construction and facilities and the relative availability of text books8.

o In the health sector, it was not possible to run econometric tests between impacts and outputs due to the absence of a recent Household Survey or Demographic &

Health Survey from which to obtain updated (and relatively disaggregated) impact data. However, a correlation analysis was performed between the main outputs

6 One possibility might have been to undertake a comparative analysis of economic performance between Mali and neighbouring WAEMU countries, based either on cluster analysis or the use of panel data. These techniques were applied in the case of the Tunisia evaluation. Due to funding constraints, their feasibility was not examined in any detail in Mali but the data constraints would certainly have been more severe than in Tunisia. 7 This is reported on page 124 of Volume III of the Synthesis Report (February 2011). 8 This analysis is described on pages 128-130 in Volume III of the Synthesis Report (February 2011).


9

(targeted by the national health policy) and the main service delivery outcomes, based on regional data so as to ensure sufficient variation. This permitted the team

to reach a positive judgement on the relevance of the national policy, showing that

recent improvements in outcomes are indeed correlated with key outputs targeted

by public spending.

19. Unfortunately, neither for education nor health was it possible to demonstrate causality rather than simple correlation. This was principally because in the absence of a recent household survey it was impossible to analyse demand-side factors, related for example to the characteristics of users of public health care/ pupils in school (family income status, level of education of parents, etc). With only supply-side factors available, it is almost impossible to draw any firm causality conclusions. In addition, despite the best efforts of the team, it was impossible to obtain the data-base of exam results, which could have been linked to administrative data from the school census in order to analyse quality issues in more depth.

2.2.3 The Approach to Step Three

20. Step Three of the methodology involved an analysis of the relative influence of GBS and/ or SBS on the government policies and interventions identified in Step Two to be among the principal determinants of the outcomes and impacts targeted by the national and sectoral strategies supported by budget support. It comprised 4 Evaluation Questions, which aimed to synthesise the analysis emerging respectively from Step One and Step Two. Essentially, this involved an analysis by the team of the results of Steps Two and One, using these results to answer five questions:

• Had the key outcomes and impacts targeted by the government strategies supported by Budget Support improved over the evaluation period? (Step Two, part a)

• Were government policies, strategies and interventions amongst the determining factors of those changes? (Step Two, part b)

• Were those government policies, strategies and interventions in turn influenced by Budget Support? (Step One, Induced Outputs)

• Was the influence of Budget Support primarily due to SBS or GBS and primarily due to funding, policy influence or strengthening of domestic capacities?

• How significant were each of these chains of influence (GBS/ SBS ! Government policies & interventions; and Government policies & interventions ! Outcomes & Impacts)?

21. The scale of influence was assessed against a simple scale: “Absent”, “Negligible”, “Weak”, “Moderately Important” and “Important”. The initial hypotheses and rankings established by the team were then checked during the Field Mission through specific questions included in relevant Focus Group discussions or through interviews with key resource persons. The final results were presented as a series of tables relating to each of the key outcome/ impact areas examined in Step Two, namely a) economic growth and poverty reduction; b) education; c) health; d) Local level governance and service delivery. An example of the analysis produced with regard to economic growth is presented in the table below.


10

Table 5: Mali - Conclusions of Step Three: Factors determining Key Economic and Social Impacts and their Relationship to the Budget Support Operations Budget Support inputs with a potential influence on the factors determining final outcomes and impacts

Degree of Influence of budget Support input. (Important / Moderately important / Weak / Negligible / Absent)

Factors determining the changes observed at the Outcome/ Impact level

Degree of Influence of the determining factor. (Important / Moderately important / Weak / Negligible / Absent)

Specific Outcome or Impact in question

NB. External factors excluded. Hence, analysis limited to the Government Interventions identified amongst the Determining factors in Step Two. (Investments/ specific types of recurrent expenditures, policy decisions, institutional reforms, etc.)

GBS dialogue and conditions on Macroeconomic policy

Negligible (in relation to the established government commitment to the WAEMU convergence criteria and the targets agreed with the IMF.)

Prudent macroeconomic management by the Authorities, notably regarding control of the fiscal deficit and the inflation rate.

Important 5% per annum average rate of real growth in GDP over evaluation period (the highest of the WAEMU zone within the period)

Budget Support finance (SBS & GBS) equivalent to up to 13.6 % of annual public spending, disbursed with an acceptable degree of predictability.

Important Reduction in the fiscal deficit after grants to an average of 0% of GDP over 2003 to 2008 (against an average of -1.6% over 2000-2002)

Important(in particular because of its moderating influence on monetary growth and the real interest rate )

GBS dialogue on the PRSP (CSLP/ CSCRP) and related strengthening of capacities for PRSP implementation

Moderately important(GBS dialogue has been important in creating a robust annual monitoring framework for the PRSP but Government commitment to the funding of the PRSP priority sectors was already well established before GBS.)

Improvement in the allocation of resources to the PRSP priority sectors

Weak(The composition of growth does not suggest that Government sector policies and programmes were determining factors in the rate of economic growth achieved.)

Budget Support finance (SBS & GBS) equivalent to up to 13.6 % of annual public spending, disbursed with an acceptable degree of predictability.

Important Increase in the level of own financed Public Investment from17% of Budget revenues in 2002 to 21 % in 2009, peaking at 28% in 2007

Moderately Important(as a counterweight to the continued stagnation of private sector investment )

GBS dialogue and conditions on PFM reform and related stimulus to PFM capacity building

Weak (PFM reforms are not sufficiently advanced to have generated significant efficiency effects.)

Improved efficiency in Budget execution, in particular for Public Investments

Absent(Faster and more efficient execution of public spending would in principle enhance the impact of public spending but there is no evidence of such efficiency effects)

GBS dialogue and conditions on the « Doing Business » indicators and related aspects of the private sector investment climate

Weak (Dialogue has not focussed on the obstacles to investment identified by the domestic private sector.).

Improvement of the Investment Climate

Negligible(The rate of private investment has remained low and the chief constraints on investment commonly identified remain present.)


11

3 Successes and Shortcomings of the Methodology

22. In this chapter, we attempt to provide an overall assessment of the relative successes and shortcomings of the methodology, as applied in Mali. Clearly, it is difficult for the evaluation team to be fully objective about these issues, and it will be important to complement the reflections made here with those of the management team and those of other researchers and evaluators who might be called upon to review the overall evaluation results.

3.1 Successful Aspects of the Application of the Methodology

23. Perhaps the most important point to state at the outset is that, notwithstanding the complexity of the evaluation task and the seriousness of the data constraints in the Malian context, it has proven possible to apply the OECD-DAC methodology, to provide evidence-based answers to the 21 Evaluation Questions and to reach robust conclusions regarding the effects of General and Sectoral Budget Support in Mali. In short, the methodology has been successfully applied and the Mali pilot exercise has demonstrated its applicability to the context of an aid dependent, lower income country, with significant data limitations. This is a fundamentally important result, which should not be forgotten.

24. In the process, a number of innovations have been introduced, which are worth documenting for potential replication in future evaluations. This remains unfinished business for the moment, because the focus of this Note has been primarily on correcting shortcomings in the methodology. Nevertheless, we would draw attention to three innovations:

• The estimation of the use of GBS as a source of “marginal revenue” (in the economics sense) to finance “marginal expenditure”.

In past evaluations, the most common assumption used in attempting to attribute specific expenditures to GBS has been that GBS as a revenue source is divided equally between all spending areas. Whilst potentially useful as a working assumption, it is clear that this lacks verisimilitude because in the counterfactual situation of no budget support being received, it seems difficult to conceive of a Minister of Finance or Budget Director proposing equal cuts in

all services as a consequence. Certain expenditures need to be given priority because they are

non-discretionary (debt servicing, pensions, financing of constitutional office holders, etc.) and others may be given priority simply because they are difficult to cut in the short term, such as salaries for established civil servants. In short, the most likely areas for cuts are those “marginal expenditures”, such as domestically financed investments, which are the last to be approved once all non-discretionary and politically or legally unavoidable expenditures are financed.

Following the same reasoning, when new sources of revenue become available - such as additional budget support – it then becomes possible to finance the most important additional

expenditures at the margin, such as more domestic investment or extra purchases of text books or more extensive road repairs or faster expansion of teaching services through

recruitment of additional contract teachers. In a Developing Country where many public spending needs often have to go unmet, it is precisely these expenditures at the margin which

make the difference between a civil service, which can do little more than sustain itself (by paying salaries and covering operating costs of buildings), and a civil service which can deliver

greater and higher quality services. Our discussions with the Budget Director in Mali and with the Directors of Finance & Administration in the health and education ministries suggested

strongly that the decisions on the allocation of budget support financing were consistent with


12

this reasoning. Two different types of graphical format were therefore used to illustrate what would be the consequences for the attribution of GBS at the aggregate and the sector level (see

Figures 2 and 3 below.)

• The use of Focus Groups to test hypotheses and triangulate information.

The use of Focus Groups in evaluation is a common and useful technique. In past Budget

Support evaluations, it has been less common – often because more time needs to be dedicated

to compile information on the core technical issues - and as a consequence there is a danger that the range of interlocutors will be limited and important voices will be lost. In Mali, the pre-

planned and structured use of focus groups was especially important.

• The use of Perception Surveys.

The Local Level Perception Survey undertaken in Mali had to be planned and executed in a rush

and unfortunately this impaired its quality and usefulness. However, the mere fact that it was

possible in a matter of 4-6 months to organise and run a survey covering 19 communes in 4 different regions is itself impressive. Such surveys cans serve much the same purpose as the use

of focus groups, only with the substantial advantage of greater scale and representativity. In all budget support evaluations, there will be important questions which available quantitative data cannot answer but where perception surveys can shed light. For example, in Mali the econometric analysis of education outcomes and impacts left two big unanswered questions: a) Why were teachers on short-term contracts (SARPE scheme teachers) achieving lower repeater rates and higher transition rates into secondary schooling than regular fully trained teachers? and b) Was the quality of teaching suffering as a result of fast expansion? A well-targeted perception survey could have shed light on these important questions. If it was known from the outset of an evaluation that such a survey was planned, then it would be possible to link it more systematically with other planned analyses - quantitative and qualitative.

Figure 2: Mali: Aggregate Public Expenditure 1999- 2009 compared to Budget Support and other Revenue Sources (% GDP)

‐5%

0%

5%

10%

15%

20%

25%

30%

1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

Dépenses en Capital sur Financement Extérieur

Dépenses en Capital sur Financement Intérieur

Biens et Services et Transferts et Subventions

Masse Salariale

Fonds Spéciaux et Budgets Annexes

Paiement de la Dette

Recettes Intérieures

Recettes et Appui Budgétaire

Recettes, Appui Budgétaire et Projets


13

Figure 3: Mali: Public Expenditure in the Education Sector, compared with GBS, SBS, Projects and other Sources of Revenue 1999 – 2009 (% GDP) 9

0

0,01

0,02

0,03

0,04

0,05

0,06

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

Dépenses en Capital sur Financement Extérieur

ABS Education

Dépenses en Capital sur Financement Intérieur

Transferts, Equipement et Autres

Fonctionnement

Masse Salariale

Recettes Intérieures

Recettes et ABG

Recettes, ABG et ABS Education

Recettes, ABG, ABS Education et Projets

9 The figures 2 and 3 are both imported directly from the Mali Budget Support Evaluation, Final report, Volume 1. Each of them shows public expenditure stacked by degree of discretionality, beginning (for aggregate expenditure) with debt servicing, then Special Funds, the wage bill, Recurrent goods & services and transfers, internally financed investments and (on the top) externally financed investments. Superimposed on this are the revenue sources, showing first what domestic revenues alone would have funded, then domestic revenues + Budget Support, then domestic revenues + Budget Support plus external project financing. The graph for education (Figure 3) follows the same principles, except that with SBS Education financing a pre‐defined set of investments (because it was explicitly earmarked), SBS features both as a set of expenditures and a revenue source. The estimated allocation of domestic revenue and GBS to the education sector is derived based on the assumption that the inter‐sectoral split of domestic revenues remains fixed from 2002 onwards, with GBS � following the �marginal revenue hypothesis� � having therefore been used to finance those increments in spending which could not have been covered by domestic revenues or SBS. This is of course a working assumption but it serves to illustrate that the levels of education expenditure actually realized would only have been possible through substantial reductions in domestic revenue allocations to other sectors in the absence of GBS, especially so in 2008, when the international financial crisis provoked a fall in domestic revenue collections. GBS avoided the need for these difficult political trade‐offs and thus helped to protect spending in education and the other priority sectors defined in the PRSP (CSCRP).


14

3.2 Addressing the Shortcomings in the Evaluation Methodology

3.2.1 Reducing the Length and Complexity of the Evaluation Process

25. The most obvious shortcoming which the Mali Budget Support evaluation has presented is that it has been a long, complex and unwieldy process:

• Assuming acceptance during June 2011 of the amendments now being incorporated in the Final Report in the light of comments received, it will have taken 20 months to complete from the start of the Inception Mission to the acceptance of the Final Report. (See Table 4 above.)

• It has produced three main reports – the Inception Report, the Desk Report and the Final Synthesis Report, each in excess of 100 pages and each with very substantial annexes, amounting in total to some 600 pages or more of written outputs.

• It has covered 31 operations by ten Development Partners, comprising four types of GBS/ SBS.

• It has addressed 21 separate Evaluation Questions.

26. Apart from the management and administration problems which this complexity creates and which themselves generate delays, such a large and unwieldy process is very difficult for policy makers to assimilate. There are many important lessons in the Mali evaluation relating to the design and management of budget support and the design and delivery of government policies and programmes, which run a high risk of simply being lost and forgotten. In part, this can be remedied by the design of more synthetic products – briefing notes, summary papers, etc – and it is hoped that some funding might be found for such products.

27. Yet, measures must be taken to simplify the evaluation task:

• In future, it would be desirable to undertake evaluations more frequently so that evaluation time periods do not exceed 5 years.

• Secondly, evaluation products should preferably be delivered separately: there is in fact a very strong logic in formally separating the outputs of Steps One, Two and Three, and then having a separate Synthesis Report.

• The number of Evaluation Questions within each step can be reduced and simplified.

• Management and quality assurance arrangements can be streamlined, so as to simplify the preparation of comments and reduce the number of document revisions.

28. We believe that improvements are needed in each of these areas and we provide below two alternative approaches as to how this might be done.

29. Yet, even with deliberate attempts to simplify the evaluation task, much of the complexity is inherent in the subject matter. Budget support is complicated and is often delivered by large numbers of Development Partners, using a range of programme design and management arrangements. Sector and Budget Support interact both in the way they influence policy dialogue and in the way they fund the budget and generate macroeconomic effects; if their individual and aggregate effects are to be properly understood they need to be evaluated together.

30. In short, whatever efforts are made to simplify processes, budget support evaluations will always be complex. This in turn means that the Evaluation team needs to be properly resourced. In Mali, the aggregate budget (excluding the Local level Perceptions Survey) was slightly over Euro 300,000, which was not sufficient and despite substantial cost over-runs by the evaluation team, there are certain tasks which have not been adequately resourced. Moreover, the Team Leader, in addition to


15

overall coordination, had to take direct responsibility for the evaluation of GBS and for the assessment of PFM reform progress. An additional senior member of the team was sorely needed.10

3.2.2 Maximising the Policy Relevance of the Evaluation

31. The second issue to address is the more careful consideration of exactly how evaluations can be organised and managed so as to feed into both country level and headquarters level policy processes. There are three types of policy agendas where GBS/ SBS evaluations can contribute substantially:

• international policy and practise on budget support design, management and evaluation;

• country level structures for the design, management, monitoring and evaluation of budget support ; and

• country level policy and budget processes at the national, local and sectoral levels.

32. The headquarters dimension has been, arguably, quite well addressed in the Mali evaluation through the link to the OECD-DAC working group on budget support evaluation. Yet, even here the links are not fully explicit: for example, no headquarters level workshop is currently planned when the evaluation results might be presented to a policy audience (rather than an audience of evaluators). No presentation or dissemination events are currently planned for the NGO community or for Parliaments or Courts of Auditors, all of which are important audiences in the international debate on budget support.

33. Regarding the country level, the Bamako dissemination conference did allow Government representatives and some in-country Development Partners to request the preparation of more operational recommendations, providing detailed proposals for future budget support processes. This will be addressed as far as is possible in the final report but it is clear that follow-up work with in-country stakeholders will be needed. Arguably, this should be financed separately from the evaluation itself but it would have been useful to plan in advance in a way that would facilitate this.

34. Perhaps the most neglected policy link is the link to the evaluation of national and sectoral policies in-country and to their development and refinement. Significant analysis has been done on education and health policy, on PFM reform strategy and on decentralisation strategies: this evaluation work should be naturally feeding in to the updating of policies in this area. However, the real lesson is that such policy-evaluation links should have been designed from the outset. The Step Two analysis should form part of a structured programme of monitoring and evaluation activities linked to the policies and strategies supported by budget support. Its timing should be planned as part of the overall programming of budget support operations. We provide below a set of recommendations on how this might be done.

10 The overall team comprised 7 people: the Team Leader (also GBS evaluator and PFM specialist), a Health and Education specialist, a Decentralisation specialist, one senior local consultant who covered the Governance area and also led the local level Perceptions Survey, one econometrician, one macroeconomist/ statistician and one local research assistant.


16

4 Two Recommended Approaches for Future Evaluations

4.1 Integrating the Evaluation Approach into Domestic Policy Processes

35. Our main critique of the OECD-DAC methodology as it has been conducted in Mali is that it views the evaluation process as an externally led function, whose main audience is at the headquarters level of Development Agencies, whereas it should actually be an important part of budget support agreements and subsequent GBS/ SBS discussions between Governments and Development Partners. Thus, it should be commissioned at the country level so as to inform ongoing dialogue on policies and institutional reforms and discussions on the design and management of budget support. The interest of headquarters should be a) to ensure that both annual monitoring and periodic evaluation of GBS/ SBS is taking place on a systematic basis; b) that evaluation approaches follows good practice and c) that the results of country evaluations are synthesised through international conferences and events, informed potentially by meta-evaluations and other cross-country comparative work. How could such a transformational shift in the approach be achieved?

36. We believe that two approaches should be promoted:

• A fully Integrated Country-led Evaluation Strategy, managed as a series of separate products corresponding to Steps One, Two and Three, whose results would be integrated by the managers of budget support processes at country level; and

• For those countries with more limited analytical and management capacity at the country level, a Composite Two Year Evaluation process could be envisaged, also covering Steps One, Two and Three but managed by a single evaluation team. For the former, the role of central level evaluation units might be more advisory and for the latter more hands-on and operational.

4.1.1 An Integrated Country Led Evaluation Strategy

37. An Integrated Country-Led Evaluation Strategy should be the desired approach in most cases. We would envisage it working as follows:

• The evaluation strategy should form part of the Memorandum of Understanding on Budget Support and should be incorporated into the financing agreements at the start of budget support programmes (ideally with all budget support donors agreeing to follow the same evaluation format). It should be clearly expressed that the purpose of the evaluation strategy would be both to inform the design of national and sector policies, supported by budget support, and to inform the design and management approach to budget support. It should not have any direct impact on annual disbursement decisions but would be expected to inform the design of new operations as well as the overall scale, scope and management arrangements for budget support.

• The GBS/ SBS Evaluation strategy would comprise different elements:

o An overall assessment covering Step One every 5 years.

o Ongoing assessments of key government policies at sectoral level, covering the objectives of the current Step Two but with each assessment covering only one area, which might be the analysis of economic growth, or of the evolution of poverty, or one sector/ one region/ one thematic area, depending on the policies of greatest importance to Government and its Development Partners. Ideally, resources might be made available for randomized


17

evaluations at the sector level regarding key policy changes proposed by recipient governments. These randomised trials would be treated as pilot studies to build an evidence base, and then once a policy is shown to work, recipient governments could use this as part of the evidence base for additional budget support. These ongoing assessments would be separately commissioned and probably undertaken at a rate of no more than one such study per year11. Teaming up with academic institutions or other institutions with established expertise in impact evaluation would be encouraged.

o These ongoing assessments would then be consolidated and linked back to budget support issues at the next 5-year assessment, through a formal Step Three exercise.

o Thus Step One and Step Three exercises would be undertaken every five years and different types of Step Two studies undertaken roughly on an annual basis, depending on the availability of data for the range of studies desired.

o Some primary data collection exercises – additional to regular household budget surveys, Demographic & health survey and other established processes – might be undertaken in order to allow for a richer process of Step Two studies. For example, Beneficiary Incidence Surveys might also be made part of the five year programme of work.

o It would be very important to integrate this programme of Step Two studies with the existing programme of studies and assessments related to the national PRSP or national development strategy, and to key sectoral strategies. In this way, budget support evaluation would be fully integrated with the evaluation of national and sectoral policies.

4.1.2 A Composite Two Year Evaluation Process

38. In smaller countries, with more limited data availability and less research capacity, it might prove difficult to manage an ongoing, high quality programme of evaluation. There may also be middle income countries – such as Tunisia, where budget support has been limited to only one or two sectors and where it might therefore be the Government’s wish to have a narrower scope of joint evaluation activities. In such countries, a composite process managed by a single team of evaluators would probably be desirable; yet in other respects many aspects of the integrated process would be retained.

• Firstly, the essential aim of trying to complete a full budget support evaluation – compromising Steps One, Two and Three – every five years should be retained. However, the timing of the evaluation would need to be decided based upon a proper analysis of data sources – especially to avoid the situation which emerged in Mali where Step Two analysis had to be undertaken despite the absence of household or other national surveys in the recent past.

• In common with the integrated approach, some separation of the individual Steps would be sought in order to avoid it becoming too unwieldy and to facilitate the assimilation of results by policy makers. We envisage a two-year process with a number of discrete products as follows:

o An initial Inception Report outlining the overall evaluation process, based on a detailed assessment of policy interests and concerns and an examination of existing data sources and planned surveys/ evaluations.

11 However, some of these Step Two studies might be quite extensive in time and scope, certainly more than has been envisaged to date for the existing methodology. For example, the step 2 methodology paper states: “The process of analysing the quality, the processing and linking of secondary data may take up to three months.” (60-70 consultant days). Depending on the quality and quantity of the data to be analysed this could easily be much more. (As an example, 3 research assistants + 1 principal researcher worked for 6 months on the World Bank’s Bangladesh study on maternal health) Moreover, if such studies were to be complemented with additional primary data collection, a time period of over a year would be necessary.


18

o A first Step Two study during Year 1.

o Initiation of any necessary primary data collection work, including possibly perception surveys/ citizen satisfaction surveys, also during Year 1.

o A Step One study during Year 2.

o Completion of a second Step Two study early in Year Two.

o A Step Three study and synthesis report towards the end of Year Two

• For the Step Two studies, it would be important to pick 1 or 2 sector-specific questions that could be reasonably expected to be answered within the timeframe. Ideally, questions would be linked to policy changes whose implementation has begun – such as specific investment programmes, introduction/elimination of fees, introduction of new service delivery approaches, etc. Exogenous events of this kind provide the best scope for causality analysis and thus give the best scope for evaluating policy impact.

4.2 Improving the Conduct of the Evaluation Process: Lessons from Mali

39. There are a number of operational lessons from the Mali experience, which should be incorporated into both of the above approaches, and indeed into other frameworks for the evaluation of budget support. Two have already been stressed, and other lessons are presented below

• The need for the evaluation team to be well resourced in terms of human and financial resources; and

• The need for the deliverable products to be presented as a series of discrete reports rather than a smaller number of big, combined reports. Thus, one would envisage:

o An integrated Inception Report, focused on methodological issues;

o Separate Step Two reports for each sector or theme assessed.

o A separate Step One report.

o A separate Step Three report;

o A short Synthesis report presenting conclusions and recommendations.

• Improving the usefulness of the Inception Phase:

o The focus should be primarily on ensuring that the available data sources and analytical methods for each Step of the evaluation are well defined, in particular for Step Two.

o There should be less focus on Evaluation Questions and judgement criteria. There should be a broadly standard set of Evaluation Questions for each of the steps, which would need to be adapted to each country context, with some relatively high-level judgement criteria identified. This should not dominate the Inception phase.

o A clear timetable of work needs to be established and agreed, including a well-defined programme for survey work (if envisaged), data collection and analysis.

• Enhancing the analytical scope of Step Two – including primary data collection

o As we have noted above, we believe that the scope of Step Two should be opened up substantially, including a range of potential statistical and other analytical approaches.

o Where econometric analysis is proposed, we see no particular reason to favour the approaches currently recommended in Step Two. The Gunning et al methodology there


19

proposed can be useful but it has a number of weaknesses and there is no special reason to believe that it is better suited to impact analysis or to budget support than alternative techniques. (Some detailed comments on the methodology are presented in annex.)

o In countries with significant limitations on data sources, it would be useful to formally plan for some primary data collection in order to enhance what is available. Within the framework of an integrated five year cycle of evaluation work, there should certainly be space for Beneficiary Incidence Surveys to be undertaken. Within composite two-year evaluations, perception surveys and client satisfaction surveys should be considered.

• Dedicating more time and attention to Step Three:

o In Mali, the Step Three analysis was done relatively quickly as the final step in the process of evaluation. The preliminary conclusions were not well discussed even within the evaluation team and there was only limited interaction with in-country stakeholders before the conclusions were committed to paper. Thus, an opportunity was missed because the Step Three framework provides a good structure for an interactive discussion on the impact of budget support. We would recommend that more time should be dedicated to this phase, particularly to consultations over preliminary conclusions.

EVA 2007/geo-acp : Evaluation des opérations d’aide budgétaire au Mali ECO Consult – AGEG – APRI – Euronet – IRAM – NCG

20

Annex One: Reflections on the Statistical Impact Evaluation

Methodology Proposed by Gunning et al.

Some Reflections on the Gunning et al. Methodology

These are some reflections on the paper ELBERS, C., J. W. GUNNING, AND K. DE HOOP (2008): “Assessing sector-wide programs with statistical impact evaluation: a methodological proposal,” World Development, 37(2), 513–520.

Statistical impact evaluation is designed for projects with a clearly defined treatment that is homogeneous across the treatment group (i.e. everyone gets the same treatment). In this paper, Gunning et al extend the method to be applicable to general budget support and sector support. The basic idea is to regress the change in an impact variable (this is the outcome of interest, e.g. exam scores) on (the change in and levels of) control variables and the change in all relevant policy interventions in the sector (or in a number of sectors). From this, one may derive the aggregate impact of, say, all schooling interventions, and further carry out a full cost-benefit analysis.

It is important (but probably impossible) to include all policy variables, otherwise there is potential omitted variable bias. Gunning et al seem to believe that non-random treatment with a policy intervention can be addressed using sample selection methods or difference-in-difference, but we consider this view too optimistic. The concern surrounding endogeneity in real world policy interventions is the key motivation driving randomised control trials! If policy interventions were automatically random, there would never be a need to design and implement randomised impact evaluations and this literature would have never even appeared! Gunning et al claim to build on this literature, but what they do has nothing to do with randomised trials and instead reverts back to regressions with endogenous policy interventions on the right hand side. So their contribution is to stress that if one includes all policy interventions in a certain sector, this should provide a good basis to gauge the effect of aid to that sector. However, how exactly to move from correlation to causal effects is not very clear. Gunning (2006), as well, mentions that, with non-random treatment but baseline data, fundamental problems of unobserved differences between treatment and control can be controlled for in a difference-in-difference framework, but Duflo and Kremer (2003) say this is not the case.

Gunning et al also argue (footnote 13) that if the question is about the impact of an intervention/ expenditure given the policy process in place that might bias the allocation across localities and/or people, then this question should be answered without controlling for selection into the sample (in other words: without addressing the endogeneity problem). There are problems with this approach. Suppose the intervention is giving scholarships to attend university. Let’s say these are allocated to clever children of rich parents. These children would have gone on to university anyway, and they have the money to do so, so the scholarship program has absolutely no causal impact on their enrolment in universities. In other words, the effect of the aid/expenditure used for the scholarships is zero. But since these children are much more likely to enrol than other secondary school graduates, it will look as if the scholarship increases tertiary enrolment. Formally, the policy intervention (receiving a scholarship) is positively correlated with unobserved attributes of the recipient that are part of the error term, because there is a selection problem (treated and untreated are different in other

EVA 2007/geo-acp : Evaluation des opérations d’aide budgétaire au Mali ECO Consult – AGEG – APRI – Euronet – IRAM – NCG

21

respects than the treatment, for instance in income or in quality of past education received). If we do not address the selection problem we would conclude the programme has had a large, positive impact on tertiary school enrolment. But it doesn’t make sense to say that this captures the effect of the intervention given the current policy process that biases the allocation of scholarships. Because even under “the current policy process that biases allocations” the causal impact is zero.

The Gunning et al method assumes full intrasectoral fungibility and zero intersectoral fungibility. This is because, for instance, education aid is linked to all education policy interventions at once. There is no tracing of specific pieces of education aid to specific education policies. If these assumptions do not hold, the estimate of the impact of sector support might be very different from the real impact. If we trace each piece of sector support to each specific policy intervention, then the assumption of full intrasectoral fungibility can be substituted for one of zero intrasectoral fungibility. Moreover, how then can general budget support be evaluated if we don’t start linking all the bits of general budget support to specific sectors and interventions? Do we just see general budget support as supporting everything the government does, i.e. all policy interventions in all sectors and regions? Gunning (2006) actually addresses this, and says that for general budget support all government interventions need to be included. Note how also this means that we are assuming that general budget support is treated in the same way as a general increase in revenue (and therefore partly financing military expenditure), which is not very believable. It also sits uncomfortably with the assumption that sector support is not fungible at the sectoral level. So the donor can make sure that education aid is used for education spending but not that general budget support cannot be used for military expenditure.

Gunning et al mention their method also allows assessing the relative impacts of each policy. But if we only estimate a linear effect for each policy intervention, then one of the policy interventions by definition must have the largest impact. The results would then suggest to shift all sectoral aid to support that policy intervention. This is a bit strange, and it emanates from the fact that a single, linear impact of each policy intervention is typically estimated, without for instance allowing for diminishing returns.

The impact of sector-wide interventions, such as administrative reform, cannot be estimated this way because they affect all people (there is no control group, or in the regression framework there is perfect collinearity with the constant). To the extent that such sector-wide interventions change inputs, they can be reflected in the effect of these inputs.

By assuming all sectoral aid feeds into expenditure that supports sectoral policy interventions, they are not only ignoring the possibility of intersectoral fungibility, but also of corruption, embezzlement, and a whole lot of political economy literature that suggests aid does not always reach recipients.

Bibliography DUFLO, E., AND M. KREMER (2003): “Use of randomization in the evaluation of development effectiveness,” Paper prepared for the World Bank Operations Evaluation Department (OED) Conference on Evaluation and Development Effectiveness in Washington, D.C., 15-16 July, 2003.

ELBERS, C., AND J. W. GUNNING (2009): “Evaluation of development policy: treatment versus program effects,” Tinbergen Institute, Discussion Paper 073/2 (revised August 2009).

ELBERS, C., J. W. GUNNING, AND K. DE HOOP (2008): “Assessing sector-wide programs with statistical impact evaluation: a methodological proposal,” World Development, 37(2), 513–520.

GUNNING, J. W. (2006): “Aid evaluation: pursuing development as if evidence matters,” Swedish Economic Policy Review, 13(2), 145–163.

joint evaluation of budget support operations ......oecd-dac working group on budget support...

Documents