institutionalisation of government evaluation: balancing trade-offs

www.3ieimpact.orgAuthor name

Institutionalisation of Government Evaluation:

Balancing Trade-Offs

Marie Gaarder, Deputy Director, 3ieMarie Gaarder, Deputy Director, 3ie

Cotonou, June 14, 2010Cotonou, June 14, 2010

International Initiative for Impact Evaluation


OverviewOverview

• Evidence – underutilised tool in policy-making

• Inception, independence and policy influence in three Latin American cases (Gaarder and Briceño, forthcoming, Journal of Development Effectiveness)

• Some key considerations for an effective evaluation framework

• How is 3ie contributing to the international effort to improve the impact of policy?


Why evaluationWhy evaluation??

Evaluation, in particular impact evaluation, is a tool that can help governments base policy on evidence of what works, rather than on prior beliefs

Evidence is an international public good

So, why is there so little of it?... ... and why has the existing evidence

not been utilized and disseminated in an optimal manner?

Although supply of good evaluators is a constraint, main limitation is lack of demand


Monitoring versus evaluationMonitoring versus evaluation

• Monitoring is used to continuously gauge whether the project or intervention is being implemented according to plan

• Evaluations assess progress towards outcomes

• Impact evaluation whether these can be attributed to the intervention

Continuous project monitoring is a sine qua non for implementing organisations to ensure efficient implementation and necessary course corrections

Some form of external, more objective or independent entity needs to be involved in overseeing project evaluation


What is ‘institutionalisation’?What is ‘institutionalisation’?

• In the context of evaluation, we consider institutionalisation a process of channelling isolated and spontaneous program evaluation efforts into more formal and systematic approaches, on the presumption that the latter provide a better framework for fully realizing the potential of the evaluation practice.

• Usually implies the creation of a governmental institution / body responsible for overseeing this process (overall or at sector level)


Evaluations have tended to be selected based on Data availability Interest of researchers and donors Availability of funds Amenability to certain evaluation methods Willingness to be evaluated (‘good project’ bias)

....rather than potential contribution to broader development strategies

Institutionalisation of quality evaluation is necessary in order to turn it into an optimal tool for policy makingoptimal tool for policy making For prioritization of evaluation agenda and scarce resources For standard-setting, comparison and benchmarking purposes For analysis of trade-offs For channelling the needs of evaluation’s clients; thus enhancing relevance

Why institutionalise evaluationWhy institutionalise evaluation??


The challenge, part IIThe challenge, part II

‘we are very keen to establish the independent evaluation office but…delays have occurred because we haven’t figured out the way of controlling it…’


InceptionInception


The Trade-OffThe Trade-Off

Independence/ Credibility

Reporting structure

Source of operational budget

Evaluation supervision

Procurement processes

Public disclosure

Policy influence

Organisational location

Independent budget line for evaluation

Enforcement supporting laws

Alliances

Ownership


MexicoMexico

• CONEVAL: Autonomous public administration entity created in 2004, under Ministry of Social Development.

• Mandate constrained to social sectors but acts as the standard setter and articulator of evaluation activities across government agencies

• Reporting to an Executive Board of six independent academics appointed by the National Commission for Social Development

• Access to public information law introduced in 2000, and CONEVAL’s General Guidelines prescribe the dissemination of all evaluation documents and results through the internet within 10 business days of their reception

• CONEVAL’s operating costs (though not the evaluations) financed through a direct budget line in the National Budget

• Social sector agencies are required by law to have an annual evaluation program agreed-upon with CONEVAL, the MOF, and the public comptroller’s office as a prerequisite for inclusion in the national budget

• In summary: has achieved both a degree of independence and policy-influence, and set-up of the evaluation system and guidelines designed to address both managerial usage and budget and planning usage


ColombiaColombia

• SINERGIA/DEPP: unit established within the National Planning Ministry• Mandate and conceptual basis are broad and involve evaluation activities across all sectors

and government levels• Inter-Sectoral Evaluation Committee (IEC) supposed to oversee evaluation agenda but has

however to date functioned on an ad-hoc basis (no provision for an extra-governmental governance body)

• No access to public information law• DEPP’s operating costs (though not the evaluations) financed from NPM budget• No enforcement-supporting law as in Mexico’s case, hence focus on generating a tradition

of utilization as a managerial tool rather than a control tool (persuasion) • In summary; maybe culture of utilization will spread more rapidly with the voluntary

approach, but (i) credibility of reported findings is questioned due to the lack of independence and public dissemination laws, and (ii) ability to enforce recommendations is lacking, thus making the system rely on voluntary adoption by voluntarily participating agencies (potential double bias).


ChileChile

• Evaluation unit within Dipres: Budget Department of the Ministry of Finance• The overall goal of the unit is to contribute to the efficiency of allocation and utilization of

public spending, contributing to better performance, transparency and accountability. • The Budget Directorate is accountable to the Congress but this has not plaid a very active

role• Since 2008 an International Advisory Panel advises on the quality of the impact evaluation

designs • The evaluations of programs and institutions are reported to Budget, Congress and the

public, and are available at Dipres’ website. • Dipres has dedicated budget line to finance the approved evaluation plan• In summary; the strongest enforcement capacity is in Chile, and this is also reflected in the

fact that the Chilean system’s information is highly utilized in budget analysis, decision making and in imposing program adjustments. However, managerial usage or ownership has been limited, given the absence of incentives for the agencies to engage in their own evaluations. (Some shortcomings with respect to the quality of the findings).


System trade-offs in Mexico, Colombia and ChileSystem trade-offs in Mexico, Colombia and Chile


• The actual usage of evaluation information is the benchmark of success, and determines the sustainability of the systems

• Unfortunately, direct measures of the effect of institutionalising evaluation upon development effectiveness are still lacking

• Two outcome type of measures have been particularly explored (although incipient) related to coverage;– Proportion of budget or universe of programs evaluated

E.g. there are 100-130 federal programs under the mandate of CONEVAL of which all are required to carry out logframe-type evaluations, in addition CONEVAL oversees directly about 15 evaluations per year, equivalent of 11% of the programs under its mandate, of which approximately 20% are impact evaluations.

– Follow-up of recommendations, commitments and actions plans derived from evaluations

101 programs were included in the CONEVAL tracking system, with 930 aspects to improve; out of these, 70% were of the responsibility of the program officers

The Chilean system reports that out of the overall commitments, 82% were fulfilled, 11% were partially fulfilled, and 6% have not been fulfilled.

Measures of successMeasures of success


Seven key measures for an effective Seven key measures for an effective evaluation frameworkevaluation framework

1. Focus on usage and clarity on a client or set of clients that are to be served, and what their interests are

2. Have a unique and broad legal mandate for evaluation

3. Immerse all impact evaluations into broader systems with complimentary evaluation instruments

4. Build local technical capacity among relevant Ministry officials, program implementers, and local researchers

5. Strengthen data collection and processing systems in order to ensure high quality of data

6. Ensure that evaluation is an integral part of programs since their inception

7. Guarantee full public disclosure through legislation on access to public information or transparency


3ie’s contribution to policy impact

• Launched Policy Window 1 – financing evaluation of interventions proposed by policy makers

• Launched Open Window 3 – new procedures including policy panel

• Financing 50 impact evaluation projects to date for close to US$15M, in all developing regions and wide variety of sectors

• Developing research agenda around how evidence can achieve the intended influence

• Synthetic reviews


Thank you!Thank you!

VisitVisithttp://www.3ieimpact.org/http://www.3ieimpact.org/

http://www.3ieimpact.org/3ie_reports.htmlhttp://www.3ieimpact.org/3ie_reports.html

institutionalisation of government evaluation: balancing trade-offs

Documents