Ten years of Evaluability at the IDB

Yuri Soares, Alejandro Pardo,

Veronica Gonzalez and Sixto Aquino

Paris, 16 November, 2010

Preliminaries

Evaluability is the “ability of an intervention to demonstrate in measurable terms the results it intends to deliver” (IDB, 2002, 2006, 2010).

In terms of Development Effectiveness, the Bank’s capacity to manage for results depends to a great extent on having operations that feature the characteristics needed for results to be measured, as well as the understanding of the main factors affecting the process by which they are generated.

IDB Experience The IDB’s Evaluation Office has produced three assessments of project

evaluability: 2001, 2005, and 2009. These exercises systematically reviewed the universe of IDB projects approved during these years. Thus, the institution now has longitudinal data on project evaluability.

The 2005 Evaluability Report recommended that evaluability standards be introduced as a criterion for project approval.

This recommendation was adopted and included in the institution’s mandate as part of its Ninth General Capital Increase (2010). The institution is currently moving toward implementing this mandate; and OVE has been instructed to perform evaluability assessments on a yearly basis.

Evaluability Dimensions The evaluability consists of nine dimensions including substantive and

formal dimensions. Substantive dimensions are those that assess the proper identification

and linkages between the conceptual elements of an intervention. Formal dimensions are those that go to the “classical” measures of

evaluability, such as the identification of indicators and baselines.

Substantive

Diagnostic

Objectives

Logic

Assumptions and risks

Formal

Output baselines

Outcome baselines

Output indicators

Outcome indicators

Monitoring and evaluation

Substantive Dimensions

Diagnosis: Evidence-based identification of the problem and its roots causes.

Objectives: Identification of what project expect to achieve. Objectives must be S.M.A.R.T. (Specific, Measurable, Agreed upon, Realistic, Temporal).

Logic: why this particular intervention and why not others? Causal chain: components create conditions produce outputs achieve outcomes

Risks: Quality of analysis in the identification of assumptions & risks. Risk Evaluation, Follow-up and Mitigation

Formal Dimensions

Outcome indicators: measures of expected results during and/or at end of project

Output Indicators: measure of expected products executed as part of the operation

Indicators must be mutually exclusive, valid, adequate, and reliable

Baselines for outcomes: Ex-ante assessments of conditions expected to change as a result of project

Baselines for outputs: Ex-ante assessments of the goods and services present prior to the project

Monitoring and Evaluation: Identification of systems and resources for data collection.

Protocol To ensure the proper application of the exercise, a protocol

was designed and implemented, consisting of three steps: Write-up of findings. Project assessments are done by peer

review. Reviewers meet, discuss the proposed operation, and produce a note reporting on findings in each evaluability dimensions.

Collegial review. The findings of the note are then discussed by a collegial group in the office. This same group reviews all projects to ensure consistency across projects.

Scoring. Once a final draft is produced, the team and collegial group agree on a scoring for each of the dimensions. The scoring scale is a 1-4 scale, with two adequate and two inadequate categories, and is based on a scoring guide.

Principles

The method adheres to a series of principles:

Quality. Reviews are done by peer reviews; these peers contain staff with knowledge of the sector and problematic.

Independence. To avoid conflicts of interest, staff who may have had involvement in a project do not participate in the review.

Accountability. A manager oversees the overall exercise and is responsible for its quality.

Consistency. All reviews are validated by the same collegial group, so as to ensure consistency across different project reviews.

Results

An Operations microscope The review tracks the evaluability of the Bank’s principal

production function: the design of lending operations. This can provide insights regarding specific problems

encountered in projects in order to improve them. For example:

Identify what efforts are needed to ensure that problem situations are accurately dimensioned and ensure that the Bank has evidence regarding the aptness of the intervention models

Identify overlooked risks that can impact the viability of the interventions in order to ensure better risk management.

Address and quantify “classical” evaluability questions, such as if outcome indicators are adequately defined, and sufficient for the scope of the operation’s objectives, and if they have baseline data. They have clear implications for the institution’s monitoring and data collection efforts.

A Institution microscope

Our experience with evaluability shows that it can also be a valuable tool to look at how the Bank operates. We have done this by analyzing the link between evaluability trends and:

changes in quality review process, variations in incentives faced by teams and managers, impact of the organizational changes in the quality of

projects, allocation of staff and resources, and fluctuations in lending portfolio

Oversight example: In 2005 and in 2009, the IDB analyzed the functioning of the institution’s quality control function. Findings included that Managers, Bank committees, and peer review instances were not providing sufficient guidance to project teams, and in almost no case were they providing oversight of evaluability-related issues of project design. Also, divisions formally responsible for quality control were mostly absent from the review process.

National systems example. In 2005 found evidence that the Institution’s bureaucratic incentives were not aligned with those of borrowers. OVE recommended the use evaluability in order determine the Bank’s value in project design in order to inform to decisions of increased use of national planning systems

Risk example. In 2005 and in 2009 the Evaluability assessment looked at how the institution managed risk at the project level. The results found that for sovereign guaranteed operations issues of risk were rarely identified, while for private sector operations risks were always front and center, but were exclusively focused on financial risks responding to the repayment related incentives

Concluding remarks

Evaluability provides CEDs with an opportunity to play an important role in assessing and influencing the way development institutions discharge their mandates

Evaluability assesses critical elements of the quality of the work of the organizations as related to their capacities for management for results

Evaluability can be used as a tool for looking at how institutions organize and for steering steer critical institutional improvement processes