1998 annual review of development...

W O R L D B A N K O P E R A T I O N S E V A L U A T I O N D E P A R T M E N T

1998 ANNUALREVIEW OFDEVELOPMENTEFFECTIVENESS

Robert Buckley

1999

The World Bank

www.worldbank.org/html/oed/ardefm98.htm Washington, D.C.

Copyright © 1999

The International Bank for Reconstruction

and Development/THE WORLD BANK

1818 H Street, N.W.

Washington, D.C. 20433, U.S.A.

All rights reserved

Manufactured in the United States of America

First printing February 1999

The opinions expressed in this report do not necessarily represent the views of the World Bank or itsmember governments. The World Bank does not guarantee the accuracy of the data included in thispublication and accepts no responsibility whatsoever for any consequence of their use. The boundaries,colors, denominations and other information shown on any map in this volume do not imply on the partof the World Bank Group any judgment on the legal status of any territory or the endorsement oracceptance of such boundaries.

The material in this publication is copyrighted. The World Bank encourages dissemination of its workand will normally grant permission promptly. Permission to photocopy items for internal or personaluse, for the internal or personal use of specific clients, or for educational classroom use is granted by theWorld Bank, provided that the appropriate fee is paid directly to the Copyright Clearance Center, Inc.,222 Rosewood Drive, Danvers, MA 01923, U.S.A., telephone 978-750-8400, fax 978-750-4470.Please contact the Copyright Clearance Center before photocopying items. For permission to reprintindividual articles or chapters, please fax your request with complete information to the RepublicationDepartment, Copyright Clearance Center, fax 978-750-4470.

All other queries on rights and licenses should be addressed to the Office of the Publisher, World Bank,at the address above or faxed to 202-522-2422.

ISSN 1019-4363

ISBN 0-8213-4474-9

i i i

v Acknowledgmentsvii Foreword, Prefacio, Préfaceix Executive Summary, Resumen, Résumé Analytiquexvii Abbreviations and Acronyms

1 1. Development Effectiveness in a Volatile World1 Institutional Weaknesses4 Broadening the Agenda to Maximize Development Effectiveness4 The New Primacy of the External Environment

7 2. Trends in Project Performance7 Outcomes More than 75 Percent Satisfactory

11 Gains in Development Effectiveness14 Institutional Development Impact—Improving but Still Weak14 Sustainability—Low and Weakening?15 Bank and Borrower Performance Improving—But Greater Gains from Borrowers

16 Outstanding Projects—7 Percent of the Total17 Sector Analysis

19 3. Development Effectiveness at the Country Level19 The Country as the Unit of Account20 Institutions, Aid, and Growth23 Adjusting to the External Environment

24 Outward Orientation and Growth24 Outward Orientation and Poverty Alleviation

27 4. Thematic Evaluations and Institutional Development27 Financial Sector

28 Capitalization and Bad Loan Exposure28 Government Interventions28 Accounting and Prudential Standards28 Enforcement Capabilities

29 Governance Issues29 Governance Institutions

33 5. Implications for the Bank and for Evaluation33 Global Risks35 Prospects for Poverty Reduction36 Implications for the Bank

36 Poverty Alleviation36 Country Strategies36 Institutional Development

39 Annexes39 Annex 1: A New Approach to Evaluating Bank Projects—The Development

Effectiveness Index42 Annex 2: Glossary of Selected Terms44 Annex 3: Data on Outcome, Sustainability, and Institutional Development Impact48 Annex 4: Report from Committee on Development Effectiveness (CODE)

C o n t e n t s

1 9 9 8 A n n u a l R e v i e w o f D e v e l o p m e n t E f f e c t i v e n e s s

i v

51 Endnotes

53 Bibliography

Boxes8 2.1 Institutional Development and Sustainability Help Capture Project Performance

11 2.2 Improvement in Africa17 2.3 Two Outstanding Projects21 3.1 Institutions and Developing Economies22 3.2 The Long-Run Expected Benefits of Adjustment24 3.3 Measuring Borrower Ownership and Its Relation to Adjustment Outcomes25 3.4 Greater Poverty and Income Inequality in Transition Economies31 4.1 Public Sector Performance Review: Integrating Public Sector Performance with a

Results-Based Management System34 5.1 What Do We Know about the Extent of Poverty?37 5.2 Poverty Alleviation and Income Growth

Figures4 1.1 Institutions and Project Performance in Low-Income Countries8 2.1 Demandingness, Complexity, Riskiness, and Outcome9 2.2 Development Effectiveness Index and Outcome: Measuring Project Performance

10 2.3 Satisfactory Outcome by Region12 2.4 Africa Region—Relative Performance Trend, Percentage Operations with

Satisfactory Outcome12 2.5 Satisfactory Outcome by Sector and Exit Fiscal Year Group13 2.6 Institutional Development Impact14 2.7 Sustainability14 2.8 Borrower Performance15 2.9 Bank Performance16 2.10 Relative Risk and Reward, by Sector, Exit FY97–9829 4.1 Performance of Civil Service Reform Interventions

Tables2 1.1 Traditional Crisis Indicators3 1.2 The Washington Consensus—Not in East Asia

15 2.1 Recently Evaluated Projects: Outstanding Performers and Poor Performers20 3.1 Performance of Bank-Supported Country Strategies and Projects21 3.2 Trends in Socioeconomic Indicators and Income, 1970–9628 4.1 Bank System Risk Exposure in East Asia

v

ACKNOWLEDGMENTS

his review was prepared by Operations Evalu-ation Department staff. The task manager wasRobert Buckley. The contributors were Will-

Silvana Valle, Geri Wise, Brigitte Wittel, and BarbaraYale provided administrative assistance.

This study was produced in the Partnerships andKnowledge Group (OEDPK) by the Dissemination andOutreach Unit. The unit is directed by ElizabethCampbell-Pagé, Task Manager, and includes CarolineMcEuen and Leo Demesmaker (editors), Kathy Straussand Lunn Lestina (desktop design and layout), andJuicy Qureishi-Huq (administrative assistance).

Tiam G. Battaile, Jr., and Ruchira Banerjee-Corcoran(OEDCM); and Deepa Chakrapani, Federico Mini,and Dinara Seytjaparova (OEDCR). Contributionswere also made by James Alm (consultant); KeithBezanson (IDS, Sussex); José María Dagnino Pastore,Louis Goreux, and Jane-Francis Kelly (consultants);Anwar Shah and Navin Girishankar (OEDCR); andSohail Malik, Kenneth Watson, and John Eriksson(consultants). Ruben Lamdany (Manager, OEDCR)and Roger Slade (Manager, OEDST) and their staffprovided comments and were helpful in many ways.Wendy Jarvie (Manager, OEDCM) supervised andhelped to structure the Review. Norma Namisato,

Director-General, Operations Evaluation: Robert Picciotto

Director, Operations Evaluation Department: Elizabeth McAllister

Manager, Corporate Evaluations and Methods: Wendy Jarvie

Task Manager: Robert Buckley

vi i

FOREWORD PREFACIO PRÉFACE

F o r e w o r d

EN

GL

IS

H

ES

PA

NO

L

FR

AN

CA

IS

The ongoing financial crisishas raised questions about theunderpinnings of developmentassistance and the role ofinternational financial institutions.A new development assistanceframework, grounded inpartnership, is emerging. That isthe backdrop for this year’s AnnualReview of DevelopmentEffectiveness by the World Bank’sOperations Evaluation Department(OED).

As in past years, the Reviewtracks the Bank’s operationalperformance based on the findingsof recent evaluations. The trendsare highly encouraging, but whencountries that have performed sowell for so long suddenly stumbleas dramatically as they did in thepast year, the meaning of project-level trends deserves carefulconsideration.

Accordingly, this Reviewdraws on the work of scholarsconvened by the Institute ofDevelopment Studies of SussexUniversity to assess theimplications of the crisis. It alsorelies on a relatively new OEDinstrument—country assistanceevaluations—to place the lessonsfrom the Bank’s project experiencein a broader context.

The Review complements theAnnual Report on PortfolioPerformance, which documents theQuality Assurance Group’s findingsabout active operations, and the

La actual crisis financiera haplanteado interrogantes acerca de losfundamentos de la asistencia para eldesarrollo y el papel que cumplen lasinstituciones financierasinternacionales. Comienza a surgir unnuevo marco para la asistencia en prodel desarrollo, basado en la formaciónde asociaciones de colaboración. Estees el telón de fondo del presenteExamen anual sobre la eficacia entérminos de desarrollo, preparado porel Departamento de Evaluación deOperaciones (DEO) del BancoMundial.

Al igual que en años anteriores,en este informe se pasa revista a losresultados de las operaciones delBanco sobre la base de lasconclusiones de las evaluacionesrealizadas recientemente. Lastendencias observadas son muyalentadoras, pero cuando los paísesque han registrado resultados muyfavorables durante mucho tiempoenfrentan de pronto gravesproblemas, como ha sido el caso en elúltimo año, es necesario analizarcuidadosamente el significado de dichaevolución al nivel de los proyectos.

En consecuencia, el presenteexamen se basa en la labor deexpertos convocados por el Instituteof Development Studies de laUniversidad de Sussex con el fin deevaluar las repercusiones de la crisis.También se apoya en un instrumentorelativamente nuevo del DEO—lasevaluaciones de la asistencia a lospaíses—con el objeto de poner las

La crise financière en courssoulève des questions quant auxprincipes directeurs de l’aide audéveloppement et au rôle desinstitutions financières internationales.Un nouveau cadre d’aide audéveloppement axé sur despartenariats fait son apparition. C’estdans ce contexte que le Départementde l’évaluation des opérations (OED)de la Banque mondiale a réalisé cetteannée son Examen annuel del’efficacité du développement.

Comme les années passées, cetExamen fait le bilan des opérations dela Banque d’après les conclusions desrécentes évaluations. Les tendancessont particulièrement encourageantes,mais lorsque des pays qui ont obtenud’excellents résultats pendant trèslongtemps ont soudain une défaillanceaussi marquée que celle qu’ils ontconnue l’an passé, la signification destendances au niveau des projets méritela plus grande attention.

En conséquence, le présentExamen s’appuie sur les travauxd’universitaires réunis par l’« Instituteof Development Studies » del’université du Sussex pour évaluer lesincidences de la crise. Il fait égalementappel à un instrument de l’OEDrelativement nouveau—les évaluationsde l’aide aux pays—pour replacerdans un contexte plus large lesenseignements tirés des projets de laBanque.

Cet Examen complète le Rapport

annuel sur la performance duportefeuille, qui rend compte des


vii i

enseñanzas derivadas de losproyectos del Banco en uncontexto más amplio.

El examen complementa elInforme anual sobre eldesempeño de la cartera, en elcual se exponen las

conclusiones del Grupo de garantía decalidad acerca de las operacionesactivas, y el Informe anual sobreevaluación de operaciones, en el que sepresenta el análisis que hace el DEO dela situación y las perspectivas de losprocesos de evaluación interna.

Robert PicciottoDirector-General, Operations Evaluation

conclusions du Groupe pour lecontrôle de la qualité sur lesopérations en cours et leRapport annuel sur l’évaluationdes opérations, dans lequell’OED fait le point de lasituation en ce qui concerne les

processus d’évaluation interne et lesperspectives qu’ils offrent.

EN

GL

IS

H

ES

PA

NO

L

FR

AN

CA

ISAnnual Report on Operations

Evaluation, which presentsOED’s assessment of the statusand prospects of internalevaluation processes.

i x

EXECUTIVESUMMARY

RESUMEN RÉSUMÉANALYTIQUE

E x e c u t i v e S u m m a r y

EN

GL

IS

H

ES

PA

NO

L

FR

AN

CA

IS

implications for the worldeconomy. Economic problems havebeen compounded by naturaldisasters, such as floods inBangladesh, in China, and inCentral America. The prospects forachieving the OECD povertyreduction targets have dimmed.

The crisis is rich in lessons forboth development practitioners andevaluators. Developing countriesnow confront a severe deteriorationin the enabling environment,highlighting the effects ofunregulated private flows and globalinterdependence and the growinginfluence of exogenous factors indetermining development impacts.

económicas. El Japón atraviesa por unarecesión que ha tenido profundasrepercusiones en la economía mundial.A las dificultades económicas se hanagregado los desastres naturales, comolas inundaciones en Bangladesh, China yCentroamérica. Las posibilidades dealcanzar las metas de reducción de lapobreza fijadas por la OCDE se handesvanecido.

La crisis deja numerosasenseñanzas para los que se dedican ala tarea del desarrollo y para quienesla evalúan. Los países en desarrollose enfrentan ahora al grave deteriorode las condiciones que hacen posibleel progreso económico, destacándoseen particular los efectos de laausencia de regulación de los flujosprivados y de la interdependencia a

El presente Examen sobre laeficacia en términos de desarrollo se da aconocer en un momento de crisis. EnAsia oriental, aproximadamente 20millones de personas han vuelto a caeren la pobreza en el último año. LaFederación de Rusia se ha visto afectadapor conmociones políticas y

This Review of developmenteffectiveness comes at a time ofcrisis. In East Asia, about20 million people have fallen backinto poverty in the last year.Russia has been beset by politicaland economic upheaval. Japan isin recession, with profound

Cet Examen de l’efficacité dudéveloppement coïncide avec unepériode de crise. En Asie de l’Est, unevingtaine de millions de personnessont retombées dans la pauvreté l’andernier. La Russie est assaillie degraves problèmes politiques etéconomiques et le Japon connaît unerécession qui a de profondes implica-tions pour l’économie mondiale. Descatastrophes naturelles telles que desinondations au Bangladesh, en Chineet en Amérique centrale ne fontqu’aggraver les problèmeséconomiques, et les chances de réaliserles objectifs de réduction de lapauvreté fixés par l’OCDE sontmaintenant réduites.

Cette crise est riched’enseignements aussi bien pour lespraticiens que pour les analystes dudéveloppement. Les pays endéveloppement se trouvent maintenantdans une situation beaucoup moinsfavorable, comme en témoignent leseffets de flux privés incontrôlés et del’interdépendance mondiale, etl’influence croissante des facteursexogènes sur les impacts dudéveloppement.

La stabilité macroéconomiquene suffit pasLes facteurs macroéconomiques nesuffisent pas à assurer une croissanceéquitable. À la différence de la crise dela dette des années 80, la crisefinancière actuelle a commencé dansdes pays dont la position budgétaireétait relativement forte et qui étaientcaractérisés par de saines politiquesmonétaires et des régimes du com-merce ouverts sur l’extérieur. Lorsque


x

A stable macro economyis not enoughSound macroeconomicconditions are not enough tosustain equitable growth.Unlike the debt crisis of the1980s, the present financial

crisis started in countries withrelatively strong fiscal situations,sound monetary policies, andoutward-oriented trade regimes.When the crisis hit, governmentbudgets in most crisis-affectedcountries were balanced or movinginto surplus, inflation was contained,interest rates were going down, andrecorded unemployment was low.

Institutions matterThe crisis showed just how costlyweaknesses in institutions can be—especially in the financial and socialsectors. Indeed, it is now clear thatstrong institutions are essential foreconomic and social stability. Poorinstitutions increase the vulnerabilityof developing and transitioneconomies to shifts in privateinvestor confidence. The importanceof institutional development goes farbeyond avoiding crises:

• For Bank-supported projects,the quality of institutions canhave important effects ondevelopment effectiveness.These effects are particularlypronounced in low-incomecountries.

• Where institutions aresystematically weak, projectsyield lower returns and entailhigher risk.

• Better institutions strengthena country’s ability to adjust.They can more than doublethe likelihood that a countryundergoing adjustment canstay the course.

ES

PA

NO

L

FR

AN

CA

ISnivel mundial, así como la

influencia creciente de losfactores exógenos en ladeterminación del impacto enel desarrollo.

La estabilidad macroeconómicano es suficienteLas condiciones macroeconómicas nobastan para mantener un crecimientocon equidad. A diferencia de lo ocurridodurante la crisis de la deuda de los añosochenta, la actual crisis financiera seoriginó en países con una situaciónfinanciera relativamente sólida, unapolítica monetaria acertada y sistemasde comercio orientados al exterior. Alproducirse la crisis, los presupuestospúblicos de la mayoría de los paísesafectados estaban equilibrados o inclusomostraban superávit, la inflación estabacontrolada, las tasas de interés estabanbajando y el desempleo registrado erareducido.

Las instituciones revisten importanciaLa crisis mostró claramente el costo quepueden significar las deficiencias de lasinstituciones, sobre todo en los sectoresfinanciero y social. En efecto, ahora nocabe duda de lo importante que escontar con instituciones sólidas paralograr la estabilidad económica y social.Cuando las instituciones son deficienteslas economías en desarrollo y entransición se vuelven más vulnerablesante los cambios en la confianza de losinversionistas privados. El desarrolloinstitucional no sólo es esencial paraevitar las crisis:

� Para los proyectos respaldados porel Banco, la calidad de lasinstituciones puede tener efectos deconsideración en la eficacia entérminos de desarrollo. Estasrepercusiones son especialmente no-tables en los países de ingreso bajo.

• Cuando las instituciones son

la crise s’est produite, les bud-gets publics de la plupart despays touchés étaient équilibrésou excédentaires, l’inflationmaîtrisée, les taux d’intérêt enbaisse et le taux de chômagedéclaré faible.

Importance des institutionsLa crise a montré combien lesfaiblesses institutionnelles peuvent êtrecoûteuses—en particulier dans lessecteurs financier et social. En fait, ilest maintenant évident que de solidesinstitutions sont indispensables à lastabilité économique et sociale. Dans lecas contraire, les économies endéveloppement et en transition sontd’autant plus vulnérables à une pertede confiance de la part desinvestisseurs privés. Le développementinstitutionnel est important, passeulement pour éviter des crises :

• Pour les projets financés par laBanque, la qualité des institutionspeut influer considérablement surl’efficacité du développement.Leurs effets sont particulièrementprononcés dans les pays à faiblerevenu.

• D’une façon générale, lorsque lesinstitutions laissent à désirer, lesprojets ont des taux de rentabilitéplus faibles et comportentdavantage de risques.

• Un pays doté de bonnes institu-tions est mieux à même des’adapter et a deux fois plus dechances de pouvoir maintenir lecap s’il est engagé dans unepolitique d’ajustement.

Une analyse de 41 pays à faiblerevenu montre que la qualité des insti-tutions n’a été jugée satisfaisante quedans un seul d’entre eux. Quarantepour cent seulement des projetssoutenus par la Banque ont un impact

EN

GL

IS

H

x i

An analysis of 41 low-income countries shows thatonly one was ratedsatisfactory on institutionalquality. Only 40 percent ofBank-supported projects havesubstantial impact on

institutional development; civilservice reforms undertaken ascomponents of structuraladjustment loans have mixedoutcomes; and public sectormanagement projects, whileimproving, have historicallyperformed below the Bankaverage. An OED evaluationshowed that Bank-supportedfinancial sector projects hadsatisfactory and sustainedoutcomes in just 50 percent ofcountries. Institutionaldevelopment is slow and difficultto achieve in a fragile institutionalenvironment and requires strongaid coordination and thedevelopment of capacity to absorbaid and reduce the risks ofoverload.

Poverty reduction and socialsafety netsA corollary lesson is that socialdevelopment should come centerstage—both in assessingdevelopment effectiveness and infinancing country assistanceprograms.

Serious reductions inemployment of 10–15 percent areestimated for Indonesia andThailand. With devaluations andthe removal of subsidies, the newlyunemployed will suffer fromdrastic losses in income and sharprises in prices. The increasinglyintegrated global environmentmeans that country susceptibility toshocks will not disappear. Muchgreater attention must be given to

EN

GL

IS

H

ES

PA

NO

L

FR

AN

CA

ISsistemáticamente deficientes, los

proyectos arrojan una menorrentabilidad y entrañanmayores riesgos.• Cuando un país cuenta coninstituciones más eficientes, tienemás capacidad de adaptación.

Gracias a éstas, al emprender ajustes,las probabilidades del país demantener el rumbo son doblementemayores.

Un análisis de 41 países de ingresobajo muestra que tan sólo uno de ellosobtuvo una calificación satisfactoria enlo que respecta a la calidad de lasinstituciones. Apenas el 40% de losproyectos respaldados por el Bancotienen un efecto considerable en eldesarrollo institucional; los resultadosde las reformas de la administraciónpública que se han emprendido comocomponentes de los préstamos paraajuste estructural han sido variados; losproyectos de gestión del sector público,si bien están mejorando, generalmentehan registrado resultados por debajodel promedio para el Banco. Unaevaluación realizada por el DEO revelóque los proyectos del sector financierorespaldados por el Banco arrojabanresultados satisfactorios y duraderostan sólo en la mitad de los paísesestudiados. El desarrollo institucionalavanza lentamente y es difícil de lograrcuando las condiciones de lasinstituciones son precarias; se requiereuna buena coordinación de la ayuda yes preciso crear la capacidad paraabsorberla y para reducir los riesgos desobrecarga.

Reducción de la pobreza y redesde protección socialUn corolario es que el desarrollo socialdebería pasar al primer plano a la horade evaluar la eficacia en términos dedesarrollo y de financiar los programasde asistencia a los países.

marqué sur le développementinstitutionnel ; les réformes de lafonction publique entreprises àtitre de composantes de prêts àl’ajustement structurel ont desrésultats mitigés ; les projetsrelatifs à la gestion du secteur

public, tout en s’améliorant, donnentdepuis toujours des résultatsinférieurs à la moyenne des projets dela Banque. Une évaluation de l’OED amontré que les projets soutenus par laBanque dans le secteur financier nedonnaient des résultats satisfaisants etsoutenus que dans 50 % des pays. Ledéveloppement institutionnel est lentet difficile à assurer dans unenvironnement institutionnel fragile etil nécessite une excellente coordinationde l’aide et la mise en place descapacités voulues pour absorber l’aideet réduire les risques de surcharge.

Réduction de la pauvreté et filetsde protection socialeUn autre enseignement à tirer est lesuivant : le développement socialdevrait jouer un rôle central, au niveauaussi bien de l’évaluation de l’efficacitédu développement que du financementdes programmes d’aide aux pays.

On estime qu’en Indonésie et enThaïlande, les suppressions d’emploissont de l’ordre de 10 à 15 %. Du faitdes dévaluations et de l’élimination dessubventions, les nouveaux chômeurssubiront les effets d’une énorme pertede revenu et d’une forte hausse desprix. L’environnement mondial étantde plus en plus intégré, les paysresteront vulnérables aux chocs. Ilfaut accorder beaucoup plusd’attention aux filets de sécuritédestinés à empêcher les pauvres etceux qui sont à la limite de la pauvretéde faire les frais de ces chocs de façondisproportionnée.

Les pays en crise ne sont pas lesseuls à connaître une aggravation des



xi i

safety nets in helping toinsulate the poor and thenear-poor fromdisproportionately bearingthe costs of shocks.

The crisis countries arenot the only ones

experiencing increasinginequality. Data for 74 countriesshow an overwhelming increasein inequality within countries inthe 1990s—49 countriesexperienced increasinginequality, while only 10 haddecreasing inequality. Thisconfirms the need to emphasizeinclusion, social development,and safety nets in the design andimplementation of reformstrategies and developmentprograms.

A country focus based onpartnershipFinancial, institutional, andsocial factors must be consideredtogether. For growth to result insustainable development requirescountry assistance strategies thatgive adequate weight tostructural factors, capacitybuilding, and social equity, andthat identify potential holes inthe boat—where structural faultsmight cause development gainsto unravel.

A credible Bank role beginswith effective projects. Thisimplies operations linked to thebroader social, civil, andeconomic environment. To scale-up successes, the Bank must workin partnership with borrowers,donors, and other stakeholders tofocus on maximizingdevelopment impact at thecountry level. To do so the Bankmust consider the important sideeffects that interrelated activities

EN

GL

IS

H

ES

PA

NO

L

FR

AN

CA

ISSe prevé una fuerte

reducción del empleo, de entre10% y 15%, en Indonesia yTailandia. Con las devaluacionesy la eliminación de subsidios, lostrabajadores que pierdan suempleo sufrirán las consecuencias

de las graves pérdidas de ingresos y delfuerte aumento de los precios. Con unentorno mundial cada vez másintegrado se mantendrá lasusceptibilidad de los países a las crisis.Se debe prestar mucha más atención alas redes de protección social a fin deevitar que la población pobre y la queestá cercana a la pobreza tenga quesoportar excesivamente el costo queentrañan las conmociones.

Los países en crisis no son losúnicos afectados por el aumento de ladesigualdad. Datos correspondientes a74 países revelan un incrementoextraordinario de la desigualdad en ladécada de 1990: ésta ha aumentado en49 países, y ha disminuido tan sólo endiez. Lo anterior confirma la necesidadde insistir en la inclusión, el desarrollosocial y las redes de protección social ala hora de diseñar y aplicar lasestrategias de reforma y losprogramas de desarrollo.

Centrar la atención en los paísesrecurriendo a las asociacionesde colaboraciónLos aspectos financieros, institucionalesy sociales deben considerarse enconjunto. Para que el crecimientoconduzca al desarrollo sostenible, en lasestrategias de asistencia a los países sedebe dar la debida importancia a losaspectos estructurales, el fortalecimientode la capacidad y la equidad social, y sedeben determinar las posiblesdeficiencias estructurales que podríanhacer desaparecer los avances en materiade desarrollo.

El Banco debe participar enproyectos eficaces; es decir, las

inégalités. Les données relativesà 74 pays révèlent que cephénomène a été très marquédans les années 90. Lesinégalités se sont en effetaggravées dans 49 pays et n’ontdiminué que dans 10. Cela

montre combien il est nécessaire demettre l’accent sur la lutte contrel’exclusion, le développement social etles filets de sécurité lorsque l’onconçoit et met en oeuvre des stratégiesde réforme et des programmes dedéveloppement.

Une aide aux pays fondée surdes partenariatsLes facteurs financiers, institutionnelset sociaux doivent être considéréssimultanément. Pour que la croissancedébouche sur un développement du-rable, les stratégies d’aide aux paysdoivent accorder un poids suffisantaux facteurs structurels, aurenforcement des capacités et à la jus-tice sociale, et identifier les lacunespotentielles d’ordre structurel quirisquent de réduire à néant les acquisdu développement.

La Banque doit opérer dans le cadrede projets efficaces. Cela veut dire que lesopérations doivent être liées au contextesocial, civil et économique dans son en-semble. Pour être plus efficace, laBanque doit opérer en partenariat avecles emprunteurs, les bailleurs de fonds etles autres parties prenantes afin demaximiser l’impact de son action sur ledéveloppement à l’échelon des pays.Pour ce faire, la Banque doit examinerles importants effets secondaires qu’ontdes activités liées entre elles sur lespolitiques et institutions des pays. Il fautégalement que tous les participantsreconnaissent leurs points faibles etpoints forts respectifs et soient en mêmetemps désireux de définir et de partagerles responsabilités en jeu. Une stratégiefondée sur un partenariat est une bonne

xiii

EN

GL

IS

H

ES

PA

NO

L

FR

AN

CA

IScan have on country policies

and institutions. It alsorequires recognition by allparticipants of their relativestrengths and weaknesses,together with a willingness todefine and share account-

ability. A partnership-basedstrategy is good policy from adevelopment perspective, and goodcorporate finance.

Much remains to be done toenhance the quality of countryassistance strategies. Wherecountry assistance evaluationshave been undertaken, OEDestimates that assistance strategieshave been satisfactory only 68percent of the time. Analysisconfirms that project outcomes arehighly dependent on the countrystrategy. For example, no countrythat had a satisfactory countrystrategy demonstrated weak projectperformance.

Project performance has improvedsubstantiallyThe percentage of Bank-supportedprojects with a satisfactoryoutcome at the end of loandisbursement increased from anaverage of 65–70 percent in the1990–96 period to an expected 75percent or higher in 1997–98,including 7 percent withoutstanding outcomes. Thisremarkable improvementdemonstrated Bank and borrowercommitment to improvingdevelopment effectiveness.

There have been majorquality improvements in two ofthe poorest-performing sectors(finance and public sectormanagement) and in Africa,particularly in agriculture. Betterborrower performance, morerealistic project designs, and

operaciones deben estarrelacionadas con las condicionessociales, civiles y económicas gen-erales. Para ampliar laproporción de proyectossatisfactorios, el Banco debetrabajar en asociación con los

prestatarios, los donantes y otras partesinteresadas, concentrándose enmaximizar el impacto en el desarrollo alnivel de los países. Para ello, debe teneren cuenta los importantes efectossecundarios que pueden producir lasactividades interrelacionadas en laspolíticas e instituciones nacionales.También es necesario que todos losparticipantes reconozcan sus fortalezasy debilidades relativas, y expresen sudisposición a definir y asumir su cuotade responsabilidad. Una estrategiabasada en una relación de colaboraciónes acertada desde el punto de vista deldesarrollo y de las finanzasinstitucionales.

Es mucho lo que queda por hacerpara elevar la calidad de las estrategiasde asistencia a los países. El DEO estimaque éstas han sido satisfactoriassolamente en el 68% de los casos. Losanalistas confirman que los resultadosde los proyectos dependen en granmedida de la estrategia aplicada. Porejemplo, en ningún país en que laestrategia de asistencia del Banco ha sidosatisfactoria se han obtenido resultadosdeficientes en los proyectos.

Mejora sustancial de los resultados delos proyectosEl porcentaje de proyectos financiadospor el Banco que registran resultadossatisfactorios al término de losdesembolsos del préstamo aumentó deun promedio de 65% a 70% en elperíodo de 1990–96 a un 75% o másen 1997–98, incluido un 7% deresultados sobresalientes. Esta notablemejora ha demostrado el empeño delBanco y de los prestatarios en

chose du point de vue nonseulement du développement,mais aussi de la situationfinancière de l’institution.

Il reste beaucoup à fairepour améliorer la qualité desstratégies d’aide aux pays. L’OED

estime que celles-ci n’ont étésatisfaisantes que dans 68 % des cas.Une analyse confirme que les résultatsdes projets dépendent largement de lastratégie dont un pays fait l’objet, etlorsque celle-ci est bien conçue, lesprojets exécutés dans le pays en ques-tion ne donnent jamais de mauvaisrésultats.

Nette amélioration des résultatsdes projetsLe pourcentage des projets soutenus parla Banque et donnant des résultatssatisfaisants une fois les décaissementsde prêts effectués est passé d’unemoyenne de 65–70 % durant la périodede 1990 à 1996 à au moins 75 %(chiffre prévisionnel) en 1997–98, 7 %des projets ayant même donnéd’excellents résultats. Ce progrèsremarquable témoigne de la volontéqu’ont la Banque et les emprunteurs derendre plus efficace l’impact de leur ac-tion sur le développement.

Des progrès considérables ont étéconstatés dans deux des secteurs où lesrésultats laissaient le plus à désirer(finances et gestion du secteur public)ainsi qu’en Afrique, en particulier dansle secteur agricole. Ces progrès tiennentà une meilleure performance desemprunteurs, à la conception plusréaliste des projets et à une meilleuregestion du portefeuille. Toutefois, ladurabilité des projets et leur impact surle développement institutionnel sontdeux points qui laissent encorebeaucoup à désirer.

Une perspective mondialeL’Examen de l’an passé concluait que



xiv

EN

GL

IS

H

ES

PA

NO

L

FR

AN

CA

ISbetter portfolio management

explain the improvedoutcomes. But sustain-ability and institutionaldevelopment impact bothremain considerably belowthose levels.

A global perspectiveLast year’s Review concluded that“the challenge is to find the rightfit between country policy andinstitutional factors and strategiesto try to improve conditionsfavorable to improved growth anddevelopment.” In a much morecomplex and hostile environment,this year’s Review reaches asimilar conclusion. It is now evenclearer that improvements inproject performance—importantthough they are—are not enough.

The architecture of the Bank’snew approach to providingdevelopment assistance has beentested by the events of the pastyear. To be sure, adjustments andrefinements in strategy must bemade, and the risks of the externalenvironment must be recognizedand internalized. Nevertheless, theBank’s new strategy formaximizing developmenteffectiveness in a volatile globalenvironment appears wellconceived. The increased emphasison partnership and povertyalleviation stressed in the StrategicCompact, and PresidentWolfensohn’s call to move “beyondprojects” in his 1998 AnnualMeetings speech, are key tomaintaining the performanceimprovements that have beenrealized in the past two years.

ImplicationsThe above diagnosis has thefollowing implications for

aumentar la eficacia en términosde desarrollo.

Se han observadoimportantes mejoras de la calidadde los proyectos en dos de lossectores con resultados menossatisfactorios (el financiero y el de

gestión del sector público), y en África,sobre todo en la agricultura. Esteprogreso se atribuye al mejordesempeño de los prestatarios, al diseñomás realista de los proyectos y a unagestión más acertada de las carteras.Con todo, tanto desde el punto de vistade la sostenibilidad como de su impactoen el desarrollo institucional, losresultados de los proyectos siguensiendo muy inferiores a estos niveles.

Una perspectiva globalEn el examen del año pasado se concluíaque el desafío consiste en encontrar lajusta medida entre la política para el paísy los aspectos institucionales, y lasestrategias para tratar de mejorar lascondiciones propicias para un mayorcrecimiento y desarrollo. Ante unentorno mucho más complejo y hostil,en el examen de este año se llega a unaconclusión similar. Ahora es incluso másevidente que no basta mejorar losresultados de los proyectos, por muyimportante que sea este aspecto.

La arquitectura del nuevo enfoquedel Banco para proporcionar asistenciapara el desarrollo ha sido puesta aprueba por los acontecimientos del añopasado. No cabe duda de que hay queajustar y afinar las estrategias, y que espreciso reconocer e internalizar losriesgos que plantean las condicionesexternas. El mayor énfasis en lasrelaciones de colaboración y en el aliviode la pobreza que se destaca en el PactoEstratégico, y el llamado que haformulado el Presidente Wolfensohn ensu discurso durante las ReunionesAnuales de 1998 en el sentido de ir“más allá de los proyectos”, son

« le problème consiste àparvenir à faire coïncider lapolitique du pays et lesstratégies et facteursinstitutionnels pour tenter decréer des conditions plusfavorables à une croissance plus

forte et à un développement plusefficace ». Dans un environnementbeaucoup plus complexe etdéfavorable, l’Examen de cette annéearrive à une conclusion similaire. Ilest maintenant plus clair encore quel’amélioration des résultats desprojets—si importante qu’elle soit—n’est pas suffisante.

La façon nouvelle dont laBanque envisage d’apporter une aideau développement a été mise àl’épreuve des événements de l’anpassé. Il est certain que la stratégiedoit être ajustée et perfectionnée etque les risques de l’environnementextérieur doivent être reconnus etinternalisés. La place accrue accordéeau partenariat et à la lutte contre lapauvreté, thèmes majeurs du Pactestratégique, et l’appel lancé par leprésident Wolfensohn pour aller« au-delà des projets » dansl’allocution qu’il a prononcée lors del’Assemblée annuelle de 1998,revêtent la plus haute importance sil’on veut que les progrès réalisés cesdeux dernières années soientdurables.

IncidencesLe diagnostic qui précède a les inci-dences suivantes du point de vue dela mesure et de l’évaluation des per-formances :

• Le suivi et l’évaluation desrésultats doivent être plustransparents, la gouvernance et laperformance institutionnellejouant un rôle de premier plan. Ilfaut accorder davantage

x v

EN

GL

IS

H

ES

PA

NO

L

FR

AN

CA

ISfundamentales para continuar las

mejoras del desempeño que sehan conseguido en los últimosdos años.

RepercusionesEl diagnóstico antes expuesto

tiene las siguientes repercusiones para lamedición y la evaluación del desempeño:

• Para hacer el seguimiento y laevaluación del desempeño serequiere mayor transparencia,poniendo en primer plano lafunción de gestión y la actuaciónde las instituciones. Se debeprestar más atención alseguimiento de los indicadoresestructurales, sociales y depobreza.

• La evaluación debe pasar a unplano más elevado y concentrarsea nivel de país, sectorial y global.

• Los sistemas de calificación que seaplican en la evaluaciones debendar más importancia en formaexplícita a las repercusiones socialesde los proyectos y programas, asícomo a los graves efectos que lasconmociones externas pueden tenersobre la población pobre.

Con respecto a las operacionesdel Banco, es preciso:

• Ampliar la proporción de proyectossatisfactorios, teniendo en cuentalos importantes efectos secundariosque pueden tener las actividadesinterrelacionadas en las políticas einstituciones de un país.

• Aumentar el apoyo para eldesarrollo institucional, sobre todode las instituciones financieras y delas redes de protección social.

• Dejar de concentrarse en el proyectoy centrar la atención en el país y enel largo plazo a la hora de diseñar yaplicar las estrategias operacionales.

d’attention au suivi desindicateurs structurels etsociaux ainsi que desindicateurs de pauvreté.• L’évaluation doit s’effectuer àun niveau plus élevé et portersur un pays ou un secteur, ou

bien avoir un caractère global.• Les systèmes de notation des

évaluations doivent accorder unpoids plus explicite à l’impact so-cial des projets et des programmeset aux effets considérables que deschocs extérieurs peuvent avoir surles pauvres.

En ce qui concerne les opérations dela Banque, il faut :

• Consolider les succès, enexaminant les importants effetssecondaires que peuvent avoir desactivités liées entre elles sur lespolitiques et institutions d’un pays.

• Renforcer le soutien audéveloppement institutionnel, enparticulier pour les institutionsfinancières et la protection sociale.

• Passer du stade de projets isolés àune approche à long terme despays au niveau aussi bien de laconception que de l’exécution desstratégies.


performance measurement andevaluation:

• Performance monitoringand assessment need greatertransparency, withgovernance and institutional

performance at center stage.More attention must be paid tomonitoring structural, social,and poverty indicators.

• Evaluation has to move to ahigher plane, focusing on thecountry, sector, and globallevels.

• Evaluation rating systems haveto give more explicit weight tothe social impact of projectsand programs and to theimportant effects that externalshocks can have on the poor.

Bank operations need to:

• Scale-up successes, consideringthe important side effects thatinterrelated activities can haveon country policies andinstitutions.

• Strengthen support forinstitutional development,particularly for financialinstitutions and socialprotection.

• Shift from a project to a long-term country focus in both thedesign and implementation ofoperational strategies.

xvii

ARDE Annual Review of Development EffectivenessARPP Annual Report on Portfolio PerformanceCAE Country Assistance EvaluationCAN Country Assistance NoteCAR Country Assistance ReviewCAS Country Assistance StrategyCBO Community-Based OrganizationCGAP Consultative Group for Assistance to the PoorCSR Civil Service ReformDAC Development Assistance CommitteeDEI Development Effectiveness IndexEAP East Asia and Pacific RegionECA Europe and Central Asia RegionGDP Gross Domestic ProductIBRD International Bank for Reconstruction and DevelopmentICRG International Country Risk GuideID Institutional DevelopmentIDA International Development AssociationIDF Institutional Development FundIDS Institute of Development Studies at Sussex UniversityIMF International Monetary FundLCR Latin America and the Caribbean RegionMNA Middle East and North Africa RegionNGO Nongovernmental OrganizationOECD Organization for Economic Cooperation and DevelopmentOED Operations Evaluation DepartmentOEDCM OED Corporate Evaluation and MethodsOEDCR OED Country Evaluation and Regional Relations GroupPER Public Expenditure ReviewQAG Quality Assurance GroupSAR Staff Appraisal ReportWDI World Development IndicatorsWDR World Development Report

abbreviations and acronyms

1

11DEVELOPMENTEFFECTIVENESS IN AVOLATILE WORLD

and owned by borrowers and other partners. Animportant new lesson is that exogenous factors are farmore influential in determining development impactsthan had been previously thought.

The past year produced a severe deterioration inthe economic conditions of the developing world.Russia’s economic reform collapsed in a year whentransition economies were finally growing after sevenyears of decline.1 The sharp reversal for some of theworld’s fastest-growing economies and the greatercaution of private investors have chilled global eco-nomic growth. The perceived risk of investing inemerging markets made a very large one-year jump.2

The crisis has induced concern about the ability ofthe global system to contain the contagion. Theeconomic turbulence and uncertainty call into questionthe consensus reported in last year’s Annual Review ofDevelopment Effectiveness. Is good macroeconomicpolicy a sufficient foundation for development effec-tiveness? Does the crisis invalidate the broadly based

he current crisis confirms two main lessons for development effectiveness—and provides

a new one. First, good macroeconomic fundamentals are necessary but insufficient for

stable and sustainable growth. In today’s global economy, sound institutions, especiallyTin the financial and social sectors, are essential to economic and social stability. Second, projects

are no longer the appropriate vehicles for development assistance unless they are connected to

balanced country assistance strategies focused on structural reform and capacity development

consensus in the development community about whatconstitutes sound economic policy?

Institutional weaknessesWith the crisis having lasted more than a year, one

fact has become evident: the costs of unregulatedmovements of private capital must be balanced againstthe risks. Indonesia, the Republic of Korea, Malaysia,the Philippines, and Thailand received net privateinflows worth almost 7 percent of their combined GDPin 1995–96. The reversal from 1996 to 1997 involved aswing of 11 percent of their combined GDP.3 Changingrisk perceptions by commercial banks, and particularlyportfolio investors (rather than foreign direct inves-tors), explain the reversal. The sudden downgrading ofcountry credit ratings sparked panic and flight amongprivate investors.

Unlike the debt crisis of the 1980s, the current crisisstarted in countries with relatively strong fiscal situa-tions, sound monetary policies, and outwardly oriented


2

Indicator Indonesia Rep. of Korea Malaysia Philippines Thailand

trade regimes (table 1.1). Except in Thailand, govern-ment budgets were balanced or moving into surpluswhen the crisis hit. Inflation was contained, interestrates were going down, and recorded unemploymentwas low. Taking into account foreign direct invest-ment, current account deficits were not excessive.

Weaknesses in economic management helped trig-ger the crisis. In all cases the capital account was themain vulnerability. Imbalances between short-termdebt and official reserves—combined with prematurefinancial liberalization and weak financial disciplinein domestic banking systems—created situations vul-nerable to speculative pressures.

Many fundamentals were sound in crisis-affectedcountries. The financial panic would not have spreadwithout weaknesses in domestic institutions. As in theSouthern Cone crisis 15 years earlier, banking disci-pline was weak and links among economic conglomer-ates, banks, and governments were too close. This ledto excessive borrowing, disproportionate real estatebooms, poor private investments, and escalating levelsof nonperforming loans. Why did not policymakers andinternational financial institutions give these weak-nesses appropriate weight? Because the lessons of thegeneral debt crisis were guiding them, not the morerelevant institutional lessons of Chile’s 1982 crisis andMexico’s 1994–95 crisis. OED’s (1990) audit report onChile’s structural adjustment loans highlighted the lackof prudential supervision of financial institutions inincreasing the economy’s vulnerability to the point ofcollapse. A key lesson of that audit was: “prudentialrules and surveillance are necessary safeguards for theoperation of domestic financial markets, rather thanunnecessary restrictions” (p. 12).

Microeconomic dysfunctions are more difficult to

spot than macroeconomic weaknesses. Relying as theydid on macroeconomic indicators, decisionmakers inthe private sector and international financial institu-tions found it difficult to argue with success. In EastAsia, as in Chile and Mexico before, credible domesticreforms, low interest rates, and good growth prospectscontributed to an explosion of private flows. Attractingthose flows were exchange rate pegs, profitable interestrate spreads, and liberalizations of the capital account.Given the “halo effect” typical of investment booms,decisionmakers overlooked the failure of Asian coun-tries to comply with basic tenets of the much-abusedWashington consensus, a listing of sound economicpractices on which most analysts agree.

Using the benchmarks of this consensus, Rodrik(1996) notes that the policies of Korea and Taiwan(China) have long been well below par. We extendRodrik’s analysis to Indonesia and Thailand in light ofOED’s recent country assistance evaluations (table1.2). Cumulatively, the analysis confirms that EastAsian countries were following policies consistent withonly 6 or 7 of the 10 tenets of the consensus. Caprio(1998) finds that banking sectors were extremely weakin Indonesia and Thailand. In contrast, according toRodrik, many Latin American economies—in particu-lar, Argentina, Bolivia, and Mexico—fulfilled most ofthe consensus conditions. The conclusion: Washingtonconsensus policies were neither the cause of highgrowth, nor the cause of the crisis.

Macroeconomic policy weaknesses were linked tocompetition policies and financial liberalization—thesequencing of which needed to be made coherent withprior institutional development and structural policyreforms. It is significant that their neglect featuredprominently in the Chile and Mexico crises. In the

Source: Reisen (1998).

TABLE 1.1: TRADITIONAL CRISIS INDICATORS

Government budget deficit (percentage of GDP)Average, 1990–94 0.4 –0.4 –0.7 –1.4 3.2Average, 1995–96 1.7 0.1 0.8 0.4 2.6

Inflation rate (change in the consumer price index)Average, 1990–94 8.8 5.3 4.1 11.1 4.6Average, 1995–96 8.7 4.7 4.4 8.3 5.8

Current account (percentage of GDP)Average, 1990–94 –2.7 –1.5 –7.4 –4.5 –7.5Average, 1995–96 –3.8 –3.4 –9.7 –5.5 –9.1

3

1. Fiscal discipline Yes, partially Yes, generally Yes Yes

2. Redirection of publicexpenditure prioritiestoward health, education,and infrastructure Yes Yes Yes Yes, in the late 1980s

3. Tax reform, includingbroadening the tax baseand cutting marginaltax rates Unclear Yes, generally Yes Yes, generally

4. Unified and competitiveexchange rates Yes, until 1996 Yes, except for brief Yes Yes, until 1991

periods

5. Secure property rights Limited President Park Yes Limitedstarted his rule in1961 by imprisoningleading businessmenand threatening toconfiscate their assets

6. Deregulation Limited Limited Limited Limited

7. Trade liberalization Limited Limited Limited Yesuntil the 1980s until the 1980s until the 1980s

8. Privatization Limited, No. Government No. Government Limited,but not an issue established many established many but not an issue

public enterprises public enterprisesduring 1950s and during 1950s and1960s 1960s

9. Elimination of barriers Yes Foreign direct Foreign direct Limitedinvestment heavily investment subject to foreign directrestricted to government control investment

10. Financial liberalization Yes Limited Limited Yes,until the 1980s until the 1980s in the 1990s

D e v e l o p m e n t E f f e c t i v e n e s s i n a Vo l a t i l e W o r l d

words of Claessens and Glaessner (1997, p. 8), “liber-alization is inexpensive, fast, and easy to implement;building institutional capacity is expensive, slow, andcomplex.” In sum, mistakes in macroeconomic policyplayed a part in the East Asian downturn, and an evenbigger part in Russia, but the more critical dysfunctionwas institutional. Financial sectors, governance stan-dards, and corporate investment regimes (and inRussia, fiscal regimes) seemed adequate as long as thebooms lasted.4 But they proved fatally flawed onceexternal conditions deteriorated. Given the unprec-

edented volume and reversibility of short-term capitalflows, weak banking institutions and ineffective regula-tory systems proved a lethal combination.

The reversal of capital flows was especially deepand disruptive where domestic interest rates werehigher than in the international markets. Financialsupervision, corporate governance, and corruption—letalone social safety nets—were too often given minimalemphasis in the “metrics” of performance monitoringand assessment. As stressed in last year’s Review,institutional development lies at the core of develop-

Source: Data on Korea and Taiwan are from Rodrik (1996); other data, OED, World Bank.

TABLE 1.2: THE WASHINGTON CONSENSUS—NOT IN EAST ASIAElements of theWashington Consensus Indonesia Rep. of Korea Taiwan (China) Thailand


4

ment effectiveness. The “silent crisis” of poverty anddestitution that affects low-income countries is deeplyrooted in capacity constraints. How important is thequality of the institutional environment for Bank-

supported projects in low-in-come countries? Very impor-tant indeed, because strongerinstitutions are associatedwith a 20 percentage pointincrease in the likelihood of aproject’s outcome being ratedsatisfactory (figure 1.1). Forthe Bank’s lending to low-

income countries over the past two years, this improve-ment would translate into a more than $1 billionincrease in effective Bank support.

Broadening the agenda to maximize developmenteffectivenessThe need to scale up from a strictly project-specificfocus to a broader, more inclusive orientation is nowwell recognized. This perspective has been realized byhard-learned lessons—for example, from experiencewith large dams. As OED (1996d) shows, large damswere viewed as being synonymous with modernizationand development in the 1950s and 1960s; but growingevidence of their adverse indirect and secondary im-

pacts turned them into targets of public criticism in the1970s and 1980s. As a result, new policies andstandards emerged that scaled-up the scope of theWorld Bank’s intervention to include avoidance ormitigation of the adverse environmental and socialconsequences of large dams and, by extension, of allprojects with significant potential adverse side effects.To move beyond isolated success stories, the full arrayof factors affecting development results must be exam-ined, and more inclusive, participatory approachesmust be developed.

Such enhanced participation is essential if thecomplementarities across sectors and activities are tobe fully exploited. However, besides developingbroader, more inclusive approaches, the Bank and itspartners must identify “holes in the boat.” Donorscannot simply focus on projects or sectors that theyknow will perform well. They must also identify pointsof stress—such as financial weaknesses—that can causegains to unravel. To achieve development effectivenessat the country level, Bank interventions must begin bydesigning effective projects. But these must link indi-vidual operations to the broader social, civil, eco-nomic, and international environment. They cannotneglect existing weaknesses that could more than offsetthe development effectiveness gains from a project. Tobe successful, the country assistance program must befirmly rooted in the borrower’s ownership of reformobjectives.

Another lesson of the crisis is that social develop-ment should take center stage in the financing ofrecovery or development programs. The shocks ofrecent years have plunged millions into absolutepoverty. The lack of formal social safety nets for theunemployed is being felt severely, partly because ofthe weakening of traditional systems that once sup-ported the poor. Cuts in public spending for the socialsectors and rises in prices associated with devaluationsand the removal of subsidies add to the burden on themiddle class and the poor. If the global objectives forpoverty alleviation of the OECD’s Development Assis-tance Committee (DAC) are to be attained, moreattention must be paid to social development andsocial safety nets.

The new primacy of the external environmentThe central emerging lesson of the crisis is that a riskyexternal environment can strangle development pros-pects. Developing countries face the prospects of

More inclusive,participatoryapproaches

must bedeveloped.

FIGURE 1.1: INSTITUTIONS AND PROJECTPERFORMANCE IN LOW-INCOME COUNTRIES

Note: The institutional quality index is composed of three vari-ables based on data from the International Country Risk Guide(ICRG) covering 1982–98. These variables are corruption, ruleof law, and bureaucratic quality. Countries are assigned incomegroups based on classifications in World Development Indica-tors 1998. The average project performance for the countrygroups are based on project-level outcome data.Source: ICRG; OED, World Bank.

Average project performance (percent satisfactory)

0

10

20

30

40

50

60

70

80

Other countriesLow-income countries

Lowinstitutional

quality

High institutional quality

5

D e v e l o p m e n t E f f e c t i v e n e s s i n a Vo l a t i l e W o r l d

continuing reductions in aid; they also confront amore uncertain environment than the developedcountries do (see Chapter 5). Private investors willgive even greater scrutiny to the country and institu-tional conditions that allow entry to global financialmarkets. That is why development effectivenessrequires a perspective that goes beyond country-levelconcerns, especially for small countries.

Particularly important for the external environ-ment are donor efforts to bolster financial systemsand to help restore the confidence that is needed forthe capital flows vital to restart growth. The WorldBank and the International Monetary Fund (IMF)now lead in the provision of information on regionaland global trends—certain to be an important globalpublic good. The Bank is especially well placed toassist in developing, processing, and understandingthe often important fragments of information withbroader implications.

A series of recent reports on international financialarchitecture laid out a range of actions to strengthenthe international financial system.5 The recommenda-tions call for enhanced transparency and greaterdiscipline in balancing market incentives and publiccontrol. They also stress the need for improved vigi-lance by international organizations. The novelty ofthese reports lies not in the proposals themselves, but intheir urging the practice of what has long beenpreached.

The following chapters review recent evidenceabout the Bank’s development effectiveness. Chapter 2describes project and sector performance trends. Chap-ter 3 considers recent evaluation lessons at the countrylevel. It uses OED’s country assistance evaluations tohelp draw out the lessons of the ongoing crisis. Chapter4 draws lessons that can be inferred from OED’sthematic studies. The final chapter discusses the impli-cations for Bank operations and evaluation.

7

22TRENDS IN PROJECTPERFORMANCE

mong evaluated projects exiting the Bank’s portfolio in fiscal 1997 and 1998, more

than 75 percent had satisfactory project outcomes. Thus the improvement in perfor-

mance presaged by last year’s Review has been sustained. There has also been a conver-

The figure for exits in the first half of fiscal 1998 is 80percent, exceeding the target of the Strategic Compact.2

While this preliminary figure is likely to be biasedupward, it represents asubstantial improvement.We estimate the percent-age of satisfactory projectsfor fiscal 1998 exits to be76 percent, only slightlyabove the level for fiscal1997.3 But it is now clearthat the Bank has moved in the past two years above theplateau in fiscal 1990–96, when the percentage satisfac-tory remained in the 65–70 percent range.

Some qualifications are in order. First, the unusu-ally large risks in the international economy call forsubstantial discounts in the long-term effects likely tobe reaped from many completed projects. Second, morethan 60 percent of the gains were in Africa, which hasaggressively implemented a portfolio restructuring

Amost of the improvement. But sustainability and insti-tutional development remain sorely neglected inproject design and portfolio management.

OED evaluates all closed projects, assessing resultslikely to be achieved in each project’s operationalphase. Since last year’s Review, OED has evaluated298 operations across all regions and sectors.1 Africahas the largest regional share of the evaluated cohort(28 percent), followed by Latin America and theCaribbean and East Asia and the Pacific, with shares ofroughly 20 percent each. Nearly half the projects are inthe finance, private sector, and infrastructure network,led by the transportation sector (30 projects). Agricul-ture projects make up an additional 21 percent.

Outcomes more than 75 percent satisfactoryThe percentage of satisfactory project outcomes hascontinued to improve in recent years. The proportion ofsatisfactory projects for fiscal 1997 exits—the latest yearfor which complete results are available—is 75 percent.

gence in performance across regions and sectors, the result of major advances in Africa and in

two of the poorest-performing sectors (finance and public sector management). Better borrower

performance and more realistic project designs, as well as better portfolio management, explain

There has been asubstantialimprovement in projectoutcomes in the lasttwo years.


8

plan, but remains the region with the highest share ofprojects at risk of not achieving their developmentobjectives.4 But a significant part of the improvementin Africa is in agriculture, which in 1993 developed anaction plan emphasizing simpler models of deliveryand greater sector coherence. So the recent gains maynot be ephemeral.

eyond theassessment ofgoal-oriented

unsatisfactory outcome.The project sought tohelp farmers to invest intheir own small wells inareas where there wasfresh groundwater—directly useful for irriga-tion. Behind the outcomerating were two factors.First, the estimated eco-nomic return fell short ofthe 10 percent minimumthreshold for satisfactoryoutcomes. Second, subsi-dies and free hook-ups tothe power grid resultedin the installation of toomany private electric

Bperformance capturedby the outcome rating,two examples showhow explicit consider-ation of institutionaldevelopment andsustainability canprovide a richerdescription of thedevelopment impact ofBank-financed projects.

In 1995 an irriga-tion project inPakistan was rated ashaving a marginally

wells. But the project wasrated as having substan-tial institutionaldevelopment because itclosed down the loss-making public institutionformerly responsible forfresh groundwater pump-ing, replacing it with amarket-based institution.Sustainability was ratedas likely for two reasons.First, the project useddiesel pumps, which donot depend on the highlysubsidized and unreliableelectric distribution sys-tem. Second, the

water-users group workedwell in coordinating thedecisions of nearly 4,000independent pump own-ers; no depletion of waterresources was expected.

Contrast this with theoutcome of the Rehabilita-tion Project of the port ofDar-es-Salaam in Tanza-nia, rated as marginallysatisfactory. Althoughthere were delays inimplementation and partof the project was not un-dertaken, the port facilitieswere converted to contain-erized operations, the

FIGURE 2.1: DEMANDINGNESS, COMPLEXITY,RISKINESS, AND OUTCOME

Note: By exit fiscal year. Broken lines (exit fiscal year 1998) in-dicate preliminary results, with less than 50 percent coverage ofexited operations.Source: OED, World Bank.

Percent substantial

40

60

80

100

Outcome

Riskiness

Complexity

Demandingness

199819971996199519941993199219911990

A third reason for caution is that these buoyantoutcome ratings are goal-sensitive—relating largely toplanned project goals. Thus their significance is con-nected to the demand, complexity, and risk of projectobjectives. In previous Reviews outcome trends weresteady or only slowly improving as the portfoliobecame increasingly ambitious. But there has been abreak in the upward-sloping curves of demandingness,riskiness, and complexity from fiscal 1996 to fiscal1997–98 (figure 2.1). This shift is admittedly modest.Still, it could indicate that the improvement in outcomeratings was achieved, at least in part, through theachievement of more modest project goals.

Other notable characteristics of the performancetrends include:

• Strong improvements among the previouslypoorest-performing groups, including financeand public sector management.

• Continued improvement in the outcome ratingsof adjustment loans, with performance levelsremaining above those of investment projects.

• Sustained progress in borrower performance,now at par with Bank performance.

• Recent improvements in Bank performance,especially in the quality of project supervision.

BOX 2.1: INSTITUTIONAL DEVELOPMENT AND SUSTAINABILITY HELP CAPTURE PROJECT PERFORMANCE

9

T r e n d s i n P r o j e c t P e r f o r m a n c e

In aggregate, this year’s evaluation results confirmthat improvements in development outcomes have beensustained and broadly based. But in the meantime, theoverall development environment has shifted in dra-matically challenging ways. This shift underscores theneed for more reliable and timely evaluation measures

and even greater use of evaluation findings. Withdemand for lending boosted by the crisis, concessionalresources declining, and Bank loans more costly toborrowers, concern with development effectivenessshould not be allowed to flag.

main project objective. Asa result, ship waitingtimes and berth timesdecreased. The institu-tional developmentcomponent, however, wasrated as negligible be-cause of poorly plannedtraining arrangements foremployees of the Tanza-nian Harbours Authority.The sustainability ofproject benefits was alsodisappointing, and ratedunlikely. Given the sizeand complexity of thefacilities installed, the lackof local basic skills was

considered a critical short-coming. It appearedunlikely that the weakmanagement of theHarbours Authority couldremedy the problem with-out additional externalsupport.

Now consider how thetwo projects would com-pare with one using anaggregate measure ofperformance thatincluded sustainabilityand institutional develop-ment, and one thatconsidered only projectoutcome. Using the

development effective-ness index, the project inPakistan scores a 6.75,well above the Bankaverage of 6.38 for fiscal1994–97. The project inTanzania receives a5.25. An analysis basedsolely on a binary classi-fication of outcome doesnot take into account thecontrasting performanceof sustainability and in-stitutional development.The descriptions of thetwo projects indicate thatthe first had better over-all performance than the

second, and that is theinformation the devel-opment effectivenessindex conveys. A crudebinary analysis basedstrictly on goal-basedevaluation (outcome)would have reversedthe ranking in favor ofthe project in Tanza-nia. These issues arediscussed more fully inAnnex 1.

FIGURE 2.2: DEVELOPMENT EFFECTIVENESS INDEX AND OUTCOME: MEASURING PROJECT PERFORMANCE

Note: By exit fiscal year. Broken lines (exit fiscal year 1998) indicate preliminary results, with less than 50 percent coverage of exitedoperations.Source: OED, World Bank.

Development effectiveness index Outcome (percent satisfactory)

5

6

7

8

9

By disbursements

By projects

199819971996199519941993199250

60

70

80

90

By disbursements

By projects

1998199719961995199419931992


1 0

FIGURE 2.3: SATISFACTORY OUTCOME BY REGION


40

60

80

100

1998199719961995199419911990

Percent satisfactory

40

60

80

100

19981997199519941993199219911990


East Asia and Pacific (EAP)

Middle East and North Africa (MENA)

South Asia (SA)

19931992

Africa (AFR)

Europe and Central Asia (ECA)

Latin America and Caribbean (LAC)

1996



40

40

60

60

80

80

100

100

Bank

East Asia and Pacific

1998

1998

1997

1997

1996

1996

1995

1995

1994

1994

1993

1993

1992

1992

1991

1991

1990

1990



40

40

60

60

80

80

100

100

Bank

Africa

1998

1998

1997

1997

1996

1996

1995

1995

1994

1994

1993

1993

1992

1992

1991

1991

1990

1990

Bank

Europe and Central Asia

Bank

Middle East andNorth Africa

Bank

Latin America and Caribbean

Bank

South Asia

1 1


Gains in development effectivenessAs part of its search for a more comprehensive measureof project performance than the outcome rating, OEDhas piloted the development effectiveness index for thisReview. It integrates current OED measures of out-comes, sustainability, and institutional developmentimpact. The measure ranges from 2 (for a project witha highly unsatisfactory outcome, which also hasunlikely sustainability and negligible institutional de-velopment impact), to 10 (where high achievements onall three measures are realized).

The index improves the presentation of performancetrends in three ways (box 2.1). First, it uses the spectrumof outcome assessments made by OED rather than thebinary assignment to satisfactory or unsatisfactory out-come. Second, it qualifies a project’s outcome judgmentby rewarding the robustness of achievements into thefuture, in some cases recognizing the lasting benefits ofsignificant achievements that fall short of expectations.Third, institutional development impact is given specialemphasis. Together these aspects of the index provide amore complete picture of trends. The new index facilitatesperformance analysis of sectors and countries. Additional

analysis is under way to confirm the robustness of thedevelopment effectiveness index, as well as its consistencyand complementarity to measures used by the Networksand the Quality Assur-ance Group. Annex 1summarizes the index’sconstruction.

While also showingimprovement over thepast two years, the devel-opment effectiveness in-dex trend is less dramaticthan the trend in out-comes (figure 2.2). Outcome ratings gains have not beenmatched by gains in project sustainability or institutionaldevelopment.

Weighted by disbursements, the data show a similarpattern of improvement, with the share of outcomes ratedsatisfactorily reaching 78 percent in fiscal 1997.5 Forfiscal 1998 exits, the preliminary satisfactory outcomeachievement rate is 84 percent, although using thedevelopment effectiveness index, the fiscal 1998 figureshows no gain over fiscal 1997, holding steady at 7.02.

BOX 2.2: IMPROVEMENT IN AFRICA

erformancegains in Africahave been driven

improvement is the suc-cess of capacity-buildingwork. There has been a16 percentage pointimprovement in the num-ber of projects havingsubstantial institutionaldevelopment impact inAfrica. For example, theNational AgriculturalServices project in Côted’Ivoire strengthened themonitoring, evaluation,and cooperative supportactivities of the Ministryof Agriculture. The Tech-nical Assistance projectin Mozambique sup-ported the country’stransition from a cen-

trally planned to a mar-ket-oriented economy,and illustrates well theimprovement in Africanpublic sector manage-ment projects. The paceof institutional develop-ment in Mozambiquewas rapid, with keyeconomic institutionssuch as the Ministry ofFinance and the centralbank significantlystrengthened, whileprogress was steady inimproving the quality offinancial reporting inboth the public andprivate sectors.

Again in Mozambique,

Pby improvements inagriculture, publicsector management, andfinance. Of the projectsexiting in fiscal1997–98, agriculturedominates the Africaportfolio (26 percentshare). Satisfactoryoutcome ratings foragriculture projectsincreased from 54percent in fiscal1993–96 to 76 percentin fiscal 1997–98.

One explanation ofthis extraordinary

a financial sectorproject more thanachieved its objectivesthrough better fiscaland monetary manage-ment and privatizationof the banking andindustrial sector. Thisresult contributed to asharp drop in inflation,and increased privateinvestment and highereconomic growth.Expansion of socialspending also resultedin a noticeableimprovement in healthand education.

Gains in outcomeratings have not beenmatched by gains insustainability orinstitutionaldevelopment.


1 2

But regional analysis shows the disbursement-weightedindex to have fallen in most regions, with the largest dropin East Asia and the Pacific (1 point, for 27 percent ofBankwide disbursements). This was offset by a largeincrease in Latin America and the Caribbean (1.15 points,for 23 percent of Bankwide disbursements). The drop inEast Asia reflects movement in China and Indonesiaespecially, which together account for roughly 80 percentof the region’s disbursements.

All regions except East Asia and the Pacific showimprovement in fiscal 1997–98 compared with long-term averages for fiscal 1990–96 (figure 2.3). Improve-ments in project outcomes were largest in Africa (box2.2) and Latin America and the Caribbean, with 14 and15 percentage point gains in the share of satisfactoryoutcomes, respectively. These improvements—particu-larly in Africa, where performance has historicallylagged behind other regions—have raised the globalaverage and reduced regional disparities (figure 2.4).6

Improvements in project performance were notice-able in most sectors (figure 2.5). The most notabledecline, among sectors with significant numbers ofrecently evaluated projects, was in industry. In thatsector there was a fall from 54 percent satisfactoryperformance in fiscal 1990–96 to a dismal 36 percentsatisfactory in fiscal 1997–98, as well as a 7 percent

FIGURE 2.4: AFRICA REGION—RELATIVE PERFORMANCETREND, OPERATIONS WITH SATISFACTORY OUTCOME


FIGURE 2.5: SATISFACTORY OUTCOME BY SECTORAND EXIT FISCAL YEAR GROUP

Note: Only sectors with at least 10 exiting projects in fiscal1997–98 are included.Source: OED, World Bank.

Percent

40

60

80

100

Bank

Other regions

Africa

Africa (AFR)

199819971996199519941993199219911990

By projects (percent satisfactory)

1997–981990–96

Social sector

Education

Multisector

Transportation

Electric power& other energy

Urbandevelopment

Population,health & nutrition

Agriculture

Water supply& sanitation

Finance

Public sectormanagement

Industry

0 20 40 60 80 100

By disbursements (percent satisfactory)

Social sector

Education

Multisector

Transportation

Electric power& other energy

Urbandevelopment

Population,health & nutrition

Agriculture

Water supply& sanitation

Finance

Public sectormanagement

Industry

0 20 40 60 80 100

1 3

drop in the average development effectiveness index.This represents the continuation of a downward trendin a sector with declining emphasis in the Bank.

The fastest-improving sectors were concentrated inthe previously poorest-performing groups. Public sectormanagement, finance, and industry were the worst-performing sectors in fiscal 1990–96 (by projects), withdevelopment effectiveness and project outcomes wellbelow the Bank average. Public sector managementand finance now show more than a 10 percentage pointincrease in the share of projects with satisfactoryoutcomes and a significant increase in the averagedevelopment effectiveness index in fiscal 1997–98.Significant improvements (by projects) were also evi-dent in agriculture; urban; transportation; and popula-tion, health, and nutrition.

Care should be taken in interpreting these sectorchanges as potentially lasting shifts in performancetrends. A comparison with performance assessments ofthe active portfolio currently under supervision pro-vides a complementary gauge.7 This provides modestsupport for the view that sector improvements arelikely to be sustained. For example, active agricultureprojects have improved to above-average performancein fiscal 1998, with lending scheduled to increase in thenear future as the Rural Action Plan is implemented.The assessments indicate declining performance for theshrinking number of industry projects. The current


portfolio data do not support the probability of sus-tained gains in the finance sector.

Adjustment loans continue to have higher averageoutcomes and sustainability than investment loans,although the gap has narrowed. The share of satisfac-tory outcomes for adjustment loans rose from 74percent in fiscal 1990–96 to 82 percent in fiscal 1997–98, compared with a risefrom 66 to 76 percent forinvestment loans. Institu-tional development perfor-mance has evened toroughly 38 percent sub-stantial for both types ofloans.

Last year’s perfor-mance results suggestedonly minor progress inIDA and blend-financed projects. This year’s resultsdisplay much stronger improvement—with IDA andblend projects among the fiscal 1997–98 exits perform-ing at par with IBRD-financed projects in outcome,sustainability, and institutional development impact.Considering outcomes alone, the share of satisfactoryprojects among IDA loans and blends was actuallyabove that for IBRD-financed projects: 79 percent werejudged satisfactory in fiscal 1997–98 compared with74 percent for IBRD loans.

FIGURE 2.6: INSTITUTIONAL DEVELOPMENT IMPACT


By projects (percent)

0 0

10 10

20 20

30 30

40 40

50 50

60 60

Negligible

Modest

Substantial

199819971996199519941993199219911990

By disbursements (percent)

Negligible

Modest

Substantial

199819971996199519941993199219911990

Improvements inproject performancewere especiallynoticeable inpreviously poorlyperforming groups.


1 4

Institutional development impact—improvingbut still weakThe first two objectives of a five-point developmentframework offered by President Wolfensohn in hisaddress at the 1998 Annual Meetings relate to institu-tional development.8 This is particularly important inlow-income countries, where the potential gains are thegreatest and, as Chapter 3 illustrates, the quality ofinstitutions the lowest.

Recent evaluations show that the institutional devel-opment impacts of Bank projects are improving, but thereis enormous scope for further improvement (figure 2.6).9

Current exits show historical highs of only 40 percent ofoperations with substantial institutional development.Between fiscal 1991 and 1995 the share of projects withnegligible institutional development rose from roughly 20percent to nearly 30 percent. That share dropped to thelowest on record, of just 15 percent, in fiscal 1997. Bydisbursements, however, the historical trend of largerprojects having greater institutional development impactsshows signs of weakening. The share of disbursementswith substantial impact was below the project-weightedaverage in two of the past three years. In Chapters 3 and 4we show the fundamental importance of institutionaldevelopment in development effectiveness. The mainfinding: the spillover effects from better monitoring andcloser attention to institutional development have beenneglected, and need much greater emphasis in Bankoperations.

Sustainability—low and weakening?The fiscal 1997–98 data send a mixed signal onsustainability10 that may well be a precursor of futuredeclines in performance attributable to a deterioratingexternal climate (figure 2.7). The proportion of projectsjudged as having likely sustainability that exited infiscal 1997 maintained an upward trend, increasing to54 percent from 46 percent in fiscal 1990–96. But theseprojects closed at the latest in June 1997, well beforethe East Asian crisis. The partial results for fiscal 1998

FIGURE 2.7: SUSTAINABILITY


FIGURE 2.8: BORROWER PERFORMANCE



Unlikely

Likely

Uncertain

By disbursements (percent)

00

2020

4040

6060

8080

Unlikely

Likely

Uncertain

199819971996199519941993199219911990 199819971996199519941993199219911990


40

60

80

100

Average

ComplianceImplementation

Preparation

199819971996199519941993199219911990

1 5


Country Project name Loan/credit number

TABLE 2.1: RECENTLY EVALUATED PROJECTS: OUTSTANDING PERFORMERS AND POOR PERFORMERS

Note: Covers the 298 projects evaluated since the last ARDE.Source: OED, World Bank.

Outstanding performersBenin Urban Rehabilitation and Management C2338Brazil Land Management I - Parana L3018Brazil Parana Municipal Development L3100Chile Second Public Sector Management L3411China Ertan Hydroelectric L3387China Integrated Regional Health Development C2009China National Afforestation C2145China Northern Irrigation - Part A C1885China Ship Waste Disposal C2391Dominican Republic Primary Education Development L3351Estonia Highway Maintenance L3731Hungary Human Resources Development L3313Lao PDR Second Telecommunications C2101Latvia Agricultural Development L3695Mexico Contractual Savings Development Program L4123Morocco Telecommunications Sector L3557Mozambique Second Economic Recovery C2628Peru Debt and Debt Service Reduction L4133Tunisia Population and Family Health L3307Vietnam Structural Adjustment C2657Poor performersEcuador Second Water Supply L2774Morocco Rural Primary Education L3026Papua New Guinea Land Mobilization L3051Rwanda Sectoral and Pre-Investment Studies C1796

In fiscal 1997–98 borrower performance improvedto 75 percent satisfactory from 66 percent satisfactoryin fiscal 1990–96 (figure 2.8).12 This increase ispronounced in IDA countries, particularly in Africaand Latin America and the Caribbean. Borrower inputshave improved dramatically in finance and publicsector management projects—rising from satisfactoryperformance in 57 percent of operations exiting infiscal 1990–96 to 84 percent exiting in fiscal 1997–98.These sectors enjoyed large gains in outcomes over thesame period, confirming the importance of borrowerinputs in project performance. Improvements of morethan 10 percentage points were found in urban andtransport operations.

The increase in borrower performance reflectsimprovements in project preparation and in compli-ance with legal covenants. Implementation perfor-mance remains the poorest-performing dimension, with

projects, exiting in the unfolding of the Asian crisis,show a decline to 50 percent of projects judged to havelikely sustainability. This decrease is largely because ofthe drop in East Asia and the Pacific projects—thosemost directly affected by the crisis—from 66 percentlikely sustainability in fiscal 1997 to 43 percent infiscal 1998. Similarly, there has been an almostdoubling of the share of active projects in the Region atrisk of not achieving their development objectives.

Bank and borrower performance improving—butgreater gains from borrowersAnalysis in past Reviews on the determinants ofsuccessful project outcomes found project-specific bor-rower performance to be the most important determi-nant of project success. Bank performance and thecountry macroeconomic policy environment werefound to be less significant.11


1 6

an unsatisfactory assessment in more than a third ofevaluated projects. The overall increase in perfor-mance puts the overall quality of borrower inputs atpar with Bank inputs. Three of every four evaluatedprojects now show satisfactory Bank and borrowerinputs.

Bank performance, by contrast, improved modestlyfor fiscal 1997–98 exits, with only a 1 percent increase inthe overall average over the 75 percent satisfactoryaverage for fiscal 1990–96 (figure 2.9).13 Lower-qualityproject identification has been offset by better supervision,while project appraisal improved slightly.14 These results,however, conceal some areas of particular improvement.Latin America and the Caribbean showed improvement

in both appraisal and su-pervision, raising theoverall Bank perfor-mance average to 80 per-cent. Conversely, bor-rower performance inEast Asia and the Pacific

projects dropped in all three dimensions in fiscal 1997–98, with appraisal performance showing the largestdecrease. Finance and public sector management andpopulation, health, and nutrition projects show improve-ments in Bank performance.

Outstanding projects—7 percent of the totalOf the 298 operations evaluated in the past year, OEDassessed 20 (7 percent) as outstanding and 4 (1 percent) asexceptionally poor (table 2.1). The 20 outstanding opera-tions all had highly satisfactory outcomes, likelysustainability, and substantial institutional developmentimpact. These projects met or exceeded their main goals,had highly innovative designs, and are replicable in othercountries or sectors. They featured strong borrowerownership, enjoyed highly satisfactory or satisfactoryquality at entry, and were well supervised.

The success of the outstanding projects can betraced to flexibility in responding to changing condi-tions—the result of consistent monitoring, good super-vision, and partnership building. Even projects withcomplex designs succeeded because of extensive andeffective Bank staff involvement and judicious techni-cal assistance (box 2.3). In contrast, only one poorperformer had satisfactory quality at entry. The char-acteristic features of poor performers are a general lackof supervision and low borrower ownership.

Sector analysisFor a successful development strategy, assessing risk (asmeasured by the variability of rewards) is as critical asassessing the expected development impact. Risks andrewards in the 12 sectors with more than 10 projectsexiting in fiscal 1997–98 are shown in figure 2.10. Theorigin of the axes corresponds to a reward equal to theaverage development effectiveness index in the 1997–98portfolio, and risk is equal to the standard deviation of theindex in that portfolio. The axes measure differences forspecific sectors relative to the Bankwide average and thestandard deviation of the development effectivenessindex. Thus each quadrant, starting from the upper-leftand moving clockwise, corresponds to one of the fourclasses: high risk–low reward, high risk–high reward,low risk–high reward, and low risk–low reward. Thecoordinates corresponding to each sector measure thesector’s risk-reward combination relative to the averageproject.

Figure 2.10 allows a comparison of averageproject results for each sector and the dispersion ofthese results around the corresponding Bankwide val-ues. However, these comparisons are not meant to ranksectors. Instead they suggest that the relative risk-reward framework may be useful for the Bank Net-works in considering potential weaknesses andstrengths of Bank interventions in specific sectors, andexternal challenges that the sectors may face. Take the

FIGURE 2.9: BANK PERFORMANCE



Average

Supervision

Appraisal

Identification

40

60

80

100

199819971996199519941993199219911990

In fiscal 1997–98borrower performance

improved to 75percent satisfactory.

1 7


industry sector, where the average development effec-tiveness index is far below the Bankwide average, andvolatility is much higher. This raises questions. Doesthe Bank have comparative advantages in supportingindustrial projects relative to the private sector? Or canthe below-par results for this sector be attributed to aconcentration of these projects in countries lackinggood infrastructure? How much of the volatility inindustry projects is the result of poor selectivity by theBank? How much can be explained by exogenousshocks (such as contagion effects in regional crises),which are likely to affect industry projects more thanless market-oriented interventions?

Similarly, the relatively low rewards of water andsanitation projects—and the relatively high confidencethat performance in this sector will be weak—raiseseveral questions. Why do this sector’s results stand insuch contrast to the rest of the Bank’s portfolio? Domeasurement problems drive the results? Or is a sectorwith such important potential effects on health andpoverty really a relatively weak performer? We do nottry to answer these questions here. The point is topresent a framework that the Bank Networks can use tocompare and contrast their sector performance with theresults in other sectors. The framework itself should bescrutinized and tested for robustness over time.

FIGURE 2.10: RELATIVE RISK AND REWARD BY SECTOR, EXIT FY97–FY98

Note: Only sectors with at least 10 exiting projects in fiscal years 1997–98 are included. While the development effectiveness indexhas desirable mathematical properties for this kind of analysis, a similar performance distribution occurs when measurement relies onproject outcome measures.Source: OED, World Bank.

Reward

-0.5

-0.4

-0.3

-0.2

-0.1

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

-1.5 -1.2 -0.9 -0.6 -0.3 0.3

Industry

Electric power and other energy sources

Water supply and sanitation

Public sector management

Population, health, and nutrition

Social sector

Urban development

Multisector

Agriculture

Education

TransportationFinance

0.5

RiskHigh risk–Low reward

High risk–High reward

Low risk–Low reward

Low risk–High reward


1 8

he importanceof borrowerownership,

legislature. These objec-tives were well exceededbecause of the dedicated,competent action of theborrower, supported by anexcellent Bank team.Capacity improvementsand the beginning ofcultural change wereachieved in several keypolicymaking agenciesand in the Library ofCongress—above-averageachievements for afreestanding technicalassistance loan.

The crucial role ofborrower commitmentand strong Bank leader-

ship is similarly illus-trated by Latvia’s Agri-cultural DevelopmentProject—the first Bank-financed investmentproject in Latvia. Itsoverall developmentobjectives were toenhance the privatizationof agriculture, agropro-cessing, and forestindustries through finan-cial and technicalassistance. Well-focusedobjectives, a simpledesign, and such innova-tive features as themobile credit officers ofthe new Agricultural

Finance Companywere importantcontributors to projectsuccess. Firmly under-pinning thesesuccesses was thegovernment’s commit-ment to the project,which was stronglyinfluenced by the ef-fectiveness and staff-ing continuity of theproject managementunit and by Banksupervision.

TBOX 2.3: TWO OUTSTANDING PROJECTS

commitment of Bankstaff, and good supervi-sion is illustrated byChile’s Public SectorManagement Loan.The loan’s main objec-tives were to increasethe efficiency and effec-tiveness of key publicmanagement agencies,remedy the govern-ment’s inadequateeconomic analysis andcoordination capabili-ties, and improvepolicymaking in the

1 9

33DEVELOPMENTEFFECTIVENESS ATTHE COUNTRY LEVEL

Country evaluations confirm that weak institutional development has been a key prob-

lem in improving development effectiveness. Risk-bearing institutions—particularly the

financial system and social safety nets—have been neglected by the development

process. In many low-income countries, channeling aid through isolated, uncoordinated enclave

projects has left capacity inadequate. Where the enabling environment is weak, projects should be

justified largely for their policy reform and capacity development impacts, with the attendant risks

reduced through judicious testing of borrower owner-ship. Country evaluations show that institution build-ing is needed to ensure that a country’s outwardorientation can safely reap the benefits of globaliza-tion—and shield the poor from its shocks.

In 1995 OED inaugurated country assistanceevaluations to assist the Board deliberations on countryassistance strategies. By now,1 17 such country assis-tance evaluations (CAEs) have been produced. Theyassess the relevance, efficacy, efficiency, sustainability,and institutional development of assistance strategies.Using the insights of independent professionals, theyare case studies of aid effectiveness in the tradition ofCassen and associates (1994) and Mosley, Harrigan,and Toye (1991). They identify lessons of experienceand draw the implications for future strategies. Morethan 90 percent of operational staff preparing countryassistance strategies found them helpful. The followingfindings emerge from an overview of CAEs:

• When graded as projects have traditionally beenevaluated by OED, CAEs rate the overall out-come of the Bank’s country strategy as satisfac-tory 68 percent of the time.2

• The rating of the Bank’s country strategy from theCAEs is a relatively strong predictor of the averageperformance of Bank-supported projects.

• The most important development issues identi-fied by CAEs are consistent with those that wouldbe inferred from other empirical analyses ofgrowth and poverty alleviation.3

The country as the unit of accountFor this Review, OED undertook a pilot analysis of thelinkages between country strategy and project perfor-mance. As a first step, all completed CAEs weresubjected to a formal rating process consistent instructure with OED project ratings, which are based onevaluative conclusions about the design, outcome, and


2 0

impact of the Bank’s assistance strategy in a country.4

OED staff preparing CAEs were asked to summarizetheir views on the outcome, sustainability, and institu-tional development of Bank country strategies as theywould on projects (table 3.1).

Project performance is strongly correlated with thequality of the country assistance strategy. None of thecountries that had a satisfactory country strategyexperienced weak project performance. In only 2 of 25periods rated did an unsatisfactory Bank countrystrategy result in satisfactory performance on projects.

CAEs’ judgments about key strategic issues aresimilar to those that would be suggested by empiricalstudies of growth and poverty.5 However, the relativelylow aggregate outcome measure, 68 percent satisfac-tory, suggests room for considerable improvement in

Bank country assistance strategies. In what follows weconsider some of the common lessons from CAEs.

Institutions, aid, and growthDevelopment assistance has achieved much (table 3.2).For low-income countries the rate of improvement onmost measures of deprivation is considerably betterthan that for high-income countries. But improvementsin growth have been less propitious (Ingram 1992). Forexample, the weighted average per capita growth ratefor low-income countries for 1980–96 (outside Chinaand India) has been negative. So low-income countriesdo not—as economic convergence models predict—catch up with high-income economies. Instead they fallfarther behind.

Poor policies have a lot to do with disappointing

TABLE 3.1: PERFORMANCE OF BANK-SUPPORTED COUNTRY STRATEGIES AND PROJECTS

Note: Average Bank-supported project outcomes over 50 percent are categorized as “satisfactory.” Country strategy ratings of 4 andabove are rated “satisfactory.” Other approaches to categorizing relatively strong and weak performance, using central tendenciessuch as mean and median, produce similar results.Source: OED, World Bank.

Côte d’Ivoire (1980–86)Morocco (1989–94)

Average project outcome performance

Albania (1992–97)Argentina (1991–96)Bangladesh (1980–96)Bolivia (1986–97)Côte d’Ivoire (1960–79)Côte d’Ivoire (1994–98)Ghana (1982–96)Jamaica (1987–96)Malawi (1995–97)Morocco (1983–88)Mozambique (1984–96)Philippines (1986–97)Poland (1986–96)Thailand (1987–96)Zambia (1994–96)

Argentina (1985–90)Côte d’Ivoire (1987–93)Jamaica (1980–86)Kenya (1990s)Malawi (1990–94)Togo (1985–90)Togo (1991–96)Zambia (1980–93)

Satisfactory Unsatisfactory

Uns

atis

fact

ory

Sati

sfac

tory

Cou

ntry

ass

ista

nce

stra

tegy

rat

ing

2 1

D e v e l o p m e n t E f f e c t i v e n e s s a t t h e C o u n t r y L e v e l

income growth. Yet even the low-income African coun-tries described as sustained policy reformers (IMF 1995)have had average growth of only 0.5 percent a year. Thegrowth rate among this group is only slightly higher thanthat realized by Europe over the 400-year pre-capitalistperiod before 1820 (Maddison 1997). So something otherthan weak macroeconomic policies is impeding growth.

There is no simple explanation for such poorperformance. But one factor, which recurs in OED’scountry evaluations, is the weakness of institutions. Forinstance, only 30 percent of low-income countries

enjoy an institutional environment rated as marginallysatisfactory—less than half the level for middle-incomecountries (box 3.1). Only 1 of 41 low-income countriesscores a satisfactory rating on institutional quality,while more than 30 percent of middle-income countriesdo.6 By contrast, OECD countries (except Korea) boastsatisfactory institutional quality ratings. Among low-income countries, a low institutional rating is morecommon than a weak policy environment: 40 percentof low-income countries and 77 percent of middle-income countries have good policy regimes.7

ince the 1980s anew developmentperspective has

ensuring the sustain-ability of developmentprograms. Yet the linksbetween developmentpractice and academicsare not strong. In addi-tion, the evaluationprofession has been slowto adapt its methods andprocesses to the newdevelopment consensus.

The papers presentedat a recent OED confer-ence (1998) illustrate thatinstitutions matter. Thepapers explore not onlyhow to get the institu-tions right but also howto assess the fit betweeninstitutions and develop-

ment challenges throughevaluative techniques.The papers show that:

• Variable combina-tions of competition,cooperation, andhierarchy are neededto achieve positivesocietal outcomes inspecific countrycircumstances.

• Getting the incen-tives right is crucialto overcoming therestrictions thatarise from the neo-classical model.

The papers argue that

S if institutional analysisis to become opera-tional, it will have toprovide greater clarityin the area of incen-tives. Institutions mat-ter because incentivestrigger motivation andaction in both thepublic and privatesectors. Incentives arethus the first buildingblock for policy designand implementation,and the evaluation ofresults. The difficultylies in aligning theincentives structurewith the collectiveinterest.

emerged. It holds thatinstitutions and eco-nomic organizationsare the key determi-nants of economic,social, and politicalprogress. Six Nobelprizes have beenawarded to scholarswho made pioneeringcontributions toneo-institutional eco-nomics. In parallel,development evalua-tors have establishedthe crucial role ofcapacity building in

Source: Picciotto and Wiesner (1998).

TABLE 3.2: TRENDS IN SOCIOECONOMIC INDICATORS AND INCOME, 1970–96 (PERCENTAGE CHANGE)

High-income 10 72 9 n.a. 66

Low-income 33 40 17 260 44

Low-income excluding

China and India 37 37 21 55 –4

Crude Infant Access todeath rate mortality rate Life safe water GNP per capita(per 1,000 (per 1,000 expectancy (% of (1987

Country group people) live births) (years at birth) population) constant $US)

Source: World Development Indicators 1998, World Bank.

BOX 3.1: INSTITUTIONS AND DEVELOPING ECONOMIES


2 2

Policies and institutions are weaker than they mightotherwise be because of inadequate donor coordination.Zambia’s country assistance strategy provides a starkexample. In 1987 the Bank concluded that Zambia’spolicy regime was not appropriate for the Bank to provide

further lending. The Bankmaintained this position un-til 1991, when it resumedlending. In the 12 yearsbefore the Bank’s with-drawal of assistance, aidaveraged more than 9 per-cent of GDP a year. During

the hiatus in Bank lending, donor assistance as a share ofGDP rose to more than 15 percent of GDP a year. Soduring a period when the Bank found the policy environ-ment such that aid was unlikely to promote developmenteffectiveness, donor assistance increased—and exceededtotal investment. This pattern of donor support wasmotivated by donors’ humanitarian concern with theproblems of a very poor country. But it did little toincrease borrower ownership, to strengthen institutions,or to reduce poverty.

CAEs of low-income countries—Albania, Ghana,

Malawi, Mozambique, and Zambia—as well as recentwork by African policymakers on the African CapacityBuilding Study (World Bank 1996a), argue that acentral problem has been lack of capacity to absorb thevolumes of aid provided. Rather than directly address-ing this, however, many development agencies, includ-ing the Bank, have established parallel methods tochannel financial assistance, ignoring the adverseeffects on capacity creation. This finding is not new,but it bears repeating. For example, Johnston and Vande Walle (1996, p. 66) argue that “aid has rarelycontributed to effective institution building as it hasbypassed local institutions in project implementationand design. The preference for enclave projects andparallel management structures to ministerial adminis-trations has been particularly destructive.”

A Danish government report on developmentcooperation issues in Tanzania (Helleiner and others1995) reaches similar conclusions about how donorpractices often undermine ownership. For example, inprimary education, the report finds that agenciesfrequently manipulate the choice of government de-partments they work with in order to achieve theirobjectives. The report contends that where the govern-

Can a countrywith a weakpolicy envi-

use the Burnside and Dol-lar (1997) measure ofmacropolicy, based onthe financial deficit,inflation, and opennessfor 1975–96. We consid-ered 43 countries forwhich data were avail-able. The Burnside andDollar policy index wasconstructed annually,and the changes in itwere used to categorizecountries:

• Durable adjusters arethose that main-tained a good indexfor at least nineyears. Not yet

durable adjusters arethose that haveadjusted for at leastthe past four yearsbut have not main-tained adjustmentlong enough (nineyears) to be classifiedas durable adjusters.

• Oscillators are thosethat do not adjust butcontinue to oscillatebetween weak andstrong policy envi-ronments. Theirpolicy index remainsvolatile over time.

The sample has 12durable adjusters, 20 not

yet durable adjusters,and 11 oscillators.Durable adjusters arefurther divided into coun-tries that adjusted andmaintained it at the firstattempt, and countrieswhere the policy indexoscillated between goodand bad before theadjustment to a goodpolicy environment wassustained. Eleven of the12 countries successful atmaintaining a goodpolicy index, accordingto the Burnside andDollar measures, did soon the first attempt.

The oscillators have

BOX 3.2: THE LONG-RUN EXPECTED BENEFITS OF ADJUSTMENT

ronment adjust? Canit shift from a bad to agood regime andsustain it over adecade or more? Or,as a number of ana-lysts have observed,do countries thatattempt to adjust notmake it—and adjustover and over again?If durable adjustmentis possible, do adjustercountries grow morerapidly?

To consider some ofthese questions, we

Lack of capacity toabsorb the aid

provided has been aproblem.

2 3

had a weak policy index,on average, for about 13years and for more than60 percent of the periodfor which data were avail-able. They also performbadly on Bank adjustmentloans—with only 49percent satisfactoryoutcomes. The averagenumber of oscillationsbetween strong and weakpolicy regimes for thisgroup is four. If we add upthe times all three catego-ries of countries haveattempted to adjust, theprobability for a countryto adjust durably is about12 percent. Of course, this

measure depends funda-mentally (and arbitrarily)on the period chosen toqualify as a successfuladjuster. It also dependson the quality of thecountry’s institutional en-vironment. For example,in countries that havesatisfactory institutionalenvironments, the prob-ability of successfullyadjusting increases to 30percent.

The threshold of nineyears is arbitrary—andconservative. If wereduce it to seven years,the odds of successincrease to 16 percent—

one in six. And as mea-sured by OED, theoutcome of adjustmentloans in not yet durableadjusters would be strongand very similar (ifslightly higher, 85 per-cent satisfactory) to thatof durable adjusters.

But regardless of thethreshold, it is clear thatthe probability of successis not high. So, for adjust-ment to have a lasting,high payoff, the gainsfrom adjustment must besubstantial. Are they?Does this turnaround togood policy make adifference? The payoff

associated with success-ful adjustment can beillustrated by the differ-ences in averagegrowth, per capitaincome, inflation, andvolatility of these vari-ables in the differentcountry classes. Insuccessful adjusters, percapita GNP grew atalmost three times therate of countries thathave not yet achieveddurable adjustment,and six times fasterthan oscillators. Theadjusters increased theirgrowth rates more thansixfold.

ment is reluctant to agree to a donor’s project, therehave been implicit threats of a reduction in generaldonor support. The report suggests that it is commonpractice for donors to pay “incentives” to governmentofficials working on their projects.

Berg (1993) points out that, in such situations,“technical cooperation takes on a role different than itstraditional one: it substitutes for and subsidizes govern-ment operating budgets. It does this directly bypayments to government staff on projects, and indi-rectly by financing experts to do operational worknormally done by government employees. This isdisadvantageous in two ways. It misuses the technicalassistance personnel resource, reducing its effectivenessfor institution building. And it is extremely costly;high-cost expatriates are hired in posts that nationalscould fill more cheaply” (pp. 213–14).

The effects of lack of donor coordination oninstitutional development can be particularly acute insmall countries, where the sheer volume of externalassistance (and the associated absorptive capacityconstraints) can hinder development effectiveness.Albania’s country assistance review (OED 1998a)found that despite supporting and helping to design

highly innovative projects overall, “IDA’s strategy . . .did not fully appreciate the risks of overload inherent ina rapidly growing and diverse portfolio and a fragileinstitutional framework” (p. 12). Donor assistance inAlbania reached more than 50 percent of GDP.

In contrast, in middle-income countries, externalassistance flows tend to be too small to have muchimpact. For instance, shortly after the 1994 devalua-tion in the 13 West African countries belonging to theCommunauté Financière Africaine, the Bank lent thelargest borrower in the zone (Côte d’Ivoire, a low-income economy) an amount equivalent to 1.5 percentof its GDP, or less than 2 percent of Bank commitmentsthat year. But to provide similar support to the threeEast Asian economies—Indonesia, Korea, and Thai-land—that ran into difficulties in 1997 would haverequired lending more than three times the scale of nettransfers that actually took place, equivalent to morethan 60 percent of total Bank lending for 1997.

Adjusting to the external environmentThe quality of lending and the support for institutionbuilding are only parts of the equation. Equally importantfor development effectiveness is how well a country



2 4

adjusts to the external environment, so that opportunitiesfor growth can be exploited and the poor can be insulatedfrom adverse shocks. CAEs provide detailed and usefulexamples of how countries adjust, or fail to do so.

Outward orientation and growthTo nurture sustained ownership, reform requires a clearunderstanding of borrower interests in the light ofpolitical economy considerations. Where the seeds ofborrower ownership are in place, lending can be auseful instrument of reform. But as CAEs show, devel-opment assistance is not science—it is art. In unstablepolicy environments, there is no substitute for case-by-case assessments, framed to distinguish risks worthtaking (Côte d’Ivoire and the Philippines) from risksthat are inappropriate (Kenya).

The expected gain from assistance is the product ofthe probability of success times the rewards of adjust-ing. The enormous gains that can be realized fromadjusting are often overlooked (box 3.2). Many studiesfocus only on whether adjustment took place—not onthe payoff. A perspective that considers only the risk offailure and not the gains from success can be mislead-ing. It is the perspective that might be used by a lenderfacing roughly the same level of loss with each failure,and a payoff that does not increase with the gains fromsuccess. This kind of decisionmaking is inappropriatefor an equity investor—or for the kind of developmentpartner described in the Strategic Compact.

What happens if a high standard is set for what isjudged to be a permanent, lasting improvement in the

policy environment? In the cases considered, onlyabout 12 percent of adjuster countries realized apermanent and major improvement in their macroeco-nomic policy environment. But for them, per capitaincome growth rates have been almost three timesthose for unsuccessful countries, and more than sixtimes higher than their pre-adjustment growth rates.

Thus a review of the broader adjustment experiencesupports a key lesson of CAEs: realization of high-payoffsuccesses requires careful analysis and cooperation,rather than the use of simple rules to determine whethersupport should be provided. This complexity does notmean that indicators are not available. For instance,information on seeming intangibles—including conceptssuch as borrower ownership—can be made operational,and obvious tests can be used to sort out the likelysustainability of investments in policy change (box 3.3).

By relying on such characteristics to guide Banksupport for adjustment lending, the probability of successcan be improved. Indeed, the probability of success inBank-supported adjustment lending has increased signifi-cantly in recent years—because the Bank has becomemore conscious of the importance of borrower ownership.Combining this insistence on ownership with a greaterconcern for institutional quality could improve perfor-mance even more. It could also increase the legitimacy ofsubsequent reform efforts.

Outward orientation and poverty alleviationIn low-income countries, capacity weaknesses arepervasive. In middle-income countries they tend to be

orrower owner-ship of adjust-ment programs

Wasty (1993) highlightsthe symbiotic relation-ship between borrowerownership and programoutcomes. It presents thefollowing performancemeasurement criteria forownership:

• Is the initiative forformulating andimplementing the ad-justment plan theborrower’s?

• Is there observableconsensus among keyministries anddecisionmakers onthe nature of thecrisis and the neces-sary actions?

• Have specificup-front actions beeninitiated before theprogram?

• Has participationtaken place withinthe society?

B The evaluationshows that measuresof ownership arestrong predictors ofoutcomes. Andthrough selected coun-try case studies, itascertains the impor-tant factors thataccount for differencesin the intensity ofborrower ownership.

BOX 3.3: MEASURING BORROWER OWNERSHIP AND ITS RELATION TO ADJUSTMENT OUTCOMES

is often cited as vitalfor making policiescredible, safeguardingagainst policy rever-sals, and ensuringthat the benefits aresustainable. An OEDevaluation of nearly100 adjustment pro-grams in 42 countriesby Johnson and

2 5

specific. The country evaluation of Poland (OED1997e) suggests that reform of the social safety netremains one of the most important items on theunfinished policy agenda. Recent reports on Thailandand Indonesia (World Bank 1998) and a growing bodyof empirical work summarized in Levine (1997) showthat in middle-income countries the institutions thatspecialize in bearing risk—financial intermediariesand social safety nets—have been neglected by thedevelopment process. Such oversight has had seriouseconomic and social consequences, as recently demon-strated in Indonesia, the Republic of Korea, Russia, andThailand.

Over the long term, openness helps poor countriesgrow faster, and assists in reducing inequalities(Edwards 1997). But openness makes it more importantto put in place well-functioning risk-bearing institu-tions. To estimate the effects of shocks on incomedistribution, OED updated data from Deininger andSquire (1996) and added estimates for other countries.The data indicate an overwhelming increase in in-

equality within countries.8 Increases in inequality were5 times more frequent than decreases: 49 countriesexperienced increasing inequality, 15 appear to haveno trend, and 10 had decreasing inequality. Inequalityis particularly acute in transition economies (box 3.4).But it is also increasing in countries that previously hadno trend or a decreasing trend.

To sum up, growth and an outward orientation arekeys to reducing poverty. But alone, they are notenough. For sustainable progress, better safety nets andbetter-targeted expenditures on those aspects of povertyfor which markets do not work are essential. As Sala-i-Martin (1997) documents, such expenditures can havestrong positive effects on growth. Similarly, OED’sSocial Dimensions of Adjustment (1996e) found that nocountry has achieved sustainable poverty reductionwithout growth. It also showed that the quality ofgrowth is critical to the distribution of benefits. Muchgreater emphasis on safety nets—and expenditures onsectors not adequately funded by market processes—are needed if poverty is to be alleviated.


BOX 3.4: GREATER POVERTY AND INCOME INEQUALITY IN TRANSITION ECONOMIES

overty has in-creased acrossEastern Europe

and there were declines in14 of 17 countries forwhich there are data. Theincome and health dimen-sions of poverty have alsodeteriorated. Nine of the17 countries with dataexperienced increases ininfant mortality, and 10 of16 countries with dataexperienced a deteriora-tion in secondary schoolenrollment.

The close relationshipbetween poverty andincome inequality distin-guishes three groups ofcountries. In the firstgroup—the Czech Repub-lic, Hungary, the Slovak

Republic, and Slovenia—inequality was historicallylow and has risen moder-ately. In the secondgroup—Poland, Romania,and the Baltics—inequal-ity was slightly higherthan in the first group atthe start of the transitionand has since increased tolevels at or above theOECD average. In thethird group—primarilyRussia and Bulgaria, andperhaps including otherCommonwealth of Inde-pendent States countries—inequality has shot upfrom moderate to levelstypical of the more

P unequal developingcountries.

Poverty and incomeinequality haveincreased in all transi-tion economies since thelate 1980s. Some in-crease in income in-equality in the region,even in the long run, isprobably anunavoidable conse-quence of the introduc-tion of market-basedrewards. But aside fromthis, structural changeand economic disloca-tion have introducedadditional inequalityand poverty.

and Central Asia. Lifeexpectancy at birth hasdeclined precipitously inseveral countries, mostnotably Russia, wherethe average life expect-ancy for men in 1995(58.3 years) was threeyears below that inIndia and a stunning sixyears lower than at thestart of transition. Thedrop in life expectancyin other countries, suchas Ukraine and theBaltics, is similarly con-centrated among men,

Source: Based on EBRD (1997), annex 2.2.

2 7

44

T

THEMATIC EVALUATIONSAND INSTITUTIONALDEVELOPMENT

he current financial crisis has far-reaching implications for development practi-

tioners, and for evaluators. A higher priority must go to monitoring financial sector

performance—and to the wide range of institutions involved in improving gover-

ment in the administration of development programs.Structural and social constraints to development

need far more scrutiny. OED’s Process Review of WorldBank Grant Programs (OED 1998j) shows thatprogress has been made on broadening the Bank’sagenda and developing instruments to nurture themany kinds of institutions that can address theseconstraints. For example, the Institutional Develop-ment Fund (IDF) and the Consultative Group to Assistthe Poorest (CGAP) promote institutional capacity-building and donor coordination. Similarly, themainstreaming of new lending instruments—such asLearning and Innovation Loans and Adaptable Pro-gram Lending—represents tangible progress in thedevelopment of stronger, more sustainable institutions.But much more needs to be done.

Although it is well known that institutionalfactors are essential ingredients of economic growthand social stability, these factors remain neglected.This chapter considers the lessons of OED thematic

evaluations for the wide range of institutional devel-opment issues involved in improving governance.

Financial sectorThe financial institutions in crisis countries violatedvirtually all the institutional norms recommended by arecent OED study on Financial Sector Reform: AReview of World Bank Assistance (OED 1998f). TheOED analysis of financial sector interventions focusedon analyzing how the elements of the Bank’s evolvingfinancial sector policy were reflected in Bank-sup-ported projects. In examining 23 countries, the studyfound a satisfactory and sustainable outcome in only12. The recommendations of the study—especially onthe timing, sequencing, and scope of regulatory inter-vention—are more relevant than ever in light of thepast year’s events. According to Reisen (1998), carefulmonitoring of financial institution conditions in thecrisis countries would have revealed serious weak-nesses in their financial systems. Consider four key

nance. Emphasis must also be given to the institutions—such as NGOs and civil society—that

help those not served by formal institutions. And to improve the effectiveness of public

expenditures, practitioners and evaluators should help to introduce results-based manage-


2 8

indicators of financial system strength developed in theOED study in the crisis countries.

Capitalization and bad loan exposureAt the outbreak of the crisis, nonperforming loans werehighest in the Republic of Korea (16 percent of totalassets), but similarly high in Thailand (15 percent) andIndonesia (11 percent; table 4.1). These figures are muchhigher than the 9 percent in Mexico in early 1995, wherethe cost of rescuing banks has been estimated at about 15percent of GDP (Caprio and Klingebiel 1996). In Indone-sia, Korea, and Thailand, capital-asset ratios were 6–10percent. So even before the crisis, nonperforming loans faroutweighed (on average) bank equity capital.

Government interventionsBanks in crisis countries may have been affected bycertain kinds of government intervention in banklending and corporate finance. The governments oftendirected lending toward particular sectors, both for-mally and informally. In addition to explicit guaran-tees, there were implicit guarantees that led to pre-sumptions of government bailout for nonperformingloans in favored sectors. This encouraged excessiveinvestment and risky lending. Once the bubble burst,the shaken investor confidence was further undermined

by the uncertain fiscal implications of honoring theseexplicit and implicit government guarantees.

Accounting and prudential standardsThe weaknesses of accounting standards in crisis coun-tries are common in many emerging markets: inconsistentfinancial reporting; limited power of auditors to examinecompany records; lax auditing and accounting standards,out of line with international good practice; lack ofpenalties for incorrect reporting of information; andtolerance of multiple accounts. In such environments,even detailed examinations by supervisors and regulatorsmay not reveal the information needed to regulateproperly or to ensure prudential soundness.

Enforcement capabilitiesAlthough some of the crisis countries had strengthenedtheir supervisory and regulatory infrastructure duringthe 1980s and 1990s, partly in response to costlybanking crises in Indonesia and Malaysia a decadeearlier, enforcement capabilities remained weak(Fischer and Reisen 1993). Bank regulators had im-posed limits on bank lending, including liquidityrequirements, exposure limits, and risk-based capitalrequirements. But according to Reisen (1998), thesestandards and ratios were poorly enforced.

Indicator Indonesia Rep. of Korea Malaysia Philippines Thailand

TABLE 4.1: BANK SYSTEM RISK EXPOSURE IN EAST ASIA

Bank system exposure to risk

(percentage of assets, end-1997)Nonperforming loans 11 16 8 6 15Capital ratio 8–10 6–10 8–14 15–18 6–10Real estate exposure 25–30 15–25 30–40 15–20 30–40Collateral valuation 80–100 80–100 80–100 70–80 80–100

Regulatory features during the 1990s

Bank lending to connected firms High HighGovernment-directed bank lending Yes Yes Yes Yes YesBank deposit insurance No No No Yes NoImportance of state-owned banks High HighAccounting standards Weak Weak Weak WeakEnforcement of regulations Weak Weak Weak Weak Weak

Incentives for capital flowsShort-term inflows Limited Limited Promoted

(promoted)Long-term inflows Limited Limited PromotedOutflows Free Limited Limited

Source: Reisen (1998).

2 9

T h e m a t i c E v a l u a t i o n s a n d I n s t i t u t i o n a l D e v e l o p m e n t

The crisis makes it urgently necessary to adopt acentral recommendation of the OED study on financialsector reform: “more resources should be allocated tomonitor and evaluate countries’ financial sector pro-grams, with performance indicators” (p. 83). The crisisalso is the reason for Bank management’s seeking specialbudget authority from the Board to support financialsector work. But above all, it confirms the serious adverseeffects of neglecting institutional development.

Governance issuesThe crisis has reinforced the already-strong evidencethat growth in per capita income is enhanced by strongproperty rights, sound legal foundations, and capablecivil servants—all operating in an effectively managedinstitutional system. There is clear evidence thatcorruption in these institutions hampers growth.1 Dobetter governance and lower corruption improve thedevelopment effectiveness of projects? Unambiguously,

yes. As measured by the development effectivenessindex, Bank-supported projects in countries with aninadequate bureaucracy are on average the weakestperformers.2 In countrieswith a well-functioningbureaucracy, projects per-form much better, withsignificantly lower risksthan average.

B a n k - s u p p o r t e dprojects in high-corruptioneconomies have had sig-nificantly lower returnswith significantly higherrisk. Corruption is almost always associated with lowbureaucratic quality, so that public sector managementprojects in corrupt countries are particularly likely to below-return/high-risk projects. Only education projectshave a high return and a low risk in countries with highcorruption. Projects in low-corruption economies nearlyalways have a higher return and a lower risk thancomparable projects in other countries.

Perhaps the most difficult governance issuesarise where the state has collapsed or failed, particu-larly in countries that have recently emerged fromconflict. In general, projects in these countries tendto have lower rewards and higher risks, reflecting theturbulence that conflict engenders. But risk-rewardperformance varies significantly across sectors and,as might be expected, many types of infrastructureprojects perform well in societies that have experi-enced destruction and civil conflict.

Another message of OED’s (1998k) study of theBank’s experience with post-conflict countries is that theBank can assist best if it avoids the overzealous pursuit offiscal rectitude. The circumstances of these countriesrequire that the first emphasis be on support for rebuildingthe institutions of government and civil society.

Governance institutionsThe Bank has long supported efforts to improve theworkings of the state. In the 1980s, Bank assistanceprimarily sought to make governments “leaner”through downsizing. In the 1990s the Bank sought a“clean state” for its clients by strengthening thecredibility of governance institutions through institu-tional reforms—that is, intra–public sector regula-tory reform and establishment of checks on arbitraryaction. For both approaches, the results have been

FIGURE 4.1: PERFORMANCE OF CIVIL SERVICEREFORM INTERVENTIONS

Note: The data reflect OED evaluations through July 1998.Source: Civil Service Reform Study, forthcoming, OED,World Bank.

OED evaluated projects

0

0

20

20

40

40

60

60

80

80

100

100

Reforming institutions

Reforming institutions

Principal agent relations

Principal agent relations

Quality of human resources

Quality of human resources

Organizations and structures

Organizations and structures

Downsizing

Downsizing

Recently completed and active projects

Unsatisfactory Lack of monitoring & evaluationSatisfactory

Percent

Percent

Do better governanceand lower corruptionimprove thedevelopmenteffectivenessof projects?Unambiguously, YES.


3 0

meager. OED recently evaluated more than 300 civilservice reform interventions supported by 124 lend-ing operations approved during 1980–97. A signifi-cant percentage of completed and ongoing opera-tions lacked the adequate monitoring and evaluation

information needed tomake meaningful assess-ments of performance.Only 33 percent of theinterventions that couldbe evaluated achievedsatisfactory outcomes(figure 4.1). Institutionaldevelopment impact wasrated as substantial inonly 16 percent of com-

pleted operations—and in only 10 percent of com-pleted stand-alone projects.

Overall project performance in the public sectormanagement sector is improving, but civil servicereform components remain among the weakest-per-forming interventions in the Bank’s portfolio. As withfinancial institutions, a central recommendation of theOED analysis is that far greater priority should begiven to integrating the use of performance indicatorsto monitor and support more effective public adminis-tration efforts. Results-based management systems canbe an effective way to focus public sector performanceon outcome measures (box 4.1).

Public expenditure analysis is central to the Bank’spolicy dialogue with member countries. It is a rapidlyexpanding aspect of the Bank’s economic and sector work,having grown from 3 reviews before 1979 to 39 in fiscal1998. Public expenditure reviews are a means for theexternal evaluation of a borrower’s fiscal policies andsector reform efforts. They provide a framework forcoordinating external assistance and assessing its effec-tiveness, and they can provide a micro foundation for theIMF’s macroeconomic framework. IDA’s deputies havestressed the importance of public expenditure reviews asinstruments for client capacity development as well as forenhancing development effectiveness by integrating re-view results with country assistance strategies.

OED’s (1998g) study of The Impact of PublicExpenditure Reviews found that quality has improvedin recent years. But it also found that public expendi-ture reviews provided good (but often dated) analysesof spending policies with little concern for cost effi-ciency or the quality of public services. Then reviews

had only a modest effect on Bank lending strategies,client expenditure policies, and aid coordination. Thestudy argued that such reviews could become signifi-cantly more effective if they were more demand-responsive, if they were better synchronized withauthorities’ budget cycles, and if they gave duerecognition to institutional constraints.

In many societies, nongovernmental organizations(NGOs) and community-based organizations (CBOs)provide a closer link to the poor than public sectorinstitutions. According to an OED (1998h) study ofNGOs in Bank-Supported Projects, 38 percent ofBank-supported projects include NGOs or CBOs intheir plans. This involvement increased to 46 percentof projects in 1997, more than doubling from 20percent in 1989. The study found that these institutionscan be particularly important in projects targeted atimproving gender equality (80 percent), the environ-ment (54 percent), and poverty alleviation (48 percent).But their capacity is often limited by erratic fundingand a lack of financial independence.

Low government capacity to work effectively withNGOs and CBOs is also important. While there aresome outstanding examples of government agenciesthat have shown a strong ability to work with NGOs,these are exceptions. The Bank’s capacity to encourageNGO and CBO involvement in projects remainslimited. And as for financial institutions and publicsector institutions, the Bank’s database and statistics onNGOs and CBOs do not provide a reliable picture oftheir involvement in Bank-supported projects. Nor dothey describe results. The database mainly recordswhether provision was made for NGO or CBOinvolvement, not the actual involvement.

Much remains to be done to develop a betterunderstanding of the task of institutions in assessing andencouraging development effectiveness. As measured bythe performance of Bank projects, there are significant,broad gains to be realized from developing and maintain-ing well-run and effectively managed public institutions.Increased vigilance on corruption—and increased reli-ance on new public sector management techniques, suchas results-based management—could have positivespillover effects on the overall quality of Bank assistance.The Bank could more systematically engage the institu-tions of civil society in addressing gender equality andpoverty alleviation. The Institutional Development Fund,the Consultative Group to Assist the Poorest, and the newlending instruments are promising vehicles for doing this.

NGOs and CBOs can beparticularly important

in projects forimproving gender

equality, protecting theenvironment, and

alleviating poverty.

3 1

T h e m a t i c E v a l u a t i o n s a n d I n s t i t u t i o n a l D e v e l o p m e n t

n OED studyof PublicExpenditure

be tabled in Parliamenttwo to three monthsbefore the budget state-ment. In contrast, publicsector values are rarelyaddressed in developingcountries, because theorientation is to “com-mand and control” ratherthan to serve the citizenry.

Authorizing environ-ment. This includesformal (budget processesand institutions) andinformal institutions ofparticipation andaccountability. Do theseinstitutions and processesprovide an enabling envi-ronment for the publicsector to meet its goals?Do the various levels ofgovernment act in thespirit of the constitutionin exercising theirresponsibilities? Whatare the checks and bal-ances against deviantbehavior? Are thereformal rules to ensurefiscal discipline? Is publicsector borrowing subjectto financial market disci-pline? How is govern-ment performancemeasured? Are outputand outcome indicatorsfor public services moni-tored? In industrial coun-

tries, institutional normsare strictly adhered to,and there are severemoral, legal, voter, andmarket sanctions againstnoncompliance. In devel-oping countries, non-compliance often isneither monitored norsubject to sanctions.

Operational capacityand constraints. What isauthorized is not neces-sarily what will getdone: available opera-tional capacity may notbe consistent with thetask at hand. Even theoperational capacity thatis available may becircumvented by thebureaucratic culture orby incentives that rewardcommand and control—and corruption. Somekey questions: Do theagencies responsible forvarious tasks have thecapacity to undertakethem? Are there bindingcontracts on public man-agers for output perfor-mance? Doesparticipation by civilsociety help alleviatesome of these con-straints? In industrialcountries, answers tomost of these questions

BOX 4.1: PUBLIC SECTOR PERFORMANCE REVIEW: INTEGRATING PUBLIC SECTOR PERFORMANCEWITH A RESULTS-BASED MANAGEMENT SYSTEM

are expected to be yes;in developing coun-tries, this will not bethe case.

The analysis andrecommendations insuch a review must beconsistent with (andrecognize) any incon-sistencies among acountry’s mission andvalues, its authorizingenvironment, and itsoperational capacity. Ifso, the review willenable the client andexternal partners,including the WorldBank, to understandbetter how to improvepublic sector perfor-mance. It will furtherserve as a catalyst tointroduce results-basedmanagement in devel-oping countries. Suchan approach to publicsector managementwould help to changebureaucratic culturefrom its emphasis oncommand and control,with arbitrary andoppressive rules, to onefocused on serving itscitizens, earning theirtrust, achieving results,and working better forless money.

AReviews (1998g)discusses how thisanalysis might givegreater emphasis tooutput measures ratherthan traditional inputmeasures. It says thatto provide relevantanalysis of publicexpenditures, a publicsector performancereview must begin bydeveloping an under-standing of threecontextual dimensionsof the country’s publicsector.

Public sector missionand values. Societalvalues and norms—asembodied in the consti-tution or in annual bud-get policy statements—may be useful points ofreference for publicsector mandates and thevalues inherent in thosemandates. In industrialcountries, the missionand values of the publicsector are spelled out ina medium-term policyframework. Forexample, in NewZealand a policy state-ment of this type must

3 3

55IMPLICATIONS FOR THEBANK AND FOR EVALUATION

he implications of the current financial crisis are sobering for the Bank and for the

evaluation profession. It has become amply clear that the value added by development

assistance programs and by evaluations would be substantially enhanced by moreTexplicit attention to exogenous factors and long-term structural constraints. There should be a

sharper focus on the measurement of poverty reduction as the acid test of development, and

better methods of assessing institutional development at both project and country levels.

The 1997 Annual Review of Development Effective-ness concluded that “the challenge is to find the right fitbetween country policy and institutional factors andstrategies to try to improve conditions favorable toimproved growth and development” (p. 51). In a muchmore complex—and hostile—external environment, thisyear’s Review reaches similar conclusions. It is now evenclearer that improvements in project performance are notenough. Broader structural and social constraints impedeproject effectiveness, but so too does the riskiness of theglobal environment for developing countries.

Before considering what these constraints mean forBank operations and their evaluation, longer-termtrends in Bank performance and development effective-ness need to be evaluated and placed in the context ofthe unprecedented events of the past year. It is particu-larly important to consider the implications of theseevents for the prospects of sustainable growth andpoverty reduction.

Global risksHow hostile the current environment is to developingcountries is shown by Euromoney’s country risk rat-ings.1 For this Review, OED calculated a GDP-weighted measure of countryrisk for developing and in-dustrial countries. The mea-sure for developing countriesshows a deterioration in thepast year to the riskiest levelsince the Latin Americandebt crisis—and one of thebiggest adverse shifts sinceWorld War II. The measurealso shows that in the after-math of the crisis, the external environment for devel-oping countries remains, unlike that for industrialcountries, at a high level of uncertainty—again, thehighest since the debt crisis. For industrial countries,

In a much morecomplex—andhostile—externalenvironment,improvements inproject performanceare not enough.


3 4

the aggregate trend is the opposite: toward less volatil-ity. Developing countries are now perceived as veryrisky investment environments. Does this matter?

Perhaps a great deal. While the relationship amongsuch aggregate measures as country risk, investment

flows, and project perfor-mance is not simple, thebroad dimensions of the rela-tionship are clear. As the debtcrisis took hold in the mid-1980s, risk measures in-creased, and Bank perfor-mance deteriorated to below70 percent satisfactory out-comes. Then, as the crisis wasresolved and the environment

improved, private capital flows increased sharply andproject performance improved, particularly in the pastfew years. These relationships are hardly precise. Butthey cannot be ascribed to coincidence. In disentan-gling some of the possible effects for developmenteffectiveness, examining the relationship to Bank per-formance is instructive.

Measures of country risk are expectations about

likely performance—expectations often not realized.The measures can change after investments have takenplace and been evaluated, either to reflect the changesthat occurred or to correct expectations. Countries cando much better than was expected—as many did in theyears before the East Asian crisis—and much worsethan expected—as in East Asia and Russia over the pastyear. How did changes in country risk affect theperformance of Bank projects exiting in fiscal 1997 and1998? Consider three types of economies: those wherecountry risk during the implementation of Bankprojects was stable, those where the country risk ratingimproved significantly during project implementation,and those where it deteriorated.

Countries whose circumstances improved accountfor more than 60 percent of the exiting projects. Inthese countries, risk perceptions improved consider-ably during implementation, as the world moved toone of the highest-ever growth rates for developingcountries. Not surprisingly, project performance inthese economies was good—the likelihood of a projecthaving a satisfactory outcome in those countries was85 percent. This is 11 percent higher than in economiesthat performed as expected when the project was

he Bank isconsidered thelargest reposi-

completed in fiscal 1996(17 countries and 5updates) and 10 (8 coun-tries and 2 updates) infiscal 1997. By 2000 theplan is to complete theremaining 22 assessmentsand 9 scheduled updates.

When the Bank’s WorldDevelopment Report onpoverty was published in1990, poverty measurescould be calculated foronly 11 countries. Thesesurveys covered the 10years leading to 1990 andtogether accounted for 40percent of the total popu-lation of the developingworld and 50 percent of

its poor. The quality andavailability of householdsurvey data for developingcountries have improvedconsiderably since then.Today 138 surveys areavailable for 69 countries.The timeliness of data hasalso improved from an av-erage lag of 11 years inthe mid-1980s to about 5years now. Even so, WorldDevelopment Indicators(World Bank 1997d)reports estimates of thepopulation living below$1 a day per person foronly 60 countries.

The most commonlyused measure—the head

count index—countspeople below the povertyline but ignores what ishappening to them andwhether they are becom-ing poorer. In the extreme,the measure actuallyimproves if the poor diefrom poverty (Sen 1976).World DevelopmentReport 1990 recognizedthat for any given increasein the incomes of the poor,the reduction in povertydepended on where thepoor were relative to thepoverty line. If they wereconcentrated just belowthe line, the increase intheir incomes would have

BOX 5.1: WHAT DO WE KNOW ABOUT THE EXTENT OF POVERTY?

Ttory of information onpoverty. But it hassystematically focusedon data collectiononly since 1991, whenit issued a directivemandating povertyassessments in bor-rowing countries. Sofar, 94 assessmentshave been done (83countries and 11updates) coveringabout 90 percent ofthe world’s poor.Twenty-two povertyassessments were

Performanceimproved

because it wasbeing swept

along by arising tide.

3 5

I m p l i c a t i o n s f o r t h e B a n k a n d f o r E v a l u a t i o n

initiated. Fortunately, outcomes appear to be morerobust with respect to unexpected deterioration incountry risk. Countries that experienced an increase inrisk, even those that had a substantial increase, did notexperience a large reduction in project performance.Overall, in deterioring economies, the likelihood of asatisfactory outcome was 73 percent.2 So at least froma preliminary analysis of performance, the deteriora-tion in conditions should not be devastating for theexisting portfolio. But the effects on future projects maybe more serious.

In some respects the recent improvement in overallperformance on Bank-supported projects has been theresult of projects being implemented in economiesundergoing improvements in broad fundamentals. Inthis light, performance improved because it was beingswept along by a rising tide. But what will happen ifthis tide has crested, and future performance is nolonger buoyed by a continually improving externalenvironment? If current forecasts of country risk areaccurate and a large portion of Bank borrowers do notexperience an improving external environment duringimplementation, overall satisfactory performance offuture projects could be reduced by as much as 5

percentage points.3 Clearly then, evaluation must givegreater prominence to the effects of the externalenvironment, particularly for sustainability.

Prospects for poverty reductionThe events of the past year confirm that the assessment ofdevelopment effectiveness should give pride of place topoverty alleviation. Chapter 3 reviewed the effects thatthe recent financial crisis and the ongoing transition fromsocialism have had on income distribution. What theseshocks mean for poverty alleviation can be seen byconsidering what they mean for the OECD DevelopmentAssistance Committee’s (DAC) goals in poverty allevia-tion, which call for a reduction in the number of people inabsolute poverty of 1 billion by 2015. To do this meansthat about 50 million people must be raised from povertyeach year for the next 20 years.6 The East Asian crisis hasalready put the DAC program nearly a year behindschedule. But this was not the only poverty-increasingshock. The increase in poverty brought about by thecollapse of safety nets in transition economies is largerthan the East Asian effect: to offset these increases willrequire another three to four years of successful effort inpoverty reduction for the DAC goals to be realized.

a bigger effect on povertythan if they were spreadmore evenly.4

It remains difficult tocompare rates ofpoverty in differentcountries. There areconceptual and practicalproblems.5 The surveysfrom which the povertydata are drawn are:

• Taken at differentpoints in time.

• Based on differentsample designs thatmay not be nation-ally representative.

• Conducted undermethodologies that

are often dissimilar.• Designed to yield a

wide variety of oftendifferent types ofinformation.

Some obtain informa-tion only on incomes,while others gatherinformation only on con-sumption. Most differ inthe depth and detailabout consumption.Methods of valuationvary considerably, withsome surveys using pricesat the nearest market,while others use farm-gate prices.

The money-metric

measure of welfareinvolves setting povertylines and denoting thehousehold cost of thelevel of welfare neededto escape poverty. Bestpractice involves adjust-ing for differences overtime or space andhousehold demograph-ics. But the data neededto do this consistentlyare inadequate andgenerally variable. Theproblems of makingpurchasing power paritycurrency adjustments forinternational compari-sons add to the lack ofcomparability. Setting

the poverty lines toohigh to include, say,countries in EasternEurope in the com-parison raises theestimates of povertyfor other countries.

If this vast heteroge-neity in the underlyingdata is not carefullycontrolled for, aggre-gation to obtainregional or globalestimates will not bevalid, and comparisonsacross countries andtime are, at best,spurious.


3 6

Together these two shocks have moved the goalposts four to five years farther away than when thetargets were established only two years ago. Muchmore must be done in improving poverty measures ifthe Bank and the donor community are to come to gripswith the full dimensions of global poverty (box 5.1).While the goals may be a useful structure, greaterdetail is needed to flesh out all the important dimen-sions of poverty reduction. Better data are needed sothat the poor truly become visible in our evaluationsand goals. More than safety nets and income growthare needed to achieve DAC’s poverty goals (box 5.2).

Implications for the BankWhat does all this mean for Bank operations andevaluations? Beyond doubt, three issues require greateremphasis: poverty alleviation, country strategies, andinstitutional development.

Poverty alleviationIn recent years poverty alleviation and social concernshave been afforded more attention in Bank strategies,as mandated by the Strategic Compact. This year’sReview finds that much closer donor coordination isessential, particularly in low-income countries. In thesecountries, declining volumes of external assistancemust be better coordinated and be more sharply focusedon poverty reduction.

OED evaluation efforts should give more emphasisto social aspects of Bank operations. Rating systemsshould give more weight to the social impact ofprojects and programs. Recent shocks have had aprofound negative effect on poverty, and we cannotexaggerate the importance of measures to address theseconcerns systematically. Safety nets should be at theforefront of poverty alleviation concerns, rather thanbeing comfortably assumed away as they have been inmany analyses of world poverty.

Country strategiesThe Bank has made progress on broadening the develop-ment effectiveness spectrum—for example, it has alreadymoved beyond projects in its strategic perspective. But asdiscussion of the external environment showed, continualimprovements in Bank and borrower performance maynot be enough to maintain the performance of recentyears, much less continue the steady improvements. Insuch a context, even the best-designed project will not

contribute to development effectiveness without a greaterunderstanding of how it fits into the broader country andinternational environment.

For evaluation, OED will reemphasize and inten-sify its shift in focus to the country as the appropriateunit for evaluation. It will also give more attention tosector performance. These new emphases will buildon—not substitute for—OED’s traditional concern withproject performance.

Institutional developmentThe events of the past year have revealed how costlyweak institutions can be. The importance of soundinstitutions has long been known. Not known was howcostly a systemic institutional failure can be. Bankoperations have made significant progress in focusingresources on this important issue. And as discussed inChapter 1, a major dimension of this work will be onfinancial institutions: recent events underscore thatmuch more needs to be done.

For evaluation, greater emphasis should be placedon the metrics of institutional development. The devel-opment effectiveness index is but a first step on thisjourney, and further work should be done towardgiving institutional development more emphasis inevaluation measures. Simpler, concrete steps areneeded. For example, today’s project supervision andcompletion reports do not require a rating of institu-tional development impact. This shortcoming shouldbe rectified. OED should make sure that its newcountry evaluation instrument gives adequate attentionto institutional development issues.

The past year has tested the Bank’s new frameworkfor providing development assistance. So far the resultsare promising. The Bank’s general strategy appears to bewell conceived. With continued adjustment and refine-ment, it should permit the Bank to help developingcountries confront a much tougher external environment.

Evaluation is central to adjusting the Bank’sapproach. For example, systematically addressing pov-erty alleviation requires making the poor visiblethrough better data and monitoring systems. Outputtargets must be linked more tangibly to policy inputsand must recognize the increased risks of the externalenvironment. With the help of evaluators, those targetsmust be embedded in the scorecards of country andsector assistance strategies.

3 7

I m p l i c a t i o n s f o r t h e B a n k a n d f o r E v a l u a t i o n

BOX 5.2: POVERTY ALLEVIATION AND INCOME GROWTH

n assessinghuman welfare,advocates of

and economic historiansis skepticism, becausethey view improved lifeexpectancy as a by-product of economicdevelopment.

But new techniques ofdisease control, usingnew knowledge ofdisease, have beensources of improved lifeexpectancy. And publicintervention has beencrucial for implementingthem. The free marketinstitutions commonlyconsidered to be behindeconomic growth havenot been responsible foradopting the newtechniques of diseasecontrol. Nor do free

market institutions ap-pear to have generatedthe new technology ofdisease control.

But perhaps economicgrowth has been neces-sary for increasing lifeexpectancy—by provid-ing the resources neededto fund public spendingon the new technology,either directly or throughinternational aid, or tofund the research respon-sible for the advance inknowledge. At most,economic growth mayhave been helpful, but itwas not required tofinance the advance inknowledge that broughtinfectious disease under

control.So higher life

expectancy cannot betaken as simply a by-product of economicgrowth or the freemarket conditions thatfoster it. Indeed, pub-lic policy initiativeshave been essential tothe improvement oflife expectancy, andthese can be—andhave been—under-taken in the absence ofeconomic growth.Thus, broadermeasures of develop-ment are important.

Ihuman developmentwould place indi-cators of socialconditions—notablylife expectancy andeducational achieve-ment—on an equalfooting with suchtraditional economicmeasures as GDP percapita and a povertyindex. Some wouldinclude indicators ofpolitical and civilliberties. To thebroader measure ofhuman development,a common reactionamong economists

Source: Easterlin (1998).

3 9

The current Bank evaluation system assesses projectresults through a set of three ordinal ratings—onproject outcome, sustainability, and institutional devel-opment impact:

• Outcome is established by answering the followingquestion: Did the project achieve satisfactory develop-ment results considering the relevance of its main statedobjectives, and the associated costs and benefits? Theoutcome rating takes into account relevance (to checkwhether the project’s objectives were consistent with thecountry’s development strategy), efficacy (to examinewhether the operation achieved its stated goals), andefficiency (to assess results relative to inputs by costs,implementation times, and economic and financialreturns). Outcome is rated on a six-point ordinal scale:highly satisfactory, satisfactory, marginally satisfac-tory, marginally unsatisfactory, unsatisfactory, andhighly unsatisfactory.

• Sustainability is defined as the likelihood, at the timeof evaluation, that the project will maintain itsresults. In assessing sustainability, evaluators focuson features (country conditions, government andeconomic policies, the political situation, and condi-tions specific to the operations, such as availabilityof funds for maintenance) that determine whether theoperation will last over its intended useful life.Sustainability is rated on a three-point ordinal scale:likely, uncertain, and unlikely.

• Institutional development impact is defined as theextent to which a project has improved an agency’sor country’s ability to use its human and financialresources effectively and to efficiently organize eco-nomic and social activities. Institutional develop-ment impact is rated on a three-point ordinal scale:substantial, modest, and negligible.

As in previous years, we report trends on eachevaluation dimension separately (see Chapter 2). In thisyear’s Review, however, we have introduced a newmeasure of overall project results—the developmenteffectiveness index is based on the three ratings. Thisinstrument allows us to take the analysis of portfoliotrends a step forward, making it possible to compareoverall average results—and their variability—acrossdifferent groups of projects.

The development effectiveness index is defined by

assigning cardinal weights to the ratings of each of thethree results-oriented counts, then combining them in asimple way. The index formula is:

ANNEX 1. A NEW APPROACH TO EVALUATING BANK PROJECTS—THE DEVELOPMENT EFFECTIVENESS INDEX

Thus, the development effectiveness index rangesfrom 2—for a project with highly unsatisfactory out-come, unlikely sustainability, and negligible institu-tional development impact—to 10—for a project withhighly satisfactory outcome, likely sustainability, andsubstantial institutional development impact. It is easyto see that outcome is the main force behind the index.Note also how the index separates between satisfactoryand unsatisfactory outcomes, where an index measureof 6 represents such a “divide.” A project with anunsatisfactory outcome will never score higher than 6,no matter what ratings it receives on the other twodimensions.

The average development effectiveness index in thefiscal 1990–98 portfolio is 6.47. The standard devia-tion is 1.85. The contribution of outcome to theaverage index in the portfolio is about 80 percent; theremaining 20 percent is almost evenly split between theother two evaluation components.

A similar cardinal measure of overall project perfor-mance was presented in last year’s Review. In buildingthat index, the aim was to establish, using subjectiveassessment, the relative importance of the three results-

DEVELOPMENT EFFECTIVENESS INDEX = OUTCOME WEIGHT +SUSTAINABILITY WEIGHT + INSTITUTIONAL DEVELOPMENT WEIGHT.

Outcome weight

Highly satisfactory 7.75 Satisfactory 6.00 Marginally satisfactory 5.25 Marginally unsatisfactory 4.50 Unsatisfactory 3.75 Highly unsatisfactory 2.00Sustainability weight

Likely 0.75 Uncertain 0.25 Unlikely 0.00Institutional development impact weight

Substantial 1.50 Modest 0.50 Negligible 0.00


4 0

oriented counts, and then combine them in an intuitivelyappealing way. But this approach, by ignoring theinformation embedded in the historically observed portfo-lio, tended to excessively penalize underperformingprojects and over-reward overperforming ones. It induceda double-counting effect that duplicated a satisfactory-unsatisfactory dichotomy, rather than conveying the extrainformation contained in the six-point classification ofoutcome results and the data about institutional develop-ment impact and sustainability.

The new measure improves on that index, makingthe inferences based on such an instrument morerobust. It does so by explicitly taking into account theinformation coming from the historically evaluatedportfolio, and by keeping to a minimum—and transpar-ently stating—the set of subjective judgments needed tochoose a specific index among the many ways ofdefining one.

Weights are assigned, using the historically evaluatedportfolio as a benchmark, and taking into account thestrong positive association among evaluation dimensions,to avoid double-counting effects and to extract the mostinformation from the empirical observations.

The fact that good (bad) ratings on one dimensionare associated with good (bad) ratings on the otherdimensions allows us to unambiguously rank morethan 70 percent of the projects in the observed portfoliousing only the ordinal information conveyed by thethree sets of ratings. Using only two clear assump-tions—premised on the Bankwide consensus about theimportance of outcome ratings and on the mostefficient way to use information embedded in theobserved portfolio—we are able to increase the total ofprojects that can be ranked to more than 81 percent.

Such rankings are then used to assign scores toeach dimension separately. Choosing a formula thatdefines the index as the sum of three scores makes iteasy to understand and calculate within the Bank, aswell as outside, and allows us to readily calculatechanges in the overall index associated with changes inthe distribution of ratings in a given group of projects(by sector, region, and so on).

Caution should be exercised in interpreting thescores on each separate dimension, since these arederived by looking at the additional informationalcontent that each result-based count contained relativeto the other two in the historically evaluated portfolio.The fact that the ratings on outcome appear to bedriving the index is the result of two forces. First, in

establishing the ranking of observed projects on whichthe index is built, we used (although parsimoniously)outcome as a tie-breaking rule. Second, the fact thatoutcome was rated on a six-point basis, as opposed tothe three-point basis used for the other two counts,resulted in outcome ratings conveying more informa-tion, and thus receiving more weight. That outcomeevaluation is based on a finer scale is a testimony to itsprominence among the ratings. We welcome thisasymmetry, because we did not have to impose anysubjective mechanism to give outcome more weight inthe determination of the index.

Although we report the percentage that each result-based count contributed to the index, this was done onlyfor completeness. After all, the index’s purpose is tosummarize information as far as possible: researchersinterested in examining performance on each count willfind it optimal to look only at the ratings in the chosendimension (rather than look at the components of theweights, which are intimately connected to one another).

Another point worth stressing about the index is thenature of its cardinality. This should be understood asinterval-scale cardinality: because it is impossible tolocate an absolute, nonarbitrary, zero point for thescale, distances between index values, although mean-ingful, are not ratios. We choose a 2–10 range becausethis makes the index readily comparable with otherindexes used within the Bank, while allowing us tospace index values in a way that may be appealing forpractitioners and nonpractitioners alike.

Although it is not of the ratio-scale nature (ratios ofindexes are not meaningful, given the arbitrary range),the cardinality of the development effectiveness indexmakes it a valuable instrument for analyzing Bankproject performance. The index makes it possible tocompare overall performance across, for instance, Sec-tors, Networks, and Regions, using the informationcontained in the project ratings. This represents a stepforward with respect to standard comparisons basedsimply on binary classification (satisfactory or unsatisfac-tory) of outcome, which leave out the sustainability andinstitutional development components of project results,and ignore the nuances embedded in the six-point scale onoutcome. This is the use of the index we are mainlyconcerned with in this report (see Chapter 2).

But the usefulness of the index is not limited to whatwe have suggested here. Using the index, means andvariances for different groups of projects—representingdifferent types of investment—can be calculated, and the

4 1

portfolio can be evaluated, using the standard tools ofportfolio theory.

The richer information embedded in the index canbe used as an alternative dependent variable inregression studies that seek to identify the factorsbehind the Bank’s intervention successes and failures(see Burnside and Dollar 1997). The index could alsobe used as an explanatory variable in the right-handside of regression analyses linking Bank efforts todevelopment results (reduced poverty or inequality,increased growth, and so on).

The guiding rationale of translating into a cardinalmeasure an ordinal system of performance ratings is tocome up with an evaluation instrument with desirableproperties, to be used to trace out the major trends,factors, and effects of Bank investments. The develop-ment effectiveness index is not meant to be a substitute

for direct cardinal measures of project performance,like the rate of return approach to project evaluation.But given the shift of development economics towardgreater attention to policy and institutions, the rate ofreturn methods are ill-fitted to capture the policyreform and institution-building components of Bankprojects. This discrepancy motivated the adoption ofan evaluation system applicable to all projects (forabout a third of Bank projects, rate of return is stillcalculated) that sacrifices the advantages of cardinalityto capture the many facets of project results. Thus thedevelopment effectiveness index represents an attemptto solve the tradeoff between these two conflictingfactors, and should not be construed as an attempt to“faithfully and precisely” measure in a cardinal waythe underlying reality.

A n n e x 1 : A N e w A p p r o a c h t o E v a l u a t i n g B a n k P r o j e c t s — T h e D e v e l o p m e n t E f f e c t i v e n e s s I n d e x


4 2

ANNEX 2. GLOSSARY OF SELECTED TERMS

Adjustment loans: Financing aimed at promotingpolicy reform. Disbursement of these funds, directed atalleviating the costs of the transition to a differentpolicy and institutional environment, is contingent onthe fulfillment of a set of conditions by the recipientcountry (usually based on macroeconomic indicators).

Bank performance: The quality of service delivered bythe Bank, especially in tasks for which it has primaryresponsibility, such as appraisal and supervision.

Borrower ownership: The extent to which the recipientcountry is involved in and committed to a project’sstrategy and goals. Ownership is greater when theborrower initiates the formulation and implementationof a project, when there is clear consensus amonggovernment officials and other decisionmakers on thecourse of action, and when there is broad publicsupport for the initiative.

Borrower performance: Defined as the assumption ofownership rights and responsibilities and delivery ofthe inputs needed to prepare and implement the project.

Contagion: Transmission of destabilizing conditionsfrom one open economy to others closely connected toit, resulting in regional crisis.

Country assistance strategy: The main vehicle forBoard review of the Bank Group’s assistance to IDAand IBRD borrowers. The strategy document describesthe Bank Group’s strategy, which is derived from anassessment of country priorities and indicates the leveland composition of assistance to be provided consistentwith the strategy and the country’s portfolio perfor-mance. The heart of the country assistance strategy isthe ongoing Bank-country dialogue and joint efforts inpreparing and implementing the strategy. Strong coun-try ownership and consultation with key stakeholders—pursued with sensitivity and the general agreement ofthe government—are crucial features of a successfulcountry assistance strategy.

DAC goals: A set of six internationally accepteddevelopment goals for the twenty-first century in theareas of poverty, gender, education, environment, andhealth. The goals were published by the OECD’s

Development Assistance Committee in a 1996 policypaper (OECD 1996).

Demandingness, complexity, and riskiness: Demand-ingness refers to the extent to which the project couldbe expected to strain the economic, institutional, andhuman resources of the government/implementingagency. Complexity refers to such factors as the rangeof policy and institutional improvements contem-plated, the number of institutions involved, the numberof project components and their geographic dispersion,and the number of cofinanciers. Riskiness refers to thelikelihood that the project, as designed, would beexpected to fail to meet relevant project objectivesefficiently. In determining project riskiness, evaluatorsconsider the extent to which the project could reason-ably have been expected at the time of projectpreparation and appraisal to face known risk factors,such as lack of borrower commitment, inadequatecounterpart funding, and war or civil disorder.

Development effectiveness: A demonstrable contribu-tion to economically sound, socially equitable, andenvironmentally sustainable growth.

Development effectiveness index: A measure of overallproject-specific results using the index aggregation ofthree OED project ratings: outcome, sustainability, andinstitutional development.

Efficacy: A measure of whether an operation achievedits physical, financial, and institutional objectives.

Efficiency: An assessment of results in relation toinputs, including costs, implementation times, andeconomic and financial returns.

Global public goods: Goods that are available for thebenefit of all countries, and for which one country’s usedoes not reduce another’s consumption. Yet no singlecountry could or would invest in these goods becausethe costs generally outweigh the aggregate benefits.

Institutional development: Improvement in the abilityof an agency or country to make effective use of humanand financial resources and to efficiently organizeeconomic and social activities.

4 3

A n n e x 2 : G l o s s a r y o f S e l e c t e d Te r m s

Macroeconomic policy: Government actions designed toaffect the entire economy rather than specific sectors ormarkets, especially with respect to the general level ofincome, employment, prices, interest rates, and balanceof payments. Policy measures are usually categorized asfiscal or monetary, depending on which instruments—taxes, public spending and debt, control of money supply,and central bank discount rates—are used.

Outcome: In project ratings, outcome refers to theextent to which a project achieved its major objectivesin a cost-efficient way. Under the results-based manage-ment framework, outcomes are the immediate effectsand changes that result from a project’s outputs (theirservices and products). For example, the outcome of ahealth publicity campaign might be a 5 percentincrease in awareness among those targeted.

Relevance: The consistency of goals with the country’soverall development strategy and the Bank’s assistancestrategy for the country.

Risk-bearing institutions: Organizations designed toshare the risks and costs of unpredictable events amonglarge groups. Examples include unemployment supportprograms, health programs, and life insurance.

Safety nets: Mechanisms that aim to alleviate theburden on the vulnerable of an unfavorable economicsituation (for example, by unemployment insurance).

Strategic Compact: The Strategic Compact betweenthe Bank’s management and Executive Board providesa long-term framework for guiding the Bank’s renewaland calculating the associated resource needs. TheCompact, approved in March 1997, adds $250 millionto the Bank’s $1.2 billion administrative budget, to beused over 30 months to improve how the Bank doesbusiness. The Compact is complemented by substantialredeployments and savings throughout the Bank, iden-tified through an ongoing review of cost-effectiveness.The Compact focuses on refueling current businessactivities, refocusing the development agenda, retool-ing the Bank’s knowledge base, and revamping institu-tional capacities.

Sustainability: The likelihood, at the time of evaluation,that a project will maintain its results in the future.

Washington Consensus: An internationally agreed setof 10 measures that are typically implemented duringpolicy reform, including fiscal discipline, financial andtrade liberalization, deregulation, taxation and publicexpenditure adjustments, and privatization.

19

98

An

nu

al

Re

vi

ew

of

De

ve

lo

pm

en

t E

ff

ec

ti

ve

ne

ss

44 TABLE 1: OUTCOME, SUSTAINABILITY, AND INSTITUTIONAL DEVELOPMENT (ID) IMPACT FOR EXIT FISCAL YEARS 1990–96,

1997, AND 1998, BY SECTOR, NETWORK, LENDING TYPE AND SOURCE, REGION AND INCOME GROUP (BY PROJECTS)

SectorAgriculture 431 27 63 36 31 51 24 76 55 39 26 23 85 54 62Education 128 8 78 54 33 19 9 74 63 47 9 8 88 38 38Electric Power & Other Energy 148 9 67 59 32 11 5 55 45 18 7 6 67 50 33Environment 2 0 100 50 0 7 3 71 57 29 1 1 100 100 0Finance 83 5 56 43 29 14 6 77 69 46 7 6 86 57 29Industry 85 5 54 41 26 7 3 57 57 57 4 4 0 0 0Mining 18 1 65 61 50 3 1 67 67 100 2 2 50 50 100Multisector 138 9 77 57 32 21 10 81 43 25 9 8 78 33 0Oil & Gas 49 3 80 53 57 4 2 75 75 25 2 2 100 50 50Population, Health & Nutrition 51 3 65 47 20 12 6 83 67 25 12 11 75 67 25Public Sector Management 80 5 54 39 28 12 6 67 67 8 9 8 100 56 56Social Sector 10 1 80 40 50 7 3 100 14 71 7 6 71 0 0Telecommunications 30 2 77 70 37 3 1 67 100 33 2 2 100 100 100Transportation 183 12 74 52 29 21 10 81 48 67 5 4 100 80 80Urban Development 77 5 66 36 21 16 7 75 44 25 6 5 100 60 40

Water Supply

& Sanitation 66 4 60 33 31 9 4 67 33 0 6 5 67 50 50

NetworkEnvironmentally & Socially Sustainable Development 433 27 63 36 31 58 27 76 55 38 27 24 85 56 59Finance, Private Sector & Infrastructure 739 47 67 49 31 88 41 71 53 40 41 36 74 54 46Human Development 189 12 74 51 30 38 18 82 55 45 28 25 7 41 22Poverty Reduction &

Economic Management 218 14 69 50 30 33 15 76 52 19 18 16 89 44 28

Note: Sust.= sustainability; sat.= satisfactory; sub.= substantial. Income group categories are derived from the World Development Indicators 1998. A full setof supplemental statistical tables providing further project evaluation results is available electronically at http://www.worldbank.org/html/oed

Exit FY 1990–96 Exit FY 1997 Exit FY 1998Sust. ID Sust. ID Sust. ID

Project Outcome (% Impact Project Outcome (% Impact Project Outcome (% ImpactNo. Share (% sat.) likely) (% sub.) No. Share (% sat.) likely) (% sub.) No. Share (% sat.) likely) (% sub.)

ANN

EX 3: DATA O

N O

UTCO

ME, SU

STAINABILITY, AN

D IN

STITUTIO

NAL D

EVELOPM

ENT IM

PACT

45

Exit FY 1990–96 Exit FY 1997 Exit FY 1998Sust. ID Sust. ID Sust. ID

Project Outcome (% Impact Project Outcome (% Impact Project Outcome (% ImpactNo. Share (% sat.) likely) (% sub.) No. Share (% sat.) likely) (% sub.) No. Share (% sat.) likely) (% sub.)

Lending typeAdjustment 225 14 74 57 35 33 15 79 58 33 12 11 92 50 50Investment 1,354 86 66 44 30 184 85 74 53 38 102 89 79 49 39Lending sourceIBRD only 821 52 71 56 35 105 48 74 61 36 47 41 75 48 43IDA/blend 758 48 62 34 27 112 52 76 47 39 67 59 84 51 39RegionAfrica 533 34 54 28 21 63 29 62 30 42 37 32 78 46 32East Asia and Pacific 278 18 83 71 42 41 19 83 66 44 22 19 81 43 52Europe and Central Asia 111 7 76 59 43 18 8 83 78 50 11 10 80 60 40Latin America & Caribbean 283 18 69 53 36 48 22 83 55 34 22 19 86 62 43Middle East & North Africa 161 10 69 47 28 20 9 70 65 25 7 6 86 43 57South Asia 213 13 68 40 29 27 12 78 63 22 15 13 73 47 33Income GroupLower 862 55 61 35 26 119 55 72 47 38 66 58 77 48 35Lower-middle 458 29 71 52 32 63 29 75 56 33 32 28 84 42 42Upper-middle 219 14 79 69 44 31 14 87 70 47 14 12 85 85 54High 40 3 82 77 44 4 2 75 100 0 2 2 100 0 100

Total/average 1,579 100 67 46 31 217 100 75 54 37 114 100 80 50 40

An

ne

x 3

: Da

ta o

n O

utc

om

e, S

us

tain

ab

ility, a

nd

In

stitu

tion

al D

ev

elo

pm

en

t Im

pa

ct

TABLE 1: OUTCOME, SUSTAINABILITY, AND INSTITUTIONAL DEVELOPMENT (ID) IMPACT FOR EXIT FISCAL YEARS 1990–96, 1997,AND 1998, BY SECTOR, NETWORK, LENDING TYPE AND SOURCE, REGION AND INCOME GROUP (BY PROJECTS) (CONTINUED)

Note: Sust.= sustainability; sat.= satisfactory; sub.= substantial. Income group categories are derived from the World Development Indicators 1998. A full setof supplemental statistical tables providing further project evaluation results is available electronically at http://www.worldbank.org/html/oed

19

98

An

nu

al

Re

vi

ew

of

De

ve

lo

pm

en

t E

ff

ec

ti

ve

ne

ss

46

Exit FY 1990–96 Exit FY 1997 Exit FY 1998ID ID ID

Disburse. Outcome Sust. Impact Disburse. Outcome Sust. Impact Disburse. Outcome Sust. Impact($millions) (% sat.) (% likely) (% sub.) ($millions) (% sat.) (% likely) (% sub.) ($millions) (% sat.) (% likely)(% sub.)

TABLE 2: OUTCOME, SUSTAINABILITY, AND INSTITUTIONAL DEVELOPMENT (ID) IMPACT FOR EXIT FISCAL YEARS 1990–96, 1997,AND 1998, BY SECTOR, NETWORK, LENDING TYPE AND SOURCE, REGION AND INCOME GROUP (BY DISBURSEMENTS)

Note: Disburse.= disbursements; sust.= sustainability; sat.= satisfactory; sub.= substantial. Disbursements are measured in real terms, deflated to fiscal 1996 USdollars. Income group categories are derived from the World Development indicators 1998. A full set of supplemental statistical tables providing further projectevaluation results is available electronically at http://www.worldbank.org/html/oed

SectorAgriculture 26,669 74 47 37 3,096 77 66 42 1,056 87 71 77Education 5,993 80 61 46 1,438 85 71 53 362 95 20 25Electric Power & Other Energy 19,350 66 67 35 1,528 87 76 40 572 90 59 48Environment 75 100 29 0 232 78 76 40 34 100 100 0Finance 9,767 60 53 38 2,336 50 45 33 847 85 74 68Industry 11,890 64 55 29 1,338 75 75 81 407 0 0 0Mining 1,273 69 80 41 106 79 79 100 270 97 97 100Multisector 21,903 86 65 44 1,488 90 57 25 760 99 34 0Oil & Gas 3,683 83 89 42 246 99 99 38 347 100 96 4Population, Health & Nutrition 1,625 79 65 36 800 94 71 22 741 80 70 16Public Sector Management 3,974 77 58 53 505 89 51 10 273 100 29 83Social Sector 674 99 89 18 435 100 6 83 181 67 0 0Telecommunications 1,306 78 79 44 382 90 100 88 91 100 100 100Transportation 16,198 83 59 32 2,047 84 44 60 675 100 58 60Urban Development 6,857 75 50 26 670 76 58 16 544 100 62 7Water Supply & Sanitation 4,574 55 29 26 1,187 65 17 0 564 59 21 24NetworkEnvironmentally & Socially Sustainable Development 26,744 74 47 37 3,328 77 67 41 1,090 87 71 74Finance, Private Sector & Infrastructure 74,898 70 59 33 9,840 73 55 44 4,318 81 58 42Human Development 8,292 81 64 42 2,673 90 60 49 1,285 82 46 16Poverty Reduction & Economic Management 25,877 84 64 46 1,993 90 55 21 1,033 99 32 22

47

TABLE 2: OUTCOME, SUSTAINABILITY, AND INSTITUTIONAL DEVELOPMENT (ID) IMPACT FOR EXIT FISCAL YEARS 1990–96, 1997,AND 1998, BY SECTOR, NETWORK, LENDING TYPE AND SOURCE, REGION AND INCOME GROUP (BY DISBURSEMENTS) (CONTINUED)

Lending typeAdjustment 41,619 78 64 43 3,717 66 52 33 1,742 97 57 65Investment 94,191 72 55 34 14,118 81 60 44 5,984 81 53 32Lending sourceIBRD only 98,619 75 64 40 11,034 75 55 43 4,979 84 52 40IDA/blend 37,192 72 43 29 6,800 83 63 40 2,747 85 60 38RegionAfrica 22,440 63 29 24 2,678 72 45 43 1,361 73 37 30East Asia and Pacific 28,468 88 81 48 4,467 88 70 55 2,089 76 31 37Europe & Central Asia 14,244 76 62 44 1,735 88 80 53 1,074 100 87 30Latin America & Caribbean 36,653 72 65 38 4,441 71 38 37 1,746 94 74 66Middle East & North Africa 10,119 71 44 29 1,618 70 52 34 519 96 34 37South Asia 23,887 70 51 32 2,894 78 73 25 936 79 69 20Income GroupLower 56,864 71 47 31 8,646 80 66 42 3,191 76 54 36Lower-middle 42,938 73 58 37 4,944 81 51 41 3,099 88 37 38Upper-middle 33,208 79 74 46 4,061 71 48 43 1,353 96 98 47High 2,801 81 79 46 184 81 100 0 83 100 0 100Total/average 135,811 74 58 37 17,834 78 58 42 7,726 84 54 39

Note: Disburse.= disbursements; sust.= sustainability; sat.= satisfactory; sub.= substantial. Disbursements are measured in real terms, deflated to fiscal 1996 USdollars. Income group categories are derived from the World Development indicators 1998. A full set of supplemental statistical tables providing further projectevaluation results is available electronically at http://www.worldbank.org/html/oed

An

ne

x 3

: Da

ta o

n O

utc

om

e, S

us

tain

ab

ility, a

nd

In

stitu

tion

al D

ev

elo

pm

en

t Im

pa

ct

Exit FY 1990–96 Exit FY 1997 Exit FY 1998ID ID ID

Disburse. Outcome Sust. Impact Disburse. Outcome Sust. Impact Disburse. Outcome Sust. Impact($millions) (% sat.) (% likely) (% sub.) ($millions) (% sat.) (% likely) (% sub.) ($millions) (% sat.) (% likely)(% sub.)


4 8

ANNEX 4. REPORT FROM COMMITTEE ON DEVELOPMENT EFFECTIVENESS (CODE)

On December 9, 1998, the Committee on DevelopmentEffectiveness (CODE) reviewed the annual summaryentitled Managing Development Effectiveness: AnOverview of FY98 Apex Evaluation Report (SecM98-941). This “chapeau” document is prepared jointly byOED and management to facilitate Board discussion byhighlighting the main conclusions and priority con-cerns arising from the FY98 Annual Review of Devel-opment Effectiveness (ARDE), the 1998 Report onOperations Evaluation (AROE), and the Annual Re-view of Portfolio Performance (ARPP).

In general, the Committee commended OED andmanagement for the excellent quality of the reports. Itwelcomed the continued improvement in the overallquality of the Bank’s portfolio, but recognized that theimprovements should be interpreted with caution andthat there are major challenges ahead. The Committeeendorsed the overall recommendations of the reportsand stressed the need to communicate general recom-mendations widely and to convert them into definitivefollow-through that will lead to poverty reduction.

First, one of the main messages emerging from thethree reports is that we are continuing to enhanceevaluation processes, and that the Bank is at an earlystage of practicing results-based management (RBM),which is both a management system and a performancereporting system. RBM provides a coherent frameworkfor learning and accountability in a decentralizedenvironment. CODE endorsed OED’s strong recom-mendation for implementing RBM, which was firstmade in the 1997 AROE. According to the 1998AROE, progress has been made in enhancing perfor-mance management and the use of lessons learned fordevelopment effectiveness in the Bank. However, thereports conclude that the Bank needs to accelerateprogress toward the implementation of full-fledgedRBM. This would entail linking corporate resourceallocation processes to results achieved and incorporat-ing evaluation information into management perfor-mance assessment. A clear statement of goals andobjectives at the corporate level is the essential precur-sor for this. The Strategic Compact has given theinstitution as a whole a clear sense of momentum anddirection, and the new development framework willhelp sharpen it. At the sectoral level, the Committeestressed the urgent need for sectors to develop strategieswith clear goals, targets, and indicators the Networks

can use in developing goals and targets for the“Corporate Scorecard.” The Committee welcomes thelaunching of a new program of sector strategy papersto assist in this regard. Furthermore, it emphasizedstrongly that the Networks must become much moreinvolved in evaluation and must contribute more to thedissemination and communication of information onresults and to the promotion of learning. At the countrylevel, the Committee noted that significant progresshas been made in enhancing the strategic focus ofCASs, but said that CASs should routinely analyzeportfolio experience, OED lessons, and QAG findings,and the link between portfolio performance and newlending should be made more explicit.

A key element of RBM is the “CorporateScorecard,” which is designed to monitor progresstoward agreed targets in the Bank’s processes, outputs,and strategies—and progress in those developmentoutcomes and goals in borrowing countries in whichthe Bank is engaged. Much remains to be done tomake the Scorecard consistent with results-based man-agement. The Committee noted management’s concur-rence with OED’s recommendation in this regard. Itlooks forward to further discussions with managementon issues such as the integration of assessment indica-tors across corporate goals and strategies and Bankperformance, and the development of procedures tolink more explicitly the performance assessments ofmanagers to the quality of the work produced in theirunits.

Second, the Committee believes that a closer linkamong strategic planning, budgeting, and evaluation isneeded. This requires more systematic, timely, anduser- friendly evaluation processes, as well as a tightercoordination among the Strategy and Resource Man-agement Vice Presidency, self-evaluation activities inOperations, and independent evaluation in OED. TheCommittee urged management to accelerate progresstoward developing an integrated work program com-bining independent and self-evaluation, as recom-mended by the Evaluation Learning Group and en-dorsed by CODE and the Board. It looks forward toreviewing a progress report on monitoring and self-evaluation in FY99.

Third, in view of the shift toward the country levelin the new development framework, the Committeeemphasized the need to strengthen partnerships with

4 9

A n n e x 4 : R e p o r t f r o m C o m m i t t e e o n D e v e l o p m e n t E f f e c t i v e n e s s ( C O D E )

borrowers and with donors. It stressed the criticalimportance of stakeholder participation in settingdevelopment goals and in the design and implementa-tion of appropriate monitoring and evaluation (M&E)systems at the project, sector, and country levels.Although progress has been made and participation hasbrought substantial benefits in enhanced stakeholderbuy-in, better implementation, and reduced risk, theCommittee was concerned that M&E and evaluationcapacity development are receiving too little attention.Evaluation in the Bank cannot substitute for aneffective system of evaluation in borrowing countries.Evaluation capacity development must be a priorityfor Bank operations. This has been a recurring themein CODE statements in the last few years, and we urgemanagement to assure action on this priority. TheCommittee looks forward to seeing the results of areview of the implementation of the 1994 EvaluationCapacity Development Task Force. It also urgedmanagement to make evaluation capacity develop-ment a major focus of partnership pilots, and a keytheme in the Bank’s forthcoming sector strategy forassisting public sector reform.

Fourth, the Committee stressed that poverty reduc-tion must continue to be the ultimate goal of all Bankactivities. Strengthening partnerships must be a keyelement of the Bank’s poverty reduction strategy. Thecrisis in East Asia has highlighted the importance ofgiving considerable attention to the strength and resil-ience of institutions, particularly in the financial andsocial protection areas. The Committee concurred withthe recommendation that the Bank should move institu-tion building to center stage and mainstream institutional

concerns fully across all Bank work. It also endorsed therecommendation that social development should becomecentral to the Bank’s work. The Committee looks forwardto hearing from management about how these recommen-dations will be implemented.

Conclusion. These reports clearly identify themajor challenges the World Bank faces and makestrong, well-founded recommendations for how thesechallenges might be met. We must now translate theserecommendations into action. The Board and manage-ment need to work together to ensure that, as aninstitution, we accelerate the pace at which we imple-ment RBM and all the attendant systemic changesrequired. The Committee recognizes that the broadsystemic changes needed to achieve a full-fledged RBMsystem in the Bank will take a sustained effort over anumber of years. However, we urge management to bemore explicit about its commitment to achieving thisgoal and how it plans to address these issues.

Working closely with management, CODE willdevelop a work program for 1999 that will, in largepart, be derived from the conclusions and recommenda-tions of the apex evaluation reports. We will discuss themost appropriate and effective ways for CODE tomonitor progress in key areas. An indicative workprogram for CODE will be circulated shortly.

Jan PiercyChairperson, CODE

5 1

ENDNOTES

Chapter 11. World Bank 1998e.

2. Based on Euromoney’s ratings for country risk.

3. IDS 1998, Chapter 1, p. 1.

4. The glossary (Annex 2) defines terms related to institu-tional development. In addition, box 3.1 illustrates the generalperspective taken by analysts who focus on institutional issues.

5. In April 1998, finance ministers and central bank gover-nors from a number of large economies formed three workinggroups—on transparency and accountability, on strengtheningfinancial systems, and on international financial crises—to dis-cuss the policy issues raised by the financial crisis. In October1998, each working group presented its findings and recommen-dations in a separate report.

Chapter 21. The terms “operation” and “project” refer to both IDA

and IBRD lending and are used interchangeably. Unless otherwisestated, all time period references relate to the fiscal year in whichevaluated operations exited the portfolio.

2. The results for fiscal 1998 are based on a sample of 114evaluated operations—40 percent of the 283 exiting operations—for which regional staff have prepared completion reports. Acomplete set of data by region and sector will be provided onOED’s Web site for reference and follow-up.

3. The likely bias arises from problems of sample representa-tion for the preliminary fiscal 1998 results. According to projectdata from the Quality Assurance Group (QAG) at exit, the cohortevaluated so far includes far fewer problem projects (16 percent)than the projects remaining to be evaluated (26 percent). Of theevaluated cohort, those rated as problem projects by QAG at exitwere only 18 percent satisfactory, while the rest were more than90 percent satisfactory. Assuming the same relationship betweenQAG and OED assessments for those fiscal 1998 exits yet to bereviewed by OED yields an estimate of 73 percent satisfactory forthe part of the cohort not reviewed here. Combining this groupwith the exits reviewed implies an overall 76 percent satisfactoryrating for the entire group of fiscal 1998 exits.

4. Data from the Annual Report on Portfolio Performance,Fiscal Year 1998 (ARPP), prepared by the Quality AssuranceGroup.

5. Disbursements are measured in real terms, deflated tofiscal 1996 US dollars.

6. The standard deviation of the regional shares of satisfac-tory projects has dropped 40 percent, from 10 percentage pointsfor fiscal 1990–96 to 6 percentage points for fiscal 1997–98.

7. Data on the active portfolio under supervision are takenfrom the Annual Report on Portfolio Performance, Fiscal Year1998 (ARPP), prepared by the Quality Assurance Group. Thebasic measure of performance used in the ARPP is the number ofprojects at risk of not achieving their development objectives.Projects at risk consist of actual and potential problem projects.Actual problem projects are those for which implementationprogress is unsatisfactory or development objectives are not likely

to be achieved. Potential problem projects are rated satisfactoryon implementation progress/development objectives, but haveother risk factors historically associated with unsatisfactoryoutcomes.

8. “The Other Crisis,” delivered in Washington, D.C., onOctober 6, 1998, emphasized the essentials of good governanceand the need to specify the regulatory and institutional funda-mentals essential to a workable market economy.

9. Institutional development impact measures the extent towhich a project has improved the ability of an agency or countryto make effective use of its human and financial resources. Thisimpact, possible even in the absence of explicit institutionaldevelopment objectives, includes both traditional impacts,through new or improved organizations, and the impact projectshave on the rules of the game governing public and private sectorbehavior.

10. Sustainability is defined as the likelihood, at the time ofevaluation, that a project will maintain its results in the future. Tojudge the sustainability of an operation, evaluators take intoaccount country conditions, government policies, and other con-ditions specific to the operation, such as availability of funds foroperation and maintenance. Basic factors behind the originalproject appraisal are also considered—such as technical, finan-cial, and economic viability; susceptibility to external shocks; andthe social, environmental, and governance environment.

11. Looking at shifts in country macroeconomic environ-ments, the fiscal 1997–98 evaluated exits have slightly moreprojects in the poorly performing country group than do thoseexiting in fiscal 1990–96 (12 percent, compared with 9 percent),with a slight decrease in projects in the lower-performing group(from 17 percent to 14 percent). The three other groupings ofhigh, lower-medium, and upper-medium performing countriesshow even less variation when analyzed for fiscal 1990–96 andfiscal 1997–98. While slight, the movement toward poorer-performing economic environments makes it unlikely that coun-try selectivity affected the improved performance results of fiscal1997–98.

12. Borrower performance is assessed for three projectprocesses—preparation, implementation, and compliance. Imple-mentation is the broadest of these three project measures, withdimensions under the government’s control—such as broadproject commitment, appointment of key staff, and counterpartfunding—as well as implementing agency factors such as man-agement, staffing, cost changes, and beneficiary participation.

13. Bank performance is assessed for three project pro-cesses—identification, appraisal, and supervision. Assessments ofthe Bank’s performance during project identification includelooking at the involvement of the government and beneficiary,whether the project is consistent with the Bank’s country assis-tance strategy, and whether there is a grounding in economic andsector work. The quality of appraisal includes the followingdimensions: technical and financial analysis, cost benefit analysis,institutional capacity analysis, and environmental and socialanalysis. Dimensions taken into account for assessing projectsupervision include progress reporting, identification/assessmentof problems, use of performance indicators, advice to the imple-


5 2

menting agency, and flexibility in suggesting/approving modifi-cations.

14. Findings that supervision improved are consistent withthe latest Quality Assurance Group self-evaluation assessments ofsupervision in Supervision Quality in FY98: A QAG Assessment(RSA2).

Chapter 31. Up until August 1998.

2. OEDCR staff were asked to rate country assistance strate-gies (quality at entry, implementation, outcome, sustainability, andinstitutional development) from their evaluations in CAEs. Wherethe evaluator thought appropriate, the country strategy was ratedseparately for different periods. For the 17 countries reviewed, CAEsidentify 25 time slices—that is, different and distinct strategies forthe country within the time period covered by the evaluations.Project ratings corresponding to the same countries and periods showthat project outcome was satisfactory 68 percent of the time. Thelatter figure is similar to the average project performance during theperiod of the CAE analysis.

3. Derived from background work on country assistancestrategies, available electronically at http://www.worldbank.org/html/oed.

4. The evaluation form follows the style of OED’s projectinformation form, and the methodology is that of OED’s method-ology for evaluating completed lending operations. See OED1997b.

5. In addition to comparing the perspectives on growth andpoverty reduction in CAEs with those of empirical models, wecompared how well these judgments on country strategy perfor-mance served as a predictor of subsequent Bank project perfor-mance. While 25 observations limited the degrees of freedom,these judgments nevertheless serve as stronger predictors ofproject performance than do macroeconomic policy measures.

6. The data on institutional quality refer to country ratingsfor bureaucratic quality, rule of law, and corruption as defined bythe International Country Risk Guide. An average of the threeratings of greater than 4 on a 1–6 scale is considered satisfactory.A rating of greater than 3.33 and less than 4.0 is consideredmarginally satisfactory.

7. This rating is based on a more limited sample of low- andmiddle-income countries. The policy index has been calculated onan annual basis for 1995, as in Burnside and Dollar (1997).

8. Using comparable data on income distribution for 45countries, Deininger and Squire (1996) show that over the 30-year period up to the early 1990s, there was no trend in within-country income inequality. In 29 countries the Gini coefficient—ameasure of income distribution—remained virtually constant, in 8it increased, and in another 8 it decreased. In the 16 countries withan increasing or decreasing trend, in 12 the change was small.Our shorter-term perspective, focusing on changes over five ormore years, suggests that in recent years a very different patternhas emerged. The trends in transition economies are an important

aspect of the changing pattern. But even in the 45 countriesconsidered by Deininger and Squire, an updating of their data anda more short- term focus shows that about five times as manycountries (24) have an increasing trend as have a decreasing trend(5 countries). Of course, given the limited amount of observa-tions, our classification was necessarily based on a heuristicapproach.

Chapter 41. See Clague (1997) and Knack and Keefer (1995). For

literature on corruption, see Bardhan (1997), World Bank(1997b), Mauro (1995), and Rose-Ackerman (1998).

2. To consider the effects that country characteristics mighthave on Bank performance, we examined performance in a widerange of countries. For example, we grouped countries by suchcharacteristics or whether they were post-conflict societies, tran-sition economies, Sub-Saharan Africa economies, had high or lowlevels of corruption, good or weak bureaucracies, and persistentlypoor policy environments.

Chapter 51. Euromoney’s semiannual country risk ratings range from 0

(most risky) to 100 (least risky). Each rating is calculated as aweighted average of nine categories of indicators representinganalytical, credit, and market indicators. The categories areeconomic data (25 percent weighting), political risk (25 percent),debt indicators (10 percent), rescheduled debt or debt in default(10 percent), sovereign credit ratings (10 percent), access to bankfinance (5 percent), access to short-term finance (5 percent),access to international bond and syndicated loan markets (5percent), and access to and discount on forfeiting (5 percent).

2. The change in risk rating between the year of approvaland the year of exit has been used to classify countries. Improve-ment refers to betterment in risk rating of more than 5 percent;performance as expected refers to countries whose rating changedby less than 5 percent; deteriorating economies are those with anincrease in risk between the year of approval and the year of exit.

3. The estimated relationship between country risk andportfolio performance during 1997–98 is used to forecast ratingsfor projects exiting in 1999. The forecast of a 5 percentage pointdeterioration in fiscal 1998 satisfactory projects is used as abaseline scenario, assuming that country risk ratings in 1999would be the same as in 1998. In a more pessimistic scenario,country risk ratings would deteriorate by 6 percentage points.

4. World Bank 1990, box 3.3, p. 47.

5. See Ravillion (1994a, 1994b, 1996) and Pollak (1991).

6. The estimated number of poor people in 2015, given theexpected increase in population and assuming a 50 percentreduction in poverty, is 900 million. This would call for liftingalmost 1 billion people out of poverty over the next 20 years, orabout 50 million people a year.

BIBLIOGRAPHY

5 3

Bardhan, Pranab. 1997. “Corruption and Develop-ment: A Review of the Issues.” Journal of EconomicLiterature 35(3): 1320–46.

Barro, Robert J. 1995. “Technological Diffusion,Convergence, and Growth.” NBER Working Paper5151. National Bureau for Economic Research,Cambridge, Mass.

Barro, Robert J., and Xavier Sala-i-Martin. 1995.Economic Growth. New York: McGraw-Hill.

Bell, Clive, and Robert Rich. 1994. “Rural Povertyand Agricultural Performance in Post-IndependenceIndia.” Oxford Bulletin of Economics and Statistics56(2): 111–33.

Berg, Elliot J. 1993. Rethinking Technical Coopera-tion: Reforms for Capacity Building in Africa. NewYork: United Nations Development Programme.

Bezanson, Keith. 1998. “Some Implications of the EastAsian Crisis for Development Effectiveness—Per-sonal Perspectives Prepared at the Request of theOperations Evaluation Department of the WorldBank.” Institute of Development Studies, SussexUniversity, Brighton.

Birdsall, Nancy. 1998. “Life is Unfair: Inequality inthe World.” Foreign Policy 111(summer): 76–93.

Burnside, Craig, and David Dollar. 1997. Aid, Poli-cies and Growth. Policy Research Working Paper1777. Washington, D.C.: World Bank.

Caprio, Gerard Jr. 1996. “Banking on FinancialReform? A Case of Sensitive Dependence on InitialConditions.” In Gerard Caprio Jr., Izak Atiyas, andJames A. Hanson, eds., Financial Reform: Theoryand Experience. Cambridge, Mass.: CambridgeUniversity Press.

———. 1998. Banking on Crises: Expensive Lessonsfrom Recent Financial Crises. Washington, D.C.:World Bank.

Caprio, Gerard Jr., and Daniela Klingebiel. 1996.Bank Insolvencies: Cross-Country Experience.Policy Research Working Paper 1620. Washington,D.C.: World Bank

Cassen, Robert, and associates. 1994. Does AidWork? Oxford: Clarendon.

Claessens, Stijn, and Thomas Glaessner. 1997. AreFinancial Sector Weaknesses Undermining the EastAsian Miracle? Washington, D.C.: World Bank.

Clague, Christopher. 1997. Institutions and EconomicDevelopment. Baltimore, M.D.: The Johns HopkinsUniversity Press.

Commander, Simon, and others. 1997. “The Causesand Consequences of Government for Growth andWell-being.” Background paper for World Develop-ment Report 1997. World Bank, Washington, D.C.

Deininger, Klaus, and Lyn Squire. 1996. “A New DataSet for Measuring Income Inequality.” The WorldBank Economic Review 10(3): 565–91.

Demery, Lionel, and Michael Walton. 1998. “ArePoverty and Social Goals for the 21st CenturyAttainable?” World Bank, Poverty Reduction andEconomic Management Network, Washington, D.C.

Easterlin, Richard A. 1998. “How Beneficent is theMarket? A Look at the Modern History of Mortal-ity.” University of Southern California. Mimeo.

Easterly, William, and Stanley Fischer. 1995. “TheSoviet Economic Decline.” The World Bank Eco-nomic Review (9)3.

EBRD (European Bank for Reconstruction and Devel-opment). 1997. Transition Report 1997: EnterprisePerformance and Growth. London.

Edwards, Sebastian. 1997. “Trade Policy, Growth, andIncome Distribution.” American Economic ReviewPapers and Proceedings 87(2): 205–10.

Fischer, Bernhard, and Helmut Reisen. 1993.Liberalising Capital Flows in Developing Countries:Pitfalls, Prerequisites and Perspectives. Washington,D.C.: Organization for Economic Cooperation andDevelopment.

Gallup, John Luke, and Jeffrey Sachs, with Andrew D.Mellinger. 1998. “Geography and EconomicGrowth.” Paper prepared for the Annual World BankConference on Development Economics, April 20–21, Washington, D.C.

Gwin, Catherine, and Joan M. Nelson. 1997. Perspec-tives on Aid and Development. Washington, D.C.:Overseas Development Council.

Hadjimichael, Michael T., and others. 1995. Sub-Saharan Africa: Growth, Savings, and Investment,1986–93. IMF Occasional Paper 118. Washington,D.C.

Helleiner, Gerald, and others. 1995. Report of theGroup of Independent Advisers on DevelopmentCooperation Issues between Tanzania and Its AidDonors. Copenhagen: Royal Danish Ministry ofForeign Affairs.

IDS (Institute of Development Studies). 1998. Papersprepared for the workshop on the East Asian Crisis,July 13–14, Sussex University, Brighton.


5 4

IMF (International Monetary Fund). 1995. “Transfers,Social Safety Nets, and Economic Growth.” IMFStaff Papers 44(1): 81–102.

Ingram, Gregory. 1992. Social Indicators and Produc-tivity Convergence in Developing Countries. PolicyResearch Working Paper 894. Washington, D.C.:World Bank.

Ito, Takatoshi. 1998. “What Can Developing CountriesLearn from East Asian Economic Growth?” In BorisPleskovic and Joseph E. Stiglitz, eds., Annual WorldBank Conference on Development Economics 1997.Washington, D.C.: World Bank.

Johnson, John H., and Sulaiman S. Wasty. 1993.Borrower Ownership of Adjustment Programs andthe Political Economy of Reform. World BankDiscussion Paper 199. Washington, D.C.

Johnston, Timothy, and Nicolas Van de Walle. 1996.Improving Aid to Africa. Washington, D.C.: Over-seas Development Council.

Keefer, Philip, and Stephen Knack. 1997. “Why Don’tPoor Countries Catch Up? A Cross-National Test ofan Institutional Explanation.” Economic Inquiry35(July): 590–602.

Knack, Stephen, and Philip Keefer. 1995. “Institutionsand Economic Performance: Cross-Country TestsUsing Alternative Institutional Measures.” Econom-ics and Politics 7(3): 207–27.

Krugman, Paul. 1994. “The Fall and Rise of Develop-ment Economics.” In Lloyd Rodwin and Donald A.Schön, eds., Rethinking the Development Experi-ence. Harrisonburg, Va.: R.R. Donnelly and Sons.

———. 1998. “The Confidence Game.” New Republic219: 23–25.

Levine, Ross. 1997. “Financial Development and Eco-nomic Growth: Views and Agenda.” Journal ofEconomic Literature 35(2): 688–726.

Levine, Ross, and David Renelt. 1991. “A SensitivityAnalysis of Cross-Country Growth Regressions.”American Economic Review 82(4): 942–63.

Levine, Ross, and Sara Zervos. 1993. “Looking at theFacts: What We Know about Policy and Growth fromCross-Country Analysis.” Policy Research WorkingPaper 1115. Washington, D.C.: World Bank.

Lipton, Michael, and others. 1998. Successes in Anti-poverty. Geneva: International Labour Office.

Maddison, Angus. 1997. “The Nature and Functioningof European Capitalism: A Historical and Compara-tive Perspective.” Banca Nazionale del LavoroQuarterly Review 50(203).

Mathieu, Nicolas. 1998. Financial Sector Reform.Operations Evaluation Department Study Series.Washington, D.C.: World Bank.

Mauro, Paulo. 1995. “Corruption and Growth.” Quar-terly Journal of Economics 110(3): 681–712.

Milanovic, Branko. 1998. Income, Inequality, andPoverty during the Transition from Planned toMarket Economy. A Regional and Sectoral Study.Washington, D.C.: World Bank.

Mosley, Paul, Jane Harrigan, and John Toye. 1991.Aid and Power: The World Bank and Policy-basedLending. London: Routledge.

OECD (Organisation for Economic Cooperation andDevelopment). 1996. “Shaping the 21st Century: TheContribution of Development Co-operation.” Paris:Development Assistance Committee.

OED (Operations Evaluation Department). 1990.Chile: Structural Adjustment Loans. Project Perfor-mance Audit Report 08851. Washington, D.C.:World Bank.

———. 1995. The Social Impact of Adjustment Opera-tions: An Overview. Report 14776. Washington,D.C.: World Bank.

———. 1996a. Argentina Country Assistance Review.Report 15844. Washington, D.C.: World Bank.———. 1996b. Ghana Country Assistance Review.Report 14547. Washington, D.C.: World Bank.———. 1996c. “Technical Assistance.” Lessons and

Practices 7. Washington, D.C.: World Bank.———. 1996d. The World Bank’s Experience with

Large Dams: A Preliminary Review of Impacts.Report 15815. Washington, D.C.: World Bank.

———. 1996e. Social Dimensions of Adjustment.Washington, D.C.: World Bank.

———. 1996f. Zambia Country Assistance Review.Report 15675. Washington, D.C.: World Bank.

———. 1997a. Côte d’Ivoire Country Assistance Re-view. Report 17015. Washington, D.C.: World Bank.

———. 1997b. “Evaluating Development Operations:Methods for Judging Outcomes and Impacts.” Les-sons and Practices 10. Washington, D.C.: WorldBank.

———. 1997c. Kingdom of Morocco Country Assis-tance Review. Report 16326. Washington, D.C.:World Bank.

———. 1997d. Mozambique Country Assistance Re-view. Report 17209. Washington, D.C.: World Bank.

5 5

B i b l i o g r a p h y

———. 1997e. Poland Country Assistance Review.Report 16495. Washington, D.C.: World Bank.

———. 1997f. “A Success and Challenge: AGETIP inSenegal.” Operations Evaluation Department Précis148. Washington, D.C.: World Bank.

———. 1998a. Albania Country Assistance Review.Report No. 18030. Washington, D.C.: World Bank.

———. 1998b. 1997 Annual Review of DevelopmentEffectiveness. Washington, D.C.: World Bank.

———. 1998c. Annual Report on Portfolio Perfor-mance Fiscal Year 1998. Washington, D.C.: WorldBank.

———. 1998d. Bangladesh Country Assistance Re-view. Report 17455. Washington, D.C.: World Bank.

———. 1998e. Bolivia Country Assistance Review.Report 17957. Washington, D.C.: World Bank.

———. 1998f. Financial Sector Reform: A Review ofWorld Bank Assistance. Report 17454. Washington,D.C.: World Bank.

———. 1998g. The Impact of Public ExpenditureReviews. Washington, D.C.: World Bank.

———. 1998h. NGOs in Bank-Supported Projects. AnOED Review. Report 18399. Washington, D.C.:World Bank.

———. 1998i. Philippines Country Assistance Review.Report 17417. Washington, D.C.: World Bank.

———. 1998j. Process Review of World Bank GrantPrograms. Report 18317. Washington, D.C.: WorldBank.

———. 1998k. The World Bank’s Experience withPost-Conflict Reconstruction. Report 17769. Wash-ington, D.C.: World Bank.

Picciotto, Robert, and Eduardo Wiesner, eds. 1998.Evaluation and Development: The Institutional Di-mension. New Brunswick, N.J.: Transaction.

Pollak, R. 1991. “Welfare Comparisons and SituationComparisons.” Journal of Econometrics 50: 31–48.

Psacharopoulos, George, and others. 1992. Povertyand Income Distribution in Latin America: TheStory of the 1980s. Latin America and the CaribbeanTechnical Department Report 27. Washington, D.C.:World Bank.

Ravallion, Martin. 1994a. “Measuring Social Welfarewith and without Poverty Lines.” American Eco-nomic Review, 84 (2): 359-64.

Ravallion, Martin. 1994b. “Poverty Comparisons.”Fundamentals of Pure and Applied Economics, vol.56. Chur, Switzerland: Harwood.

Ravallion, Martin. 1996. “Issues in Measuring andModeling Poverty.” The Economic Journal (Septem-ber). Blackwell.

Ravallion, Martin, and Shaohua Chen. 1997. “WhatCan New Survey Data Tell Us about Recent Changesin Distribution and Poverty?” The World BankEconomic Review 11(2): 357–81.

Ravallion, Martin, and Gaurav Datt. 1994. Growthand Poverty in India. Washington, D.C.: WorldBank.

Reisen, Helmut. 1998. “Domestic Causes of CurrencyCrises: Policy Lessons for Crisis Avoidance.” Paperpresented at the Institute of Development Studies EastAsian Crisis Workshop, Sussex University, July 13–14, Brighton.

Rodrik, Dani. 1996. “Understanding Economic PolicyReform.” Journal of Economic Literature 34(March): 9–41.

Rose-Ackerman, Susan. 1998. “Corruption and Devel-opment.” In Boris Pleskovic and Joseph E. Stiglitz,eds., Annual World Bank Conference on Develop-ment Economics 1997. Washington, D.C.: WorldBank.

Sachs, Jeffrey D., and Andrew Warner. 1995. “Eco-nomic Reform and the Process of Global Integra-tion.” Brookings Papers on Economic Activity.Washington, D.C.

Sala-i-Martin, Xavier. 1997. “I Just Ran Four MillionRegressions.” NBER Working Paper Series 6252.Cambridge, Mass.: National Bureau for EconomicResearch.

Sen, Amartya. 1976. “Poverty: An Ordinal Approachto Measurement.” Econometrica 46: 437–46.

Squire, Lyn. 1993. “Fighting Poverty.” American Eco-nomic Review 83(2): 377–92.

UNDP (United Nations Development Programme).1998. Poverty in Transition? New York: RegionalBureau for Europe and the Commonwealth of Inde-pendent States.

Working Group on International Financial Crises.1998. Report prepared by representatives of Argen-tina, Australia, Belgium, Brazil, Canada, France,Germany, Hong Kong SAR, Italy, Japan, Korea,Mexico, the Netherlands, Singapore, South Africa,the United Kingdom, and the United States. Meetingsheld in Washington, D.C., April 1998.

World Bank. 1990. World Development Report 1990:Poverty. New York: Oxford University Press.


5 6

———. 1996a. “Partnership for Capacity Building inAfrica.” A report of the Working Party on the Impactof Bank Policies, Instruments, and Operational Prac-tices on Capacity Building in Africa. Washington,D.C.

———. 1996b. World Development Report 1996:From Plan to Market. New York: Oxford UniversityPress.

———. 1997a. Annual Report 1997. Washington, D.C.———. 1997b. Helping Countries Combat Corruption:

The Role of the World Bank. Washington, D.C.———. 1997c. “The Strategic Compact: Renewing the

Bank’s Effectiveness to Fight Poverty.” Washington,D.C.

———. 1997d. World Development Report 1997.Washington, D.C.

———. 1998a. Assessing Aid. A Policy ResearchReport. Washington, D.C.

———. 1998b. “Country Assistance Strategies: Retro-spective and Outlook.” Managing Directors Opera-tions Policy and Strategy Group, Washington, D.C.

———. 1998c. East Asia: The Road to Recovery.Washington, D.C.

———. 1998d. World Development Indicators 1998.Washington, D.C.

———. 1998e. World Development Report 1998/99:Knowledge for Development. New York: OxfordUniversity Press.

———. 1998f. “Supervision Quality in FY98: A QAGAssessment (RSA2).” Quality Assurance Group,Washington, D.C.

The Operations Evaluation Department (OED), an in-dependent evaluation unit reporting to the World Bank’sExecutive Directors, rates the development impact andperformance of all the Bank’s completed lending opera-tions. Results and recommendations are reported to theExecutive Directors and fed back into the design andimplementation of new policies and projects. In addition tothe individual operations and country assistance programs,OED evaluates the Bank’s policies and processes.

Operations evaluation studies, World Bank discus-sion papers, and all other documents are available fromthe World Bank InfoShop.

Summaries of studies and the full text of the Précisand Lessons & Practices can be read on the Internet athttp://www.worldbank.org/html/oed/index.htm

How To Order OED PublicationsDocuments listed with a stock number and price codemay be obtained through the World Bank’s mail orderservice or from its InfoShop in downtown Washington,DC. For information on all other documents, contactthe World Bank InfoShop.

Ordering World Bank PublicationsCustomers in the United States and in territories notserved by any of the Bank’s publication distributors maysend publication orders to:

The World BankP.O. Box 960Herndon, VA 20172-0960Fax: (703) 661-1501Telephone: (703) 661-1580The address for the World Bank publication database onthe internet is: http://www.worldbank.org (select publi-cations/project info).E-mail: [email protected] number: (202) 522-1500Telephone number: (202) 458-5454

The World Bank InfoShop serves walk-in customersonly. The InfoShop is located at:

701 18th Street, NWWashington, DC 20433, USA

All other customers must place their orders throughtheir local distributors.

Ordering by E-MailIf you have an established account with the World Bank,you may transmit your order by electronic mail on theInternet to: [email protected] Please include youraccount number, billing and shipping addresses, the titleand order number, quantity, and unit price for each item.

OPERATIONS EVALUATION DEPARTMENT PUBLICATIONS

1999

Appréciation de l’efficacité du développement:L’évaluation à la Banque mondiale et à la Société financiére internationale

Determinar la eficacia de las actividades de desarrollo:La evaluació en el Banco Mundial y la Corporación Financiera Internacional

Lessons of Fiscal Adjustment

Monitoring & Evaluation Capacity Development in Africa

Nongovernmental Organizations in World Bank–Supported Projects: A Review

Philippines: From Crisis to Opportunity

Public Sector Performance — Lessons from Urban Transport

1998

1997 Annual Review of Development Effectiveness

Assessing Development Effectiveness: Evaluation in the World Bank and the International Finance Corporation

Evaluation and Development: The Institutional Dimension (Transaction Publishers)

Financial Sector Reform: A Review of World Bank Assistance

India: The Dairy Revolution

Rebuilding the Mozambique Economy: Assessment of a Development Partnership

Public Sector Performance — The Critical Role of Evaluation

The World Bank’s Experience with Post-Conflict Reconstruction

1997

1995 Evaluation Results

Agricultural Extension and Research: Achievements and Problems in National Systems

Fiscal Management in Adjustment Lending

Mainstreaming Gender in World Bank Lending: An Update

Paddy Irrigation and Water Management in Southeast Asia

Poland Country Assistance Review: Partnership in a Transition Economy

Reforming Agriculture: The World Bank Goes to Market

Zambia Country Assistance Review: Turning an Economy Around

1996

1994 Evaluation Results

Industrial Restructuring: World Bank Experience, Future Challenges

Lending for Electric Power in Sub-Saharan Africa

Social Dimensions of Adjustment: World Bank Experience, 1980–93

The Aga Khan Support Program: A Third Evaluation

OED SERIES

OPERATIONS EVALUATION DEPARTMENT

ENHANCING DEVELOPMENT EFFECTIVENESS THROUGH EXCELLENCE AND INDEPENDENCE IN EVALUATION.

The Operations Evaluation Department (OED) is an independent unit within the World Bank; it reports directly to theBank’s Board of Executive Directors. OED assesses what works, and what does not; how a borrower plans to run andmaintain a project; and the lasting contribution of the Bank to a country’s overall development. The goals of evaluationare to learn from experience, to provide an objective basis for assessing the results of the Bank’s work, and to provideaccountability in the achievement of its objectives. It also improves Bank work by identifying and disseminating thelessons learned from experience and by framing recommendations drawn from evaluation findings.