quality at entry in cy99 a qag...

QUALITY AT ENTRY IN CY99 A QAG ASSESSMENT

June 7, 2000

QUALITY ASSURANCE GROUP

46140

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

ACRONYMS

AFR Africa Region APLs Adaptable Program Loans CMU Country Management Unit CPIA Country Policy and Institutional Assessment DO Development Objective EA Environmental Assessment EAP East Asia and Pacific Region ECA Europe and Central Asia Region EMT Energy, Mining and Telecommunications Department ESSD Environmentally and Socially Sustainable Development Network FPSI Finance, Private Sector and Infrastructure GEF Global Environmental Facility HDN Human Development Network IRIS Integrated Records and Information Services ISG Information Solutions Group LACI Loan Administration Change Initiative LCR Latin America and the Caribbean Region LILs Learning and Innovation Loans M&E Monitoring and Evaluation MNA Middle East and North Africa Region MP Montreal Protocol PAD Project Appraisal Document PSAL Programmatic Structural Adjustment Loan PTI Program of Targeted Interventions PREM Poverty Reduction and Economic Management Network QEA Quality-at-Entry Assessment QERs Quality Enhancement Reviews RSA Rapid Supervision Assessment SAL Structural Adjustment Loan SAP Systems, Applications, Products

TABLE OF CONTENTS Page

Executive Summary ........................................................................................................................... i I. Objectives and Approach.............................................................................................................1 II. Assessment Findings ...................................................................................................................4 A. Summary Findings..................................................................................................................4 B. Bank-wide Results ..................................................................................................................5 C. Results by Quality Dimension ................................................................................................8 D. Treatment of Safeguard Policies...........................................................................................13 E. Results by Region.................................................................................................................14 F. Results by Network/Sector ...................................................................................................17 G. Results by Instrument ...........................................................................................................19 H. Bank Inputs and Processes....................................................................................................20 I. Follow-up to QEA2 ..............................................................................................................23 III. Conclusions and Recommendations..........................................................................................24 Annexes 1. List of Operations in the Sample.................................................................................................27 2. Approach Paper...........................................................................................................................28 3. Main Contributors .......................................................................................................................34 4. Detailed Tables............................................................................................................................35 5. Guidance Questionnaire ..............................................................................................................42 6. Environment Aspects ..................................................................................................................53 7. Poverty and Social Development Aspects...................................................................................57 8. Financial Management Aspects ..................................................................................................62 Figures 1. Bank-wide Results.........................................................................................................................5 2. Trends in Quality at Entry.............................................................................................................7 3. Principal Objectives ......................................................................................................................7 4. The QEA3 Quality Diamond.........................................................................................................8 5. The Quality Dimension Results: 1997 and 1999...........................................................................9 6. Performance on Poverty Impact..................................................................................................10 7. Application of Safeguard Policies in Investment Projects ..........................................................13 8. Results by Region........................................................................................................................14 9. AFR+ECA vs. Rest of the Bank..................................................................................................15 10. Results by Network ................................................................................................................17 11. Results by Sector....................................................................................................................18 Text Boxes Box 1 QEA1 and QEA2 Revisited ....................................................................................................3 Box 2 Highly Satisfactory Operations...............................................................................................6 Box 3 Poverty and Social Development: Elements of Best Practice ..............................................11 Box 4 MNA’s Quality Risk Assessment Process............................................................................16 Box 5 Improving the Quality of the Africa Region’s Operations ...................................................17 Box 6 Strengths and Weaknesses by Sector....................................................................................19 Box 7 Peer Review at its Best: China Renewable Energy Development Project............................22 Box 8 QERs – Using QAG Methodology to Enhance Quality .......................................................24

EXECUTIVE SUMMARY

As part of its efforts to help enhance the quality of the Bank’s operational work, QAG assessed the quality at entry of a random sample of 80 operations approved during CY99. The sample covered 28% of the 280 operations approved during the year, spread over 39 CMUs and 12 Sectors. Main Findings

• Overall quality at entry improved for the third consecutive year. In line with the target of the Strategic Compact, nine out of ten operations and 93% of dollar commitments were rated satisfactory or better. Improvements were strongest on financial aspects and institutional analysis, and in the use of participatory processes in project design.

• Improved performance was noted in five of the six Regions and three of the four Networks. There was a convergence of results among Regions, with even the lowest performer reaching an 80% satisfactory or better level. In contrast to 1998, there were no differences in quality as a result of project size, length of project preparation, the timing of Board approval or country income level.

• Less satisfactory performance is increasingly a localized phenomenon. Two CMUs and one Sector representing about 15% of the QEA3 sample account for 60% of the less than satisfactory cohort. This suggests that the effort to eliminate all sub-standard projects can be focused for targeted improvements.

• The share of highly satisfactory projects grew in 1999 from 10 to 17 percent. Particularly noteworthy was the large share of APLs, GEF and Montreal Protocol operations in this category (accounting for 7 of the 13 highly satisfactory operations in the sample).

• While overall quality results were encouraging, three areas in particular remain problematic and require concerted attention: poverty and social impact analysis, risk analysis and monitoring and evaluation. These three areas were rated satisfactory in only 50% to 70% of operations. High-quality poverty analysis is critical to the Bank’s mission. So too are good M&E mechanisms to track performance and impact. The quality of risk analysis has not improved over the three years of the review.

• Greater attention to managing quality-at-entry has clearly paid off. Some Regions such as LCR have consistently strong results and others such as MNA have shown dramatic improvements over the past three years; their success is due in large measure to unique organizational and process approaches to quality. Throughout the Bank, sector managers are taking more direct responsibility for quality assurance. Nevertheless, Bank inputs and processes were still judged deficient in one out of five cases.

Quality at Entry Assessment in CY99 Page ii

Conclusions and Recommendations

Operational quality-at entry has improved steadily during the period of the Strategic Compact. This has been achieved at a time when the requirements for project preparation have become more stringent and funds for project preparation have become constrained. Task teams and managers should be commended for balancing these dimensions so well. At the same time, one out of ten projects is not yet satisfactory and only one in six is highly satisfactory. The Bank must continue to strive for consistently excellent performance in a difficult environment. This suggests the following agenda for the coming year:

A. Consolidate the Progress in Quality Assurance: To sustain the gains made over the past three years in quality assurance:

� The peer review system needs to be tightened with more independent selection of reviewers and systematic attention to their comments and to task team responses;

� Sector and country managers must become even more directly involved in the oversight of quality; and

� Quality Enhancement Reviews (QERs), which have been found very useful by staff and managers, should be generalized to all high risk operations.

Work is already underway by Regions and Sector Boards to implement these recommendations.

B. Ensure Good Quality Across the Board:

� For Regions and Networks with sub-standard performance in 1999, quality assurance programs should be reinforced to ensure improved results in CY00 (Action Plans have already been prepared by AFR and ECA and are under preparation by the Finance Sector);

� The application of environmental, social, and fiduciary policies to LILs, adjustment and programmatic lending needs to be clarified (a memorandum clarifying the treatment of environmental, social and fiduciary aspects in adjustment operations was issued in June 2000; follow-up work on several dimensions is also underway to provide additional guidance to staff); and

� Adequate funding needs to be ensured for mandated fiduciary and safeguard compliance without depleting task budgets (being addressed as part of the FY01 budget guidelines).

C. Attack the Persistent Problems: Four areas have shown little improvement and need focused attention:

� Better guidance and incentives need to be provided to task teams to help address candidly country, project and sectoral risks (a high level Task Force is currently examining all aspects of the Bank’s risk management framework and is expected to make recommendations, inter alia, to improve the effectiveness of efforts at the project level);

� Monitoring and evaluation needs to be improved (an M&E Task Force initiated after RSA3 has just completed its work and will be presenting its recommendations to CODE in June 2000);

� Poverty and social impact analysis needs to be strengthened – operations should articulate better expected poverty and social impacts and, where relevant, provide

Quality at Entry Assessment in CY99 Page iii

mechanisms to maximize and monitor those outcomes. The PTI categorization should also be revisited (the Poverty Group in PREM is taking the lead on implementation of this recommendation); and

� To stem erosion of institutional memory, there is an urgent need to restore an appropriate system of records management (ISG and Regions are implementing an action plan to address this issue).

Some of these recommendations are budget neutral. However, others will involve significant incremental cost. Regions are allocating increased funding in FY01 for fiduciary and safeguard requirements. Generalization of QERs to high risk operations will have an estimated additional cost of about $1 million (excluding Task Team time), but should be off-set in part by savings such as the phasing-out of the peer review system, and more efficient project processing. The M&E Task Force is currently estimating the incremental annual costs of implementing their proposed action plan. Rebuilding the system of record management will be a multi-year task; ISG estimates the capital cost at $5 million for FY01 and provision has been made in the capital budget.

I. OBJECTIVES AND APPROACH

1. The Quality Assurance Group carried out its third annual Quality at Entry Assessment (QEA3) from September 1999 through March 2000. The assessment covered a random sample of 80 operations (Annex 1) drawn from 280 operations approved by the Executive Directors in calendar year 1999. This report presents the findings and recommendations emerging from the assessment. 2. Quality at Entry Assessments have three main objectives:

• Promote accountability for quality by providing senior management with indicators of progress on Bank-wide, Regional and Sectoral quality of operational work.

• Catalyze systemic change by improving understanding of the key determinants of operational quality and disseminating assessment findings to appropriate units in the Bank.

• Learn from experience and help identify examples of best practice to emulate. 3. The QEA3 assessments involved a two-stage process. In Stage I, operations were assessed in a “rapid” evaluation by panelists (including a “generalist”, a sector specialist, and specialized thematic reviewers for environmental, social/poverty and financial management areas) through reviewing all relevant documents and an interview with the Task Team. Stage I helped to identify outlier operations requiring more thorough examination. In Stage II, all such outlier operations were subject to in-depth reviews by customized panels (typically grouping 4-5 panelists including one or more technical specialists, an implementation specialist, a sector economist and a country specialist). All panel findings were discussed with the concerned staff and managers before being finalized. The assessment methodology used in QEA3 is broadly the same as used in the first two exercises (Annex 2, 1999 Approach Paper). This year, there were two significant enhancements to the methodology:

• Focus on Poverty. This year’s questionnaire contained more specific questions on a project’s poverty focus;

• Enhanced Consultation with Task Teams Prior to Assessment. In 1999, Panels interviewed the task teams prior to making an initial assessment. Last year, interviews took place only after the initial assessments. Specialized reviewers for environment, social dimensions and finance also participated in each interview.

4. As in the past, operations were assessed for overall likelihood of achieving their development objectives, based on quality along eight dimensions:

• Appropriateness of Project Concept, Objectives and Approach • Quality of Technical and Economic analysis • Treatment of Environmental aspects • Attention to Poverty and Social aspects • Attention to Financial Management aspects • Attention to Institutional Capacity • Readiness for Implementation, and • Analysis of Risks and Sustainability

5. All operations were also rated for the quality and cost-effectiveness of the related Bank inputs and processes.

Quality at Entry Assessment in CY99 Page 2

6. Individual operations were rated on a four-point scale:

Highly Satisfactory: The operation responds well to the client’s needs, has a high probability of meeting its development objectives, and complies with Bank policies and standards. In addition, the operation is a useful model, because it contains elements of best practice, is unusually innovative or is especially responsive to client circumstances.

Satisfactory: The operation responds well to the client’s needs, is likely to meet its development objectives, complies with Bank policies and standards, and involves risks commensurate with rewards.

Marginal: The operation can be rated satisfactory only if some significant deficiencies are corrected.

Unsatisfactory: Major redesign or a substantially different approach is necessary for the operation to be rated satisfactory.

7. A total of 109 senior staff, managers and consultants participated as panelists in the assessments. In addition, a total of 26 thematic specialists served as panelists focusing on the areas of environment, social and poverty aspects and financial management. A list of main contributors to this assessment is shown in Annex 3. Altogether 18 of the 80 sample operations were subject to in-depth reviews (this included five adjustment operations with complex issues that were taken up for in-depth reviews directly). Of the 13 Stage I operations that went into Stage II, three were upgraded from marginally satisfactory to satisfactory, two were downgraded from satisfactory to marginally satisfactory and one from marginally satisfactory to unsatisfactory, the other seven remained unchanged in terms of overall ratings. 8. Every effort has been made to ensure the QEA3 results are objective, credible and consistent. Nevertheless, in interpreting and using the assessment results, it is important to keep in mind the following:

• The QEA3 sample of 80 projects provides results with a high degree of confidence at the aggregate level (90% confidence level with ± 5% margin of error). The confidence drops significantly, however, at the disaggregated level, especially for smaller Regions and Sectors. Care should be taken in interpreting results for smaller cohorts.

• QEA methodology is not meant to provide a definitive judgment on any individual operation. Given the nature of the assessment (desk-based, with minimal borrower/beneficiary consultation) any operation-specific findings are to be seen primarily as an alert that more thorough scrutiny may be warranted; by itself, results from any single assessment are not adequate for staff/manager accountability.

• The Guidance Questionnaire has proven very robust for assessing the quality of investment projects. It is being used extensively in Quality Enhancement Reviews and other quality improvement efforts. It has proven to be somewhat less suited to reviews of adjustment operations, which have different analytical requirements. Moreover, appropriate standards for the treatment of certain aspects such as the environment and social analysis in adjustment lending is still being defined.


• The results of QEA3 are comparable to QEA2 even though certain questions were covered more rigorously than in the past, including environment, poverty analysis, management inputs and the impact of adjustment lending on the natural and social environment.

• Tracking of QEA1 and QEA2 (Box 1) confirms the robustness of results at the aggregate level.

BOX 1: QEA1 AND QEA2 REVISITED

How good are the QEA ratings as predictors of results on the ground? While a definitive answer must await ICR and OED evaluations some 5-6 years down the line, as an interim proxy, we have revisited the CY97-98 cohort of 530 operations including the 210 operations in the QEA1 and QEA2 samples. Based on the Business Warehouse data as of May 1, 2000, the current status of this cohort is as follows:

Category No.DO Prob. Act. Prob. At Risk

A. QEA1+ 2 Samples Overall Ratings- Sat. or better 178 (85%) 11 20 24- Less than Sat. 32 (15%) 22 38 41

Subtotal QEA1+2 Samples 210 (100%) 12 22 26

B. Not in QEA1+2 Samples 320 7 16 20

Total CY97-98 Approvals 530 9 18 23

Current Status (%)

Four observations are worth noting:

• The aggregate QEA ratings are a good predictor of current portfolio performance, vis-à-vislikelihood of achieving development objectives.

• Operations rated less than satisfactory for quality at entry are indeed much more problem prone, notwithstanding the extra supervision attention that the QEA ratings may have prompted. However, considering the relatively large, albeit offsetting errors, QEA ratings are not a robust instrument for staff/managerial accountability at the individual project level.

• Differences between the sampled and non-sampled part of the cohort suggest that being part of the QAG sample may itself be leading to changes in staff behavior, including greater realism in PSR ratings.

• Detailed analysis of the underlying QEA data suggests some scope for improving robustness of the QEA methodology by assigning greater weight to two aspects of project quality--capacity of the implementing agencies and readiness for implementation. Sharpening tools for ex ante assessment of borrower ownership should also be helpful.


II. ASSESSMENT FINDINGS

A. Summary Findings

9. Based on the data from the sample questionnaires, reinforced by observations by the 18 in-depth panels, the main findings of QEA3 are as follows:

• The overall quality-at-entry continued to improve in 1999. Operations rated satisfactory orbetter reached 89%, compared to 86% in 1998, and 82% in 1997. There were more highly satisfactory operations than last year (17% vs. 10%). In terms of dollar commitments, 93% of the cohort was satisfactory or better as compared with 80% in 1998 and 92% in 1997. While statistically the QEA3 results are within the margin of error of results from previous assessments, the consensus of “veteran” panelists and the specialized reviewers is that the general quality of the cohort is markedly better than when the QEA program was launched three years ago.

• All Regions except EAP showed improvements over last year, and two Regions—LCR and MNA—had 100% satisfactory results. For LCR, it represents continued strong performance over three years, while for MNA it represents a major turnaround, from worst to best over the same period. There is also a notable convergence of results. In 1997, there was a 43% divergence between the best and poorest performing Regions. In 1999, the difference is only 20%.

• Less satisfactory performance is increasingly a localized phenomenon. Two CMUs and one Sector representing about 15% of the QEA3 sample account for 60% of the less than satisfactory cohort. This suggests that the effort to eliminate all sub-standard projects can be focused for targeted improvements.

• Three of the four Networks showed gains. At the sector level, Agriculture improved markedly. There were, however, two notable backsliders in 1999: Transportation and Finance.

• Three of the six GEF/MP projects were rated highly satisfactory. APLs had the best overall investment lending ratings, with 4 out of 11 considered highly satisfactory. Together, the APLs, GEF and Montreal Protocol (MP) operations account for over 50% of the highly satisfactory operations, i.e. twice their share of the overall sample.

• All but one of the 11 adjustment operations were rated satisfactory or better, although none were rated highly satisfactory. Dollar commitments were 93% satisfactory or better, the same as investment operations. However, attention to the potential negative environmental and social consequences of the reform program was felt to be inadequate in a number of the adjustment operations. Part of it was probably due to the lack of clarity on treatment of these aspects in adjustment lending.

• On quality dimensions, seven of the eight aspects reviewed showed improvements. Better results were noted generally on the financial and institutional aspects. Even though environment slipped slightly, it is due in large part to stricter criteria being applied.

• Safeguard and fiduciary compliance improved overall, but the goal of fully satisfactory compliance has not yet been reached; safeguard compliance was less than fully satisfactory in 11% of applications, although only 1% (2 operations) were unsatisfactory. Remedial action is being taken on both operations. On the other hand, some projects were found to have excessively zealous compliance on fiduciary aspects. Some LILs in particular had financial and procurement requirements of standard operations, which effectively defeated their purpose.


• Treatment of social development aspects improved significantly over QEA2 with 80% rated satisfactory or better on social dimensions. Particularly notable was the increased use of stakeholder participation in project design. About two-thirds of all operations with potential poverty impact were found to have satisfactory poverty analysis and well-designed activities targeted at poverty; another 20% could have benefited from better presentation of poverty reduction linkages but in the rest of the operations (10), greater attention to poverty issues would most likely have led to changes in project design.

• The relatively weak dimensions of project quality were Risk Assessments and M&E. Project risks were under-stated in many instances (64% satisfactory) and monitoring and evaluation of overall project outcomes were often inadequate (78% satisfactory for Monitoring and 64% satisfactory for Evaluation).

• In contrast to 1998, there were no significant quality differences among operations as a result of project size, length of project preparation or date of Board presentation. There was also no significant difference as a result of country riskiness (using CPIA indicators as a proxy) or income levels.

• Despite improvement in the last two years, Bank inputs and processes are still judged deficient in one out of five cases. There are continuing problems with the institutional records’ management, and peer review process.

B. Bank-wide Results

10. The overall cohort of operations reviewed in 1999 improved over the previous year. Operations rated satisfactory or better improved to 89% of the total, compared to 86% in 1998 and close to the target of 90%. In terms of commitments, the results are even better, with 93% satisfactory or better compared to 80% last year when a few large operations had poor ratings. As indicated in Figure 1, there were more projects rated highly satisfactory and fewer rated marginally satisfactory or unsatisfactory.

FIGURE 1: BANK -WIDE RESULTS% Satisfactory or better

A. By Number of Operations B. By Commitments

12 1017

7076

72

18 14 11

0

20

40

60

80

100

Percent

Highly sat. Sat. Marg/Unsat.

QEA1 QEA2 QEA3

13 198

79

61

85

9

20

7

0

20

40

60

80

100

Percent

Highly sat. Sat. Marg/Unsat.

QEA1 QEA2 QEA3

11. The highly satisfactory operations are listed in Box 2 together with the elements of excellence for which they were noteworthy. The average loan size of these operations was roughly one-half that of the overall portfolio, suggesting they were especially focused. Marginally satisfactory


operations typically suffered from lack of experience. Several operations had inexperienced task team leaders without adequate management support. Other operations were in countries or sectors where the Bank itself was inexperienced. Often these translated into unrealistic goals, inadequate assessment of institutional capacity and underestimated project risks. Several operations were also sent to the Board prematurely for various “country relationship” reasons. Two CMUs and one Sector representing about 15% of the QEA3 sample account for 60% of the less than satisfactory operations, suggesting that the bulk of the quality deficiencies are localized problems.

BOX 2: HIGHLY SATISFACTORY OPERATIONS

Albania Community Works x xArgentina Water Sector Reform (APL) x x x xArmenia Dam Safety x x x xBangladesh Municipal Services x x x x x x x xBenin First Decentralized City x x x x x x xBhutan Rural Access Roads x x x x xChina Production Sector (MP) x x x x x x x xChina Renewable Energy Dev. (GEF) x x x x xGhana Second CommunityW&S (APL) x x x x x x xIndia AP Power APL I x x x xMexico Rural Dev. APL II x x x x x xNepal Road Maintenance & Dev. x x x x x x xPoland Rural Environ. Protection (LIL/GEF) x x x x

12. Unlike 1998, this year’s results show no patterns related to elapsed time of project preparation, timing of Board presentation or loan size (Annex 4). There was also no significant difference as a result of country riskiness (using CPIA indicators as a proxy) or income level. 13. The 1999 results continue a trend of steady improvement in quality-at-entry over the past decade, as shown in Figure 2.


FIGURE 2: TRENDS IN QUALITY AT ENTRY % Satisfactory or better

55

66 67

7882

86 89

0

20

40

60

80

100

Per

cent

FY91* CY93* FY96* FY97(I)* CY97 CY98 CY99

* ECON Studies

14. In 1999, for the first time the Guidance Questionnaire asked Review Panels to identify the extent to which operations addressed selected objectives on a 4-point scale ranging from High to Negligible. One purpose was to use panelists’ judgement for making a reality check on the gap, if any, between stated objectives and those addressed by the operations. The questions were also intended to assess whether operations with multiple significant objectives were more likely to be rated lower than those with a single major objective. 15. The results showed the following principal objectives for the 80 operations reviewed:

FIGURE 3: PRINCIPAL OBJECTIVES

% of Operations Institutional Development 78 Economic or Sector Policy Reform 68 Infrastructure Development 55 Poverty Reduction 48 Environment 27 Other 3

16. As might be expected, operational objectives varied significantly among Regions (see Annex 4). Poverty alleviation was an over-arching objective in IDA-financed projects. Almost 80% of operations in Africa and 50% in South Asia had poverty reduction as the principal goal. Infrastructure investment was the major objective in middle income countries, while environment was most frequently cited as the objective in higher income countries. Institution-building and policy reform were almost universal—found in more than two out of every three projects, reflecting the recognition that institutions and good policies matter fundamentally in every country, regardless of income.


17. The analysis of the number of objectives did not reveal any clear outcomes about the relationship between complexity and quality. Operations with more than one significant objective were no more likely to be less than satisfactory than operations with a single significant objective. These results may reflect more on the somewhat imprecise nature of the question asked than on inherent link between project complexity and quality-at-entry (see Annex 3). C. Results by Quality Dimension 18. Projects in the QEA process are assessed along eight dimensions of project quality and Bank inputs and processes (see Guidance Questionnaire in Annex 5). Figure 4 provides a composite profile of the QEA3 sample along these dimensions. The outer diamond represents what the profile would be if the Bank were to achieve 100% quality on each of the nine dimensions. The middle diamond represents the current average of all QEA3 operations rated satisfactory or better overall, while the innermost diamond depicts the profile of QEA3 operations rated less than satisfactory overall. Apart from environmental and financial management aspects, the innermost diamond is significantly different from the middle diamond on each of the quality dimensions. 19. There are several implications of these results. First, even among operations rated satisfactory or better, there is scope to do better. Second, less than satisfactory operations are not hard to identify. They show deficiencies on almost every dimension. Third, environmental and financial aspects rate well even in marginally satisfactory operations, suggesting that good practices in those areas are becoming mainstreamed.

FIGURE 4: THE QEA3 QUALITY DIAMOND % Satisfactory or better on each quality dimension

20

50

30

60

20

10

1010

87

90

81

91

91

92

92

92

100

71

R1

R2

R3

R4

R5R6

R7

R8

R9

Overall Satisfactory or Better Overall Less than Satisfactory

R 1 : T he P roje c t C onc e pt , O b je c t ive s a nd A ppr oa c h R 6 : Ins tit ut io na l C a pa c ity A na lys is R 2 : Te c hnic a l a nd E c o n om ic A spe c ts R 7 : R e a dine ss for Im p le me nta t ion R 3 : Env iro nme n ta l A spe c ts R 8 : R isk A sse ssm e nt a nd Susta ina b ility R 4 : P ove rty a nd S oc ia l A spe c ts R 9 : B a nk Inp uts a nd P roc e sse s R 5 : Fina nc ia l M a na ge m e nt A spe c ts


20. Figure 5 compares results for all operations between 1997 and 1999 along all dimensions of quality.

FIGURE 5: RESULTS BY QUALITY DIMENSION 1997 AND 1999 % Satisfactory or better on each quality dimension

8591

80

8895

90

71

80 77

89

74

83

76

82

72 73 75

81

0

20

40

60

80

100

Per

cent

R1 R2 R3 R4 R5 R6 R7 R8 R9

QEA1 QEA3

R 1 : T he P roje c t C onc e pt , O b je c t ive s a nd A ppr oa c h R 6 : Ins tit ut io na l C a pa c ity A na lys is R 2 : Te c hnic a l a nd E c o n om ic A spe c ts R 7 : R e a dine ss for Im p le me nta t ion R 3 : Env iro nme nta l A spe c ts R 8 : R isk A sse ssm e nt a nd Susta ina b ility R 4 : P ove rty a nd S oc ia l A spe c ts R 9 : B a nk Inp uts a nd P roc e sse s R 5 : Fina nc ia l M a na ge m e nt A spe c ts

21. What stands out is the overall improvement during the two years on all quality dimensions, with the exception of Environment. The quality of Institutional Analysis improved by 9 percentage points. Social and Poverty Aspects also improved by 9 percentage points and Financial Management by 12 percentage points. 22. Environmental Aspects. The apparent decline in the overall satisfactory ratings for environmental aspects reflect two factors—a stricter assessment in 1999 due to the participation of environmental specialists on every project and the application of environmental consideration explicitly to adjustment operations for the first time in QEAs. The QEA3 results suggest that the EA process for investment lending is now largely mainstreamed within the preparation and appraisal process. However, 3 of the 11 SALs reviewed were rated marginally satisfactory on environment, because the Panels judged that the Bank ought to have been more cognizant of potential impacts on the environment of the policy agenda, including privatization, energy reform, port rationalization (see Annex 6 on Environment Aspects). The panelists assessments also revealed a diversity of views on what constitutes appropriate treatment of environmental issues and suggested that current policy (as articulated in OD8.60 and OP4.01) may not be widely understood. To help clarify current operational policy in this area, Management has issued an Operational Memorandum summarizing existing policy on adjustment lending, which also covers current policy in the environmental area; follow-up work on several dimensions is also underway to provide additional guidance to staff in adjustment lending.1

1 These issues will be taken up in the Adjustment Lending Retrospective, and feed into the conversion of OD8.60 into OP8.60, scheduled for FY01.


23. Social and Poverty Aspects. The treatment of social and poverty aspects registered significant improvement in CY99. About 80% of operations were rated satisfactory or better for social dimensions, compared to 67% in QEA2. Particularly noteworthy was the growth in the use of participatory process in design of investment operations. In contrast, 6 of the 11 adjustment operations were rated less than satisfactory for participation and stakeholder analysis. Generally speaking, operations were better at identifying the social and poverty issues than at diagnosing problems and designing appropriate actions. About one-third of the operations were deficient in analysis of potential adverse impacts of projects on the poor, and almost half were less than satisfactory in specifying potential outcomes and monitorable indicators. The Africa Region, with the majority of the Bank’s poverty-targeted projects also had the lowest rating for social aspects (63%) while ECA (70%) was also below average. 24. As shown in Figure 6, poverty issues were relevant to 74 of the 80 operations in the cohort. A total of 50 operations (including 7 adjustment operations) out of these 74 demonstrated significant and well-designed activities aimed at poverty reduction; another 14 could have benefited from better presentations of poverty reduction linkages, but in at least 10 operations, greater attention to poverty issues would most likely have led to changes in project design. The review suggests that more guidance is needed on the analysis and design of poverty interventions. With the introduction of countrywide PRSPs, the focus should be on what needs to be done at the project-specific level in terms of impact assessment, analysis, design of interventions and M&E mechanisms. Some elements of best practice in treatment of poverty and social development aspects are outlined in Box 3. 25. A total of 26 operations were formally categorized as Program of Targeted Interventions (PTI). The analysis also shows that being a PTI did not increase the quality of the poverty analysis. Roughly one-third of both PTI and non-PTI operations had less than satisfactory poverty analysis. A number of panelists and task managers questioned the utility of the PTI classification. With 90% of projects found to have some potential poverty impact, the utility of tracking specific operations through the PTI mechanism may have disappeared.

80 operations 6 NA on Pov. Impact

74 operations w/Pov Impact

33 Satisfactory 15 Less than satisfactory17 Satisfactory 9 Less than satisfactory

26 PTI 48 Non-PTI

FIGURE 6: PERFORMANCE ON POVERTY


BOX 3: POVERTY AND SOCIAL DEVELOPMENT - ELEMENTS OF BEST PRACTICE

The panelists found the following elements of best practice in treatment of poverty and social development:

• Comprehensive poverty analysis which disaggregates poverty impacts and follows poverty reduction strategies aimed at enhancing opportunities, empowerment and security of the poor (e.g. Sierra Leone CRRP, Samoa Infrastructure Asset Management).

• Formulation of social development outcomes based on social analysis and participation and linked to the project’s development objectives (e.g. Poland Rural Environment LIL).

• Attention to potential adverse impacts on livelihoods and social services and clear mitigation measures (e.g. India AP Power APL).

• Organization and institutional analysis including the “rules of the game” governing decision-making and resource allocation (e.g. Indonesia SSAL).

• Institutionalized mechanisms for participation in project implementation and monitoring with greater transparency and accountability (e.g. Côte d’Ivoire Pilot Literacy Project, West Bank & Gaza CDP II).

• Systematic monitoring and evaluation of social outcomes with clearly defined performance indicators and feedback mechanisms (e.g. Mexico Rural Development APL).

26. The review found that only 27 of the operations explicitly addressed gender issues. At least another 16 could have benefited from explicit treatment of gender, i.e. more than one-third of the projects that could have benefited from gender analysis neglected to do so. There appears to be a need to further disseminate the Bank’s policy on “The Gender Dimension of Development” (OP 4.20), issued in October 1999, and to provide guidelines to ensure that country teams have adequate awareness and understanding of the provisions of the new gender policy. 27. Risk Management. Development is by nature a risky business and QAG has gone to considerable length to ensure that panels do not penalize operations where the risks are commensurate with the returns. And yet, unlike other quality aspects that have shown steady improvement, there has been little progress over the past three years in the treatment of risk management and the scores are now well below the Bank’s average for other quality dimensions.

28. This year’s panels have again identified the absence of candid treatment of risks and measures to manage them as a central issue. The lack of candor is especially noticeable when considering that such risky undertakings as institutional development, economic and sector reforms, and poverty reduction are the principal objectives of many of this year’s cohort. And yet, only three investment operations of the 69 in the cohort were recognized explicitly as High Risk in the PADs.

29. While use of the logical framework, which was intended, inter alia, to help with risk assessment, is now being routinely followed on most projects, as yet this tool has not had an impact on the quality of risk assessment. Having failed to properly identify and quantify the risks, there is little scope for designing appropriate measures for dealing with them or exploring alternative approaches to minimize them. This failure to correlate risks with project design options, particularly during the conceptual stage of a project, is a serious weakness of current risk management practices in the Bank. As noted by one panel: “A major risk is inherent in the proposed rapid devolution of program implementation responsibility to the provinces, given the severe capacity constraints and the difficulty of inducing good staff to move to provincial headquarters. Yet there is little reference to this risk in the PAD.” Another panel reports: “The


PAD table on risks appears to be an after thought and consequently it does not adequately inform management of the risks inherent in the design of this project.”

30. Improving the quality of projects at entry will clearly require a better assessment of risks. The link between risk assessment and quality at entry is highlighted by the fact that nine of the ten projects rated less than satisfactory this year were also found to have less than satisfactory rating for risk assessment. To help address country, project and sector risks candidly, task teams need to be provided with better guidance, perhaps through a more explicit methodology for risk assessment. A Risk Management Task Force is currently examining all aspects of the Bank’s risk management framework and is expected to make recommendations, inter alia, to improve the effectiveness of efforts at the project level.

31. Monitoring and Evaluation. There is no discernible improvement in this year’s M&E results compared with the previous two assessments. A deficiency identified by a number of panels is the failure to reach agreement with government on arrangements for M&E before Board presentation. This failure is being construed as reflecting government’s ambivalence toward M&E and a reluctance to incur the added costs associated with introducing and operating a proper system. Another shortcoming is the absence of appropriate and clearly defined monitoring indicators focusing on impact and outcome instead of just inputs and outputs. Arrangements for evaluating the impact and measuring the outcome are generally less satisfactory than arrangements for monitoring implementation. Of particular concern is the failure to ensure adequate arrangements for monitoring and evaluating the poverty and social impacts on about half the operations where poverty is considered to be an important objective. 32. Among Networks, shortcomings are mainly found in HDN and PREM. More than one in four projects in these two networks received less than satisfactory rating for both monitoring and evaluation, and just one project out of 33 in the two networks was judged highly satisfactory on both aspects. By contrast, the percentage of projects with highly satisfactory arrangements for M&E was 33% in ESSD and 26% in FPSI. Furthermore, just 3 projects out of 47 in these two networks had ratings for M&E that were less than satisfactory. A breakdown by regions shows shortcomings with monitoring arrangements to be mainly concentrated in AFR and ECA (more than 30% of the projects rated less than satisfactory). On the other hand, all regions had at least 25% of the projects with less than satisfactory arrangements for evaluation. An M&E Task Force (initiated after RSA3) has just completed its work and will be presenting its recommendations to CODE in June 2000. 33. Financial Management. The quality of financial management arrangement in projects approved in CY99 was judged satisfactory or better in 89% of projects compared to 77% in CY98. There are also a number of encouraging signs of change. Overall, the progress achieved in 1999 is significant. However, four key issues on financial management arose during the QEA3 process: (i) the need to link financial arrangements to an assessment of country and project risks, (ii) consistency in approach to meet minimum acceptable standards; (iii) tailoring financial requirements to project scale and complexity, and (iv) the treatment of financial issues in adjustment operations. 34. Financial arrangements should take project risks into account. Where the perceived level of corruption is high, the mitigation strategy should be laid out clearly. 35. While attention to financial management has increased dramatically in the past two years, and the number of financial specialists has more than tripled (from 24 to 91), the requirements regarding financial assessments and mandatory project sign-off are still interpreted differently


among Regions and need to be applied more consistently to ensure adherence to minimum acceptable standards. 36. Financial management arrangements should be appropriate for the scale and complexity of projects. Getting the right balance is not always easy. At one extreme, a $2.5 million LIL had a 14-page financial management analysis of LACI-compliance. On the other hand, a $575 million project (with Bank financing of $90 million), covering most of a sector in a country could have benefited from a more explicit analysis of past experience and strategies to address known problems. The Financial Management Board is completing a set of guidelines for small projects such as LILs. 37. While all 11 adjustment operations in the QEA3 cohort meet the Bank’s minimum requirements, two of them could have been significantly improved by greater consideration of risks in the design of the operation. Additional guidance for financial management in adjustment operations is currently being developed.

D. Treatment of Safeguard Policies

38. The Bank’s experience with each of its 10 safeguard policies was reviewed in QEA3, with the following results:

FIGURE 7: APPLICATION OF SAFEGUARD POLICIES IN INVESTMENT PROJECTS

Policy Number of

Applications Number

Marginally Satisfactory Number

Unsatisfactory

Environmental Assessment 58 5 - Natural Habitats 19 2 - Involuntary Resettlement 18 3 1 Pest Management 13 1 1 Indigenous Peoples 10 - - Forestry 8 - - Projects in International Waters 5 1 - Cultural Property 3 1 - Dam Safety 3 - - Projects in Disputed Areas 3 - - TOTAL 140 13 2

39. The results indicate that safeguards are now widely addressed in Bank operations. Three-quarters of the investment projects reviewed were subject to environmental assessment, and other safeguard policies were applied in 82 instances, with broadly satisfactory results. In only 9% of the applications was the treatment found to be less than satisfactory, and in only 2 cases—one each for pest management and involuntary resettlement, was there failure to address the safeguard aspects—with a rating of unsatisfactory for non-compliance. In both cases remedial action is being taken as a result of the QEA3 review. 40. Adjustment lending is not subject to the safeguard and fiduciary requirement of investment lending. They are expected to ensure appropriate consideration of environmental impacts of the policy reform. The treatment of environmental and social aspects was considered marginally satisfactory in about one-third of the 11 operations. The appropriate application of safeguard policies to adjustment lending has just been clarified (see paragraph 22).


E. Results by Region

41. As indicated in Figure 8, all Regions except EAP showed improved performance in 1999.

FIGURE 8: RESULTS BY REGION % Satisfactory or better

58

82 84

92

100

88 89

73

80

100 96 100

57

75

100

8883

8882

8689

0

20

40

60

80

100

Perc

ent

AFR EAP ECA LCR MNA SAR BANK

QEA1 QEA2 QEA3

42. Two aspects were particularly notable. First, two Regions—LCR and MNA—had 100% satisfactory results. This continued the tradition of LCR with the best overall results for all three years. (In fact, only one LCR project out of 54 has been rated less than satisfactory). For MNA, this represented a turnaround from poorest performer in 1997, to best performer in 1999. Second, there is a continuing convergence among Regions. In 1997, the difference between the top and bottom performers in overall satisfactory ratings was 43%. In 1999, the difference had closed to only 20% --with lowest overall performance at 80% compared to 100% for the two best. Despite this converging trend, two Regions, Africa and ECA, are performing below the level of the other four Regions on each of the quality dimensions (see Figure 9).


FIGURE 9: AFR+ECA VSREST OF THE BANK (ROB) % Satisfactory or better for each quality dimension

84

88

90

88

94

93

88

94

91

72

59

72

77

82

86

87

87

67

R1

R2

R3

R4

R5R6

R7

R8

R9

ROB AFR+ECA

R 1 : T he P roje c t C onc e pt , O b je c t ive s a nd A ppr oa c h R 6 : Ins tit ut io na l C a pa c ity A na lys is R 2 : Te c hnic a l a nd E c o n om ic A spe c ts R 7 : R e a dine ss for Im p le me nta t ion R 3 : Env iro nme nta l A spe c ts R 8 : R isk A sse ssm e nt a nd Susta ina b ility R 4 : P ove rty a nd S oc ia l A spe c ts R 9 : B a nk Inp uts a nd P roc e sse s R 5 : Fina nc ia l M a na ge m e nt A spe c ts

43. What accounts for LCR’s consistent performance and for MNA’s improvement? There are several factors—client capacity, staff experience, and perhaps most significant, the unique structure of the country departments in LCR. Each CD has four sector leaders, one for each Network. The sector leaders are responsible for ensuring operations conform to CAS priorities, meet the requisite standards of excellence, and serve as safeguard against “supply driven” projects. They ensure adequate management attention is devoted to project quality. 44. MNA’s improvements can also be traced to focused management attention in the past year. Embarrassed by the results in 1997 and 1998, the MNA team adopted an intensive screening system to identify the “quality risk” in projects. As described in Box 4, each new project entering the pipeline is examined for potential risks on quality—special borrower circumstances, inherent complexity of the problem and staff experience. Depending on the outcome of the review, the project is assigned to one of four different preparation review processes, ranging from normal peer review/internal reviews to an external Quality Enhancement Review (QER). The results are reviewed by senior management every six months. In addition, MNA has launched a customized “quality training” for its staff/managers.


BOX 4: MNA’SQUALITY RISK ASSESSMENT PROCESS

After two consecutive years of sub-par results, the MNA Region embarked in April 1999 on a comprehensive Quality at Entry Action Program. A key feature of the program is a systematic “quality at entry risk” assessment for every project in the pipeline. Managers in sector units evaluate risk factors from a country, sectoral, project or project team perspective and validate these with the Country Director. Project quality risks are categorized as low, medium or high. Each project is then assigned by the Sector Unit a specific review process for project preparation depending on the level of risk. Processes used vary between units but can be grouped under four categories:

• ER (Enhanced Peer Review): Sector leaders nominate or screen peer reviewers and ensure their comments are recorded and dealt with. The Regional Director of Quality assurance participates periodically especially by bringing the best quality peer reviewers from other Regions.

• PR (Panel Review): For medium-risk projects, the ER process is supplemented by an in-depth review by one or two independent specialist staff or consultants prior to the PAD Review. Their review includes an assessment of the document quality and readiness of the project for implementation.

• NR (Network Review): For projects with special Network interests (e.g. technical innovation), the ER process is supplemented by a Network review prior to project appraisal. Where Network review is not possible, an informal review would take place within the Region.

• Quality Enhancement Review: For high-risk or especially interesting operations, the Region organizes a formal QER with help from the concerned Network and QAG.

The process encourages appropriate attention especially to technical design and analysis issues and the assignment of the right task manager and teams early in the project preparation cycle. The result: MNA had 100% satisfactory quality-at-entry in 1999.

45. More recently, other Regions have tightened quality review processes. The Africa Region, for example, has announced new procedures for improving the quality of operations (Box 5). EAP has already reviewed its two projects rated marginally satisfactory in 1999 and concluded that they were essentially "orphans" in terms of the managerial attention they received, compounded by relatively inexperienced Task Management and the speed with which they were delivered. EAP has adopted new procedures for the review and risk rating of proposed projects, similar to those in MNA, which should help to avoid such outcomes in future. And ECA has embarked on an intensive quality at entry review to ensure at least 90% of its projects in CY00 are satisfactory or better.


BOX 5: IMPROVING THE QUALITY OF THE AFRICA REGION’SOPERATIONS

Improvement of operational quality in the Africa Region has been substantial and steady, but the Region’s quality at entry performance continues to lag behind rising Bank-wide averages. The Region is taking steps to close the gap. The Region has reinforced its quality support apparatus and recently merged its Operations Support and Knowledge and Learning Groups, with a view to better incorporate best practices and lessons into operational work. The Region is implementing an action plan to improve the capacity, effectiveness and coordination of units involved in the environmental, social and poverty, and institutional aspects of operations. The Region is also setting up a special team to ensure compliance with the Bank’s safeguard policies. To support Sector Managers in their quality management function the region will increase the amount of time for quality assurance activities at the disposal of sector managers and lead specialists and fund this extra time out of a Regional account. The Region is also taking steps to increase the accountability for quality through the OPE process, the performance award system and specific follow-up of QAG assessments.Regarding capacity development, the Region is launching, with network support, a large-scale training and learning sharing program aimed at providing support to task teams. The Region has been a heavy user of the recently introduced "on demand" program of quality enhancement reviews (QERs).

F. Results by Network/Sector

46. Three of the four Networks showed improvement in 1999, as shown in Figure 10. Most notable changes this year were ESSD’s rather dramatic improvement and FPSI’s decline. ESSD’s improvement was led by the agriculture sector (improving from 64% last year to 100% satisfactory rating this year). FPSI’s decline was a result of deterioration in Transportation and Finance/Private Sector Development.

FIGURE 10: RESULTS BY NETWORK % Satisfactory or better

83 79

100

8490

83

69

85 8790 89

92

0

20

40

60

80

100

Per

cent

ESSD FPSI HD PREM

QEA1 QEA2 QA3


The sample sizes under QEA3 are not large enough to draw robust conclusions at the sectoral level for most sectors. Combining the QEA3 results with those of QEA1 and QEA2 provides, however, a useful, longer-term perspective on comparative performance of different sectors (Figure 11). Box 6 depicts comparative strengths and weaknesses of sectors measured in terms of significant (approximately 10%) difference from Bank-wide averages on different questions as assessed by QEA panelists.

FIGURE 11: RESULTS BY SECTOR (QEA1+2+3) % Satisfactory or better

82

94

81 81 84

100

85

73

85 87 88 88

0

20

40

60

80

100

Per

cent

Agric

Env

EMT

Fin&

PSD

Transp

Urban

WS&

SEd

uc

Health

Soc Pr

otM

ulti

PSM


BOX 6: STRENGTHS AND WEAKNESSES BY SECTOR

Strengths Weaknesses Agriculture Stakeholder participation

Attention to social impacts Financial management systems First year program readiness Assessment of project risks

Environment Clarity of objectives Appropriateness of project approach Use of sector knowledge Stakeholder participation

Impact evaluation

Education Poverty focus Consideration of alternative designs Institutional analysis M&E arrangements Risk assessment and mitigation

Health Borrower ownership Stakeholder participation

Appropriateness of Project approach Learning from experience Clarity of objectives Appropriateness of project conditionality Peer review process

Social Protection Learning from experience Use of sector knowledge Poverty targeting Risk assessment

Stakeholder participation

Finance Clarity and realism of objectives Appropriateness of project approach Adequacy of sector knowledge Stakeholder participation M&E of social and poverty impacts Peer review process

Transportation Learning from experience Clarity of objectives Risk assessment Peer review system

Financial management systems

EMT Clarity and realism of objectives Appropriateness of project approach

Stakeholder participation Attention to social impacts Capacity of executing agencies Risk assessment and mitigation

Urban Appropriateness of project approach Use of sector knowledge Economic rationale Institutional analysis M&E arrangements Risk assessment and mitigation Peer review system

Financial management systems

Public Sector Management

Appropriateness of project conditionality

Use of sector knowledge Disaggregation of social impact (e.g. by gender) Stakeholder participation Risk assessment and mitigation

G. Results by Instrument

47. As in 1998, the newest lending instruments—LILs and APLs—performed well. APLs accounted for 4 out of the 13 projects rated highly satisfactory although they constituted only 14% of the cohort. No LILs or APLs were less than satisfactory. They had 100% satisfactory ratings on the critical dimensions of (i) project concept, objectives and approach, and (ii) economic and


technical analyses, suggesting that they were particularly well-conceived solutions to problems. APLs were the only instrument with 100% satisfactory ratings on institutional aspects, a critical factor in long term development effectiveness, and LILs were alone with 100% for risk analysis, again appropriate for an instrument designed explicitly to mitigate risk. However, regarding APLs, more attention needs to be paid to the definition of specific triggers for the subsequent phases. In the case of one large APL the in-depth panel highlighted the lack of “sufficient precision” in the definition of “triggers” in the PAD and Legal Agreement as one of the key weaknesses of that operation. 48. Projects financed by the Global Environmental Facility and the Montreal Protocol also had above-average results. Three of the six GEF/MP operations were rated highly satisfactory. These projects typically have very clear and explicit objectives and systematically adequate resources for project preparation. 49. Adjustment Operations. The 11 adjustment operations accounted for 49% of the sample in terms of dollar commitments. All but one of those operations were rated satisfactory or better,although none were rated highly satisfactory. In terms of dollars, 93% of the commitments were for satisfactory or better operations, an improvement over 79% of the previous year's sample. 50. The adjustment operations often pose a dilemma to country teams - the need to balance client responsiveness with adequate treatment of risk. QAG reviews of the last two years have identified risk management to be inadequate in a number of adjustment operations. Specifically, with the short time available between tranches, the ability to adapt the operation to changing circumstances can often be substantially compromised. This issue should be the subject of further thought and guidance to task teams. While Country teams/Regional management should remain ultimately responsible for striking an appropriate balance between client responsiveness, project scope, and implementation concerns, institutional mechanisms in the Bank should be strengthened to ensure the tradeoffs are considered and the resulting risks are explicitly discussed and evaluated at an early stage. 51. An operation-by-operation approach to reforms has been found to suffer from an inherent disadvantage that the reforms supported are those that happen to be achievable at the particular time that the operation is processed. The recently introduced Programmatic SAL (PSAL) is designed specifically to support capacity building and institutional reforms. Such a framework that provides a clear and predictable roadmap should be explored more systematically. 52. As indicated in paragraph 22, the appropriate treatment of environmental and social aspects in adjustment operations is not well understood by Task Teams. Consequently, about one-third of adjustment operations were found to be marginally satisfactory in assessing the potential negative impacts of policy reform in these areas. A memorandum clarifying current policy requirement has just been issued by Management.2

53. The overall results and the debate surrounding certain quality dimensions have raised questions of whether the current QAG Guidance Questionnaire is adequately adapted to adjustment operations. SALs have different analytical and processing requirements than investment operations. The questionnaire will be adapted further for adjustment operations before the next round.

2 These issues will be taken up in the Adjustment Lending Retrospective, and feed into the conversion of OD8.60 into OP8.60, scheduled for FY01.


H. Bank Inputs and Processes

54. The moderate improvement in this year’s quality at entry occurred despite a slight decline in the Bank Inputs and Processes. This seemingly incongruous result is a reflection of panels "raising the bar" on management performance by rating “Inputs and Processes” less than satisfactory in all but two cases where the management performance was rated less than satisfactory. In the panels’ view, a more active involvement by management could have raised the overall quality of these operations. This judgment can be illustrated by an example from the findings of a Stage II panel. This panel rated an APL operation as overall satisfactory but found the management’s performance and the Bank Inputs and Processes marginally satisfactory. In the panel’s judgment the less than satisfactory rating was justified because the managers failed to ensure that the operation adhered to the regional management’s directives for phasing of the APL, establishing clear milestones for each phase of the APL, and preparing a supervision plan.

55. Nine out of the ten projects in this year’s cohort with less than satisfactory rating overall, plus seven other projects with an overall satisfactory rating, were rated less than satisfactory on the Bank Inputs and Processes. A close symmetry exists between the regional ratings of this parameter and the overall ratings, with ECA and Africa having a satisfactory rating of less than 75 percent, and the other regions ranging between 88 and 100 percent satisfactory rating.

56. Use of Resources. The adequacy of budget resources for project preparation is a perennial issue. The growth in requirements in a constrained budgetary environment has led to the view that the average Bank project preparation budget of about $300,000 is inadequate. The recent Task Force on meeting the Bank’s fiduciary and safeguard requirements agreed, indicating that an additional $5 million is needed for project preparation alone. The QEA3 results do not provide a clear picture whether budget resource constraints have influenced quality at entry. Stage II panelists took the greatest in-depth look at budget resource questions. They found that inadequate budgets were not a critical constraint to delivering good quality on operations (although there is no specific question on the adequacy of budgets, only on their effective use). By contrast, the highly satisfactory outcome for 50% of GEF/MP operations was attributed by panelists, in part, to the availability of ample resources for preparation.

57. Management Contribution. This year's results have again confirmed the wide spread belief that active management input to quality assurance translates into higher quality at entry, especially where the task team is inexperienced in relation to the operation's complexity. Major findings include:

• SAR and EAP have the highest percentages of projects with highly satisfactory contributions to quality by management (38% and 30% respectively) while MNA and LAC have almost no instances where the management contribution was less than satisfactory.

• The percent of projects considered to have received less than satisfactory quality assistance from management in the Africa and ECA regions was 42% and 25% respectively.

• Many sector managers have assumed a more active role in quality enhancement with promising results and at least two of the Networks (ESSD and HDN) have appointed dedicated quality assurance and/or compliance staff.

At least one region and a number of sector managers throughout the Bank have started using a formal process to evaluate the strength of the task team in relation to the operation's complexity and the capacity of the borrower. There is clearly scope for more managers adopting this or a similar system.


58. Peer Review Process. Panels continue to voice concerns about the efficacy of the peer review system. This year’s finding show that:

• The quality of guidance given by peer reviewers is deemed adequate (it now stands at 90% satisfactory rating).

• There is evidence of failure to make appropriate use of reviewers’ advice (only 83% satisfactory). All but one of the ten projects rated less than satisfactory failed to make satisfactory use of peer advice.

The Africa Region recently issued instructions requiring staff to attach to all invitations to PCD and PAD/Decision meetings (as well as minutes) a record of advice sought and given and action taken as a result, with an explanation of why any advice has not been used. In addition, a number of units in the Bank are supplementing the peer review system with a QER process and all regions are increasingly using QERs for complex and difficult projects. 59. Regions should give serious consideration to systematically supplementing the current peer review system with QER for high risk operations. Irrespective of which quality enhancement process is used, all regions should adopt a similar requirement to that introduced by the Africa region for recording the advice received (see Box 7 for an example of “best practice” for recording such advice).

BOX 7: PEER REVIEW AT ITS BEST: CHINA RENEWABLE ENERGY DEVELOPMENT PROJECT

The China Renewable Energy Development Project is an excellent example where the use of peer reviews not only influenced the quality of the project design and analysis but was recorded explicitly to aid in project processing and for eventual evaluation of the effectiveness of the advice.

As a joint GEF/World Bank operation, the project review process functioned under GEF rules which include a broad inter-agency review process as well as a review by an expert external to the Bank who is on a Roster maintained by the GEF. The GEF procedures also require that all reviewers’ comments be submitted in writing and the task team’s response formally recorded. Their comments were clustered around nine different subject areas (such as capacity building, market for PV systems, etc). Their comments and the team’s response on each item were summarized in a matrix (of five pages) circulated prior to the Appraisal Review meeting to help explain the rationale for the project concept and approach as appraised.

The external expert reviewer functioned as an independent “quality assurer”, participated in project pre-appraisal and provided comments on project assumptions, designs, integrity, individual components, documentation clarity and quality. While there was some initial confusion and anxiety about his role, the overall result was an improved project and is replicable, according to the task team. The perspectives of the independent quality assurer and the task team were recorded and attached to the peer review matrix.


60. Project Documentation. Many task teams complained that the PAD format makes it difficult to present a coherent description of the project’s basic story line: rationale, content and approach. Echoing this complaint, several panelists expressed unhappiness with the inadequacy of the project documentation (particularly the PAD) for making meaningful judgment on the operations’ substantive content and any assessment of the quality assurance process. Problems concerning documentation highlighted over the past two years include:

• Inadequate explanation of how go/no go decisions were taken, and on what basis, and what type of guidance was provided by country management.

• Supporting documents such as PIPs, intended to replace the SAR, are incomplete or not available in time.

• In the long run, inadequate documentation, including records management, could seriously affect the Bank's ability to administer operations (particularly APL and programmatic operations).

61. Despite these complaints and shortcomings, there were just as many operations with highly satisfactory rating for quality of project documentation as there were operations with less than satisfactory documentation. Inadequate documentation is particularly prevalent in the ECA region (8 out of the 20 regional projects were assessed less than satisfactory). In the EAP region on the other hand, nearly half the projects were judged to have highly satisfactory documentation, though even here there were four projects with less than satisfactory documentation. These results indicate that where regions and task teams pay attention to documentation, satisfactory results can be obtained even with the PAD format. Task teams should refer to operations with highly satisfactory rating for documentation with a view to improve the substantive presentation of operations in the PAD and to ensure that key documents are catalogued and available to current and future task members.

62. In order to improve the quality of project documentation, ISG and OCS are focusing on providing task teams with necessary tools and templates to facilitate the creation of an electronic project file. This file would contain links to documents generated internally as well as those received externally from implementing agencies and partners. This work is proceeding under the Integrated Records and Information System (IRIS) program and will be rolled out commencing in FY01.

I. Follow-up to QEA2

63. Quality Enhancement Reviews. The QEA2 Report recommended adoption of a number of measures to further enhance project quality. The most explicit was the suggestion that the Regions, in collaboration with the Networks, strengthen quality enhancement prior to appraisal to help overcome weaknesses in the peer review system and management oversight. This has led to the introduction of Quality Enhancement Reviews (QERs). 64. QERs are “on-demand” technical assistance reviews to task teams to suggest ways of improving the identification and preparation of specific projects using the QAG methodology (see Box 8). As of March 2000, 40 QERs had been completed and another 14 were being planned. Feedback from the participants has been very positive and suggests the mechanism is particularly useful when carried out early in the project cycle by staff primarily from outside the Region and under well-defined terms of reference covering the specific issues for which the task team is seeking guidance. Next year’s QEA4 exercise will explicitly track the quality-at-entry of projects


that have had QER reviews (there was only one such project in this year’s sample, and it was rated satisfactory).

BOX 8: QERS – USING QAG METHODOLOGY TO ENHANCE QUALITY

Feedback from Bank staff and managers after the first Quality at Entry exercise in CY97 suggested the QAG methodology could enhance quality if it were provided “on demand” to task teams during project identification and preparation.

A pilot program of Quality Enhancement Reviews (QERs) was carried out in FY99 and expanded in FY00. Customized panels of three to five persons with different expertise spent one to two days reading documentation, interviewing staff and borrowers, and accessing information from inside and outside the Bank to make suggestions for improving project quality. The QERs, which were voluntary, normally took place between the Project Concept Document stage and negotiations when design options are still open.

The benefits of the QERs so far have been:

• Inputs from a diversified group of experts from inside and outside the Bank lead to insights and advice that add value and have led to modifications in project design and analysis, changes in lending instruments, and specific measures to increase beneficiary impact.

• Having a QER panel work full-time on issues of importance to task teams produces results, even if the QER is of limited duration. Some QERs have been as short as half a day.

• The fact that the QER is voluntary and the initiator provides the terms of reference for the panel increases the chance that good advice will be incorporated into project design.

• Envisaged as a one-time enhancement, project teams are beginning to experiment with reassembling the panel for short periods of time at different points in project preparation and implementation, sometimes as a substitute for the peer review system.

• The broad representation of staff and managers on QER panels is adding to the “quality culture”in the Bank.

III. CONCLUSIONS AND RECOMMENDATIONS

65. The preceding analysis suggests the Bank has made genuine progress towards better quality under the Strategic Compact. Overall operational quality-at-entry continues to improve. Management attention to quality has become a recognized, regular business responsibility. The treatment of fiduciary and safeguard aspects of projects has improved substantially. Sub-standard performance is increasingly a localized phenomenon. At the same time, certain important project design elements are not done consistently well and the increasing burdens on task teams threaten to overwhelm this progress. While mindful that the ultimate quality of projects remains the Borrower’s responsibility, the challenges for the Bank’s own work in the coming year are to:

• Consolidate the gains achieved in quality assurance;

• Ensure good quality is achieved in every operational unit; and

• Tackle the continuing areas of analytical deficiency.


66. Specifically, the agenda for quality at entry in the coming year should include the following: A. Consolidate the Progress in Quality Assurance:

• The peer review system should be tightened whereby peer reviewers are selected independently of the task team, submit their comments in writing and the task team responses are also recorded in writing;

• Sector and country managers must become even more directly involved in the oversight of operational quality; and

• Quality Enhancement Reviews (QERs), initiated after QEA2, which have been found very useful by staff and managers, should be generalized to high risk operations.

Work is already underway by Regions and Sector Boards to implement these recommendations.

B. Ensure Good Quality Across the Board:

• The Regions and Networks with sub-standard performance in 1999 should reinforce their quality assurance programs in CY00. They should look specifically at the approaches adopted by LCR and MNA (Action Plans already prepared by AFR and ECA and are under preparation by the Finance Sector);

• The application of environmental, social and fiduciary policies to LILs, adjustment and programmatic lending needs to be clarified (a memorandum clarifying the treatment of environmental, social and fiduciary aspects in adjustment operations was issued in June 2000; follow-up work on several dimensions is underway to provide additional guidance to staff)3;

• Adequate funding for fiduciary and safeguard compliance is needed without depleting task budgets for other critical analyses. The FY01 budget guidelines require Regions to allocate increased funding for fiduciary and safeguard compliance. Close monitoring of these guidelines’ implementation is needed.

C. Attack the Persistent Problems. Four areas have shown little improvement and need focused attention:

• Better guidance and incentives need to be provided to task teams to address candidly country, project and sectoral risks (a high level Task Force is currently examining all aspects of the Bank’s risk management framework and is expected to make recommendations, inter alia, to improve the effectiveness of efforts at the project level);

• Monitoring and evaluation needs to be improved (an M&E Task Force initiated after RSA3 has just completed its work and will be presenting its recommendations to CODE in June 2000);

• Poverty and social impact analysis needs to be strengthened. Operations should articulate better expected poverty and social development outcomes and, where relevant, provide mechanisms to maximize and monitor those outcomes. The PTI categorization should

3 These issues will be taken up in the Adjustment Lending Retrospective, and feed into the conversion of OD8.60 into OP8.60, scheduled for FY01. Work is also underway to take stock of and assess the Bank’s experience with programmatic investment lending, and to explore further the appropriate application of safeguard and fiduciary policies to such approaches.


also be revisited. (The Poverty Group in PREM is taking the lead in helping implement this recommendation); and

• To stem erosion of institutional memory, there is an urgent need to restore an appropriate system of records’ management (ISG and Regions are implementing an action plan to address this issue).

Some of the above recommendations are budget neutral. However, others will involve significant incremental cost. Regions are allocating increased funding in FY01 for fiduciary and safeguard requirements. Generalization of QERs to high risk operations will have an estimated additional cost of about $1 million (excluding Task Team time), but should be off-set in part by savings such as the phasing-out of the peer review system, and more efficient project processing. The above-mentioned Task Force on M&E is currently estimating the incremental annual costs of implementing their proposed Action Plan. Rebuilding the system of records’ management will be a multi-year task. ISG estimates the capital cost at $5 million for FY01 and provision has been made in the capital budget.

quality at entry in cy99 a qag...

Documents