ZAMBIA : NATIONAL RURAL WATER SUPPLY AND

SANITATION PROGRAM 1. BASIC INFORMATION

a. Basic project data

Project title: National Rural Water Supply and Sanitation Program

Project code: P-ZM-E00-009 Instrument number(s): 2100150013198

Project type: Public Operation Sector: Water And Sanitation

Country: Zambia Environmental categorization (1-3) : 2

Processing Milestones Key Events Disbursement and Closing date

Date approved: 27/112008 Cancelled amount: Original disbursement deadline:

31/03/2012

Date signed: 22/12/2008 Supplementary financing: NA Original closing date:31/12/2011

Date of entry into force :

29/01/2009

Restructuring: nA Revised disbursement deadline:

30/11/2015

Date effective for 1st

disbursement: 29/01/2009

Extensions (specify dates):

31/12/2011 to 30/06/2013

30/06/2013 to 31/12/2014

31/12/2014 to 31/03/2015

Revised closing date: 30/11/2015

Date of actual 1st : 18/04/2009

b. Financing sources

Financing source/ instrument

(MUA)

Approved amount

(MUA) :

Disbursed amount

(MUA) :

Percentage disbursed

(%):

Loan: ADF 12,063,968.00 12,063,968.00 80.43%

Grant:

Government:

Other (ex. Co-financiers):

TOTAL : 12,063,968.00 12,063,968.00 80.43%

Co-financiers and other external partners: -

Execution and implementation agencies: Ministry of Local Government and Housing

c. Responsible Bank staff

Position At approval At completion

Regional Director - Kennedy Mbekeani (OIC)

Sector Director Andrianarison Rakotobe Mohamed El Azizi

Sector Manager Sering Jallow Osward Chanda

Task Manager Mecuria Assefaw Herbert Chinokoro

Alternate Task Manager

PCR Team Leader Alex Gomani

PCR Team Members Herbert Chinokoro

d. Report data

PCR Date : 8/12/2015

PCR Mission Date: From: 26/10/2015 To: 3/11/2015

PCR-EN Date: 26/06/2018

Evaluator/consultant : Daniel VAN ROOIJEN Peer Reviewer/Task Manager: Ram Janakiram

PCR EVALUATION NOTE FOR PUBLIC SECTOR OPERATIONS

2. PROJECT DESCRIPTION Summary from Appraisal Report including addendum/corrigendum or loan agreement, and taking into account any modification that

occurred during the implementation phase.

The project entitled “National Rural Water Supply and Sanitation Program” was developed to support the

government of Zambia to improve water supply, sanitation and hygiene in rural areas of Zambia. The project

consisted of 4 major components; 1) improving access to water supply, improving access to sanitation and

hygiene, 3) support to improve the performance of the Rural Water Supply and Sanitation sub-sector, and 4)

programme management. The project agreement became effective in January 2009. The initial completion date of

31 December 20011 was extended three times with a final revised completion date of 30 November 2015.

a. Rationale and expected impacts:

The project rationale was that investments to improve rural access to water supply, sanitation and hygiene

practices combined with institutional capacity strengthening of the RWSS sector will reaching the MDG goal for

water as well as improving health and quality of live, and reducing poverty of the rural population of Zambia. The

expected impact from the project was 3-fold; 1) improved access to water and sanitation, 2) reduced infant and

maternal mortality, and 3) reduced incidence of worm infestation and diarrheal diseases among infants and

children in rural communities. The people expected to benefit from the project are the total rural population of

Zambia, all local authorities working in the RWSS sector and schools and associations, the private sector and

development agencies. The project deadline was extended 3 times during the execution phase, adding about 3

years to the planned duration of 5 years.

The main problems addressed by this project were (as stated in the appraisal report):

1. The water supply and sanitation infrastructure in many of the rural areas in Zambia were dilapidated;

2. Ineffective operation and maintenance leading to a large number of dysfunctional water points throughout the

rural parts of Zambia

3. The capacity of the water supply production and sanitation systems were inadequate to meet the demand of

the existing population and to cater to future growth of Zambia;

4. Low priority and declining budgetary allocation to rural water supply in the past were major constraints to

ensure the provision of secure water supply to the population

5. Un-coordinated government and donor projects often led to duplication of efforts

6. Over concentration of activities in a few provinces due to donor preferences for particular districts or

provinces

The expected impact from the project was 3-fold; 1) improved access to water and sanitation, 2) reduced infant

and maternal mortality, and 3) reduced incidence of worm infestation and diarrheal diseases among infants and

children in rural communities. The people expected to benefit from the project are the total rural population of

Zambia, all local authorities working in the RWSS sector and schools and associations, the private sector and

development agencies

b. Objectives/Expected Outcomes:

The main objective envisaged in the program was to provide equitable and sustainable access to water supply and

sanitation to meet basic needs for improved health and poverty alleviation for Zambia’s rural population and

contribute towards achievement of MDG for water and sanitation of respectively 75% and 60% coverage by 2015.

The specific objectives of the programme were to: 1) increase access to safe, reliable and convenient quantities of

water supply from present 37% to 55% of the rural population by 2010 and to 75 percent by 2015, 2) to increase

access to proper sanitation from 13% to 33% of the rural population by 2010 and to 60 percent by 2015. The

program also had comprehensive gender objectives aiming to increase participation women in trainings and

increasing the proportion of women in construction as well as operation and maintenance of water supply and

sanitation infrastructure.

The long term expected outcomes are:

- Improved access to water and sanitation

- Reduced infant and maternal mortality, and;

- Reduced incidence of diarrhoea diseases and worm infestation in infants and children in rural

communities.

The expected outcomes in the medium term are:

- Increased proportion of the rural population with adequate and convenient supply of clean water;

- Increased number of rural families having constructed their own improved sanitation facilities and

effectively using them;

- All rural schools with access to adequate sanitation facilities;

- Improved awareness and attitudes on environmental cleanliness and personal hygienic practices;

- Effective RWSS planning and programming leading to equitable, transparent and efficient allocation of

resources;

- Rural Communities and LAs are gender responsive;

- The profile of the WSS sector raised in national planning and allocation of resources and increased

stakeholder participation and buy in into the RWSS programme, and;

- Improved functionality of rural water supply facilities.

c. Outputs and intended beneficiaries:

The programme has the following 8 outputs:

1. 1,585 new boreholes constructed;

2. 655 existing boreholes rehabilitated;

3. 10 water schemes reticulated;

4. 272 hand dug wells constructed;

5. 871 public sanitation facilities constructed;

6. improved capacity of MLGH and Districts in order to manage RWSS more effectively;

7. 24,400 VWASHE members trained, and;

8. 448 area pump menders trained.

The programme aims at providing safe water supply and sanitation in all rural areas of Zambia to people who are

currently lacking safe access. Other beneficiaries included the central and local government institutions will be

enhanced, the artisans trained and assisted to establish repair outlets for water supply and sanitation systems, and

the private sector. The prioritization and selection criteria for districts/provinces, took into account the baseline

access level of water supply and sanitation, population size and poverty indicators, such as infant mortality rate.

The Luapula Province and Northern Province were selected, based on their low coverage of water supply and

sanitation and worst poverty rates, relative to Zambia’s other provinces. Within these two provinces fifteen

districts were selected for programme intervention.

d. Principal activities/Components:

The appraisal report and PCR lists the following four components and associated activities:

(i) Water Infrastructure Development consisting of the following activities:

Rehabilitation, improvement, and new construction of rural water schemes;

Provision of rural sanitation for communities and public institutions

(ii). Sanitation and Hygiene Education

Community management training and capacity building for planning and management of water supply and

sanitation facilities and sanitation and hygiene education.

(iii).Institutional Support and Capacity Building

Develop and/or revise policies guidelines, tools, checklists;

Capacity building: human resources development, organization and operational capacity strengthening and

organizational systems improvements, gender mainstreaming and HIV/ AIDS mitigation at national and

district levels

Planning, support and supervision through technical support (TA) teams and NGOs

(iv).Program Management

Develop and maintain a comprehensive RWSS Information Management System (IMS);

Establish water sector indicators and benchmarks which include gender specific indicators;

Collect and analyze data and information generated for NRWSSP operations;

Develop O&M system and spare parts supply chain;

Program management capacity at national and district levels.

3. PROJECT PERFORMANCE ASSESSMENT

RELEVANCE

a. Relevance of the project development objective: The project’s development objectives, to improve access to water supply and sanitation and hygiene practices, is considered

very relevant with regards to the governments goals with regards to reducing poverty and improving health outcomes of the

rural population of Zambia. The project scope fits well with strategies and priorities by the Bank (e.g. water policy on

integrated water resources management). It is also well embedded into the water vision and sector strategies, principles and

policies by the Government of Zambia. The vision in Zambia’s Water policy states: “To optimally harness water resources

for the efficient and sustainable utilisation of this natural resource to enhance economic productivity and reduce poverty.” In

addition, the programme is aligned with the water act and other sector strategies.

The evaluator concurs with the PCR rating for this criterion and considers it as satisfactory (3).

b. Relevance of project design (from approval to completion):

According to the PCR the program was well designed to meet the objective of assisting the Borrower to improve

access for the rural population to safe, reliable and convenient quantities of water supply of quality meeting

national standards and to increase the access of the rural population to improved sanitation and improved health

& hygiene practices. The program components did reflect this well as they were all necessary and sufficient to

achieve the objective; and took into account lessons learnt from the past projects in order to enhance its impact on

the population. The main design elements included:

(i) formulation of the program through a participatory approach that involved representatives of all

stakeholders (women, communities, NGOs, private sector, local governments, Cooperating Partners

(CPs) and central government) and

(ii) formulation of the NRWSSP in line with the national strategies of the FNDP

(iii) the adoption of joint sector review mechanism for monitoring sector performance to reduce government

transaction costs, and

(iv) Adoption of a common operations and maintenance system and supply chain for spare parts.

(v) The community-based approach for implementation, and O&M was expected to contribute to

sustainability by creating community ownership.

The programme was designed in a straightforward and logic manner, consisting of separate components for water

supply, sanitation and hygiene promotion, institutional sector strengthening, and programme management. The

design is relevant with regards to the impact it is expected to make. From literature it is known that improvements

of access to water supply and sanitation has direct positive outcomes for health and poverty reduction. The

programme also follows a clear course based on previous interventions in the water and sanitation sector and

includes a clear gender focus. The multiple extensions granted during implementation phase would question the

quality of planning at project design phase. The substantial under-estimation of unit-costs for construction of

water supply schemes raises the question on what basis the initial calculations at programme design were done.

The risks assessment is adequate and the risks considered in the appraisal report are realistic and inclusive.

Environmental and social protection measures were anticipated through the development of an environmental and

social review template for the sub-project to be developed under the programme. Environmental protection

measures are centred on the protection of water sources from pollution and preventing the development of

breeding grounds for mosquitoes, flies and other insects at the water and sanitation sites. The appraisal reports

mentions the special attention to gender objectives as part of social protection, however there is no information

stating how priorities of vulnerable groups (such as the poor, elderly, disabled people, and children) are

considered in the programme or its sub-components. There is no clear indication that a pro-poor design was used.

The evaluator concurs with the PCR rating for this criterion and considers it as satisfactory (3).

EFFECTIVENESS

c. Effectiveness in delivering outputs:

Based on the available information, it can be concluded that the project delivered with mixed results on its 8

intended outputs. While for some outputs, the target was met or exceeded, there are a few critical outputs on

water supply and public sanitation that did not perform as planned.

In the PCR, the justification in the narrative section of output 3 on construction of water schemes gives

inadequate sensitization about available water supply technologies as the prime reason why the target was not

met. However, it is not explained or justified why sensitization was inadequate. Under output 4, construction of

hand-dug wells, there is a lack of clarity on the exact reasons why just 44% of the target was met. Similar for

Output 5 on construction of public sanitation facilities, whereby less than 70% of the target was met, there is a

lack of information on the reasons why the target was not met, despite a number of project extensions. The

training components of the programme seems to have delivered well against the targets.

This criterion on effectiveness in delivering outputs is considered unsatisfactory (2) which is lower than the

PCR rating of satisfactory. The reviewer agrees with this revised rating of unsatisfactory (2).

d. Effectiveness in delivering outcomes:

The effectiveness in delivering results at the outcome level is mixed. An important outcome of the project is the

people benefiting from access to improved water supply and sanitation infrastructure and services. However,

there is a lack of information on the methods used to calculate the number of beneficiaries. The outcome level

indicators are shown as national access rates in %. Besides the national level outcome reporting, outcome 3 and 4

show progress at the regional level in the two regions of intervention (Luapula Province and Northern Province),

which is considered very useful to demonstrate actual progress made in these two regions which can be attributed

to the ADB-funded programme. There is insufficient detail providing reasons why access to public sanitation

facilities was not fully achieved against target. Also, there is no indicator on the number of people benefiting from

household sanitation, as a result of the use of the CLTS approach. Behavioural change and sensation as well as a

better quantification of capacity strengthening at the individual staff level could have been captured at the

outcome level reporting, for example through the use of a baseline and end line survey.

This criterion on effectiveness in delivering on outcomes is considered satisfactory (3) which is the same as

the PCR rating.

e. Project development outcome:

Based on the information available, it is evident that substantial progress has been made with regards to

improving access to water and sanitation in the target areas (2 provinces) in which the programme has focussed. It

is however not clear to what extent improved health and poverty alleviation, as well as behavioural change with

regards to hygiene practices was realised for Zambia’s rural population. Despite the fact that RWSS units have

been established and X no. of members have been trained, there is a lack of evidence demonstrating that local

authorities have increased their capacity to manage RWSS more effectively. The evaluator rates this criterion

as unsatisfactory (2), which is lower than the PCR rating of satisfactory.

f. Beneficiaries:

There is insufficient information detailing the methods used to calculate the number of beneficiaries for

sanitation. The calculation of beneficiaries was clarified by the task manager during final review, as follows:

The national rural program gives the population served per water point which is 250 people/BH or hand dug well. Water

reticulated schemes serve 3000 people at each scheme.

New BHs 1857 *250 = 461,750

Rehab 607*250 = 151,750

Hnd dug wells 121*250 = 30,250

Schemes 5*3000 = 15,000

Total beneficiaries in 658,750 (of clean water supplies)

The PCR includes a gender figure, however “>50%” is not an exact figure. It would be useful to know the exact

figure for future reference and for comparison with other projects.

g. Unanticipated additional outcomes (positive or negative, not taken into consideration in the project

logical framework):

The evaluator would generally agree with the additional outcomes listed in the PCR. However, some of these

would have already been known at project design phase and should therefore be considered as anticipated. These

are dignity for girls through access to improved sanitation, hygiene promotion and health outcomes for children.

The successful use of CLTS is listed as an unanticipated additional outcome, however achievements in household

access sanitation appears not to have been quantified which is considered a missed opportunity. Some of the

unanticipated additional outcomes listed could have been better documented, quantified and used as success

stories of the programme. The inclusion of two negative outcomes is welcomed, as this helped to a more balanced

and realistic report.

EFFICIENCY

h. Timeliness:

The project agreement became effective in January 2009. The initial completion date of 31 December 20011 was

extended three times with a final revised completion date of 30 November 2015. The 3 years in additional to the

planned duration of 5 years resulted in a total project duration of about 8 years. The ratio of planned and actual

implementation time is 0.62, which is in the category of be (<0.75 and ≥0.50) resulting in a rating of

unsatisfactory (2) for this criterion.

i. Resource use efficiency:

The calculated proportion between the median physical implementation rate (95.65%) and commitment rate

(80.43%) is 1.19. The short explanation given in the PCR is considered sufficient. This corresponds to a highly

satisfactory rating (4), which is higher than the PCR rating for this criterion (3).

j. Cost-benefit analysis:

The economic rate of return at project approval stage was 28% compared to 21% upon project completion. The

calculated ratio between the ERR at completion and the ERR at appraisal is 0.75 which is within the category

giving a satisfactory score (ratio within category of <1 and ≥0.75). Overall, the evaluator generally agrees with

the calculations made to arrive at the ERR in Annex 10 of the appraisal report. The subsequent satisfactory rate

given satisfactory (3) which is higher than the PCR score (2).

k. Implementation progress:

The information made available reports of late submission of documents to the ADB, delays and unfinished

construction of water and sanitation infrastructure, etc. Field reports from the bank and external audit reports

show a large number of areas and examples in which the executing agency has filed to comply with covenants

and procedures. At various times during the implementation phase, some of the issues raised were, poor book

keeping record, lack of community involvement to ensure ownership, issues with tendering of works, poor

financial management, poor M&E, project staffing issues, and motor bikes not having insurance. The evaluator

would rate this criterion as unsatisfactory (2), which is lower than the PCR rating of satisfactory.

SUSTAINABILITY

l. Financial sustainability:

The PCR narrative for this section uses the experience with the V-WASHE Committees as good evidence of

financial sustainability. There seems to be a lack of systematic assessment of financial feasibility of water supply

schemes, for example through carrying out a willingness and capacity to pay for improved water services. The

positive example of Chipowe village is given, stating their relatively high contribution. Based on additional

information provided by the task manager, it is clear that the project experience in Chipowe village is similar to

other villages in other district where the programme has had an intervention. Based on additional information

provided by the task manager, it is clear the way local authorities manage their budgets to cover major

maintenance works (for costs exceeding ZMK 500). Only water supply is mentioned when it comes to financial

sustainability, however it is unclear what payment modalities exist for the sanitation facilities constructed at

public places such as market centres. The evaluator would rate this criterion as satisfactory (3) which is the

same as the PCR score of satisfactory.

m. Institutional sustainability and strengthening of capacities:

Based on the information available, staff in charge of running the water supply and sanitation systems in the two

provinces of intervention have received proper technical trainings in system operation and maintenance and have

been provided with tools and equipment. In addition there seems to be a good account of capacity building at the

level of local authorities whom seem to be better equipped to carry out their responsibilities. The D-WASHE

Committees, V-WASHE Committees and Area Pump Members are considered good entities to ensure community

participation at the local level. The evaluator rates this criterion as satisfactory (3) which is in line with the

PCR score.

n. Ownership and sustainability of partnerships:

The PCR narrative for this section raises a very valid point on the urgent need to ensure that communities are

sensitized on the multiple and long-term benefits of improved water supply and sanitation service delivery for

their lives. At least one of the audit reports mentioned vandalism in a number of cases which could be a sign of a

lack of community involvement and ownership. It will be essential for all development agencies working in the

WASH sector in Zambia, to use a common approach for interventions in the area of water supply, sanitation and

hygiene promotion. Zambia was the first African country in which the CLTS approach was successfully rolled

out, however it is very critical to coordinate efforts to maintain ownership and sustain associated benefits from

villages being free from open defecation. The role of media and civil society organisations can be instrumental.

The evaluator concurs with the PCR score and rates this section as satisfactory (3).

o. Environmental and social sustainability:

An environmental and social management plan has been developed for the programme and sets out the possible

environmental and social risks of the interventions. From the project appraisal report it can be noted that the

executing agency with support from the Bank has developed an Environmental and Social Streamlining Template

for Water and Sanitation to provide guidance for preparing the environmental and social impact assessment,

mitigation and monitoring at the local level, to be used during the design phase of sub-projects. These include

arrangements for water quality testing, consultations with the communities, the mobilisation of the various

committees, monitoring and arrangements for taking mitigation measures. The evaluator concurs with the PCR

score and rates this section as satisfactory (3).

4. PERFORMANCE OF STAKEHOLDERS

a. Bank performance:

The bank has overall performed satisfactory with regards to supervising this programme. The bank made use of

previous lessons learned from similar projects, for example the modality of using small contractors instead of

local authorities for construction was based on previous lessons learned. The Bank could perhaps have played a

stronger role in making sure that soft elements of interventions had been better captured in the monitoring and

evaluation framework, for example, capacity building and hygiene promotion and sensitization around benefits of

improved water supply.

The performance of the Bank is rated as Satisfactory (3).

b. Performance of the Borrower:

The Banks mission reports and external audit reports reflect a large number of issues during project

implementation which have negatively affected the performance of the borrower. Some of the issues are in the

areas of financial administration, reporting, staffing, supervision of consultants and recruitment of staff and

contractors. A number of critical activity areas could have been better utilised, for example gender mainstreaming

and sensitisation campaigns, use of mass media. There is a lack of evidence of sufficient measures put in place to

ensure financial sustainability of infrastructure and service put in place by the project. Based on the above, the

performance of the Borrower is rated as unsatisfactory (2) which is lower than the PCR score of

satisfactory (3). The reviewer concurs with the assessment made and the rating of unsatisfactory 2.

c. Performance of other stakeholders:

The performance of other stakeholders, being the contractors, is considered poor. The available information

clearly indicates that there have been large delays in implementation, the quality the works have been poor and

sometimes work was left unfinished. Based on the above, the performance of the Borrower is rated as

unsatisfactory (2), the same as the PCR score of unsatisfactory (2).

5. SUMMARY OF OVERALL PROJECT PERFORMANCE

a. Overall assessment:

The project’s development objectives, to improve access to water supply and sanitation and hygiene practices, is

considered very relevant with regards to the governments goals with regards to reducing poverty and improving

health outcomes of the rural population of Zambia. The programme was designed in a straightforward and logic

manner, consisting of separate components for water supply, sanitation and hygiene promotion, institutional

sector strengthening, and programme management. The programme also follows a clear course based on previous

interventions in the water and sanitation sector and includes a clear gender focus. Based on the information

available, it is evident that substantial progress was made with regards to improving access to water and sanitation

in the target areas (2 provinces) in which the programme has focussed.

The multiple extensions granted during implementation phase would question the quality of planning at project

design phase. The substantial under-estimation of unit-costs for construction of water supply schemes raises the

question on what basis the initial calculations at programme design were done. The risks assessment is adequate

and the risks considered in the appraisal report are realistic and inclusive. Special attention was planned to be

given to gender objectives as part of social protection, however there was no information stating how priorities of

vulnerable groups (such as the poor, elderly, disabled people, and children) are considered in the programme or

its sub-components. The Banks mission reports and external audit reports reflect a large number of issues during

project implementation which have negatively affected the performance of the borrower. There seems to be a lack

of systematic assessment of financial feasibility of water supply schemes and qualitative achievements such as

capacity building and hygiene promotion and sensitization have been insufficiently captured in progress

reporting.

Overall the evaluation considers the project satisfactory albeit with a lower average score (2.63) than the

PCR (2.88)

b. Design, implementation and utilization of the M&E (appreciation of the evaluator):

There is no evidence of a solid monitoring and evaluation framework in place including qualitative achievements

such as capacity building and hygiene promotion and sensitization.

Based on additional feedback from the task manager, it should be noted that the Bank was supporting specific

activities in the national program and not all activities. Development of the M&E framework was not part of it,

just providing M&E training was the scope as at that time DANIDA was providing support to the development of

an IMS system for rural water which was not completed and was not functional.

The gender reporting is not accurate, the “>50%” is not an exact figure and should have been specified. There is

also no evidence than any system that did exist was used. Overall score: 1.42

6. EVALUATION OF KEY LESSONS LEARNED AND RECOMMENDATIONS

a. Lessons learned:

The three lessons learned with respect to the programme are not all considered valid. The first lesson learned is

actually a recommendation. With regards to the second lesson learned; it is noted that despite trainings giving for

local authorities, there are still capacity constraints noted at the district and national level. This would raise the

questions, whether the contents of these trainings have been appropriate, to what extend these trainings given

were sufficient and if the right agencies and individual were targeted. The third lesson learned also contains a

recommendation which is reformulated and added under the section of recommendations.

The Evaluation’s opinion of each lesson is given in summary below:

1. NOT VALIDATED, REFORMULATED AS A RECOMMENDATION

2. VALIDATED

3. VALIDATED

The evaluator has no additional lessons learned to add.

b. Recommendations:

The four key-recommendations made in the PCR are all considered relevant and valid, however some are quite

project specific. In addition, the evaluator would recommend to have a better M&E system in place that captures

progress made in capacity strengthening, hygiene promotion with or without CLTS and additional dimensions of

access to water supply (distance, time, water quality, etc).

The Evaluation’s opinion of each recommendation is given in summary below:

1. VALIDATED

2. NOT VALIDATED, TOO PROJECT SPECIFIC

3. VALIDATED

4. NOT VALIDATED, TOO PROJECT SPECIFIC

The evaluator would like to add the following 2 recommendations:

1) There is need for improved coordination between the key stakeholders to avoid compromising the outcomes of

the national program of this size and importance (follows from third lesson learned).

2) The evaluator would recommend for future projects with a similar design to have a better M&E system in

place that captures progress made in capacity strengthening, hygiene promotion and additional dimensions of

access to water supply.

7. COMMENTS ON PCR QUALITY AND TIMELINESS The overall PCR rating is based on all or part of the criteria presented in the annexe and other: The quality of the PCR is rated as highly

satisfactory (4), satisfactory (3), unsatisfactory (2), and highly unsatisfactory (1). Le timeliness of the PCR is rated as on time (4) or late

(1). The participation of the Borrower, co-financier, and the bank’s external office(s) are rated as follows: Very Good (4), Good (3), Fair

(2), Poor (1).

The extent of quality and completeness of the PCR evidence and analysis to substantiate the ratings of the various

sections is considered satisfactory. The extent of objectivity of PCR assessment score is considered satisfactory.

The extent of internal consistency of PCR assessment ratings; inaccuracies; inconsistencies; is considered

satisfactory. The extent of identification and assessment of key factors (internal and exogenous) and unintended

effects (positive or negative) affecting design and implementation is considered satisfactory. The adequacy of

treatment of safeguards, fiduciary issues, and alignment and harmonization is considered satisfactory. The extent

of soundness of data generating and analysis process (including rates of returns) in support of PCR assessment is

considered satisfactory. The overall adequacy of the accessible evidence (from PCR including annexure and other

data provided is considered satisfactory. The extent to which lessons learned (and recommendations) are clear and

based on the PCR assessment is considered satisfactory. The extent of overall clarity and completeness of the

PCR is considered satisfactory.

The quality of the PCR is rated as satisfactory (3.00). The narrative and numeric scoring are generally consistent

with each other and sufficient explanation has been provided.

The timeliness of the PCR is rated as on time (4), since it was produced within 6 months after project closure.

8. SUMMARY OF THE EVALUATION This is a summary of both the PCR and IDEV ratings with justification for deviations/comments. Appropriate section of the PCR

Evaluation should be indicated in the last column in order to avoid detailed comments. The evaluator must provide a reasonable

explanation for each criterion the PCR rating is not validated by IDEV. Consequently, the overall rating of the project could be “equally

satisfactory”.

Criteria PCR PCREN Reason for disagreement/

Comments

RELEVANCE 3 3 Satisfactory.

Relevance of project development objective 3 3 Satisfactory.

Relevance of project design 3 3 Satisfactory.

EFFECTIVENESS 3 2 Unsatisfactory.

Development objective (DO) 3 2 Unsatisfactory. Based on the

information available, it is evident that

substantial progress has been made with

regards to improving access to water

and sanitation in the target areas (2

provinces) in which the programme has

focussed. It is however not clear to what

extent improved health and poverty

alleviation, as well as behavioural

change with regards to hygiene practices

was realised for Zambia’s rural

population. Despite the fact that RWSS

units have been established and X no. of

members have been trained, there is a

lack of evidence demonstrating that

local authorities have increased their

capacity to manage RWSS more

effectively.

EFFICIENCY 2.50 2.75 Satisfactory.

Timeliness 2 2 Unsatisfactory.

Resource use efficiency 3 4 Highly satisfactory. The calculated

proportion between the median physical

implementation rate (95.65%) and

commitment rate (80.43%) is 1.19

(which is is ≥1). The short explanation

given in the PCR is considered

sufficient.

Cost-benefit analysis 2 3 Satisfactory. The economic rate of

return at project approval stage was 28%

compared to 21% upon project

completion. The calculated ratio

between the ERR at completion and the

ERR at appraisal is 0.75 which is within

the category giving a satisfactory score

(ratio within category of <1 and ≥0.75).

Overall, the evaluator generally agrees

with the calculations made to arrive at

the ERR in Annex 10 of the appraisal

report.

Implementation progress (IP) 3 2 Unsatisfactory. The information made

available reports of late submission of

documents to the ADB, delays and

unfinished construction of water and

sanitation infrastructure, etc. Field

reports from the bank and external audit

reports show a large number of areas

and examples in which the executing

agency has filed to comply with

covenants and procedures. At various

times during the implementation phase,

some of the issues raised were, poor

book keeping record, lack of community

involvement to ensure ownership, issues

with tendering of works, poor financial

management, poor M&E, project

staffing issues, and motor bikes not

having insurance.

SUSTAINABILITY 3 3 Satisfactory.

Financial sustainability 3 3 Satisfactory. The PCR narrative for this

section uses the experience with the V-

WASHE Committees as good evidence

of financial sustainability. There seems

to be a lack of systematic assessment of

financial feasibility of water supply

schemes, for example through carrying

out a willingness and capacity to pay for

improved water services. The positive

example of Chipowe village is given,

stating their relatively high contribution.

Based on additional information

provided by the task manager, it is clear

that the project experience in Chipowe

village is similar to other villages in

other district where the programme has

had an intervention. Based on additional

information provided by the task

manager, it is clear the way local

authorities manage their budgets to

cover major maintenance works (for

costs exceeding ZMK 500). Only water

supply is mentioned when it comes to

financial sustainability, however it is

unclear what payment modalities exist

for the sanitation facilities constructed at

public places such as market centres.

Institutional sustainability and strengthening

of capacities 3 3 Satisfactory.

Ownership and sustainability of partnerships 3 3 Satisfactory.

Environmental and social sustainability 3 3 Satisfactory.

OVERALL PROJECT COMPLETION

RATING 2.86 2.69

Bank performance: 3 3

Borrower performance: 3 2 Unsatisfactory. The Banks mission

reports and external audit reports reflect

a large number of issues during project

implementation which have negatively

affected the performance of the

borrower. Some of the issues are in the

areas of financial administration,

reporting, staffing, supervision of

consultants and recruitment of staff and

contractors. A number of critical activity

areas could have been better utilized, for

example gender mainstreaming and

sensitization campaigns, use of mass

media. There is a lack of evidence of

sufficient measures put in place to

ensure financial sustainability of

infrastructure and service put in place by

the project.

Performance of other shareholders: 2 2 Unsatisfactory.

Overall PCR quality: - 3 Satisfactory.

9. PRIORITY FOR FUTURE EVALUATIVE WORK: PROJECT FOR PERFORMANCE

EVALUTION REPORT, IMPACT EVALUTION, COUNTRY/SECTOR REVIEWS OR

THEMATIC EVALUATION STUDIES:

- Project is part of a series and suitable for cluster evaluation

- Project is a success story

- High priority for impact evaluation

- Performance evaluation is required to sector/country review

- High priority for thematic or special evaluation studies (Country)

- PPER is required because of incomplete validation rating

Major areas of focus for future evaluation work:

a) Performance evaluation is required for sector/ country review

b) Cluster evaluation (institutional support)

c) Sector evaluation (budgetary support or public finance management reforms)

Follow up action by IDEV: Identify same cluster or sector operations; organize appropriate work or consultation mission to

facilitate a), b) and/or c).

Division Manager clearance Director signing off

Data source for validation:

Task Manager/ Responsible bank staff interviewed/contacted (in person, by telephone or

email)

Documents/ Database reports

Attachment:

PCR evaluation note validation sheet of performance ratings

List of references

Appendice 1

PROJECT COMPLETION REPORT EVALUATION NOTE Validation of PCR performance ratings

PCR rating scale:

Score Description 4 Very Good – Fully achieved with no shortcomings 3 Good – Mostly achieved despite a few shortcomings 2 Fair – Partially achieved. Shortcomings and achievements are roughly balanced 1 Poor – very limited achievement with extensive shortcomings

UTS Unable to score/rate NA Non Applicable

Criteria Sub-criteria PCR

work

score

IDEV

review Reasons for deviation/comments

RELEVANCE Relevance of the

project development

objective (DO) during

implementation

3 3

Satisfactory.

Relevance of project

design (from approval

to completion)

3 3

Satisfactory.

OVERALL RELEVANCE SCORE 3.0 3.0 Satisfactory.

EFFECTIVENE

SS*

Effectiveness in delivering outcomes

Outcome 1: Increased

access to water supply

in target area (%)

No

score 2

Target was not met. There is insufficient

information on why the calculation of costs of

water supply schemes

Outcome 2: Increased

access to sanitation in

target area (%)

No

score 3 Target almost met.

Outcome 3: No. of

additional people with

access to water in

target area

No

score 3 Satisfactory.

Outcome 4: No. of

additional people with

access to improved

sanitation from

constructed sanitation

facilities in schools,

health centers and

No

score 2

Details stating the reasons why access to public

sanitation facilities was not fully achieved

against target was later provided in written by

the task manager. The ‘Annex 1’ which contains

these reasons was not available to the evaluator

at the time of the evaluation.

Criteria Sub-criteria PCR

work

score

IDEV

review Reasons for deviation/comments

markets.

Outcome 5: % of

functional VWASHE

Committees with

Women holding key

positions

No

score 3 Satisfactory.

Outcome 6: Improved

functionality of rural

Water supply

facilities.

No

score 3 Satisfactory.

Effectiveness in delivering output

Output 1: 1,585 new

boreholes constructed

No

score 4

Highly satisfactory. Target was exceeded.

Output 2: 655 existing

boreholes

rehabilitated

No

score 3

Satisfactory. Target almost reached.

Justification considered adequate.

Output 3: 10 water

schemes reticulated

No

score 2

Unsatisfactory. Target not met. The justification

in the narrative section gives inadequate

sensitization about available water supply

technologies as the prime reason why the target

was not met. However, it is not explained or

justified why sensitization was inadequate.

Output 4: 272 hand

dug wells constructed

No

score 2

Unsatisfactory. Target not met. Also, there is a

lack of clarity on the exact reasons why just

44% of the target was met.

Output 5: 871 public

sanitation facilities

constructed

No

score 2

Unsatisfactory. There is a lack of information

on the reasons why the target was not met,

despite a number of project extensions.

Output 6: improved

capacity of MLGH

and districts in order

to manage RWSS

more effectively

No

score 3 Satisfactory.

Output 7: 24,400

VWASHE members

trained

No

score 3 Satisfactory.

Output 8: 448 area

pump menders trained

No

score 3 Satisfactory.

Development objective (DO)

Development

objective rating 3 2

Unsatisfactory. Based on the information

available, it is evident that substantial progress

has been made with regards to improving access

to water and sanitation in the target areas (2

provinces) in which the programme has focused.

Criteria Sub-criteria PCR

work

score

IDEV

review Reasons for deviation/comments

It is however not clear to what extent improved

health and poverty alleviation, as well as

behavioral change with regards to hygiene

practices was realized for Zambia’s rural

population. Despite the fact that RWSS units

have been established and X no. of members

have been trained, there is a lack of evidence

demonstrating that local authorities have

increased their capacity to manage RWSS more

effectively.

Beneficiaries

Beneficiary 1:

658,750

beneficiaries of clean

water supplies

97.1% 3

Satisfactory. The PCR includes a gender figure,

however “>50%” is not an exact figure. It

would be useful to know the exact figure for

future reference and for comparison with other

projects.

Beneficiary 2:

234,309 beneficiaries

accessing improved

public sanitation

facilities

65.8% 3

Satisfactory. It is unclear why household

sanitation was not captured under beneficiary

reporting.

Unanticipated outcomes (positive or negative not considered in the project logical

framework) and their level of impact on the project (high, moderate, low) Institutional

development

No

score

moder

ate Capacity strengthening of local authorities.

Gender

No

score high

It would be useful to know the exact figure of

%women for future reference and for

comparison with other projects.

Environment &

climate change

No

score -

Poverty reduction No

score -

Private sector

development

No

score moder

ate

Private firms and artisans have benefited from

construction of water supply and sanitation

infrastructure.

Regional integration No

score -

Other (specify)

EFFECTIVENESS OVERALL

SCORE

EFFICIENCY Timeliness (based on

the initial closing

date)

2 2 Unsatisfactory.

Resource used

efficiency

3 4 Highly satisfactory. The calculated proportion

between the median physical implementation

Criteria Sub-criteria PCR

work

score

IDEV

review Reasons for deviation/comments

rate (95.65%) and commitment rate (80.43%) is

1.19 (which is ≥1). The short explanation given

in the PCR is considered sufficient.

Cost-benefit analysis

2 3 Satisfactory. The economic rate of return at

project approval stage was 28% compared to

21% upon project completion. The calculated

ratio between the ERR at completion and the

ERR at appraisal is 0.75 which is within the

category giving a satisfactory score (ratio within

category of <1 and ≥0.75). Overall, the

evaluator generally agrees with the calculations

made to arrive at the ERR in Annex 10 of the

appraisal report.

Implementation

progress (from the

IPR)

3 2 Unsatisfactory. The information made available

reports of late submission of documents to the

ADB, delays and unfinished construction of

water and sanitation infrastructure, etc. Field

reports from the bank and external audit reports

show a large number of areas and examples in

which the executing agency has filed to comply

with covenants and procedures. At various times

during the implementation phase, some of the

issues raised were, poor book keeping record,

lack of community involvement to ensure

ownership, issues with tendering of works, poor

financial management, poor M&E, project

staffing issues, and motor bikes not having

insurance.

Other (specify)

OVERALL EFFICIENCY SCORE 2.50 2.75 Satisfactory.

SUSTAINABILIT

Y

Financial

sustainability

3 3 Satisfactory. The PCR narrative for this section

uses the experience with the V-WASHE

Committees as good evidence of financial

sustainability. There seems to be a lack of

systematic assessment of financial feasibility of

water supply schemes, for example through

carrying out a willingness and capacity to pay

for improved water services. The positive

example of Chipowe village is given, stating

their relatively high contribution. It is unclear to

what extend Chipowe village can be considered

representative for the other district where the

programme has had an intervention. It is unclear

to what extend local authorities have sufficient

budgets in place and earmarked to cover major

Criteria Sub-criteria PCR

work

score

IDEV

review Reasons for deviation/comments

maintenance works (for costs exceeding ZMK

500). Only water supply is mentioned when it

comes to financial sustainability, however it is

unclear what payment modalities exist for the

sanitation facilities constructed at public places

such as market centres.

Institutional

sustainability and

strengthening of

capacities

3 3 Satisfactory.

Ownership and

sustainability of

partnerships

3 3 Satisfactory.

Environmental and

social sustainability

3 3 Satisfactory.

*The rating of the effectiveness component is obtained from the development objective (DO) rating in the latest

IPR of the project (see Guidance Note on the IPR).

The ratings for outputs and outcomes are determined based on the project’s progress towards realizing its targets,

and the overall development objective of the project (DO) is obtained by combining the ratings obtained for

outputs and outcomes following the method defined in the IPR Guidance Note. The following method is applied:

Highly satisfactory (4), Satisfactory (3), Unsatisfactory (2) and Highly unsatisfactory (1).

Criteria Sub-criteria PCR

Work

score

IDEV

review Reasons for deviation/comments

BANK

PERFORMANCE

Proactive identification and

resolution of problems at

different stage of the project

cycle

No

score 2

Unsatisfactory. The Bank could have been

more pro-active, especially in taking steps

to reduce delays.

Use of previous lessons

learned from previous

operations during design and

implementation

No

score 3

Satisfactory. The modality of using

small contractors instead of local

authorities for construction was based

on previous lessons learned.

Promotion of stakeholder

participation to strengthen

ownership

No

score 3

Satisfactory.

Enforcement of safeguard and

fiduciary requirements

No

score 3

Satisfactory.

Design and implementation of

Monitoring & Evaluation

system

No

score 3

Satisfactory. Based on additional

feedback from the task manager, it

should be noted that the Bank was

supporting specific activities in the

national program and not at all

activities related to M&E.

Quality of Bank supervision

(mix of skills in supervisory

teams, etc)

No

score 3

Satisfactory.

Timeliness of responses to

requests

No

score 2

Unsatisfactory. Some responses took

longer than required.

OVERALL BANK PERFORMANCE

SCORE 3.00 2.71

Satisfactory

BORROWER

PERFORMANCE

Quality of preparation and

implementation

No

score 2

Compliance with covenants,

agreements and safeguards

No

score 2

Provision of timely counterpart

funding

No

score 3

Responsiveness to supervision

recommendations

No

score 2

Measures taken to establish

basis for project sustainability

No

score 2

Timeliness of preparing

requests

No

score 3

OVERALL BORROWER PERFORMANCE

SCORE 3.00 2.33 Unsatisfactory

PERFORMANCE

OF OTHER

STAKEHOLDERS

Timeliness of disbursements

by co-financiers

No

score 2

Functioning of collaborative

agreements

No

score 2

Quality of policy dialogue

with co-financiers (for PBOs

only)

No

score 2

Quality of work by service

providers

No

score 2

Responsiveness to client

demands

No

score 2

OVERALL PERFORMANCE OF OTHER

STAKEHOLDERS 2.00 2.00 Unsatisfactory

The overall rating is given: Very Good, Good, Fair and Poor.

(i) Very Good (HS) : 4

(ii) Good (H) : 3

(iii) Fair (US) : 2

(iv) Poor (HUS): 1

DESIGN, IMPLEMENTAION AND UTILIZATION OF MONITIRING AND

EVALUATION (M&E)

Criteria Sub-criteria IDEV

Score Comments

M&E DESIGN M&E system is in place, clear,

appropriate and realistic 2

Monitoring indicators and

monitoring plan were duly

approved

2

Existence of disaggregated gender

indicator 2

The gender reporting is not accurate, the

“>50%” is not an exact figure and should

have been specified.

Baseline data were available or

collected during the design 3

Other, specify

OVERALL M&E DESIGN SCORE 2.25

M&E

IMPLEMENTA-

TION

The M&E function is adequately

equipped and staffed 1 No evidence of an M&E officer in place.

OVERALL M&E IMPLEMENTATION SCORE 1

M&E

UTILIZATION

The borrower used the tracking

information for decision 1 No evidence that this was used/in place.

OVERALL M&E UTILIZATION SCORE 1

OVERALL M&E PERFORMANCE SCORE 1.42

PCR QUALITY EVALUATION

Criteria

PCR-

EVN (1-

4)

Comments

QUALITY OF PCR

1. Extent of quality and completeness of the

PCR evidence and analysis to substantiate the

ratings of the various sections

3

2. Extent of objectivity of PCR assessment

score

3

3. Extent of internal consistency of PCR

assessment ratings; inaccuracies;

inconsistencies; (in various sections; between

text and ratings; consistency of overall rating

with individual component ratings)

3

4. Extent of identification and assessment of key

factors (internal and exogenous) and unintended

effects (positive or negative) affecting design

and implementation

3

5. Adequacy of treatment of safeguards,

fiduciary issues, and alignment and

harmonization

3

6. Extent of soundness of data generating and

analysis process (including rates of returns) in

support of PCR assessment

3

7. Overall adequacy of the accessible evidence

(from PCR including annexure and other data

provided)

3

8. Extent to which lessons learned (and

recommendations) are clear and based on the

PCR assessment (evidence & analysis)

3

9. Extent of overall clarity and completeness of

the PCR

3

Other (specify)

PCR QUALITY SCORE 3.0 No further comments.

PCR compliance with guidelines (PCR/OM ; IDEV)

1. PCR Timeliness (On time = 4; Late= 1) 4 PCR report was completed within 6 months

after project completion.

2. Extent of participation of borrower, Co-

financiers & field offices in PCR preparation

No information

3. Other aspect(s) (specify)

PCR COMPLIANCE SCORE

*** rated as Very Good (4), or Good (3), or Fair (2), or Poor (1)

References

African Development Bank Group, Policy for integrated water resources management, April 2000

African Development Bank Group, Staff Guidance on Project Completion Reporting and Rating, Quality

Assurance and Results Department (ORQR), Quality Assurance Division (ORQR.2), August 2012

African Development Bank Group, Policy for integrated water resources management, April 2000

Project Completion Report for National Rural Water Supply and Sanitation Program Zambia. December

2015

Government of the Republic of Zambia, National Water Policy, Ministry of Energy and Water

Development, February 2010.

Government of the Republic of Zambia, Water Act Ministry of Legal Affairs, 2006

Various other internal reports, memos, including trip reports, progress reports, etc