zambia - nkana water supply and sanitation project · b. relevance of project design (from approval...

ZAMBIA - Nkana Water Supply and Sanitation Project 1. BASIC INFORMATION

a. Basic project data

Project title: Nkana Water Supply and Sanitation Project

Project code: P-ZM-E00-008 Instrument number(s): 2100150018345

Project type: Public Operation Sector: Water and Sanitation

Country: Zambia Environmental categorization (1-3) : 2

Processing Milestones Key Events Disbursement and Closing date

Date approved: 27/11/2008 Cancelled amount: Original disbursement deadline:

31/12/2013

Date signed: 22/12/2008 Supplementary financing: NA Original closing date: 31/12/2013

Date of entry into force :

29/01/2009

Restructuring: NA Revised disbursement deadline:

31/03/2015

Date effective for 1st

disbursement: 01/09/2009

Extensions (specify dates): 31/12/2013 to

30/09/2014, 30/09/2014 to 31/03/2015

Revised closing date: 31/03/2015

Date of actual 1st : 01/09/2009

b. Financing sources

Financing source/ instrument

(MUA)

Approved amount

(MUA) :

Disbursed amount

(MUA) :

Percentage disbursed

(%):

Loan: ADF 34,998,407.20 34,998,407.20 99.99%

Government: 3,820,000 3,820,000 89.10%

Other (ex. Co-financiers):

TOTAL : 38,818,407.20 38,818,407.20 98.92%

Co-financiers and other external partners:

Execution and implementation agencies: Nkana Water and Sewerage Company

c. Responsible Bank staff

Position At approval At completion

Regional Director - Kennedy Mbekeani (OIC)

Sector Director Andrianarison Rakotobe Mohamed El Azizi

Sector Manager Sering Jallow Osward Chanda

Task Manager Mecuria Assefaw Herbert Chinokoro

Alternate Task Manager -

PCR Team Leader - Alex Gomani

PCR Team Members - Herbert Chinokoro

Monia Moumni (peer reviewer)

Aicha Mohamed Moussa (peer

reviewer)

Kelvin Banda (peer reviewer)

d. Report data

PCR Date : 30/11/2015

PCR Mission Date: From: 19/10/2015 To: 23/10/2015

PCR-EN Date: 29/06/2018

Evaluator/consultant : Daniel VAN ROOIJEN Peer Reviewer/Task Manager: Ram Janakiram

PCR EVALUATION NOTE FOR PUBLIC SECTOR OPERATIONS

2. PROJECT DESCRIPTION Summary from Appraisal Report including addendum/corrigendum or loan agreement, and taking into account any modification that

occurred during the implementation phase.

The project aimed to increase improved access to water supply and sanitation to residents in three urban centres of

Kitwe, Kalulushi and Chambishii in the Copperbelt Region of Zambia. The Government of Zambia was the

borrower of the ADF loan, with the Nkana Water and Sewerage Company (NWSC), under the MLGH, being the

Executing Agency of the project. Most of the funding was provided by the African Development Fund through a

loan with a small contribution provided by the Government of Zambia. The project agreement was signed in

December 2008 and closed in March 2015 after 2 extensions.

a. Rationale and expected impacts:

The project rationale was born from the current physical conditions of old water and sewerage infrastructure in the

three urban centres in whit substantial water losses occurring in the water supply systems. In order to keep up with

the growing population and subsequent demand for water and sewerage infrastructure, the need for rehabilitation

and expansion of the current systems was considered essential. Another reason to develop this project was the

understanding of the important factor of water and sanitation services for the status of human health. The expected

impact envisaged by the project was to increase the proportion of the population with access to adequate water

supply and sanitation services as well as improvement of the health conditions of the entire population.

b. Objectives and Expected Outcomes: The overall project objective was to improve the access to adequate water supply and sanitation services in Kitwe,

Kalulushi and Chambishi Towns.

Discrepancies exist between the outcomes listed in the appraisal report and outcomes listed in the PCR. The

appraisal report has listed 4 outcomes while the PCR lists 5 more outcomes included which are in the area of the

performance water service delivery (outcome 4-8) and the functioning of so called resident development

committees (outcome 9). The 4 outcomes as listed in the project appraisal report were:

1) Increased proportion of the residents of the three urban centres benefiting from improved water supply services

2) Increased number of the people in the three urban centres benefiting from improved sanitation facilities

3) Reduced incidence of waterborne diseases

4) Improved management of water and sanitation services

Beneficiaries of these outcomes, as listed in the log-frame of the appraisal report, are:

- Urban and peri-urban populations of Kitwe, Kalulushi and Chambishi

- The two local authorities

- Nkana Water and Sewerage Company (NWSC)

c. Outputs and intended beneficiaries: Expected outputs were:

o Detailed designs prepared

o Water supply system rehabilitated, augmented and extended

o Sewerage system rehabilitated, augmented and extended

o Increased awareness on the need for improved public health and personal hygiene

o Gender mainstreamed and integrated in the training and communication materials

o Community organizations in semi urban trained and able to manage public water supply points

o Water kiosks constructed

o On-site sanitation facilities constructed

o Water supply facilities provided and rehabilitated in selected public institutions

o Sanitation facilities provided and rehabilitated in selected public institutions

o Support to improved solid waste management

o M&E system including gender specific indicators developed

o Performance Contract signed between NWSC and MLGH

o Enhanced institutional capacity of NWSC to manage water and sewerage services

o Enhanced institutional capacity of municipal authorities to manage solid waste services

Beneficiaries of these outputs, as listed in the log-frame, were:

- Nkana Water and Sewerage Company (NWSC)

- Kitwe City Council (KCC)

- Local authorities

- Contractors

- Residents of the three towns

- Community organizations

- NGOs

- Ministry of Local Government and Housing (MLGH)

- Ministry of Energy and Water Development (MEWD)

- Health Centres

- Schools

d. Principal Components and/or activities:

The project has the following 5 components:

1. Water supply works

2. Sewerage works

3. Sanitation and Hygiene Education

4. Institutional Support to NWSC

5. Project Management

Principal activities, as listed in the log-frame of the appraisal report, are:

1. Engineering services

2. Construction works for water supply system

3. Construction works for the sewerage system

4. Sensitization and promotion of sanitation and personal hygiene for the urban poor in the project areas

5. Promotion of use of public water points and on-site sanitation in semi/peri urban areas

6. Water supply and sanitation facilities in public institutions

7. Institutional Support

The main activities by project components are provided below:

Component One : Water Supply Works: Kitwe, Kalulushi and Chambishi

1.Rehabilitation of Bulangililo Water Treatment Plant (WTP),

2. Rehab & expansion of East Nkana WTP for an additional 32,000 m3/d;

3.Rehabilitation of the existing water transmission and distribution systems in Kitwe;

4.New intake works on Mwambashi river and a 10 km pumping main to existing 7 Shaft WTP (15,000 m3/d),

refurbishment of the plant, and rehabilitation and extension of distribution system;

5.Transmission from the mine raw water mains to a new treatment plant with a 10,000 m3/d capacity, and

Rehabilitation of distribution system.

Component Two : Sewerage Works in Kitwe, Kalulushi and Chambishi

1.Rehabilitation of waste water collection and, transportation system and treatment plants (82,560 m3/d) in Kitwe

and expansion of treatment capacity by 5,000 m3/d;

2.Rehabilitation of existing sanitation facilities in Kalulushi including 10,000 m3/d waste stabilization ponds;

3.Rehabilitation of existing sanitation facilities in Chambishi including 5,000 m3/d waste stabilization ponds

Component Three: Sanitation and Hygiene Education

1. Carry out sanitation promotion, hygiene education and support for onsite household sanitation.

2.Provision of water and sanitation facilities for public institutions (schools, health centers) and Community

capacity building, and support related to solid waste management in the three urban and peri-urban areas covered

by the project.

Four: Institutional Support to NWSC

1.Improvement of NWSC’s MIS, tariff/affordability study,

2.Support for Consumer Watch Groups, workshop and

office rehabilitation, leakage control, and provision of

logistics, including operational and specialized utility vehicles

Five: Project Management

1.Provision of engineering services for detailed designs, procurement and supervision, including assessment of

alternative sustainable sources of water in the project area,

2.Support to PIT for project management and Project Audit

3. PROJECT PERFORMANCE ASSESSMENT

RELEVANCE

a. Relevance of the project development objective:

The project’s development objective, to improve the access to adequate water supply and sanitation services in

Kitwe, Kalulushi and Chambishi Towns, is considered very relevant with regards to the government’s goals with

regards to reducing poverty and improving health outcomes of the population of Zambia. The project scope fits

well with strategies and priorities by the Bank (for example its policy on integrated water resources management).

It is also well embedded into the water vision and sector strategies, principles and policies by the Government of

Zambia. The vision in Zambia’s Water policy states: “To optimally harness water resources for the efficient and

sustainable utilisation of this natural resource to enhance economic productivity and reduce poverty.” In addition,

the programme is aligned with the water act and other sector strategies. An important strategy, published by

Zambia’s Ministry of Local Government and Housing in the final year of the project implementation period, is

the national urban and peri-urban sanitation strategy. This strategy aims to “provide adequate, safe and cost-

effective sanitation services to 90 percent of the urban population by 2030”. The project’s scope will allow it to

greatly contribute to this goal. The evaluator concurs with the PCR rating for this criterion and considers it

as highly satisfactory (4). The reviewer concurs with the assessment and rating for this criterion - 4 –

Highly satisfactory.

b. Relevance of project design (from approval to completion):

The project design is considered comprehensive, logic and relevant in light of the development objectives and

sector strategies set by the government of Zambia. The appraisal report demonstrates that risks have been

assessed in an adequate manner and environmental and social protection measures have been well incorporated.

The design has taken into account the lessons learned from a precious urban water and sanitation project financed

by the Bank for Kitwe Town. The inclusion of knowledge management and water supply performance indicators

is much welcomed. Participatory approaches in project design (and implementation), as well as a strong focus on

water demand management with community participation is recognized as very relevant to ensure that benefits

from the project are sustained after project completion. The evaluator concurs with the PCR rating for this

criterion and considers it as satisfactory (3).

Additional design elements incorporated in the project are:

The NWSSP was designed as a stand-alone operation which was expected to contribute to the

development of a program approach for development of water supply and sanitation services in all urban

and peri-urban areas of the country

Based on priorities identified through a pre-feasibility study carried out by the Borrower encompassing

technical, economic and financial analyses.

Included conditions on reporting on the key financial indicators of the company to ensure sustainability

of the project outputs and outcomes.

Whenever possible and without sacrificing the expected outputs, standard and less expensive solutions

were selected. For example, uniformity of types of equipment and fittings like pumps and valves, meters

and other fittings, etc.

EFFECTIVENESS

c. Effectiveness in delivering outputs:

Discrepancies exist between the outputs listed in the appraisal report and outputs listed in the PCR. The appraisal

report has listed 15 outputs while the PCR lists 18 outputs. This evaluation will focus on the 18 outputs listed in

the PCR. Some of the outputs listed in the project appraisal report such as output 4.1 on “increased awareness” or

4.3 on “ability to manage water points” focus on behavioural change and capacity strengthening and hence should

be considered as project outcomes instead of outputs.

The PCR narrative is limited to the civil works carried out by contractors being compliant to the contracts. It does

not cover other outputs listed.

Regarding output 2 and 3 on water production and wastewater treatment, the electricity disruption affecting water

production is considered as a valid reason which lies outside the power of the project. Output 4 (Number of

operable public water points) does not capture whether the water points are in operation or not. Instead of

“operable water points”, this should have been “functioning water points”, in order to monitor actual water

access. Regarding output 5 (% public water points operated by women), the reported figure on progress was not

well calculated. Since target was not reached, the value should be below 100%. Regarding output 7 on the

percentage of female headed households benefiting from onsite sanitation, the narrative uses an absolute number

while the indicator is used as %. The reported figure on progress was not well calculated. With current value 30%

against target of 43.3% the target was not reached. The value on progress should be below 100%. Regarding

output 8 on water supply to schools and health centres, there is unfortunately no details provided on the no. of

schools vs the no. of health centres. Regarding sold waste management (output 10), it would have been good to

get more details on the type of solid waste management (what, how, where, etc.). Regarding output 14, it is

unclear how progress of 84% was calculated. Based on the no. of teachers and health inspectors combined, the

progress has exceeded the target (>100%) which is not reflected in the PCR.

Based on the below reflection on output-level project performance, the evaluator would rate this criterion

as satisfactory (3) which is in line with the PCR score.

d. Effectiveness in delivering outcomes:

Discrepancies exist between the outcomes listed in the appraisal report and outcomes listed in the PCR. The

appraisal report has listed 4 outcomes while the PCR lists 5 more outcomes included which are in the area of the

performance water service delivery (outcome 4-8) and the functioning of so called resident development

committees (outcome 9). This evaluation will focus on the 9 outcomes listed in the PCR.

While the PCR narrative rightly points at the important livelihoods opportunities which the project brought in

terms of building skills in construction and casting sanitation platforms, these software gains have not been

included in the M&E framework or captured in the list of outcomes.

The increase in access coverage of water supply services is commendable. It would have been good to know if

water supply was also expanded to unplanned settlements, as was done for sanitation. For sanitation, it is not

entirely clear what proportion of people benefiting from improved sanitation is at the household level. Regarding

disease incidence, the reduction in disease incidence can likely be attributed to the project interventions. It would

have been good to see the numbers for all years in order to see whether an actual positive trend can be observed.

The fact that there has not been any major disease outbreak since 2012 is definitely a good sign, however in order

to conclude something on attribution this should always be compared with other urban centres that did not have

an intervention. The progress made on increasing metered connections deserves praise. Regarding unaccounted

for water, the reported progress made of 176% should be negative since recent value is worse than the baseline

value. It is based on a wrong calculation. Baseline is 42%, target is 25% and current value is 44%. This would

lead to the following calculation: Progress against target = (recent value – baseline value) / (target value – baseline) = (44% - 42%) / (25% - 42%) = -12%

Regarding outcome 6 on increasing the collection ratio, the stated reasons for not meeting the target are well

justified. Regarding average hours of water supply (outcome 7); the reported progress made of 67% should be

negative (-33%) since the most recent value is worse than the baseline value. The target of 24hours water supply

on average is considered not realistic and should have been lowered at project design phase. Regarding outcome 8

on water quality; it would have been good to have included a separate indicator for water treated for consumption

and treated wastewater because each water category have different water quality requirements and subsequent

standards.

Based on the below reflection on outcome-level project performance, and given the shortcomings, lack of

robust evidence and data to determine the number of beneficiaries, (due in large part to very weak and

non-functional M&E system), absence of assessment of the outcomes of the capacity building program, the

evaluator would rate this criterion as unsatisfactory (2) which is lower than the PCR score.

e. Project development outcome:

Despite some incomplete or inaccurate reporting at the outcome and output level, the overall progress made with

regards to the development objective is considerable. Substantial gains have been made with both the expansion

of water service provision as well as sanitation infrastructure in the 3 towns. It would have been good to include

indicators for behavioural change with regards to hygiene and capacity strengthening in the log-frame, in order to

capture progress in these important soft interventions areas. This would have been relevant with regards to

sustainability as well.

However, given the 2 ranking for outcomes, this criterion is ranked unsatisfactory (2).

f. Beneficiaries:

The PCR has listed 7 categories of beneficiaries. Some overlap seem to exist in the report, for example when

comparing row 2 and row 4 on beneficiaries of water supply and sanitation in public places. There is also a lack

of consistency in the use of peri-urban or urban population, which makes it difficult to ascertain whether peri-

urban or urban population is referred to. The gender proportion has been included for all beneficiary groups,

however they have not been disaggregated for each group.

g. Unanticipated additional outcomes (positive or negative, not taken into consideration in the project

logical framework):

The PCR reports 4 unanticipated additional positive outcomes with regards to health, private sector development,

gender equality and social equality for disabled people which were not included in the original log-frame. These

outcomes are considered valid and relevant to the development objective of the project.

EFFICIENCY

h.Timeliness: The project period lasted 6.22 years instead of the planned 5 years which is just over a year longer than planned.

The resulting ratio of planned implementation time (PAR) and actual project implementation time from the date

of effectiveness is 0.80 which is within the category (<1 and ≥0.75) and therefore considered satisfactory. The

PCR provides comprehensive details as to why the project implementation took longer than anticipated and the

reasons as to why the delays have occurred.

However, the reasons provided in the PCR are inadequate to justify the delays in implementation as mentioned

above. More proactive steps could have been taken to reduce the delays. The PCR has rated this indicator as 2

which is appropriate. The reviewer recommends to retain this rating.

i. Resource use efficiency:

The project shows overall good evidence of physical implementation versus resources used. The median

percentage physical implementation of 100%, compared to a total commitment rate of 98.92%. The calculated

proportion between the median physical implementation rate and commitment rate is 1.01 which falls into the

category of ≥1 and corresponds to a rating of highly satisfactory (4), which is higher than the PCR rating of

satisfactory (3) for this criterion.

j. Cost-benefit analysis:

The appraisal report mentions that the Economic Internal Rate of Return of 16.9%. The PCR states that the EIRR

has risen from 16.9% to 17.8% upon project completion. The methodology and calculations carried out as part of

the economic analysis is not detailed in the PCR itself. Reference is made to an annex which is not available to

the evaluator. It is therefore difficult to assess the validity and assumptions made for analysing the economic costs

vs benefits of this project based on the information provided. Therefore, it is difficult for the evaluator to compare

and evaluate the ratio of anticipated EIRR at appraisal versus the EIRR at project completion.

However, the economic analysis carried out at appraisal makes the following assumptions:

(i) UFW is projected to be reduced from 43% to 25% as a result of the project,

(ii) Incremental sanitation revenue estimated at 25% of water revenue,

(iii) Average tariff is projected to increase from ZMK 1350/ m3 to reach ZMK 2500/ m3 by 2021

(iv) Incremental operations and maintenance costs are estimated at 15% of the incremental revenue.

(v) Health benefits from improved water supply and sanitation services, as well as economic value of

sludge produced from the improved facilities are considered

(vi) Economic life of the investment is estimated over 25 years.

Since the above assumptions were made at appraisal in 2008, it would be useful to demonstrate how realistic

these assumptions were, deviations from these estimates and provide the relevant evidence and data to support

this.

k. Implementation progress:

As noted earlier substantial progress was made during the project implementation. Sufficient evidence has been

given of progress at the output, outcome and impact level.

The PCR narrative reports that Procurement and financial management remained consistent with Bank systems

and procedures. However, a number of audit reports reported on issues found with compliance to financial

protocols and procedure, poor financial management including incomplete financial reporting, poor record

keeping and improper use of accounting software.

The monitoring and evaluation framework has not sufficiently captured progress made in the areas of capacity

building, institutional strengthening and behavioural change. Social and environmental safeguards were

sufficiently accounted for in the appraisal report. The PCR reports that an environmental and social management

plan and resettlement policy framework was put in place and monitored during project implementation. It would

have been useful if key data from this framework would have been included in the results reporting, for

evaluation. The evaluator rates this criterion with unsatisfactory (2), which is lower than the PCR score of

satisfactory (3).

SUSTAINABILITY

(l).Financial sustainability:

The PCR narrative for this section is comprehensive and demonstrates a realistic view on issues with operational

management of the (peri-) urban water supply schemes. It is not clear whether the Nkana water and sewerage

company has received technical support in terms of water demand modeling and scenario analysis of non-

revenue water in light of overall financial sustainability of the water supply systems. As the narrative rightly

points out, it will be critical to periodically carry out regular tariff adjustments if necessary. Community

sensitisation is a critical aspect for making sure consumers are willing to pay for improved water services. There

is no mention of payment modalities of public sanitation facilities, despite public sanitation being one of the core

intervention areas of the project. From literature it is known that users of sanitation facilities have a capacity and

willingness to pay for using these services. Therefore it should have been included in the overall assessment of

financial sustainability of the project outcomes. The evaluator would concur with the PCR rating for this

criterion and rates it as unsatisfactory (2).

The appraisal report mentions that at full capacity, the incremental revenue (ZMK 47 billion) generated from the

project will more than offset the incremental recurrent costs (ZMK 6.4 billion). The incremental revenues as well

as the cost savings generated from the project will more than offset the incremental recurrent costs. Consequently,

there will not be any need for budgetary support or subsidies from GRZ to the operations and maintenance of the

water supply and sanitation system in the project area. (Source; Appraisal report, August 2008, Section 4.4.

Sustainability section). Was this incremental revenue achieved? Evidence and supporting data is needed to

validate this achievement.

m. Institutional sustainability and strengthening of capacities:

The PCR report has discussed the various activities carried out that have led to strengthening technical capacities

of the water and sewerage company and government agencies as well as community groups. The project also

supported the enactment of the Zambia Environmental Management Agency Act of 2011, which is commendable

and will help manage the environment. While all these elements are positive and would help ensure institutional

sustainability, these outcomes should have been measured and included in the M&E framework. Simple surveys

could have captured changed capacities at the organisation level, community level, as well as at the individual

level. This is considered essential and without this kind of evidence the evaluator is unable to conclude the rate to

which institutional sustainability is ensured. Therefore the evaluator is unable to rate this criterion.

n. Ownership and sustainability of partnerships:

Despite a comprehensive narrative provided, the information available for true verification of this criterion

remains somewhat limited. The evaluator agrees with the justification provided with regards to efforts made to

ensure ownership and sustainability of partnerships, especially by involving the communities at the various

project stages. This criterion is rated as satisfactory (3), in line with the PCR score.

o. Environmental and social sustainability:

The PCR reports that Environmental and Social Management Plan and Resettlement Policy Framework was put

in place and monitored during project implementation. As noted earlier, it would have been useful if key data

recorded as part of this plan and framework would have been included in the results reporting, for evaluation. The

project has reported strong involvement of communities at various stages of the project which is considered key

to ensure social sustainability. To ensure social acceptance it will important to ensure that water tariffs are

appropriate and if possible pro-poor. The gains made in terms of increased wastewater treatment capacities and

water saved through reduction of physical losses are very likely to have a positive effect on the environment. The

evaluator concurs with the PCR score and rates this criterion as satisfactory (3).

4. PERFORMANCE OF STAKEHOLDERS

a. Bank Performance:

The bank has overall performed well during the design, implementation and completion phase of this project.

Regular field visits have been carried out and performance including elements of adherence to Banks protocols

have been evaluated during the project by external evaluators. Perhaps more attention could have been given to

timeliness of implementation of activities and a stronger supervision on financial administration after issues

reported in the audits. There is no evidence that a mid-term review has taken place in this project, which is

considered essential in project supervision. The evaluator would rate the Banks performance as satisfactory

(3) which is in line with the PCR rating.

h. Borrower Performance

The project Nkana Water and Sewerage Company Government was the executing agency and the Republic of

Zambia was the borrower. Performance of the Borrower is considered satisfactory (3), despite various issues

raised on financial management and compliance to procedures and protocols and despite high staff turnover as

reported by the water and sewerage company. The borrower could have done a better job at measuring progress

made with regards to behavioural change and hygiene practices among communities as well as strengthening of

staff capacities at the water utility.

c. Performance of other stakeholders:

Other stakeholders are mainly the contractors that were signed to carry out the major civil works for construction

of water supply and sewerage infrastructure as well as the water points and sanitation facilities. No major issues

were reported with regards to their performance apart from an incidental delay on the Kitwe Water Works. The

evaluator concurs with the PCR rating and rates the performance of the contractors as satisfactory (3).

6. SUMMARY OF OVERALL PROJECT PERFORMANCE

a. Overall assessment:

The project’s development objective, to improve the access to adequate water supply and sanitation services in

Kitwe, Kalulushi and Chambishi Towns, is considered very relevant with regards to the government’s goals with

regards to reducing poverty and improving health outcomes of the population of Zambia. The project design is

considered comprehensive, logic and relevant in light of the development objectives and sector strategies set by

the government of Zambia. A number of discrepancies exist between the outputs and outcomes listed in the

appraisal report and outputs and outcomes listed in the PCR. Despite some incomplete or inaccurate reporting at

the outcome and output level, the overall progress made with regards to the development objective is

considerable. Some overlap seem to exist in the PCR with regards to reporting on beneficiaries reached. The

project has performed well in terms of timeliness, resource use efficiency and creating ownership. The project

could have done better with regards to progress of implementation and ensuring financial sustainability. Overall,

the bank, borrower and contractors have performed well in this project.

The program is satisfactory with regards to the dimensions of relevance of program development objective and

program design. The project has been successful in delivering the water supply and sanitation infrastructure. With

regards to the program objectives in the remaining three of the four dimensions – viz. effectiveness, efficiency

and sustainability, the overall program performance is partly satisfactory (2.7) – due in large part to a lack of

sufficient and relevant evidence to support the assessment and achievements of the project development

objectives.

b. Design, implementation and utilization of the M&E (appreciation of the evaluator):

The monitoring and evaluation framework is considered somewhat limited in design since it did not capture

progress made in the areas of capacity building, institutional strengthening and behavioural change. For example,

while the PCR narrative rightly points at the important livelihoods opportunities which the project brought in

terms of building skills in construction and casting sanitation platforms, these software gains have not been

included in the M&E framework or captured in the list of outcomes.

6. EVALUATION OF KEY LESSONS LEARNED AND RECOMMENDATIONS

a. Lessons learned:

The evaluator fully concurs with the 2 lessons learned as listed in the PCR. These are on 1) better planning at

design phase to reduce risks during implementation and 2) stronger community involvement. On the second

lesson learned, the evaluator would note that this involvement would apart from savings in hardware costs also

benefit community acceptance of (changes in) tariffs, and could enable the adoption of a pro-poor tariff structure.

In addition to the above, the inclusion of a more comprehensive monitoring and evaluation framework, covering

software elements, should be considered as a lesson learned.

One of the activities in this project was to document the knowledge and experiences gained during the

implementation of this project which would add value and build up lessons for the design and management of

similar projects within the communities, replicated throughout the country. It would be helpful to elaborate in

more detail the specific lessons learnt from this project which were utilized in the design and implementation of

similar projects in Zambia.

The Evaluation’s opinion of each lesson learned is given in summary below:

1. VALIDATED

2. VALIDATED

The evaluator has no additional lessons learned to be suggested for inclusion.

b. Recommendations:

The evaluator fully agrees with the 4 recommendations listed in the PCR. It is a good sign that all agencies

targeted to consider these evaluations are the national agencies and government bodies. In addition to these

recommendation the evaluator would add that it is important for the water and sewerage company to keep track of

consumer satisfaction for example through the use of continued surveys or a hotline or other tools. Water demand

plans recommended should include scenario modeling of water demand based on realistic population growth

scenarios.

The Evaluation’s opinion of each recommendation is given in summary below:

1. VALIDATED

2. VALIDATED

3. VALIDATED

4. VALIDATED

The evaluator would like to add the following 2 recommendations:

1) Implementation risks in future projects with a similar design could be reduced through: (i) a thorough

institutional analysis, including capacity gap analysis with a training plan to enhance and strengthen sector

institutions and agencies both in public and private sector,

2) In any development project, a simple monitoring system needs to be included, that covers both

implementation progress as well as outcome monitoring, which establishes baseline values no later than at

appraisal stage.

7. COMMENTS ON PCR QUALITY AND TIMELINESS:

The PCR quality is rated 2.90 based on the scoring of the 9 criteria listed in Annex 1.

The extent of quality and completeness of the PCR evidence and analysis to substantiate the ratings of the various

sections was considered satisfactory. The extent of objectivity of PCR assessment score was considered

satisfactory. The extent of internal consistency of PCR assessment ratings; inaccuracies; inconsistencies was

considered as unsatisfactory. The outputs and outcomes listed in the PCR are not consistent with those listed in

the log-frame of the project appraisal report. The extent of identification and assessment of key factors (internal

and exogenous) and unintended effects (positive or negative) affecting design and was considered satisfactory.

The adequacy of treatment of safeguards, fiduciary issues, and alignment and harmonization was considered very

satisfactory. The extent of soundness of data generating and analysis process (including rates of returns) in

support of PCR assessment was considered satisfactory. The overall adequacy of the accessible evidence from

PCR including annexure and other data provided was considered unsatisfactory. However, not all annexes were

accessible. The extent to which lessons learned (and recommendations) are clear and based on the PCR

assessment (evidence & analysis) was considered highly satisfactory. The extent of overall clarity and

completeness of the PCR was considered satisfactory.

PCR timeliness was considered very late (1) since the PCR report was completed on 29 June 2018 and the revised

project completion date was 31 March 2015 – which was much more than 6 months after project completion.

8. SUMMARY OF THE EVALUATION

Criteria PCR PCREN Reason for disagreement/

Comments

RELEVANCE 3.5 3.5 Relevance of project development objective 4 4 Highly satisfactory.

Relevance of project design 3 3 Satisfactory.

EFFECTIVENESS 3 2

Development objective (DO) 3 2 Unsatisfactory. Although, the overall

progress made with regards to the

development objective is considerable,

the PCR only reports on 4 of the 9

outcomes anticipated in the Project

Appraisal Report. Taking the wider set

of outcomes into consideration reduces

the performance. For instance, it would

have been good to include indicators for

behavioral change with regards to

hygiene and capacity strengthening in

the log-frame, in order to capture

progress in these important soft

interventions areas. This would have

been relevant with regards to

sustainability as well.

EFFICIENCY 2.75 2.67 Timeliness 2 2 Unsatisfactory. No disagreement

Resource use efficiency 3 4 Highly satisfactory. The project shows

overall good evidence of physical

implementation versus resources used.

The median percentage physical

implementation of 100%, compared to a

total commitment rate of 98.92%. The

calculated proportion between the

median physical implementation rate

and commitment rate is 1.01 which falls

into the category of ≥1 and corresponds to a rating of highly satisfactory (4),

which is higher than the PCR rating of

satisfactory (3) for this criterion.

Cost-benefit analysis 3 UTR

Implementation progress (IP) 3 2 Unsatisfactory. The PCR narrative

reports that procurement and financial

management remained consistent with

Bank systems and procedures. However,

a number of audit reports reported on

issues found with compliance to

financial protocols and procedure, poor

financial management including

incomplete financial reporting, poor

record keeping and improper use of

accounting software. The monitoring

and evaluation framework did not

sufficiently capture progress made in the

areas of capacity building, institutional

strengthening and behavioural change.

Social and environmental safeguards

were sufficiently accounted for in the

appraisal report. The PCR reports that

Environmental and Social Management

Plan and Resettlement Policy

Framework was put in place and

monitored during project

implementation. It would have been

useful if key data from this framework

would have been included in the results

reporting, for evaluation.

SUSTAINABILITY 2.75 2.67 Financial sustainability 2 2 Unsatisfactory.

Institutional sustainability and strengthening

of capacities 3 UTR The PCR report has discussed the

various activities carried out that have

led to strengthening technical capacities

of the water and sewerage company and

government agencies as well as

community groups. While all these

elements are positive and would help

ensure institutional sustainability, these

outcomes should have been measured

and included in the M&E framework.

Simple surveys could have captured

changed capacities at the organisational

level, community level, as well as at the

individual level. This is considered

essential and without this kind of

evidence the evaluator is unable to

conclude to what extend institutional

sustainability is ensured.

Ownership and sustainability of partnerships 3 3 Satisfactory.

Environmental and social sustainability 3 3 Satisfactory.

OVERALL PROJECT COMPLETION

RATING 3 2.71

Bank performance: 3 3 Satisfactory.

Borrower performance: 3 3 Satisfactory.

Performance of other shareholders: 3 3 Satisfactory.

Overall PCR quality: - 2.90 Satisfactory.

9. PRIORITY FOR FUTURE EVALUATIVE WORK: PROJECT FOR PERFORMANCE

EVALUTION REPORT, IMPACT EVALUTION, COUNTRY/SECTOR REVIEWS OR

THEMATIC EVALUATION STUDIES:

- Project is part of a series and suitable for cluster evaluation

- Project is a success story

- High priority for impact evaluation

- Performance evaluation is required to sector/country review

- High priority for thematic or special evaluation studies (Country)

- PPER is required because of incomplete validation rating

Major areas of focus for future evaluation work: (i) Contribution of this project to the design of the National Urban Water Supply and

Sanitation Program (NUWSSP) which was expected to be ready in early 2009 and current status of this program

(ii) Assessment of financial, institutional, and environmental sustainability especially of Nkana Water and

Sewerage Company (NWSC)

(iii) Revised economic and financial analysis based on relevant and reliable data

(iv) Reduction in water borne diseases, mortality rates. Health related costs of households and the labor burden for

women as a result of this project.

a) Performance evaluation is required for sector/ country review

b) Cluster evaluation (institutional support)

c) Sector evaluation (budgetary support or public finance management reforms)

Follow up action by IDEV: Identify same cluster or sector operations; organize appropriate work or consultation mission to

facilitate a), b) and/or c).

Division Manager clearance Director signing off

Data source for validation:

Task Manager/ Responsible bank staff interviewed/contacted (in person, by telephone or

email)

Documents/ Database reports

Attachment:

PCR evaluation note validation sheet of performance ratings

List of references

Appendice 1

PROJECT COMPLETION REPORT EVALUATION NOTE Validation of PCR performance ratings

PCR rating scale:

Score Description 4 Very Good – Fully achieved with no shortcomings 3 Good – Mostly achieved despite a few shortcomings 2 Fair – Partially achieved. Shortcomings and achievements are roughly balanced 1 Poor – very limited achievement with extensive shortcomings

UTS Unable to score/rate NA Non Applicable

Criteria Sub-criteria PCR

work

score

IDEV

review Reasons for deviation/comments

RELEVANCE Relevance of the project

development objective

(DO) during

implementation

4 4 Highly satisfactory.

Relevance of project

design (from approval to

completion)

3 3 Satisfactory.

OVERALL RELEVANCE SCORE 3.50 3.50

EFFECTIVENE

SS*

Effectiveness in delivering outcomes

Outcome 1: Percentage

of residents benefiting

from adequate water

supply services

No

score 4

Highly satisfactory. The increase in access coverage

of water supply services is commendable. It would

have been good to know if water supply was also

expanded to unplanned settlements, as was done for

sanitation.

Outcome 2: Percentage

of residents benefiting

from improved

sanitation facilities

No

score 3 Satisfactory.

Outcome 3: Incidence of

diarrhoea and dysentery

No

score 3

Satisfactory. The reduction in disease incidence can

likely be attributed to the project interventions. It

would have been good to see the numbers for all

years in order to see whether an actual positive trend

can be observed. The fact that there has not been any

major disease outbreak since 2012 is definitely a

good sign, however in order to conclude something

on attribution this should always be compared with

other urban centres that did not have an intervention.


work

score

IDEV


Outcome 4: Increased

Metered connections

No

score 3 Satisfactory and well justified.

Outcome 5:

Unaccounted for water

(UFW)

No

score 2

Unsatisfactory. The reported progress made of 176%

should be negative (-12%) since the most recent

value is worse than the baseline value. (see

discussion in section 3d)

Outcome 6: Increase

Collection ratio

No

score 3

Satisfactory. The stated reasons for not meeting the

target are well justified.

Outcome 7: Average

daily hours of supply

No

score 2

Unsatisfactory. The reported progress made of 67%

should be negative (-33%) since the most recent

value is worse than the baseline value. The target of

24-hours water supply on average is considered not

realistic and should have been lowered at project

design phase.

Outcome 8: Quality of

both water supply and

treated waste water

No

score

2

Unsatisfactory. It would have been good to have

included a separate indicator for water treated for

consumption and treated wastewater because each

water category have different water quality

requirements and subsequent standards.

Outcome 9: Functional

Resident Development

Committees (RDCs) and

Increased women

participation from 30%

to 50%.

No


Effectiveness in delivering output

Output 1: Audit report

submission

No


Output 2: Quantity of

water produced and

distributed in Kitwe,

Kalulushi & Chambishi

(m3)

No

score 3

Satisfactory. The electricity disruption affecting

water production is considered as a valid reason

which lies outside the power of the project.

Output 3: Quantity of

waste water treated

(estimate) (m3)

No

score 3

Satisfactory. The electricity disruption affecting

wastewater treatment is considered as a valid reason

which lies outside the power of the project.

Output 4: Number of

operable public water

points

No

score 2

This output does not capture whether the water

points are in operation. Instead of “operable water

points”, this should have been “functioning water

points”, in order to monitor actual water access.

Output 5: % public

water points operated by

women

No

score 3

The reported figure on progress was not well

calculated. Since target was not reached, the value

should be below 100%.

Output 6: Number of

improved onsite

sanitation facilities

reaching vulnerable

groups

No

score 3 Very good. Satisfactory.

Output 7: % Female

headed households

No

score 2

Unsatisfactory. Narrative uses an absolute number

while the indicator is used as %. The reported figure


work

score

IDEV


benefiting from onsite

sanitation

on progress was not well calculated. With current

value 30% against target of 43.3% the target was not

reached. The value on progress should be below

100%.

Output 8: Number of

new water supply

facilities provided and

rehabilitated in schools

and health centres

No

score 3

Satisfactory. There is unfortunately no details

provided on the no. of health centres.

Output 9: Number of

new sanitation facilities

provided and

rehabilitated in schools

and health centres

No

score 3

Satisfactory. No details are provided on the no. of

health centres.

Output 10: Provision of

solid waste management

equipment

No

score 3

In order to evaluate this, it would have been good to

get more details on the type of solid waste

management (what, how, where, etc.)

Output 11:

Rehabilitation of offices

and provision of

equipment

No

score 3

Satisfactory.


specialized utility

vehicle

No

score 3

Satisfactory.


logistic and utility

vehicles

No

score 3

Satisfactory.

Output 14: No. teachers

and health inspectors/

CDOs trained in

sanitation and hygiene

promotion and

HIV/AIDs as per PAR

No

score 3

Not clear how progress of 84% was calculated.

Based on the no. of teachers and health inspectors

combined, the progress has exceeded the target

(>100%) which is not reflected in the PCR.

Output 15: 50% RDC

women trained S & H

and management

No


Output 16: 7,000

community members

trained in S&H

HIV/Aids

No


Output 17: Reduce girls

absence from schools by

60% per month

No

score 2 Target not reached. Unsatisfactory.

Output 18: Clearing of

Dambo’s

No


Development objective (DO)

Development objective

rating

3 2 Unsatisfactory, there is incomplete or inaccurate

reporting especially at the outcome level, where only

4 of the 9 target outcomes are considered by the

PCR.


work

score

IDEV


Beneficiaries

Beneficiary 1: 704,143

people with improved

access to water supply

in Kitwe, Kalulushi and

Chambishi towns

105% 3 Satisfactory.


people accessing

improved water supply

and sanitation at public

places, schools and

health centres

105% 3 Satisfactory. Overlap exists with the fourth row of

beneficiaries.


people with improved

access to sanitation in

Kitwe, Kalulushi and

Chambishi towns

84% 3 Satisfactory.


people accessing

improved water supply

and sanitation at public

places such as markets

and bus stops, schools,

and health centres

100% 3 Satisfactory. This is overlapping with the second row

of beneficiaries.


people benefitting

from 1,654 sanitation

facilities in Twaiteka,

Ipusukilo and Kamatipa.



Community members

sensitized in WASH,

HIV and malaria


Beneficiary 7: 625

teachers and 4 Health

inspectors

84% 3 Satisfactory.

Unanticipated outcomes (positive or negative not considered in the project logical

framework) and their level of impact on the project (high, moderate, low) Institutional

development

No

score

modera

te

Gender No

score high Project has well defined gender objectives.

Environment & climate

change

No

score high

Reduction of environmental pollution through higher

wastewater treatment capacity.

Poverty reduction No

score

modera

te

Private sector

development

No

score high

Regional integration No

score -


work

score

IDEV


Other (specify)

EFFECTIVENESS OVERALL

SCORE

3 2

EFFICIENCY Timeliness (based on the

initial closing date) 2 2 No diasgreement

Resource used

efficiency 3 4

Highly satisfactory. The project shows overall good

evidence of physical implementation versus

resources used. The median percentage physical

implementation of 100%, compared to a total

commitment rate of 98.92%. The calculated

proportion between the median physical

implementation rate and commitment rate is 1.01

which falls into the category of ≥1 and corresponds

to a rating of highly satisfactory (4), which is higher

than the PCR rating of satisfactory (3) for this

criterion.

Cost-benefit analysis 3 UTR

Regarding calculations and assumptions made,

reference is made to an annex which is not available

to the evaluator. It is therefore difficult to assess the

validity and assumptions made for analysing the

economic costs vs benefits of this project based on

the information provided. Therefore, the evaluator is

unable to compare and evaluate the ratio of

anticipated EIRR at appraisal versus the EIRR at

project completion.

Implementation progress

(from the IPR)

3 2 Unsatisfactory. The PCR narrative reports that

Procurement and financial management remained

consistent with Bank systems and procedures.

However, a number of audit reports reported on

issues found with compliance to financial protocols

and procedure, poor financial management including

incomplete financial reporting, poor record keeping

and improper use of accounting software.

Other (specify)

OVERALL EFFICIENCY SCORE 2.75 2.67 Satisfactory.

SUSTAINABILITY Financial sustainability 2 2

Unsatisfactory.

Institutional

sustainability and

strengthening of

capacities

3 UTR

The PCR report has discussed the various activities

carried out that have led to strengthening technical

capacities of the water utilities and government

agencies as well as community groups. While all

these elements are positive and would help ensure

institutional sustainability, these outcomes should

have been measured and included in the M&E

framework. Simple surveys could have captured

changed capacities at the organisation level,

community level, as well as at the individual level.

This is considered essential and without this kind of


work

score

IDEV


evidence the evaluator is unable to conclude to what

extend institutional sustainability is ensured.

Ownership and

sustainability of

partnerships

3 3

Satisfactory.

Environmental and

social sustainability 3 3

Satisfactory.

*The rating of the effectiveness component is obtained from the development objective (DO) rating in the latest IPR of the

project (see Guidance Note on the IPR).

The ratings for outputs and outcomes are determined based on the project’s progress towards realizing its targets, and the

overall development objective of the project (DO) is obtained by combining the ratings obtained for outputs and outcomes

following the method defined in the IPR Guidance Note. The following method is applied: Highly satisfactory (4),

Satisfactory (3), Unsatisfactory (2) and Highly unsatisfactory (1).


Work

score

IDEV


BANK

PERFORMANCE Proactive identification and

resolution of problems at different

stage of the project cycle

No

score

2 Unsatisfactory. Perhaps more attention

could have been given to timeliness of

implementation of activities and a stronger

supervision on financial administration

after issues reported in the audits.

Use of previous lessons learned

from previous operations during

design and implementation

No

score

3 Satisfactory.

Promotion of stakeholder

participation to strengthen

ownership

No

score

3 Satisfactory.

Enforcement of safeguard and

fiduciary requirements

No

score

3 Satisfactory.

Design and implementation of

Monitoring & Evaluation system

No

score

2 Unsatisfactory. The Bank could have

played a stronger role in ensuring that the

monitoring and evaluation framework

sufficiently captured progress made in the

areas of capacity building, institutional

strengthening and behavioural change.

Quality of Bank supervision (mix

of skills in supervisory teams, etc)

No

score

3 Satisfactory.

Timeliness of responses to

requests

No

score

3 Satisfactory.

OVERALL BANK PERFORMANCE

SCORE

2.71

BORROWER

PERFORMANCE

Quality of preparation and

implementation

No

score

3 Satisfactory.

Compliance with covenants,

agreements and safeguards

No

score

2 Unsatisfactory. The audit reports raised a

number of issues with regards to financial

management and compliance to procedures

and protocols

Provision of timely counterpart

funding

No

score

3 Satisfactory.

Responsiveness to supervision

recommendations

No

score

3 Satisfactory.

Measures taken to establish basis

for project sustainability

No

score

3 Satisfactory. Some issues reported with

regards to financial sustainability.

Timeliness of preparing requests No

score

3 Satisfactory.

OVERALL BORROWER PERFORMANCE

SCORE

2.83

PERFORMANCE

OF OTHER

STAKEHOLDERS

Timeliness of disbursements by

co-financiers

No

score

3 Satisfactory. No major issues were

reported with regards to their performance.

Functioning of collaborative

agreements

No

score

3 Satisfactory.

Quality of policy dialogue with

co-financiers (for PBOs only)

No

score

3 Satisfactory.

Quality of work by service

providers

No

score

3 Satisfactory.

Responsiveness to client demands No

score

3 Satisfactory.

OVERALL PERFORMANCE OF OTHER

STAKEHOLDERS

3

The overall rating is given: Very Good, Good, Fair and Poor.

(i) Very Good (HS) : 4

(ii) Good (H) : 3

(iii) Fair (US) : 2

(iv) Poor (HUS): 1

DESIGN, IMPLEMENTAION AND UTILIZATION OF MONITIRING AND

EVALUATION (M&E)

Criteria Sub-criteria IDEV

Score Comments

M&E DESIGN

M&E system is in place, clear,

appropriate and realistic 2

Unsatisfactory. Discrepancies exist between

outcomes and outputs listed in appraisal

report vs PCR, which is not sufficiently

explained / clarified.

Monitoring indicators and

monitoring plan were duly

approved

3 Satisfactory.

Existence of disaggregated gender

indicator 3

Satisfactory. With regards to beneficiary

reporting, the gender proportion was included

for all groups, however they were not

disaggregated for each group.

Baseline data were available or

collected during the design 3 Satisfactory.

Other, specify

OVERALL M&E DESIGN SCORE 2.75

M&E

IMPLEMENTA-

TION

The M&E function is adequately

equipped and staffed 2 Unsatisfactory.

OVERALL M&E IMPLEMENTATION SCORE 2

Criteria Sub-criteria IDEV

Score Comments

M&E

UTILIZATION

The borrower used the tracking

information for decision UTR

Not clear from the information whether this

was done.

OVERALL M&E UTILIZATION SCORE UTR

OVERALL M&E PERFORMANCE SCORE 2.38 Unsatisfactory.

PCR QUALITY EVALUATION

Criteria

PCR-

EVN (1-

4)

Comments

QUALITY OF PCR

1. Extent of quality and completeness of the PCR

evidence and analysis to substantiate the ratings of

the various sections

3 Satisfactory.

2. Extent of objectivity of PCR assessment score 3 Satisfactory.

3. Extent of internal consistency of PCR assessment

ratings; inaccuracies; inconsistencies; (in various

sections; between text and ratings; consistency of

overall rating with individual component ratings)

2 Unsatisfactory. The outputs and outcomes listed in

the PCR are not consistent with those listed in the

log-frame of the project appraisal report

4. Extent of identification and assessment of key

factors (internal and exogenous) and unintended

effects (positive or negative) affecting design and

implementation

3 Satisfactory.

5. Adequacy of treatment of safeguards, fiduciary

issues, and alignment and harmonization

3 Satisfactory.

6. Extent of soundness of data generating and

analysis process (including rates of returns) in

support of PCR assessment

3 Satisfactory.

7. Overall adequacy of the accessible evidence (from

PCR including annexure and other data provided)

3 Satisfactory. However, not all annexes were

accessible.

8. Extent to which lessons learned (and

recommendations) are clear and based on the PCR

assessment (evidence & analysis)

3 Satisfactory.

9. Extent of overall clarity and completeness of the

PCR

3 Satisfactory.

Other (specify)

PCR QUALITY SCORE 2.90

PCR compliance with guidelines (PCR/OM ; IDEV)

1. PCR Timeliness (On time = 4; Late= 1) 1 PCR report was completed more than 6 months

after project completion.

2. Extent of participation of borrower, Co-financiers

& field offices in PCR preparation

3 Satisfactory.

3. Other aspect(s) (specify)

PCR COMPLIANCE SCORE 2

*** rated as Very Good (4), or Good (3), or Fair (2), or Poor (1)

References

African Development Bank Group, Policy for integrated water resources management, April 2000

African Development Bank Group, Staff Guidance on Project Completion Reporting and Rating, Quality

Assurance and Results Department (ORQR), Quality Assurance Division (ORQR.2), August 2012

Project Completion Report for Nkana Water Supply and Sanitation Project. November 2015

Government of the Republic of Zambia, National Water Policy, Ministry of Energy and Water

Development, February 2010.

Government of the Republic of Zambia. National Urban and Peri-Urban Sanitation Strategy (2015 - 2030)

Ministry of Local Government and Housing (MLGH). November 2015

Government of the Republic of Zambia, Water Act Ministry of Legal Affairs, 2006

Various other internal reports, memos, trip reports, progress reports and audit reports