final report final evaluation of eu water facility wash ......final evaluation of eu water facility...
TRANSCRIPT
Final Report
Final Evaluation of EU Water Facility WASH Programme in 4 districts and 4 Municipalities
30 July 2010
Index
Confidential –All Rights Reserved Ernst & Young – 2010
Final Evaluation of EU Water Facility WASH Programme in Four districts and Four Municipalities – Final Report
Page 1
Index
Final Report ............................................................................................................................ 1
Index ...................................................................................................................................... 1
List of Abbreviations ................................................................................................................ 2
Executive Summary ................................................................................................................. 4
1. Project and Evaluation Objectives ......................................................................................... 6
1.1. Project Objectives .................................................................................................... 6
1.2. Evaluation Objectives ............................................................................................... 7
2. Evaluation Methodology ....................................................................................................... 8
3. Limitations of the Evaluation ................................................................................................ 9
4. Findings and Discussion of Findings ..................................................................................... 10
4.1. Project Description ................................................................................................. 10
4.2. Origin, context and key assessment ......................................................................... 11
4.3. Summary of project implementation ........................................................................ 12
4.4. Changes in context and in the key assessment areas ................................................. 14
4.5. Progress in achieving the objectives ........................................................................ 14
4.6. Financial Execution ................................................................................................ 16
4.7. Issues and action required ...................................................................................... 17
4.8. Cross-cutting and other issues................................................................................. 17
4.9. Main conclusions and Recommendations .................................................................. 19
Annexes ............................................................................................................................... 22
Annex 1 – Terms of Reference ................................................................................................ 23
Annex 2 – List of people interviewed ....................................................................................... 30
Annex 3 – EU WF Final Evaluation Outline 3 ............................................................................. 33
Annex 4 – List of Documents Reviewed .................................................................................... 35
Annex 5 – Guideline of semi-structured interviews to UNICEF .................................................... 37
Annex 6 – Guideline of semi-structured interviews to DNA ......................................................... 39
Annex 7 – Guideline of semi-structured interviews to DPOPH ..................................................... 41
Annex 8 – Guideline of semi-structured interviews for Municipalities .......................................... 43
Annex 9 – Guideline of semi-structured interviews for Water Committees ................................... 45
Annex 10 – Guideline of semi-structured interviews for Schools ................................................. 47
Annex 11 – Details on Section 4.5 ........................................................................................... 49
Annex 12 – PPT for the Final Evaluation Report ........................................................................ 53
List of Abbreviations
Confidential –All Rights Reserved Ernst & Young – 2010
Final Evaluation of EU Water Facility WASH Programme in Four districts and Four Municipalities – Final Report
Page 2
List of Abbreviations
ACP African, Caribbean and Pacific
AIDS Acquired Immune Deficiency Syndrome
CF Common Fund
CLTS Community Led Total Sanitation
DAS Department of Water and Sanitation
DC
DNA
Demonstration Centres
National Directorate of Water
DPOPH Provincial Directorate of Public Works and Housing
DRA Demand Responsive Approach
EC European Commission
EUWF European Union Water Facility
FIPAG
GIS
Fundo de Investimento e Património de Agua
Geographic Information System
GoM Government of Mozambique
HH Household
HIV Human Immunodeficiency Virus
IWRM Integrated Water Resource Management
JMP Joint Monitoring Programme
MDG Millennium Development Goals
NGO Non-Governmental Organization
NWP National Water Policy
O&M Operation and Maintenance
ODF Open Defecation Free
PEC Participation and Community Education
PES Plano Económico e Social
PESAR Plano Estratégico de Agua Rural
PESOD Plano Económico e Orçamento Distrital
PHAST Participatory Hygiene Water and Sanitation Transformation
PRONASAR Programa Nacional de Agua e Saneamento Rural
PRSP Poverty Reduction and Strategy Paper
SDPI District Services of Planning and Infrastructures
SINAS Sistema de Informação Nacional de Agua E Saneamento
SWAPs Sector-wide Approaches
UNICEF United Nations Children's Fund
List of Abbreviations
Confidential –All Rights Reserved Ernst & Young – 2010
Final Evaluation of EU Water Facility WASH Programme in Four districts and Four Municipalities – Final Report
Page 3
UTAS Unidade de Tratamento de Água e Saneamento
WASH Water, Sanitation and Hygiene
WC Water Committee
WSSD Water and Sanitation Sustainable Development
ZAMWAT Zambeze Water
Executive Summary
Confidential –All Rights Reserved Ernst & Young – 2010
Final Evaluation of EU Water Facility WASH Programme in Four districts and Four Municipalities – Final Report
Page 4
Executive Summary
Under the European Union Water Facility (EUWF), the European Commission (EC) and UNICEF
Mozambique signed a four year programme Contribution Agreement on July 21, 2006 targeting
water, sanitation and hygiene in eight vulnerable areas in Mozambique. The overall objective of the
EUWF was to contribute to poverty reduction and sustainable development through the
achievement of the Millennium Development Goals (MDGs) and Water and Sanitation Sustainable
Development (WSSD) targets in African, Caribbean and Pacific (ACP) countries.
In line with the Government of Mozambique (GoM) and UNICEF Cooperation Programme 2007-
2009, the overall objective of the EUWF WASH Programme was “to contribute to achieving the
Millennium Development Goals (MDG) and Poverty Reduction and Strategy Paper (PRSP) targets for
water and sanitation in vulnerable rural and peri-urban areas of selected provinces and districts of
Mozambique”
The Specific Objectives of the Programme were “to provide increased access to safe and affordable
water, sanitation and hygiene facilities for vulnerable communities in four rural districts and four
peri-urban areas of Sofala and Zambezia Provinces, which have been selected due to high
vulnerability indicators, including diarrhoea prevalence, cholera cases and HIV/AIDS prevalence”.
According to the logical framework the project was expected to achieve the following results: (i)
National Water Policy (NWP) guidelines disseminated and sector planning, monitoring and
evaluation strengthened; (ii) Provincial, municipal and district level capacity to plan, manage,
monitor and evaluate the WASH sector using NWP guidelines and DRA (Demand Responsive
Approach) strengthened; (iii) Participatory and sustainable Water Committees (WC) put in place and
operational, as well as hygiene practices and HIV/AIDS awareness improved in all project districts
and municipalities strengthened; (iv) Access to safe and affordable water supply, sanitation and
hygiene facilities improved at communities levels and schools in all eight district projects; and (v)
Project management and technical assistance timely provided on a yearly basis. In particular, by the
end of the Programme it was expected that:
water points;
dren with safe access to sanitation and
would had benefited from
hygiene promotion.
Executive Summary
Confidential –All Rights Reserved Ernst & Young – 2010
Final Evaluation of EU Water Facility WASH Programme in Four districts and Four Municipalities – Final Report
Page 5
The main objective of this study was to evaluate the EUWF WASH Programme. UNICEF, the EC and
the GoM were keen on evaluating the performance, impact on target population and partner’s
performance along the Programme implementation.
The evaluation was carried out through desk review of the main project documents. In addition,
personal interviews using interview guides and outline discussed and agreed with UNICEF; direct
observation; informal conversation; and focus group discussions. The evaluation team visited two
out of four districts (Dondo and Nicoadala) and three out of four municipalities (Beira, Dondo and
Quelimane) from Sofala and Zambézia provinces, in the project areas as to assess in depth, the
different aspects of the project activities. Field visits included spot check on various project outputs
i.e. water points, latrines; dish racks; hand washing facilities; etc. and discussion with the WC
members and other key informants.
This evaluation report is based on the format provided by UNICEF included in Section 7: Monitoring,
Review and Reporting - CRIS “Implementation Report“ of the European Commission - Aid Delivery
Methods – Project Cycle Management Guidelines (Volume 1 – March 2004). It provided the main
headings, such as, project description; origin, context and key assessment; summary of project
implementation; changes in context and in the key assessment areas; progress in achieving
objectives; financial execution; issues arising and action required; and cross-cutting and other
issues.
Main findings from the evaluation team were: (i) the capacity building component was rolled out as
per the Programme Plan, with nearly all guideline targets, trainings and workshops achieved and
with strong results in capacity strengthening at national and provincial levels as well as in private
sector. Capacity building at district and municipal level, especially in planning and monitoring
programme activities, had been overall good but with some challenges in district, municipalities and
with NGO (Non-Governmental Organization) staff. In terms of WASH infrastructure provision
component in the target areas, in all cases it exceeded the targets set for infrastructure provision
and the number of people covered by the programme. Especially the construction and rehabilitation
of water sources as well as motivating households to construct their own latrines (without subsidy)
have exceeded expectations in terms of end beneficiaries covered.
At institutional level, the Programme has contributed to the realization of decentralized governance
including planning, implementing and monitoring of interventions. The sense of ownership of the
Programme by (sub) national authorities has been strong.
The total project budget for the four year period 2006-2010 was EUR 5,570,279.20, of which EC
and UNICEF contributed fifty per cent (EUR 2,785,139.60) each. In terms of expenditure, by June
2010 EUR 5,080,676.68 has been spent, which corresponds to 91% of the funds available.
Project and Evaluation Objectives
Confidential –All Rights Reserved Ernst & Young – 2010
Final Evaluation of EU Water Facility WASH Programme in Four districts and Four Municipalities – Final Report
Page 6
1. Project and Evaluation Objectives
1.1. Project Objectives
Under the European Union Water Facility (EUWF), the European Commission (EC) and UNICEF
Mozambique signed a four year programme Contribution Agreement on July 21, 2006 targeting
water, sanitation and hygiene in eight vulnerable areas in Mozambique. The overall objective of the
EUWF was to contribute to poverty reduction and sustainable development through the
achievement of the Millennium Development Goals (MDGs) and Water and Sanitation Sustainable
Development (WSSD) targets in African, Caribbean and Pacific (ACP) countries.
In this context, and in line with the Government of Mozambique (GoM) and UNICEF Programme of
Cooperation 2007-2009, the overall objective of the EUWF was “to contribute to achieving the
Millennium Development Goals (MDG) and Poverty Reduction and Strategy Paper (PRSP) targets for
water and sanitation in vulnerable rural and peri-urban areas of selected provinces and districts of
Mozambique”
The Specific Objectives of the Programme were “to provide increased access to safe and affordable
water, sanitation and hygiene facilities for vulnerable communities in four rural districts and four
peri-urban areas of Sofala and Zambezia Provinces, which have been selected due to high
vulnerability indicators, including diarrhoea prevalence, cholera cases and HIV/AIDS prevalence”.
According to the logical framework the project was expected to achieve the following results:
► NWP guidelines disseminated and sector planning, monitoring and evaluation strengthened;
► Provincial, municipal and district level capacity to plan, manage, monitor and evaluate the
WASH sector using NWP guidelines and DRA is strengthened;
► Participatory and sustainable community management structures engaged and operational,
and hygiene practices and HIV/AIDS awareness improved in all project districts and
municipalities strengthened;
► Access to safe and affordable water supply, sanitation and hygiene facilities improved at
communities and schools in all eight project districts;
► Project management and technical assistance timely provided on a yearly basis.
Project and Evaluation Objectives
Confidential –All Rights Reserved Ernst & Young – 2010
Final Evaluation of EU Water Facility WASH Programme in Four districts and Four Municipalities – Final Report
Page 7
In particular, by the end of the Programme it was expected that: (i)
safe and sustainable water points; (ii) 25,000 families (125,000 people) with access to safe
sanitation facilities (Household latrines);
sanitation and hygiene facilities (latrines, hand washing, urinals); and
with hygiene promotion.
1.2. Evaluation Objectives
The main objective of this study is to evaluate the EUWF WASH Programme. UNICEF, the EC and the
GoM were keen on evaluating the performance, impact on target population and partners’
performance in the Programme implementation.
The information gathered in this assignment will contribute to the improvement of the Monitoring
and Evaluation system within the WASH sector. Additionally, lessons learnt in the course of the
evaluation would be very useful for similar ongoing WASH programmes in Mozambique, such as the
One Million Initiative. Moreover, the information collected through the study will help to assess
whether recommendations and conclusions from the previous Programme evaluations of (EU
Monitoring visit and Sustainability Checks) were integrated in this approach.
According to the Terms of Reference provided by UNICEF, specifically the evaluation was:
► To make an assessment of how far has the Programme Monitoring and Evaluation
framework been used by stakeholders;
► To make an assessment of the adequacy and appropriateness of the training provided to the
end-users;
► To make an assessment of the sustainability of water, sanitation and hygiene facilities;
► To make an assessment of the functioning of the WCs put in place; and
► To make an assessment of the involvement and awareness of the end-users in the
management of water points and shared sanitation facilities.
The detailed Terms of Reference for the evaluation are provided in Annex1.
Evaluation Methodology
Confidential –All Rights Reserved Ernst & Young – 2010
Final Evaluation of EU Water Facility WASH Programme in Four districts and Four Municipalities – Final Report
Page 8
2. Evaluation Methodology
The evaluation was carried out through desk review of the main project documents (please refer to
Annex 2). In addition, personal interviews using interview guides and outline discussed and agreed
with UNICEF (please refer to Annex 3); direct observation; informal conversation; and focus group
discussions (kindly refer to the list of interviewees in Annex 4 and semi-structured interviews guides
to the target group in Annexes 5, 6, 7, 8, 9 and 10).
Key informants consulted included project staff at national level and as well as in target provinces,
districts and municipalities, officers from the Provincial Directorate of Public Works and Housing
(DPOPH) and District Services of Planning and Infrastructures (SDPI).
The evaluation team visited two out of four districts (Dondo and Nicoadala) and three out of four
municipalities from Sofala and Zambézia provinces, in the project areas as to assess in depth, the
different aspects of the project results.
Field visits included spot check on various project outputs i.e. water points, latrines; dish racks;
hand washing facilities; etc. and discussion with the WCs members and other key informants.
This evaluation report is based on the format provided by UNICEF included in Section 7: Monitoring,
Review and Reporting - CRIS “Implementation Report“ of the European Commission - Aid Delivery
Methods – Project Cycle Management Guidelines (Volume 1 – 2004).
Limitations of the Evaluation
Confidential –All Rights Reserved Ernst & Young – 2010
Final Evaluation of EU Water Facility WASH Programme in Four districts and Four Municipalities – Final Report
Page 9
3. Limitations of the Evaluation
Limitations of the evaluation were related to the scope of the work, timeframe and budget. The
main limitations were the following:
► It was not included in the scope of the work a detailed assessment of the activities
performed by the programme. The evaluation focus mainly on the achievement of the
expected results;
► An in depth assessment of the end-users was also not included in the scope of the work. The
evaluation focuses on the impact of the project but did not perform a survey on the end-
users;
► The report format was agreed after the field work, which poses an enormous challenge for
the evaluation team;
► Limited timeframe and budget did not allow a more in-depth evaluation.
Findings and Discussion of Findings
Confidential –All Rights Reserved Ernst & Young – 2010
Final Evaluation of EU Water Facility WASH Programme in Four districts and Four Municipalities – Final Report
Page 10
4. Findings and Discussion of Findings
The findings are presented and discussed based on the CRIS “Implementation report” format. The
format provides the main headings, such as project description; origin, context and key assessment;
summary of project implementation; changes in context and in the key assessment areas; progress
in achieving objectives; financial execution; issues arising and action required; and cross-cutting and
other issues.
4.1. Project Description
Overall objective: The Overall Objective of the project was to contribute towards the achievement
of MDG and Poverty Reduction and Strategy Paper (PRSP) targets for water and sanitation in
vulnerable rural and peri-urban areas of selected provinces and districts of Mozambique.
Purpose: The program aimed at providing improved access to safe and affordable drinking water,
sanitation and hygiene facilities for vulnerable communities in 4 rural districts and 4 municipalities.
Results: The project results were (i) National water policy guidelines disseminated and sector
planning, monitoring and evaluation strengthened; (ii) Provincial, municipal and district level
capacity to plan, manage, monitor and evaluate the WASH sector using NWP guidelines and DRA
strengthened; (iii) Participatory and sustainable community management structures engaged and
operational, as well as hygiene practices and HIV/AIDS awareness improved; and (iv) Access to safe
and affordable water supply, sanitation and hygiene facilities improved at communities levels as well
as in schools; and (v) Project management and technical assistance timely provided on a yearly
basis. In particular, the project expected results by 2010 were: (i) 150,000 people with access to
safe and sustainable water points; (ii)
sanitation facilities (Household latrines); (iii) 120 schools and 45,000 schoolchildren with safe
access to sanitation and hygiene facilities (latrines, hand washing, urinals); (iv) 120,000 people had
benefited from hygiene promotion.
Main activities: (i) provide capacity building which included guidelines, trainings and workshops to
strengthen capacity at national and provincial levels as well for private sector; (ii) provide capacity
building at district and municipal level, especially in planning and monitoring programme activities;
(iii) provision of WASH infrastructure in the target areas. Especially the construction and repair of
water sources as well as motivating households to construct their own latrines (without subsidy);
Findings and Discussion of Findings
Confidential –All Rights Reserved Ernst & Young – 2010
Final Evaluation of EU Water Facility WASH Programme in Four districts and Four Municipalities – Final Report
Page 11
(iv) provide assistance at the institutional level, to the realization of decentralized governance
including planning, implementing and monitoring of interventions.
Location and length: Districts of Nicoadala, Milange, Buzi and Nhamatanda and municipalities of
Beira, Dondo, Mocuba and Quelimane in Sofala and Zambézia provinces. The length of the
programme was four years.
Cost and key inputs: Budget for the four year period 2006-2010 was EUR 5,570,279.20, of which
EC and UNICEF contributed fifty per cent each. Key inputs were: (i) staff from WASH, Education and
Health Sections, and (ii) supplies (cars, motorcycle, computers and other equipment).
4.2. Origin, context and key assessment
Rationale/justification for the project: One of the major constraints affecting the water sector is
the high percentage of water points out of order (30 percent). This is due to lack of capacity in
Operation and Maintenance (O&M) linked to the weak supply chain, and to hydro-geological and
climatic factors, such as wells drying out. The capacity development initiative aimed at improving
the overall management of activities, both during and after the external funding period, in line with
the national policies and strategies of decentralisation, integrated water and sanitation and
Integrated Water Resource Management (IWRM) principles, community involvement and the use of
appropriate technologies. This will not only ensure the sustainability of development activities but in
addition the potential for synergistic benefits to this project was also expected to be exploited. The
promotion of appropriate water and sanitation technology options and the establishment of local
centres for spare parts (in direct partnership with the private sector) would contribute to the
sustainability of development activities. This would be complemented with an intensive application
of participatory methodologies for community development, improved hygiene practices and
community based management, which will enhance demand in providing better services and was
expected to promote community ownership.
Main conclusions arising from the assessment of the project context: The assessment was
conducted based on problem analysis. As a result, the four key blocks defined for this project were
identified as crucial to ensure an integrated and coordinated intervention in the sector, such as: (i)
NWP dissemination and overall sector planning, monitoring and evaluation; (ii) Provincial and
district level planning, monitoring and evaluation; (iii) Sustainability of WASH infrastructures and
Findings and Discussion of Findings
Confidential –All Rights Reserved Ernst & Young – 2010
Final Evaluation of EU Water Facility WASH Programme in Four districts and Four Municipalities – Final Report
Page 12
hygiene promotion; and (iv) Access to affordable water and sanitation facilities at community and
school levels.
In that time, the Government has recognized these priorities as crucial to attain the MDGs on WASH
by 2015 and by doing so to contribute to reducing poverty in accordance with the latest PRSP
(2006-2009). The project would strengthen ongoing programmes in the Provinces of Zambezia and
Sofala. This EUWF project would build on the good progress already made in water and consolidate
the actions developed in sanitation and hygiene in schools and communities. The provincial capacity
to manage medium level projects (1 to 2 million USD/year) had been tested with success and a new
project would be crucial to consolidate this capacity and improve the gaps already existing (better
planning and M&E skills, school and household sanitation and hygiene). In addition, this project
would focus on peri-urban areas (Quelimane, Mocuba and Beira) and would consolidate the
experience of integrating WASH improvement in schools with overall community management
capacities.
4.3. Summary of project implementation
Main features of the implementation highlighting main developments: The project aimed at
strengthening the planning, management and monitoring and evaluation (M&E) capacity at national,
provincial, municipal and district levels. It provided training for hundreds of sanitation, health and
education workers at the provincial and district level as well as the training of mechanics in the
maintenance of hand pumps and community activists and teachers in the promotion of good
hygiene practices. The project also supported the creation of sanitation clubs in schools and
awareness programmes on hygiene promotion in the classroom. Another important component of
the project included activities directly linked to the service delivery of water, sanitation and hygiene
facilities.
Problems encountered: in Table below, all major problems encountered and the solutions given are
presented:
Problems encountered Solutions given
(i) weak capacity of the private sector to deliver
timely, creative and high quality services;
(i) provided training to the local construction
companies, a new contract management process for
drilling of borehole was successfully introduced;
(ii) shifted from the inclusion of geophysical surveys in
the contracts of the supervising engineers to include
them in the contract with the drilling contractors,
whose effect was that the many negative boreholes
Findings and Discussion of Findings
Confidential –All Rights Reserved Ernst & Young – 2010
Final Evaluation of EU Water Facility WASH Programme in Four districts and Four Municipalities – Final Report
Page 13
Problems encountered Solutions given
drilled before, were not any longer paid by UNICEF but
accountable to the contractor themselves;
(ii) low sustainability of the constructed and
rehabilitated WASH facilities, particularly in relation
to WC and the O&M of the facilities;
Strong focus on capacity building and strengthening
the Water User Groups in the Annual Work Plan for
2010. A technical manual was developed for WCs to
improve their performance, accountability and
transparency;
(iii) limited human resource capacity both at district
and municipality levels;
hired technicians for WASH in each district and
municipalities to be later integrated in SDPI;
(iv) lack of quantitative indicators related to NGO
contract management to effectively measure the
social mobilisation and hygiene promotion outputs of
NGO partners.
continued strengthening the monitoring and
evaluation systems through the SINAS/JMP
initiatives.
Lessons learned: (i) New results-based contract management process for drilling of boreholes has
been successful in reducing the cost per borehole and ensured the quality of infrastructure;
(ii) The establishment of WC requires more consideration in terms of follow-up training and
monitoring to ensure continued sustainable use of the water supply infrastructure as well as
providing more guidance for management of facilities. The experience has been that sequencing of
PEC with the construction of the water points is crucial, as well as monitoring of important
indicators regarding the organisation of the WCs. The results based contracts for PEC, implemented
from 2008 onwards, are anticipated to significantly improve the monitoring of the WCs, however,
this remains an area of concern. In addition, more guidance has to be given over continued periods
of time to WCs that experience problems with establishing proper systems of tariffs and collection
of revenue, which has a knock-on effect on proper and timely maintenance of water points. One of
the measures taken is the development of a Maintenance Manual to guide WC in management of
their water and sanitation facilities and assure transparency and accountability to improve financial
management as well;
(iii) The need for more flexibility and adjustments during the project implementation, taking into
account the changes in the context and the sector dynamic;
(iv) Regarding to the project impact, it was clear that rural and peri-urban areas have different
characteristics and specifications, and therefore future interventions need to be more geographical
focused for a better definition of intervention models;
(v) The coordination process among the implementation partners at central and provincial level
(UNICEF, DNA and DPOPH) requires time and should be defined and shared in advance to
accommodate potential administrative constrains;
Findings and Discussion of Findings
Confidential –All Rights Reserved Ernst & Young – 2010
Final Evaluation of EU Water Facility WASH Programme in Four districts and Four Municipalities – Final Report
Page 14
(vi) Provinces, municipalities and districts have shown the capacity to manage directly funds
allocated by the project for the local level (around 89% of funds were directly managed by the
provinces, municipalities and districts); and
(vii) The programme established sanitation clubs, trained teachers and students on sanitation and
hygiene promotion. These activities have not only being done in the schools but also in the
surrounded areas of the school. In this context the sanitation clubs mobilized the communities on
sanitation and hygiene promotion, as well. This can be considered as an added value for the
programme, and future interventions may consider this integrated approach.
4.4. Changes in context and in the key assessment areas
From 2007 the rural water and sanitation decentralization process was arranged. The district was
made responsible for water coverage maintenance ensuring the functionality of existing water
points through repair of non-functional water points. It was also introduced the PEC Zonal which is
managed at district level and is being accepted as a way to sustainable water and sanitation. This
approach contributed to reduce water point’s breakdowns and increased the water coverage.
The PEC ZONAL and CLTS approach had contributed to increase the willingness of community and
citizen to have improved water and sanitation facilities, such as improved toilet and tap in house. It
also contributed to trigger the innovation of the communities on latrine construction.
The collaboration between UNICEF and FIPAG on pipe extension had conducted to a situation that
the pressing over the water points with hand pump and with taps in the community reduced because
a considerable number of families decided to have their own taps in their house. These facts
mentioned above were relevant because they contributed to the sustainability of WASH facilities.
4.5. Progress in achieving the objectives1
The programme is expected to have contributed towards the measured increases in coverage;
however, it is not assumed that it is the only attributing factor as several other interventions might
have been contributing to increase coverage as well. Almost all planned outputs have been realized
or exceeded. In both provinces, HH with access to safe water sources increased over the target
defined in the programme, i.e, it was expected an increase of 6%, having been reached an average
1 This section is based on Final Donor Report for European Commission SC/06/450 produced by UNICEF – June 2010.
Findings and Discussion of Findings
Confidential –All Rights Reserved Ernst & Young – 2010
Final Evaluation of EU Water Facility WASH Programme in Four districts and Four Municipalities – Final Report
Page 15
of 12%. In terms of safe sanitation facilities, in average the target has been achieved, although the
increase in Zambézia had been of 3% against 5% defined as target.
An assessment of the progress in achieving each of the expected results is presented below:
Result 1: District Government and Municipality were trained on NWP guidelines which improved
their skills to disseminate and promote it at the community level. The improvement of their skill
leaded to better provision of water and sanitation services as they improved their planning and
monitoring and evaluation skills. The WCs have improved their skills in water and sanitation
infrastructures management, as the WCs members have received training and refreshments on
basic management skills and also have received support from the district government and
municipality in terms of their legitimacy as legal community representative. The sustainability check
conducted by Austral Cowi in 2009, show that the levels of sustainability in Milange and Nicoadala
were highest compared to the other districts (sustainability index of 51,78% in Milange and 49,88%
in Nicoadala). The same situation was found in Dondo (90%), which has also one of the highest levels
of sustainability (please refer to Ernst & Young – Sustainability Check of WASH Activities in 9
Districts).
Regarding to the WASH data bank, it were established in 2007 with indicators for the target
provinces. As part of the process for developing the MDG Road Map, the Strategic Plan for the Rural
Water Supply and Sanitation (PESA-ASR, 2006-2015) was developed with active participation from
WES stakeholders from sub-national and national levels, including the project targeted districts and
municipalities.
Result 2: According to UNICEF, around 89% of the programme funds (cash and goods) were directly
managed by provinces, districts and municipalities, from which 3%; 55% and 31%, respectively,
exceeding the 70% defined by the project. The project has provided training on e-SISTAFE and on
Procurement rules and procedures to strengthen the management capacity of the funds.
In the four districts and four municipalities the planning process took place. The plans are available
and based on the assistance received the staff have developed capacity to plan and implement the
activities.
Result 3: All water points have WC in place. Although there are monthly paying schemes working in
the districts, in urban areas the payments schemes doesn’t work well because there are alternative
sources of water available. The current percentages of households in programme areas with access
to safe water sources is in average 60%, being Nicoadala 48.2%, Dondo 81% and Quelimane (60%).
Findings and Discussion of Findings
Confidential –All Rights Reserved Ernst & Young – 2010
Final Evaluation of EU Water Facility WASH Programme in Four districts and Four Municipalities – Final Report
Page 16
This average represents three out of eight target areas2, which does not allow for a conclusion.
However, the target of 90% set by the programme was too ambitious as the baseline (DNA 2003) in
the two selected provinces (Sofala and Zambézia) was in average 38%.
Result 4: The Programme Specific Objectives had by and large been achieved, namely: (i) over
296,000 people have access to safe drinking water, through construction and rehabilitation of 313
water points; (ii) approximately 215,000 people (nearly 39,000 families) in rural and peri-urban
areas had access to safe sanitation facilities; (iii) around 58,450 learners in 167 schools in poor
rural and peri-urban areas had access to safe school WASH facilities; and approximately 134,000
people in rural and peri-urban areas.
Result 5: Project management and technical assistance was timely provided on a yearly basis for
the implementation of the project.
For more details vide Annex 11.
4.6. Financial Execution
The total project budget for the four year period was EUR 5,570,279.20, of which 50 per cent (EUR
2,785,139.60) was funded by the EC and 50 per cent by UNICEF. The budget allocation was in line
with the need to spend the resources in capacity building and in the provision of WASH facilities. A
large portion of the total budget, around 83%, was allocated to two core project items of
expenditure, namely (i) subcontracting related to construction activities; and (ii) other cost category
(self-help construction of HH latrines, PEC and support training programmes (PEC, DRA, PHAST);
support preparation of MDG progress report; support the establishment of provincial data banks and
GIS. The remaining 17% were allocated to equipment and supplies, human resources, administrative
costs. In terms of expenditure, by June 2010 EUR 5,080,676.68 has been spent, which
corresponds to 91% of the available funds. Some over and under deviations of the budget occurred
but they were all necessary and timely agreed with EC, namely: (i) human resources due to the fact
that the time that additional WASH staff has spent to the EU project has been billed accordingly; (ii)
Latrines (self-help construction of san plats) due to the introduction of CLTS which has costs
significantly low; (iii) M&E (provincial databanks) due to existence of alternative funds and (iv)
services (geophysical studies for boreholes) due to the new contractual arrangement introduced.
2 Data from Beira, Milange, Mocuba, Búzi and Nhamatanda.
Findings and Discussion of Findings
Confidential –All Rights Reserved Ernst & Young – 2010
Final Evaluation of EU Water Facility WASH Programme in Four districts and Four Municipalities – Final Report
Page 17
4.7. Issues and action required
The main issue was the sustainability. In assessing the sustainability it was important to highlight
that the two main project focuses were capacity building and provision of water and sanitation
facilities. A variety of people at provincial, district and municipalities level had been trained, and
they were prepared to continue providing services to the communities. The activities that would be
sustained include the WCs and the maintenance of the water points.
Although limited human resource capacity exists in some selected areas, it was expected that the
trained community members would continue sensitizing their and other communities in sanitation
and hygiene matters. Therefore, the improvements in the living conditions of the target population
would be maintained.
Evidences from the field visits indicated that these activities would be continued. The staff of
DPOPH/DAS, Municipalities and District has acquired knowledge and skills to implement the
activities including in their annual plan. However, high staff rotation within the Government
counterparts may represent a challenge for the sustainability, as a result of the absence of
“institutional memory” that lies on the technician.
Sustainability levels of water, sanitation and hygiene facilities could be considered satisfactory. WCs
put in place were working, but they seemed to require a close monitoring, because in some places
visited the percentage of families paying for the services was very low, which may pose a big
challenge for the future sustainability.
The sustainability relied also on the need to strengthen the capacity of the private sector to deliver
timely and creative services at local level, which also might require additional support.
4.8. Cross-cutting and other issues
Poor women and children are most at risk and frequently made vulnerable from lack of water and
sanitation facilities, that is why the project addressed the gender equality and child issues during its
implementation. Water sector institutions, including WC, are generally dominated by men who often
resist the equal representation of women in decision making bodies. For this reason and the
recognition of the fact that women play a central role in the provision, management and
safeguarding of water, the project addressed the gender equality in its implementation.
Findings and Discussion of Findings
Confidential –All Rights Reserved Ernst & Young – 2010
Final Evaluation of EU Water Facility WASH Programme in Four districts and Four Municipalities – Final Report
Page 18
That was verified through the following aspects: (i) Most of WC have women playing a relevant role
as a president or chief of maintenance group; (ii) In some communities women were trained as
mechanics, which repair the water points. In schools, children play a role in the sanitation clubs
established in the schools. The project worked with communities and schools to provide sanitary and
hygiene education for both boys and girls ensuring that both were responsible for cleaning and
toilet maintenance. In schools, boys and girls play the same roles in the sanitation and hygiene
promotion within established schools sanitation clubs. These sanitation clubs were also mobilizing
the surrounded communities on the hygiene promotion acting as transforming agent.
An environmental assessment has not been carried out in the selected areas. However, the
potential negative environmental implications are considered negligible. It was, therefore, felt that
water supply interventions will not impact negatively on groundwater resources. The major
contribution in the environment issues was through the sanitation component, as follows: (i)
stimulating the HH to construct latrines and innovate new types of latrines; (ii) CLTS have
contributed to create a healthy and sustainable environment; (iii) The implementation of Integrated
Water Resources Management approach which has contributed to develop in each province an
integrated Master Plans, ensuring sustainable use of water resources; (iv) The construction of wells
and boreholes, finished to a standard to protect groundwater supplies was well practised in
Mozambique. The type of hand pump employed, Afridev, usually restricts abstraction rates to less
than one litre per second and their disbursement is widespread; (v) To improve routine water quality
testing and treatment, more affordable and community friendly tools and materials were introduced
(H2S Test Kits), integrated with the participatory community education campaigns for safer hygiene
practices.
The HIV/AIDS has been an important issue addressed by the programme. It is now widely accepted
that availability of safe water and sanitation does not automatically lead to improvement in health if
not combined with hygiene promotion and appropriate hygiene behaviours. This was particularly
important for People Living with HIV/AIDS. In a HIV/AIDS context it was important that water supply
pits and latrines were easily accessible and close to where needed to reduce the burden of long-
distance water collection for care-givers and also to reduce the risk of girls and women being raped
while fetching water, and thus reducing vulnerability of HIV infection. Activists were trained under
the Programme to promote sanitation and hygiene and also trained to promote HIV/AIDS awareness
in the communities, allowing them to change their behaviour.
Findings and Discussion of Findings
Confidential –All Rights Reserved Ernst & Young – 2010
Final Evaluation of EU Water Facility WASH Programme in Four districts and Four Municipalities – Final Report
Page 19
4.9. Main conclusions and Recommendations
The findings presented and discussed under five traditional practice of evaluation of development
aid formalised by the OECD-DAC criteria, namely: Relevance of the project, Efficiency in terms of
implementation, Effectiveness in achieving the results, Impact the project had on the lives of the
beneficiaries and Sustainability of the results, are presented below.
Relevance
The programme was highly relevant. It was implemented in areas selected due to their high
vulnerability indicators, including diarrhoea prevalence, cholera cases and HIV/AIDS prevalence, of
four rural districts and four peri-urban areas of Sofala and Zambézia Provinces. In addition, its
objectives are aligned with the GoM main policy documents regarding to the sector, namely PARPA
and National Water Policy.
Efficiency: The programme set up and implementation procedures were adequate for achieving of
project objectives. The implementation model was adequate and the involvement of provincial,
district, municipality and water committees creates awareness on the project implementation and
results. There was also effective backstopping from UNICEF through monitoring visits that provided
support in relevant areas of the project implementation. The use of both human resources and
funds was efficient. It has spent around 91% of the budget but has exceeded the majority of the
expected results.
Effectiveness
From the review of the project achievements after four years of implementation, the EU Water
Facility WASH Programme has been a success. It has been particularly strong in delivering results in
terms of infrastructure provision in the target areas, where in all cases it exceeded the targets set
for infrastructure provision and the number of people to be covered by the action.
Especially the construction and rehabilitation of water sources as well as motivating households to
construct their own latrines (without subsidy) have exceeded expectations in terms of end
beneficiaries covered.
The Programme Monitoring and Evaluation framework has been used by stakeholders, specifically
the Government at all levels and other implementing partners in their planning and projects
activities.
Findings and Discussion of Findings
Confidential –All Rights Reserved Ernst & Young – 2010
Final Evaluation of EU Water Facility WASH Programme in Four districts and Four Municipalities – Final Report
Page 20
Impact
The project has shown potential to have a great impact on the living conditions of the selected
population, in which almost all planned outputs have been realized or exceeded. At household level,
the main impact of the project will be related to the increase of social status as a result of having
decent improved latrine. In addition to that, HH members, mainly women, have more time available
to participate in other activities as a result of available water supply close to the HH. Based on the
improved access to water and sanitation conditions which leads to a clean environment, less
sickness is expected in the home as well as an increased personal hygiene.
The benefits mentioned can also indirectly impact on the reduction of HH expenditure with health
care, and therefore, in a more disposable income to pay for essentials such as food, soup and
clothing.
However, a specific assessment of the project impact in the selected areas is recommended. The
evaluation team considers too early to make any judgement on the project impact at this stage.
Sustainability
There is a high probability that the benefits provided by the programme may continue after its
termination, taking into account that it has made a great investment in capacity building of local
communities and in the provision of water and sanitation facilities.
However, this is a great challenge for all stakeholders involved, mainly the water management
committees, the local service providers and the local authorities, which may require additional
technical assistance and support.
Taking into account the conclusions above, it is recommended that:
► Better coordination of monitoring and evaluation activities, especially at central level
counterparts;
► Prioritize WASH activities on peri-urban areas, which seem to be neglected in the majority of
interventions. The need for these services has been growing;
► Increase the direct support provided to the counterparts in order to ensure a more closed
implementation of the planned activities;
► Improve information flows about the programme among the stakeholders, specifically regarding
the involvement of central level counterparts;
Findings and Discussion of Findings
Confidential –All Rights Reserved Ernst & Young – 2010
Final Evaluation of EU Water Facility WASH Programme in Four districts and Four Municipalities – Final Report
Page 21
► Share lessons learned with other provinces in order to disseminate best practice being
implemented by the programme;
► Improve the articulation and/or coordination with other complementary programs, in order to
harmonize the interventions at all levels;
Annexes
Confidential –All Rights Reserved Ernst & Young – 2010
Final Evaluation of EU Water Facility WASH Programme in Four districts and Four Municipalities – Final Report
Page 22
Annexes
Annexes
23
Annex 1 – Terms of Reference
Annexes
24
UNICEF Mozambique
TERMS OF REFERENCE:
Final Evaluation of EU Water Facility WASH Programme in 4 Districts
(Buzi, Nicoadala, Nhatanda, Milange) and 4 Municipalities (Beira, Dondo,
Quelimane, Mocuba)
Submitted by: WASH Section
1. Background and Context
Within the framework of the partnership between the European Commission, UNICEF and the
Government of Mozambique, the EU Water Facility WASH Programme – Improvement of the
Livelihoods of Rural and Peri-urban Poor through Increased Access to Safe and Sustainable Water,
Sanitation and Hygiene Facilities in Sofala and Zambezia Provinces- has been implemented since July
2006. This intervention aims to contribute towards the achievement of the MDG and Poverty
Reduction Strategy Paper (PRSP) targets for water and sanitation in vulnerable rural and peri-urban
areas of underserved provinces in terms of safe water and sanitation coverage. The Programme, which
has duration of 4 years, will end in July 2010 and has the following expected results:
National water policy guidelines disseminated and sector planning, monitoring and evaluation
strengthened.
Provincial, municipal and district level capacity to plan, manage, monitor and evaluate the
WASH sector using NWP guidelines and DRA is strengthened.
Participatory and sustainable community management structures put in place and operational,
and hygiene practices and HIV/AIDS awareness improved in all project districts and
municipalities is strengthened.
Access to safe and affordable water supply, sanitation and hygiene facilities improved at
communities and schools in all eight project districts
Project management and technical assistance timely provided on a yearly basis.
In particular, by the end of the Programme it is expected that the following achievements will have
been made:
150,000 people with access to safe and sustainable water points.
25,000 families (125,000 people) with access to safe sanitation facilities (Household latrines)
120 schools and 45,000 schoolchildren with safe access to sanitation and hygiene facilities
(latrines, hand washing, urinals).
120,000 people benefited with hygiene promotion.
By March 2010, the following results were reported:
132.220 people with access to safe and sustainable water points.
73.664 families with access to safe sanitation facilities (household latrines).
127 schools with safe access to sanitation and hygiene facilities.
134.327 people benefited from hygiene promotion.
Annexes
25
2. Justification
The external final evaluation is a pre-requisite of the European Union cooperation and is part of the
Programme Monitoring and Evaluation framework, which was drawn up jointly by UNICEF, the
European Commission and the Government of Mozambique at the beginning of the Programme. This
assignment is in line with planned activities of UNICEF Mozambique, as it is part of the WASH
Section Programme Annual Work Plan for 2010.
3. Purpose and Objective
The main objective of this study is to evaluate the EU Water Facility WASH Programme, an initiative
that aims to improve access of rural and peri-urban poor people to safe and sustainable water,
sanitation and hygiene facilities in 4 districts and 4 municipalities from Sofala and Zambezia
provinces. The Programme will finish in July 2010, after 4 years of implementation, and UNICEF, the
European Commission and the Government of Mozambique are keen on evaluating its performance,
impact on target population people and performance of partners in the Programme implementation.
The information gathered in this assignment will contribute to the improvement of the Monitoring and
Evaluation system within the WASH sector. Lessons learnt in the course of the evaluation will be very
useful for similar ongoing WASH programmes in Mozambique, such as the One Million Initiative.
Moreover, the information collected through the study will help to assess whether recommendations
and conclusions from previous evaluations of the Programme (EU Monitoring visit and Sustainability
Checks) were integrated in its approach.
4. Methodology and Technical Approach
The proposed methodology for this end of Programme evaluation is a combination of qualitative data
collection methods, such as literature and programme documents review, in-depth individual
interviews with key informants (technical staff involved in the programme implementation,
community leaders, etc.), focus group discussions and direct observations in the field.
The assignment should consider the following key questions:
How far has the Programme Monitoring and Evaluation framework been used by
stakeholders?
Have end-users been adequately trained?
Is sustainability of water, sanitation and hygiene facilities satisfactory?
Are water management committees put in place working?
Are end-users aware of and involved in the management of water points and shared sanitation
facilities?
Due to the time and financial limitations, a process of sampling will be done in which the following
districts and municipalities will be visited. These include Beira and Dondo Municipalities and
Nhamatanda District in Sofala Province and Quelimane Municipalitiy and Nicoadale Distric in
Zambezia Province.
The consultant is expected to develop the questionnaires and sampling protocol and have these pre-
approved by UNICEF.
Annexes
26
5. Activities and Tasks
Desk review of programme documents such as Programme Proposal, Progress Reports,
Sustainability Check, etc. UNICEF will provide all these documents.
In depth interviews and focus group discussion with Programme staff based in Maputo
(UNICEF, DNA-DAR, NGOs which carried out the social mobilization process) and in the
field.
In depth interviews and focus group discussion with other key stakeholders, such as
community leaders, members of Water Management Committees, directors and teachers from
schools were water, sanitation and hygiene facilities were implemented, etc.
Direct observation of Programme outcomes in the field.
Draft of the final report.
Meeting with main stakeholders to present and discuss results
Presentation (Power Point) of main results.
Revise draft report based on stakeholders’ feedback.
6. Deliverables and Timeframe
Deliverables
The consultant or consultancy firm will be expected to produce the following deliverables:
Draft of the final report
A stand-alone Power Point presentation of the evaluation methodology and results with
complete speaking notes that will enable a user to comprehend the assessment without a
presenter needing to speak.
A separate Lessons Learnt document.
Final Report including an executive summary and abstract (based on specific format to be
provided by UNICEF).
All these documents must be presented in English and Portuguese language, ensuring adequate level
of both languages, and in digital format (five CDs) and hard copy (five copies).
Timeframe
The time given for completion of this evaluation is 4 weeks, of which 1 is anticipated to be of
fieldwork in Zambezia and Sofala Provinces. Below there is a detailed timeframe of the Evaluation:
Tasks
Week
1 2 3 4
Issuance of Contract
Literature review
In depth interviews with key
stakeholders (Maputo)
In depth interviews with key
stakeholders (Sofala and
Zambezia)
Data analysis
Draft Final Report
Annexes
27
Meeting with key stakeholders to
discuss preliminary results
Stakeholders analysis of the
Report
Incorporate stakeholders feed
back
Final report
7. Management, Organization and Schedule
This assignment is a critical component of the monitoring and evaluation framework designed at the
beginning of the EU Water Facility WASH Programme by the Directorate of Water of the
Government of Mozambique (DNA), UNICEF and the European Commission. The consulting firm
will be contracted by UNICEF Mozambique, and UNICEF will provide the consulting firm a
stakeholders’ contacts list. The consulting firm will be responsible for scheduling and attending
meetings with relevant authorities and institutions, such as local authorities and institutions at the
district and municipal level, NGOs involved in the Programme implementation, DNA staff, etc.
UNICEF will designate a focal point within the WASH Section, who will liaise with the consultant or
consulting firm during the assignment implementation.
As specified in Section 6 of this document, this assignment has duration of 4 weeks.
7. Budget and Remuneration
The total cost of this evaluation will be no more than $20,000. Timeframe is critical for the
successful completion of this assignment and, therefore, consulting firms must ensure that they
have the right staff to carry out duties, particularly in the fieldwork stage.
9. Qualifications and Specialized Knowledge
The consulting firm should have sound experience in evaluation of development projects/programmes
in Mozambique. In particular, experience within the WASH Sector is highly desired.
10. Conditions of Work
The table below indicates standard UNICEF guidance for conditions of work in which the consultancy
will be carried out:
Condition of work
To be provided by UN
Agencies
Remarks
Yes No
Field Trip Transport
X
Annexes
28
International Travel
X
Visa for internationally
recruited
X
DSA
X
Security Clearance
X
Accidental Death and
Dismemberment and
Accident Medical
coverage
X
Health Insurance
X
Office Space
X
Computer
X
Secretarial Services
X
Photocopier services
X
Other supplies
X
11. Remarks:
Please add any additional information you might found required.
Technical Evaluation Criteria:
Example of technical criteria and relative points:
Technical Criteria Technical Sub-criteria Maximum Points
Overall Response Completeness of response Overall concord between RFP requirements and proposal
Maximum Points 10
Company experience
Range and depth of experience with similar projects Number of customers, size of projects, number of staff per
project
Annexes
29
Client references
Maximum Points 15
Key Personnel Key personnel: relevant experience and qualifications
Maximum Points 20
Proposed
Methodology and
Approach
Technologies used - compatibility with UNICEF Project management, monitoring and quality assurance process Innovation approach
Maximum Points 25
Financial Proposal 30
Total Maximum 100
30
.
Annex 2 – List of people interviewed
31
Nome Posição Intituição Local
Samuel Godfrey WASH Section Head UNICEF Maputo
Angelina Xavier WASH Specialist UNICEF Maputo
Samuel Manhiça WASH Specialist UNICEF Maputo
Carlota Muianga WASH Specialist UNICEF Maputo
Julieta Felicidade DAR Head DNA Maputo
Idalina Alfai Staff DNA Maputo
Renato Solomon Staff DNA Maputo
Abel Florêncio Substituto do Chefe DPOPH/DAS Sofala
Benardete Manga Técnico DPOPH/DAS Sofala
Jerimias Liando Vereador Município da Beira Sofala
Julieta Nhamposse Técnico Município da Beira Sofala
Francisco António Presidente do comité de Água Bairro Nhangau - Município da Beira Sofala
Salomão Pedro Cobrador do fontanario Cemitério do Régulo - Municipio da Beira Sofala
Manual Dimo Gestor de Fontanario EPC de Macaronhe Sofala
Gente António Presidente do comité de Água Bairro Njalane-Municipio da Beira Sofala
Castelo Manuel Director Pedagógico EP1 de Macuti - Beira Sofala
Jorgina Inácio Coordenadora do Núcleo de Saneamento EP1 do Matadouto - Beira Sofala
Gervásio Brito Gestor do Balneário Público bairro maraza - Municipio da Beira Sofala
Maurio Safrao Cobrador do balneário Público bairro Matadouro - Municipio da Beira Sofala
Humberto Lalala Balneário público Bairro Miquejo - Municipio da Beira Sofala
Manuel Augusto Mulima
Técnico Água e Saneamento - representante do Chefe Município de Dondo Sofala
Vilares Patricio Fiscal Município de Dondo Sofala
Araujo Peixe Presidente do comité de Água Bairro Nhamawabwe/ Und C - Municipio de Dondo Sofala
Carlota Augusto Cartoto Presidente do comité de Água
Bairro Macharote - Municipio de Dondo Sofala
Paulo Erbezia Chefe do Núcleo de Saneamento Centro Primário de Acomodação - Dondo Sofala
Celestino João Director da Escola Centro Educacional de Dondo Sofala
Silva Mário Vereador de planificação - substituto do presidente Município de Quelimane Zambézia
Armando Jorge Técnico de Água e Saneamento Municipio de Quelimane Zambézia
Xavier Luis Fagima Director da escola EPC de Nhanhibua -Municipio de Quelimane Zambézia
Maria da Conceição R.J. Tebueira Drectora da escola
EPC de Morrupué - Municipio de Quelimane Zambézia
Amina Suluvai Chefe de Promoção de Higiene Bairro santagua - Municipio de Quelimane Zambézia
Castro conhece Chefe de promoção de Higiene
Presidente do conselho comunitario de promoção de Higiene no bairro Manhaua Zambézia
32
Nome Posição Intituição Local
Carvalho Branquinho Roupa Gestor do centro Centro de demostração Zambézia
Virgílio Augusto Amizade Chefe do SDPI
Governo do distrito de Nicoadala Zambézia
Ailton Mulessiua Técnico do SDPI Governo do distrito de Nicoadala Zambézia
Fernado Alilo Técnico do DAS DPOPH/DAS Zambézia
Guilhardo Perreira Vergonha Director da Escola
EPC de Namitangurinho - distrito de Nicoadala Zambézia
Bonifácio António Presidente do comité de Água Manta mutuela Zambézia
Maria Madalena Chefe do grupo de Manutenção Fitimela A Zambézia
Direscente Armando Mtumula Presidente do comité de Água Mudidi Zambézia
Pilares Laveque Presidente do comité de Água Assumate Zambézia
33
Annex 3 – EU WF Final Evaluation Outline 3
34
Expected Results Indicators What to assess Research tool Information source
1.1.NWP guidelines disseminated in four
districts and four municipalities by 2007
Do districts and municipalities plans for the WASH
Sector follow NWP guidelines? Are Districts and
Municipalities staff familiar w ith NWP guidelines?
Have they attended any training about NWP?
Meetings;
Interview s
DNA (DAR &GPC),
DPOPH, Municipalities
and Districts
1.2.National data bank established and
operational by 2009
What has been done from 2007 to now ? What are
the next steps planned and the timeframe? Meetings DNA (GPC)
1.3. MDG Road Map prepared for the
sector by 2006.
What have been the steps taken so far? Planes
estrategicos??? Meetings DNA (GPC &DAR)
2.1. 70% of project funds managed
directly by provinces and district
authorities by 2009
What percentage of the programme funds (cash and
goods) w as directly managed by by provinces,
municipalities and districts managing?
Desk Review
UNICEF Financial
Reports; Meetings
UNICEF staff (Samuel
godfrey & Teofilo
Nhapulo
2.2. Four districts and four municipal
plans prepared and implemented in all
project areas by 2009
Did the planning process take place? Are there any
plans? Have they develop capacity to plan and
implement? Interview s
Municipalities, Districts
(SDPI), DPOPH
2.3. DRA principles applied in all project
areas by 2009
2.4. Tw o provincial data banks
established & operational by 2009
Are those data banks operational? If so, w hen w ere
they established? Who use them? What service do
they provide? If not, w hy? Were they established at
some point?
On site-observation;
Interview DPOPH
3.1. 90% of w ater points w ith monthly
w ater paying schemes managed by
w ater committees (# of w omen) in all
districts by 2009 ;
Are the w ater committes put in place in all w ater
points? Are the monthly paying schemes w orking? If
not, w ere they put in place at some point during the
project? Why they didn't w ork?
Meetings,
Interview s,
SDPI technical staff,
Municipalities technical
staff, Water
Committees members,
UNICEf staff
3.2. frequency of hand pumps break
dow n in one year reduced to a maximum
of tw o in project areas;
In average, w hat is the frequency of breakdow ns in
hand pumps? Are there records of breakdow ns?
How do w ater committes pay for the hand pumps
reparation? Are there trained artisans available to the
communities?
Water committes
members
3.3. three days as the maximum number
of days that it takes to repair a pump in
project areas;
Are there w ater points not being used for long
periods ? What happen in those cases? Water committes members
3.4. 90% of project areas household
using safe w ater sources by 2009;
What is the current percentage of households in
programme areas w ith access to safe w ater
sources?
UNICEF staff, off icers
from DNA, DPOPH and
SDPI, Municipalities
3.5. 60 % of w omen using HH latrines by
2009 in project areas
3.6. 70 % of school children (girls and
boys) practising hand w ashing after
defecation by 2009 in project areas
4.1. 150,000f people w ith access to safe
and sustainable w ater points in project
areas by 2009;
How many w ater points w ere constructed under the
programme framew ork? How many people do they
serve? % of those w aterpoints that are w orking
now ?
UNICEF staff, off icers
from DNA, DPOPH,
SDPI, Municipalities
4.2. 25,000 families w ith access to safe
sanitation facilities (HH latrines) by 2009 in
project areas;
Number of latrines constructed under the programme
framew ork? Are these latrines being used?
UNICEF staff, off icers
from DNA, DPOPH,
SDPI, Municipalities
4.3. 120 schools and respective
schoolchildren w ith safe access to
sanitation and hygiene facilities (latrines,
hand w ashing, urinals) by 2009 in project
areas;
Number of w ater points and latrines constructed in
schools under the programme framew ork? Are these
infraestructures operational? Are students using
them?
UNICEF staff, DPOPH,
Directors of schools,
Municipalities
5.1. At least 90% of project expenditure
level yearly ensured; What w as the yearly expenditure of the programme?
UNICEF staff, f inancial
records of the
programme
5.2. At least 85% of activities are
implemented by 2009;
In 2009, how many activities w ere implemented?
Which ones? And by the end of the programme?
What activities w ere not implemented? Why?
UNICEF staff, DNA,
DPOPH, SDPI and
Municipalities
Result 1. National w ater policy
(NWP)guidelines disseminated and
sector planning, monitoring and
evaluation strengthened by 2009
Result 2. Provincial, municipal and
district level capacity to plan, manage,
monitor and evaluate the WASH sector
using NWP guidelines and DRA
strengthened by 2009
Result 3. Participatory and sustainable
community management structures put
in place and operational, and hygiene
practices and HIV/AIDS aw areness
improved in all project districts and
municipalities by 2009
Result 4. Access to safe and
affordable w ater supply, sanitation and
hygiene facilities improved at
communities and schools in all eight
project districts by 2009
Result 5. Project management and
technical assistance timely provided on
a yearly basis up to 2009
35
Annex 4 – List of Documents Reviewed
36
1. UNICEF Mozambique, June 2010 - Final Report for the European Commission - SC/06/450 -
Improvement of the Livelihoods of Rural and Peri-urban Poor through Increased Access to Safe and Sustainable Water, Sanitation and Hygiene Facilities in Sofala and Zambezia Provinces, Mozambique
2. European Commission, 2006 - Detailed Grant Application Form. ACP-EU WATER FACILITY ACTIONS IN ACP COUNTRIES for Non-state Actors (ACP and EU), Public Bodies (EU) and International Organisations
3. UNICEF Mozambique, June 2010 - EU Water Facility II Final Evaluation Outline 1
4. UNICEF Mozambique, June 2010 - EU Water Facility II Final Financial Report
5. UNICEF Mozambique, June 2010 - Minutes of EU Water Facility II Final Evaluation Meeting
6. UNICEF Mozambique, July 2007 - Progress Report for the European Commission - SC/06/450 - Improvement of the Livelihoods of Rural and Peri-urban Poor through Increased Access to Safe and Sustainable Water, Sanitation and Hygiene Facilities in Sofala and Zambezia Provinces, Mozambique
7. European Commission, July 2008 – Aid Delivery Methods: Project Cycle Management
Guidelines (Volume 1)
8. UNICEF Mozambique, March 2009 - Progress Report for the European Commission - SC/06/450 - Improvement of the Livelihoods of Rural and Peri-urban Poor through Increased Access to Safe and Sustainable Water, Sanitation and Hygiene Facilities in Sofala and Zambezia Provinces, Mozambique
9. UNICEF Mozambique, June 2010 - Final Report for the European Commission - SC/06/450 - Improvement of the Livelihoods of Rural and Peri-urban Poor through Increased Access to Safe and Sustainable Water, Sanitation and Hygiene Facilities in Sofala and Zambezia Provinces, Mozambique
10. UNICEF Mozambique, March 2009 - Progress Report for the European Commission - SC/06/450 - Improvement of the Livelihoods of Rural and Peri-urban Poor through Increased Access to Safe and Sustainable Water, Sanitation and Hygiene Facilities in Sofala and Zambezia Provinces, Mozambique
11. Ernst & Young, November 2009 – Sustainability Check for the WASH
12. Austral Cowi, October 2009 - Sustainability Check for the WASH in Zambézia Province
37
Annex 5 – Guideline of semi-structured interviews to UNICEF
38
Result 1
National WP
1.Describe the involvement of your institution in the dissemination of WP?
2.There was any follow up activity?
Result 2
Funds management
1. What was the role of UNICEF in all this process?
2. What percentage of the programme funds (cash and goods) was directly managed by provinces and municipalities and district managing?
Result 3
Safe water sources
3. What is the current percentage of households in programme areas with access to safe water sources?
4. What are the factors affecting the access to safe water sources?
Result 4
People with access to safe and sustainable water points
5. How many water points were constructed under the programme framework?
6. How many people do they serve?
7. How many waterpoints are working now?
Families with access to safe sanitations facilities
8. Number of latrines constructed under the programme framework?
9. Are these latrines being used? If not why? Is there any change in the approach?
10. What type of latrines were constructed?
Schools with access to safe sanitation and Hygiene facilities
11. Number of water points and latrines constructed in schools under the programme framework?
12. Are these infraestructures operational?
13. Are students using them? If not, why they are not using?
Result 5
Project expenditure
14. What was the yearly expenditure of the programme?
39
Annex 6 – Guideline of semi-structured interviews to DNA
40
Result 1
National WP
1. Describe the involvement of your institution in the dissemination of WP?
2. There was any follow up activity?
3. Do districts and municipalities plans for the WASH sector follow the WP?
4. Are Districts and Municipalities staff familiar with WP guidelines?
5. Who implemented/facilitated the training?
6. How did you get feedback about the dissemination of WP?
Data bank
7. When the data banks were established? How was the process of its establishment?
8. How was the situation before the establishment of data bank?
9. What is the actual situation of data bank implementation?
10. What are the next steps planned and the timeframe?
MDG
11.Are you aware about the MDG for your sector?
12.What have been the steps taken so far by your institution?
1. Please provide evidences for the steps taken?
Result 2
Result 3
Safe water sources
14. What is the current percentage of households in programme areas with access to safe water sources?
15. What are the factors affecting the access to safe water sources?
Result 4
People with access to safe and sustainable water points
16. How many water points were constructed under the programme framework?
17. How many people do they serve?
18. How many waterpoints are working now?
Families with access to safe sanitations facilities
19. Number of latrines constructed under the programme framework?
20. Are these latrines being used?
21. What type of latrines were constructed?
Result 5
41
Annex 7 – Guideline of semi-structured interviews to DPOPH
Result 1
National WP
1. Describe the involvement of your institution in the dissemination of WP?
2. There was any follow up activity?
3. Do districts and municipalities plans for the WASH sector follow the WP?
4. Are Districts and Municipalities staff familiar with WP guidelines?
5. Have they attended any training about WP?
6. Who implemented/facilitated the training?
7. How did you get feedback about the dissemination of WP?
Data bank
8. When the data banks were established?
9. How was the situation before the establishment of data bank?
10. What is the actual situation of data bank implementation?
11. What are the next steps planned and the timeframe?
Result 2
Plans and projects implemented by prov and districts
12. Did the planning process take place?
13. Are there any plans?
14. Have they develop any capacity to plan and implement?
15. What type of assistance have you received for this process?
Data bank
16. Are these data banks operational?
17. Who use them?
18. What is the service do they provided?
19. Did you get any training on data bank operation?
Result 3
Safe water sources 20. What is the current percentage of households in programme areas with access to safe water sources?
21. What are the factors affecting the access to safe water sources?
Result 4
People with access to safe and sustainable water points
22. How many water points were constructed under the programme framework?
23. How many people do they serve?
24. How many waterpoints are working now? What has been done to increase the figures?
Families with access to safe sanitations facilities
25. Number of latrines constructed under the programme framework?
26. Are these latrines being used?
27. What type of latrines were constructed? What has been done to increase the figures?
Schools with access to safe sanitation and Hygiene facities
28. Number of water points and latrines constructed in schools under the programme framework?
29. Are these infraestructures operational?
30. Are students using them? If not, why they are not using?
Result 5
Annex 8 – Guideline of semi-structured interviews for Municipalities
Result 1
National WP
1. Describe the involvement of your institution in the dissemination of WP?
2. There was any follow up activity?
3. Do districts and municipalities plans for the WASH sector follow the WP?
4. Are Districts and Municipalities staff familiar with WP guidelines?
5. Have they attended any training about WP?
6. Who implemented/facilitated the training?
7. How did you get feedback about the dissemination of WP?
Result 2
Plans and projects implemented by prov and districts
8. Did the planning process take place?
9. Are there any plans?
10. Have they develop any capacity to plan and implement?
11. What type of assistance have you received for this process?
Result 3
Water points
12. Are the water committes put in place in all water points?
13. Are the monthly paying schemes working? If not, were they put in place at some point during the project? Why they didn't work? 14. How many women are involved in management? Sustainability ..
Safe water sources
15. What is the current percentage of households in programme areas with access to safe water sources?
16. What are the factors afecting the access to safe water sources?
Result 4
People with access to safe and sustainable water points
17. How many water points were constructed under the programme framework?
18. How many people do they serve? 19. How many water points are working now?
Families with access to safe sanitations facilities
20. Number of latrines constructed under the programme framework?
21. Are these latrines being used? 22. What type of latrines were constructed? Sustainability .. Latrinas familiares ...Balneários ... Drenagem Como o programa ajudou a aumentar a capacidade de saneamento … conhecer a rede?
Schools with access to safe sanitation and Hygiene facilities
24. Are these infraestructures operational? 3. Are students using them? If not, why they are not using?
Result 5
Annex 9 – Guideline of semi-structured interviews for Water Committees
Result 1
National WP
1. Have they attended any training about WP?
2. Who implemented/facilitated the training?
Result 2
Result 3
Water points
3. Are the water committes put in place in all water points?
4. Are the monthly paying schemes working? If not, were they put in place at some point during the project? Why they didn't work?
5. How many women are involved in management?
Hand pumps break down
6. In average, what is the frequency of breakdowns in hand pumps? Are there records of breakdowns?
7. How do water committes pay for the hand pumps reparation?
8. Are there trained artisans available to the communities?
9. If not, who fix the pumps?
Days to repair
10. Are there water points not being used for long periods ? What happen in those cases?
11. Why there are not being used?
12. Were do you get the spair parts?
Result 4
People with access to safe and sustainable water points
13. How many waterpoints are working now?
Families with access to safe sanitations facilities
14. What type of latrines were constructed?
Schools with access to safe sanitation and Hygiene facilities
15. Are students using them? If not, why they are not using?
Result 5
Annex 10 – Guideline of semi-structured interviews for Schools
Result 1
Result 2
Result 3
Result 4
Schools with access to safe sanitation and Hygiene facilities
1. Number of water points and latrines constructed in schools under the programme framework?
2. Are these infraestructures operational? 3. Are students using them? If not, why they are not using? 4. What has been done to ensure the operations and maintenance of water points and latrines? (O que tem feito para garantir a operacionalidade das escolas?) 5. Como é que feita a manutenção e limpeza das latrinas nas escolas e a capacitação dos professores e crianças?
Result 5
Annex 11 – Details on Section 4.5
Result 1: District Government and Municipality were trained on NWP guidelines which improved
their skills to disseminate and promote it at the community level. The District Government and
Municipality promote regular visits to the communities in order to hear their view points, inform
them about the opportunities that the demand responsive approach offers and establish, in
collaboration with their authorities, the channels for requesting funds. The improvement of their
skill leaded to better provision of water and sanitation services as they improved their planning and
monitoring and evaluation skills.
At the community level the managing body is the water and sanitation committee. This body was
responsible for the following up the operation and maintenance of the water source; it is also
responsible for the stocks of spare-parts; controlling user’s contributions, promotion and
dissemination of sanitation and hygiene messages in the communities. The WCs are increasingly
improving their skills in water and sanitation infrastructures management, as they receive training
and refreshments on basic management skills and also support from the district government and
municipality in terms of their legitimacy as legal community representative. The WC also plays an
important role in the improvement of community participation.
Regarding to the WASH data bank, it were established in 2007 with indicators for the target
provinces. In order to gather, analyze and disseminate the information, were undertaken the
following activities: (i) conducted the GIS/Mapping in all districts; (ii) were established the National
Information System for Water and Sanitation (SINAS), implemented under DNA leadership. This
system promotes synergies between Government and sector partners in terms of defining strategic
options and sector priorities as well as further enhancing sector harmonisation; (iii) A training
workshop on WES data analysis and trends towards the achievement of country MDG targets had
been held in 2008; (iv) The training provided basis for review and refinement of sector performance
indicators and definitions aligned with the international M&E guidelines for WASH; and would also
support the sector to strengthen monitoring to track sector progress.
As part of the process for developing the MDG Road Map, the Strategic Plan for the Rural Water
Supply and Sanitation (PESA-ASR, 2006-2015) was developed with active participation from WES
stakeholders from sub-national and national levels, including the project targeted districts and
municipalities. This strategic document leads to the establishment of the Sector Wide-Approach
(SWAps) for the rural water and sanitation – the National Rural Water Supply and Sanitation
Programme (PRONASAR) – and the Common Fund (CF) for the PRONASAR. Both PRONASAR and CF
form the basis for harmonized sector programming, funding, monitoring and evaluation.
Result 2: Around 89% of the programme funds (cash and goods) were directly managed by
provinces (3%), municipalities (31%) and districts (55%), exceeding the 70% defined by the project.
In the four districts and four municipalities the planning process took place. The plans are available
and based on the assistance received their staff have developed capacity to plan and implement the
activities.
In Zambézia province the first data bank was established in 1998 but it didn’t work for a long time.
In 2002 was established a second data bank which was improved by DFID/UNICEF in 2006. To
operate the data bank was indicated a staff to operate it and received training in Maputo carried out
by DNA. Between 2006 and 2008 the staff received training on job training type through WE
Consults agency which helped on data’s interpretation. In time being the data bank is operational
and the information is provided to the provincial government, ONGs, DPOPHs to plan their activities.
The Sofala data bank was established in 2008 by ZAMWATER programme funded by EU. This data
bank never worked because the computers memories were stolen. In 2009, with the support of
UNICEF/EUWF the new memories were bought, but they still need to be installed and the staff
trained.
Result 3: All water points have WC in place. The situation is good in the districts but improvements
are required in the peri-urban areas. Today, in peri-urban areas, there are a variety of alternative
sources of water available, such as water taps and network supply taps provided by FIPAG. When
the project was designed, there were not expected that the network would cover such areas.
Currently, in the visited locations, breakdowns in hand pumps are almost inexistent. Therefore, no
records of breakdowns were found. However, the WC still collecting monthly payments for future
hand pumps reparation. There are trained artisans available to the communities. All water points
visited by the evaluation team are being used.
The current percentages of households in programme areas with access to safe water sources is in
average 60%, being Nicoadala 48.2%, Dondo 81% and Quelimane (60%). This average represents
three out of eight target areas3, which does not allow for a conclusion. However, the target of 90%
set by the programme was too ambitious as the baseline (DNA 2003) in the two selected provinces
(Sofala and Zambézia) was in average 38%.
3 Data from Beira, Milange, Mocuba, Búzi and Nhamatanda.
Result 4: The Programme Specific Objectives had by and large been achieved.
The programme result of 150,000 people with access to safe and sustainable water points was
exceeded thus over 296,000 people have access to safe drinking water, through construction and
rehabilitation of 313 water points. All the water points visited were operational.
The objective of providing access to safe sanitation facilities (Household latrines) for 25,000
families (125,000 people) has been exceeded thus approximately 215,000 people (nearly 39,000
families) in rural and peri-urban areas had access to safe sanitation facilities. All six latrines visited
in Dondo by the evaluation team were operational, and the same number in Nicoadala was also
operational.
The expected result of 120 schools and 45,000 school children with safe access to sanitation and
ited with hygiene
promotion, have been also exceeded. Around 58,450 learners in 167 schools in poor rural and peri-
urban areas had access to safe school WASH facilities. The visited facilities in eight schools were all
operational and being used.
The programme result of 120,000 people benefiting from hygiene promotion had been also
exceeded; reaching approximately 134,000 people in rural and peri-urban areas. These people have
increased knowledge on proper hygiene behaviour through hygiene promotion activities.
Result 5: Project management and technical assistance was timely provided on a yearly basis for
the implementation of the project. It was also provided support for the implementation by assisting
responsible for managing the financial aspects of the programme at the provinces as well
supporting the procurement of equipment in line with UNICEF rules and regulations. Monthly field
visits have been carried out in each project area in coordination with national, provincial, municipal
and district teams.
Annex 12 – PPT for the Final Evaluation Report