TERMS OF REFERENCE

For the Independent Final Evaluation of Action Against Hunger’s

Reducing Vulnerability through Disaster Risk Management (DRM) in Khyber Pakhtunkhwa (KPK) Province, Pakistan - Phase 3,

Summary TableProject Name Reducing Vulnerability through Disaster Risk Management

(DRM) in Khyber Pakhtunkhwa (KPK) Province, Pakistan - Phase 3,

Contract Number PAK 3044-14/0009Partners (if applicable) NA.Location (country/ies, region/s)

Charsadda, Nowshera, Dir Lower, Dir Upper, Khyber Pukhtoonkwa Pakistan

Sector(s) Disaster Risk Management (DRM).Duration 27 Months (But actual implementation period is 13 Months) Starting Date 01st December 2014Ending Date 31st December 2016Intervention/Country Office Language

English

Donor and Contribution/s Royal Norwegian Embassy Country Office administering the Programme/Project

ACF Pakistan

Responsible Action Against Hunger HQ

ACF USA

Evaluation Type Independent Final Project EvaluationEvaluation Dates 1st December to 10th Jan 2017

1

1. BACKGROUND of the Project

1.1. Map of Intervention AreaNote: Charsadda, Nowshera, Lower Dir and Upper Dir are the Targetted districts.

1.2. Rational for the Project The persistent disasters, combined with structural poverty factors, unequal access to basic services, and a lack of government and community capacity to fully recover from the disasters while absorbing the displaced, has resulted in a systemic cycle of increasing vulnerability amongst communities. While past NGO initiatives in the project area have succeeded in training government staff, creating awareness of DRM, and developing DRM plans in a small percentage of communities, large areas remain uncovered. In addition, those plans that have been created are not well integrated in to district and provincial planning, with district level government actors involved in DRM lacking the capacity to manage and utilize gathered information effectively. The need exists to continue building resilience at the community level, in parallel with government capacity building and strengthening linkages between governments, NGOs, and communities.

2

Flood is the major disaster affecting the population in the assessed districts in the past years. A base line survey assessing the structural and non-structural knowledge of location community on mitigation measures carried out in September 2013 reveal that people strongly believe that the "Act of God" is the main reason for natural disasters in their area. Moreover, it shows that no proper early warning systems are established at community level and people receive information by observing the rising level of the river. 76.9% of people interviewed said that they built Pakka (strong) infrastructure to mitigate the effect of disaster but 93% did not understand the non-structural measures such as land use policy, reforestation, early warning system. 69 % of respondents said that district disaster management unit (DDMU) was not working properly and the remaining respondents did not know the existence of the DDMU. Evacuation plans were also assessed through this survey. Only 35.7% of the respondents said they had already identified the evacuation centre to be used during the disaster. ACF in implementing DRR project in the flood effected targeted districts of Khyber Pakhtunkhwa (KPK) Province, this is consecutive third phase of the project funded by Royal Norwegian Ministry.

1.3. Project Objectives

General Objective:To continue reducing the vulnerability and improve the resilience to disaster of KPK province, Pakistan.

Specific Objective:To enable communities to prepare for and mitigate the impacts of disasters using participatory methods aimed at strengthening the self-reliance capacities of the population and reinforcing or adapting infrastructure.

The project log frame is attached in annex I.

1.4. Current Status of the ProjectThe project soft and hard activities are currently ongoing as per initial planning and tentatively completed by mid of December 2016. Initially the project was to be completed by September 2016, but the No Objection Certificate (NOC) seeking from National and Provincial Government took more time than expected and it lead to delays in the implementation, a no cost extension has been approved by RNE and thus the project will be ending in December 2016. Besides that, a few project activities have been redesigned due to parallel government campaigns / interventions across the province like Plantation, and predefined some disaster resilience infrastructure in the proposal has been changed to need based infrastructures, while the rest of the interventions are as proposed initially. 2. PURPOSE AND OBJECTIVES OF THE EVALUATION

2.1. Rational for the Evaluation This evaluation is conducted as an exercise of accountability towards the donor and the beneficiaries at its final implementation stage. The evaluation will be conducted by a third party independent evaluator.

2.2. Objectives of the Evaluation To determine how effectively and efficiently project outputs/outcomes are achived and has

contributed to the overall and specific objectives of the project. To assess the project interventions are likely to be sustainable beyond the project lifetime.

3

To examine the preparedness level of communities (in community and schools) to cope with and district authorities to respond to any future emergencies.

To identify the good practices and lesson learnt for future programming To examine the cross cutting issues like enivornment, Gender equality, coordination,

advocacy and participation

2.3. Users of the Evaluation Direct users: ACF Pakistan Mission, ELA unit - UK, ACF USA (HQ), ACF field teams; Indirect users: ACF International Network, Royal Norwegian Embassy, Government of KPK,

NDMA, PDMA, Provincial government, district and local government, UN agencies and DRR Clusters, INGOs and NGO Consortiums as well as humanitarian learning platforms (such as ALNAP).

2.4. Use of the Evaluation The learnings from the evaluation will be used to developed new strategies at National,

Provincail and District level (if needed). These learning will also be helpful to improve organizational strategy / approach towards DRR and design projects and interventions as per the contextual requirement.

The learning and/or successes on the ground will be dissiminated within ACF networks and learnings will be reaplicated in other similar contaxts.

These learnings will also support to define advocacy and communication contents with different stakeholders.

The evaluation will measure the potential issues faced and recommendations for future, that will guide future programing.

3. EVALUATION SCOPE

3.1. Elements covered by the evaluation This evaluation will be foucsed on the four intervention districts of Khyber Pakhtoonkhwa

(KPK) Province, including Charsadda, Nowshera, Dir upper and Dir lower. The evaluator will consider the goals and objectives of the project in comparison to the project interventions at field level.

The project evaluation will be based on the Development Assistance Committee DAC criteria (Design, Coherence, Coverage, Relevance and Appropriateness, Effectiveness, Efficiency, Sustainability and Likelihood of Impact) that will be followed by the evaluator.

Identify unintended changes, both positive and negative, in addition to the expected results;

3.2. Elements not covered by the evaluation NA.

3.3. Cross-cutting issues Throughout the evaluation process, gender concerns should be addressed in line with the

ACF Policies. All data should be disaggregated by sex and age. Different needs of women, men and of marginalised groups targeted by the project should be considered throughout the evaluation process. Moreover, the community participation should be emphasised and how ACF ensures that communities were involved throughout the programme cycle.

4. EVALUATION APPROACH AND QUESTIONS

4

As per Action Against Hunger Evaluation Policy and Guidelines1, Action Against Hunger adheres to the Development Assistance Committee (DAC) criteria for evaluating its interventions. Specifically, Action Against Hunger uses the following adapted criteria: Design, Coherence, Coverage, Relevance and Appropriateness, Effectiveness, Efficiency, Sustainability and Likelihood of Impact.The evaluator must complete the DAC criteria rating table (Refer to Annex V) and include it as part of the final evaluation report.

Evaluation questions have been developed to help the evaluator assess the project intervention against these criteria. (Refer to Annex V). The evaluator may adapt the evaluation criteria and questions, but any fundamental changes should be agreed between the evaluation manager and the evaluator and reflected in the inception report.

5. EVALUATION METHODOLOGY

The evaluation methodology will be proposed by the evaluator after thorough study of TOR's requirment. The evaluatior is expected to use mixed methods approach and/or Most Significant Change Technique (MSCT), collecting both qualitative and quantitative information from targeted household and communities. Moreover for triangulation of the information, interviews of other key stakeholders i.e Provincial Disaster Management Athourity, District Disaster Management Athourity, District Disaster managerment officer (DDMO), Deputy Commissioner officer / Assisstant Commissioner officer (ACO/DCO) , Executive District officer (EDO - Education), Targetted Schools Management (Prinicples, Teachers), University of Peshawar (UOP), DRR Clusters, UN Agencies, project key staff etc

5.1. Evaluation BriefingPrior to the evaluation taking place, the evaluator is expected to attend an evaluation technical briefing with the Evaluation and Learning Unit (ELA) ACF-UK focal person and in country with PQA Coordinator.

5.2. Desk reviewThe evaluator will undertake a desk review including the project documents and proposals, progress reports, outputs of the interventions (such as publications, Research studies by students, baseline and endlines, communication materials, etc.), results of any internal planning process and relevant materials from secondary sources.

5.3. Action Against Hunger HQ InterviewsAs part of the evaluation, the evaluator will interview HQ stakeholders to get preliminary information about the project intervention office being evaluated (DRR Advisor etc.). Briefings by telephone must be agreed in advance.

5.4. Inception ReportAt the end of the desk review period and before the field work, the evaluator will prepare a brief inception report based on the format provided. The report will be written in English and will include the following sections:

Key elements of the Terms of Reference (TORs) to demonstrate that the evaluator will adhere to the TORs;

The methodological approach to the evaluation include an evaluation matrix in annex to specify how the evaluator will collect data to answer the evaluation questions, pointing out the limitations to the methodology if any and the choice of sites per field visit;

1http://www.alnap.org/resource/6199

5

The data collection tools; A detailed evaluation work plan and; Statement of adherence to Action Against Hunger Evaluation Policy and outline the

evaluation report format. The inception report will be discussed and approved by the mission level focal person (PQA Coordinator) while ELA unit (ACF-UK) will support in technical advisory role. The inception report will also be shared with HQ (ACF-USA) Technical staff for validation and approval.

5.5. Field MissionPrimary data collection techniquesThe evaluator will interview key project stakeholders (expatriate/national project staff, local/national representatives, local authorities, humanitarian agencies, or donor representatives, communities) as per the list in Annex X. The evaluator will use the most suitable format for these interviews as detailed in the inception report. The evaluator is also expected to collect information directly from beneficiaries/Communities. For Triangulation of the findings, the evaluator could also conduct Focus Group Discussions (beneficiaries, non-beneficiaries, key informants – teachers and community leaders, government staff etc.) and surveys.

Field visitsThe evaluator will visit the project sites and the facilities provided to the beneficiaries (if any) according to select methods described in the inception report.

Secondary data collection techniques: Desk reviewThe evaluator will further review complementary documents and collect project monitoring data or of any other relevant statistical data (as and when required).

Debriefing and stakeholders workshopThe evaluator shall facilitate a learning workshop in country to present preliminary findings of the evaluation to the project and key stakeholders (ACF management and field staff, Local, Provincial actors and donor representative i); to gather feedback on the findings and build consensus on recommendations; to develop action-oriented workshop statements on lessons learned and proposed improvements for the future.

5.6. Evaluation Report The evaluation report shall follow the following format and be written in English:

Cover Page; Summary Table to follow template provided Table of Contents Executive Summary must be a standalone summary, describing the intervention, main

findings of the evaluation, and conclusions and recommendations. This will be no more than 2 pages in length

Background Information Methodology describe the methodology used, provide evidence of triangulation of data and

presents limitations to the methodology Findings includes overall assessment of the project against the evaluation criteria, responds

to the evaluation questions, all findings are backed up by evidence, cross-cutting issues are mainstreamed and; unintended and unexpected outcomes are also discussed

Conclusions: Conclusions are formulated by synthesizing the main findings into statements of merit and worth, judgements are fair, impartial, and consistent with the findings

Lessons Learnt and Good Practices presents lessons that can be applied elsewhere to improve intervention or country office performance, outcome, or impact and; identify good

6

practices: successful practices from those lessons which are worthy of replication; further develop on one specific good practice to be showcased in the template provided in Annex VI

Recommendations should be as realistic, operational and pragmatic as possible; that is, they should take careful account of the circumstances currently prevailing in the context of the action, and of the resources available to implement it both locally. They should follow logically from conclusions, lessons learned and good practices. The report must specify who needs to take what action and when. Recommendations need to be presented by order of priority

Annexes: These should be listed and numbered and must include the following: logical framework provided in Annex I, Good practice template provided in Annex VI, Evaluation Criteria Rating Table provided in Annex V, list of documents for the desk review, list of persons interviewed, data collection instrument, evaluation TORs

The whole report shall not be longer than 30 pages, 50 pages including annexes. The draft report should be submitted no later than 10 calendar days after departure from the field. The final report will be submitted no later than the end date of the consultancy contract. Annexes to the report will be accepted in the working language (i.e. English) of the country office.

5.7. Debriefing with the ELA Unit – ACF UK The evaluator should provide a debriefing to the ELA unit – ACF UK to discuss any issues related to the quality of the evaluation report.

5.8. Debriefing with Action Against Hunger HQThe evaluator should provide a debriefing with the relevant Action Against Hunger HQ on her/his draft evaluation report, and on the main findings, conclusions and recommendations of the evaluation. Relevant comments should be incorporated in the final report.

6. KEY DELIVERABLES

The following are the evaluation outputs the evaluator will deliver to the evaluation manager:

Outputs DeadlinesInception Report 15th Dec-2016Stakeholders workshop 28th Dec-2016Draft Evaluation Report 06th Jan - 2017Final Evaluation Report 17th Jan – 2017

All outputs must be submitted in English and under Word Document format.

The quality of the inception report and the evaluation report will be assessed by the ELA unit (ACF-UK) focal person and mission level PQA coordinator. The evaluator is expected to follow the format, structure and length as defined under section 5.4 and 5.6 above.

7. MANAGEMENT ARRANGEMENTS AND WORKPLAN

These evaluation TORs have been developed in a participatory manner based on inputs from relevant stakeholders and approval from Donor.

The evaluator will directly report to the evaluation manager ii. The evaluator will submit all the evaluation outputs directly and only to the evaluation manager. The evaluation manager will do a quality check (ensure required elements are there) and decide whether the report is ready for

7

sharing. The evaluation manager will forward a copy to key stakeholders for comments on factual issues and for clarifications. The evaluation manager will consolidate the comments and send these to the evaluator by date agreed between the evaluation manager and the evaluator or as soon as the comments are received from stakeholders. The evaluator will consider all comments to finalize report and will submit it to the evaluation manager who will then officially forward to relevant stakeholders.

7.1. Tentative Work planNOTE: Consultants are expected to work 6 days a week (either Sundays/Fridays or whatever day the field office has off will not be paid) during their consultancy contract. Travel days are not paid as they are not working days as such.

Activities Evaluator Working Days

Dates

Evaluation briefing with the evaluation manager (in country) 0.5 11th DecInterviews with HQ / ELA Unit 0.5 12th Dec

Desk review, preparation of field work and prepare Inception Report

3 15th Dec

Inception Report approval 1 (Not paid) 16th Dec

In country interviews with ACF staff (Key project staff) 1 17th Dec

Field Travel, Field work, collection and analysis of secondary data and meeting with stakeholders

Min. 10 17th – 26th Dec

Stakeholders Workshop in country 1 28th Dec

Evaluation debriefing with the evaluation manager 0.5 29th Dec

Evaluation debriefing with HQ 0.5 29th Dec

Draft Report Min. 5 06th Jan 2016

Action Against Hunger-UK: Quality check and initial review by the evaluation manager, circulate draft report to key stakeholders, consolidate comments of stakeholders and send to evaluator

Min. 8 14th Jan 2017

Final report on the basis of stakeholders, country office, HQ, and Action Against Hunger -UK comments

Min. 3 17th Jan 2017

Total: 25 days

7.2. Profile of the evaluatorThe evaluation will be carried out by an international / Pakistani evaluation consultant with the following profile: Knowledge about the Community Based Disaster Risk Management (CBDRM)/ Water, Sanitation

and Hygiene (WaSH) projects in context of Pakistan and particularly in KPK. Significant field experience in the evaluation of humanitarian / development projects; Relevant degree / equivalent experience related to the evaluation to be undertaken; Significant experience in coordination, design, implementation, monitoring and evaluation of

programmes. Programmes (Well versed with OECD-DAC evaluation Criteria). Good knowledge and experience of gender equality and non-discrimination;

8

Good communications skills and experience of workshop facilitation; Ability to write clear and useful reports (required to produce examples of previous work); Fluent in English, however for Pakistani national’s fluency in Urdu and Pushto language would be

an asset. Understanding of donor requirements; Ability to manage the available time and resources and to work to tight deadlines; Independence from the parties involved. Applications from Pakistani nationals are strongly encouraged

Others: Consultant will be responsible for his own health insurance Consultant will be responsible for its own equioment

8. LEGAL AND ETHICAL MATTERS

The ownership of the draft and final documentation belong to the agency and the funding donor exclusively. The document, or publication related to it, will not be shared with anybody except Action Against Hunger before the delivery by Action Against Hunger of the final document to the donor.

Action Against Hunger is to be the main addressee of the evaluation and its results might impact on both operational and technical strategies. This being said, Action Against Hunger is likely to share the results of the evaluation with the following groups: Donor(s); Governmental partners; Various co-ordination bodies.

For independent evaluations, it is important that the consultant does not have any links to project management, or any other conflict of interest that would interfere with the independence of the evaluation.

8.1. Intellectual Property RightsAll documentation related to the Assignment (whether or not in the course of your duties) shall remain the sole and exclusive property of the Charity.

9. Dead line for Proposal:Interested candidates/ institutes should submit their technical & financial proposal along with updated profile on or before December 07th , 2016 till 5:00 PM. Organization have right to reject all those proposals which will deliver after deadline. Technical & Financial Proposal need to submit in a sealed envelope with the clearly marked “Propsal for RNE Final Evaluation” at our office address as mentioned below, Logistics Department- Action Against Hunger, 3rd Floor, 65 West Executive Heights, Fazal e Haq Road, Blue Area, Islamabad. Tel: +92-51-2806241-2

The minimum content of the proposal shall be:

- CV of the proposed consultant- At least three references- At least 2-3 previous evaluation report for similar work- Financial offer with a rate in PKR or USD/day for the consultancy fees

9

10. ANNEXES TO THE TORs

I. Log frame Project Log frameII. Evaluation Criteria and Detailed Evaluation Questions

III. List of Project documents for the desk reviewIV. List of people to be interviewedV. Evaluation Criteria Table

VI. Good Practice Template

10

Annex I: Project Log frame

11

Annex II: Evaluation Criteria and Detailed Questions

To assess the project intervention against each evaluation criteria, the evaluator will respond to the following evaluation questions:

Design: Have local capacities been considered and integrated into the project’s resilience approach? Were project objectives are clear and realistic Did a regular collection of monitoring data apart from the initial concept and design? Have

systems been put in place by ACF to improve the monitoring and the delivery of support services?

To what extent have previous applicable project evaluation recommendations been considered in the design of this project?

To what extent did the project design take into account gender issues as per the ACF Gender Policy? To what extent the project mainstream gender equity in the design and delivery of activities (e.g. integrating women’s specific needs, Needs of persons with disabilities & oldage and capacities in the resilience approach)?

What measures have been taken to ensure transparency and accountability? How could ACF improve its communication with communities in order to improve transparency and accountability?

Relevance/Appropriateness: Are the interventions aligned with local needs and priorities as well as government priorities. Was the intervention in-line with donor policies and priorities? What were the challenges faced (operational and programmatic) and how could they

mitigated? Coherence:

Is the project implementation strategy in-line with the national strategies and policies to ensure consistency?

Does the project design fit within and complement existing initiatives by other organizations?

Are the project interventions being align with ACF country strategy and DRR strategy? To what extent has ACF managed to coordinate effectively among different stakeholders?

Coverage: To what extent were the most vulnerable members of the target population effectively

covered by the project? Were the criteria and indicators defined in the project suitable to identify the vulnerable

population? Is the geographic coverage being relevant and as per government priorities? What are the remaining gaps or continuation of intervention that need to be covered within

the target area?

Efficiency: How economically were the resources/inputs converted into results? Is the current operational model cost effective and efficient to achieve the results?

Effectiveness: To which extent does the project reach the planned targets against the proposal indicators

in a timely manner? What were the main challenges? What was the level of participation of the communities during the targeting process and the

implementation of the program?

12

How has the project responded to positive and negative factors (both foreseen and unforeseen) that arose throughout the implementation process? Has the project team been able to adapt the implementation process in order to overcome these obstacles without hindering the effectiveness of the project?

How effectively were the project stakeholders (especially PDMA, DDMA, NDMA and academia) engaged in the project interventions? And how effective was the capacity support?

Sustainability and likelihood of impact: Does the project design include a sustainability strategy involving local communities and

beneficiaries as well as other relevant stakeholders? What measures, and with what success, did ACF ensure that all interventions were

sustainable (including training, quality hardware, integration with government departments and resilience)?

Assess and evaluate ACF’s project exit strategy. To what extent is there ownership and buy-in of the approach? Is it likely to be

continued once the program ends?

13

Annex III: List of project documents for the desk review

The following documents will be reviewed by the evaluator during the desk review phase:

DocumentsAction Against Hunger Country StrategyNational Strategy in Action Against Hunger relevant sectors (DRR, WaSH, etc.)Cluster Strategy in Action Against Hunger relevant sectors (DRR, WaSH, etc.)Any prior assessments for DesignIntervention proposal and budget (and amendments if applicable)Partnership Agreement/Memorandum of understanding with University of Peshawar Logical frameworkMonitoring planProject/Activity Scope Document (ASD/PSD)Activity Progress ReportsField visits reportsProject implementation planTraining and supervision reportsBeneficiaries monitoring reportsBaseline and endline reportsSurveys and research studiesIntervention Villages and Beneficiries ListAny other relevant document

Annex IV: List of people to be interviewed

The evaluator will interview the following stakeholders2:

Internal

Position and OrganisationCountry DirectorDeputy Country Director-ProgDeputy Country Director-SupportWASH CoPQA CoField Co - KPKGrants coFinance Co Logistic CoHR CoSenior Project ManagerPQA Supervisor KPKSupport unitsProgram Team

2 This list is non-exhaustive.

14

External

Name Position and Organisation CoTo be confirmed Clusters (Nut / FSL / WaSH)

To be confirmed DOHTo be confirmed PPHI /To be confirmed PDMATo be confirmed UN agenciesTo be confirmed Military IntelligenceTo be confirmed Agriculture department

Based on samplingTarget Communities andBeneficiaries

Annex V: Evaluation Criteria Table

The evaluator will be expected to use the following table to rank the performance of the overall intervention using the DAC criteria. The table should be included in annex of the evaluation report.

Criteria Rating(1 low, 5 high)

Rationale

1 2 3 4 5DesignRelevance/AppropriatenessCoherenceCoverageEfficiencyEffectivenessSustainability Likelihood of Impact

Guidance for rating the evaluation criteria:

Rating Definition

1. Unsatisfactory Performance was consistently below expectations in most areas of enquiry related to the evaluation criteria. Overall performance in relation to the evaluation criteria is not satisfactory due to serious gaps in some of the areas. Significant improvement is needed. Recommendations to improve performance are outlined in the evaluation report and Action Against Hunger will monitor progress in these areas.

2. Improvement needed

Performance did not consistently meet expectations in some areas of enquiry– performance failed to meet expectations in one or more essential areas of enquiry. Some improvements are needed in one or more of these. Recommendations to improve performance are outlined in the evaluation

15

report and Action Against Hunger will monitor progress in these key areas.

3. On average meets expectations

On average, performance met expectations in all essential areas of enquiry and the overall quality of work was acceptable. Eventual recommendations over potential areas for improvement are outlined in the evaluation report.

4. Meets expectations

Performance consistently met expectations in all essential areas of enquiry, and the overall quality of work was fairly good. The most critical expectations were met.

5. Exceptional Performance consistently met expectations due to high quality of work performed in all essential areas of enquiry, resulting in an overall quality of work that was remarkable.

Annex VI: Good Practice Template

The evaluation is expected to provide one (1) key example of Good Practice from the intervention/country office. This example should relate to the technical area of intervention, either in terms of processes or systems, and should be potentially applicable to other contexts whereAction Against Hunger operates. This example of Good Practice should be presented in the Executive Summary and the Main Body of the report.

Title of Good Practice(Max. 30 words)

Innovative Features and Key Characteristics

(What makes the selected practice different?)

Background of Good Practice

(What was the rationale behind the good practice? What factors/ideas/developments/events lead to this particular practice being adopted? Why and how was it preferable to other alternatives?)

Further explanation of chosen Good Practice(Elaborate on the features of the good practice chosen. How did the practice work in reality? What did it entail? How was it received by the local communities? What were some of its more important/relevant features? What made it unique?)

Practical/Specific Recommendations for Roll Out(How can the selected practice be replicated more widely? Can this practice be replicated (in part or in full) by other Action Against Hungerprogrammes? What would it take at practical level? What would it take at policy level?)How could the Good Practice be developed further?

(Outline what steps should be taken for the practice to be improved and for the country office to further capitalise on this good practice)

16

i Subject to their availabilityii PQA CoordinatorELA Unit Focal Person – ACF UK while in field it will be PQA Coordinator.