Decentralized evaluation for evidence-based decision makingWFP Office of Evaluation

Decentralized Evaluation Quality Assurance System (DEQAS)

Quality Checklist for Decentralized Evaluation Terms of ReferenceVersion April 2017

[Title of the decentralized evaluation]

Overall

General Comments/Status

Title: Clear without being too long, reflecting the subject of the evaluation

Length: does not exceed 15 pages (excluding annexes) Accessibility: o TOR written in a clear and understandable languageo TOR adequately emphasises the strategic and/or

operational focus of the decentralized evaluation.o TOR provides a good substantive overview of the

subject of the evaluation. o TOR provides sufficient information to stakeholders

on how the evaluation will unfoldo TOR provides sufficient information to the evaluation

T O R Q C V e r s i o n N o v e m b e r 2 0 1 5 P a g e 1 | 16

team on what is expected from them.

Editing

o Template has been followed and all its elements are included in the TOR (titles)

o Acronyms are spelt out the first time they are used. o Paragraphs and pages are numbered electronically. o Cross-references are used.o Tables and diagrams are used as relevant and are

numbered.o When data or quotes are used, TOR provides sources

of the data/quotes (either directly below the table/graph or in footnotes)

1. Introduction

Overall: The introduction should provide key information about the purpose of the TOR and the facts relevant to the subject of the decentralized evaluation.

Expected Content Assessment criteria Comments/Status

Purpose of TOR Factual information about

the subject of the evaluation (i.e. activity/operation/thematic area/country portfolio/transfer modality/pilot

Expected content is provided.

Clearly indicates 1. Subject of evaluation2. Name of commissioning

office3. Period covered by the

T O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 2 | 16

project/other being evaluated)

The period covered by and scope of the evaluation

evaluation4. Scope of the evaluation

2. Reasons for the Evaluation

Overall: Concise explanation of why the evaluation is being conducted, and for who.

Expected Content Assessment criteria Comments/Status

2.1 Rationale

Specify why the evaluation is to be undertaken

Specify why it is needed at this time

Specify how the evaluation will be used by the WFP commissioning office and other stakeholders

Valid rationale Clarity on how the timing

of the evaluation meets the stated needs in decision-making processes

2.2 Objectives

Specify the objectives of the evaluation (e.g. accountability and learning)

Nuance the standard text in an additional

Standard text on accountability and learning used

Additional objectives added where appropriate and/or

T O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 3 | 16

paragraph to specify whether more weight is placed on accountability or on learning and why

the accountability and learning objectives contextualised

Text has been nuanced as required if more emphasis on one objective

2.3 Stakeholders and Users

Specify the key internal and external stakeholders and what their stakes/interests are in the evaluation

Specify intended primary and secondary users of the evaluation, both internal to WFP and external where applicable

Indicate how beneficiaries’ perspectives (men, women, boys and girls) will be included in the evaluation process, including in relation to accountability to affected populations, and how this will be addressed

All relevant stakeholders, including beneficiaries, are identified/listed

Demonstrates understanding of stakeholder interests and concerns

Identification of users is linked to the reasons for and objectives of the evaluation

Beneficiaries are identified as stakeholders

Clear analysis of how WFP commitments to accountability to affected populations will be upheld

Stakeholder analysis is gender-responsive

T O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 4 | 16

3. Context and Subject of the Evaluation

Overall: Comprehensive description of the evaluation context and information on the nature of the evaluation subject

Expected Content Assessment criteria Comments/Status

3.1 Context

Inclusion of information about the context within which the subject of the evaluation has been/is being implemented, including: Poverty, food security Government policies,

priorities and institutional capacity

Humanitarian issues, including social protection programme/s, migration patterns and host community/social tensions

Gender equality and empowerment of women (GEEW) dimensions of the context

International assistance in the area

Key external events Other WFP work in the

area and Work of other key actors

Contextual information is focussed and concise.

Information is relevant and important to understanding the context for the subject of the evaluation

The section focuses on trend data that is relevant, is coherent with scope and important to the subject of the evaluation.

T O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 5 | 16

3. Context and Subject of the Evaluation

3.2 Subject of the Evaluation

Type of activity/operation/ thematic area/transfer modality/pilot project/other subject of evaluation

Geographic scope of the evaluation subject

Relevant dates: Approval date; start date; expected end date

Main partners (Government; NGOs; Bilateral; Multilateral)

Planned outputs at design

1. Beneficiary numbers (planned and revised) disaggregated by sex and age/activity

2. Amount of transfers (food, cash, vouchers), outputs of technical assistance (TA)/capacity development activities)

3. Other outputs

Information is relevant and important to understanding the subject of the evaluation:

1. What it is?2. When it was designed? 3. What are the key inputs ($

value, technical assistance (TA)/capacity development activities)

4. What are the key objectives outcomes and planned activities?

5. What are the planned outputs (beneficiaries, C&V $)

6. What is the total level of funding to the evaluation subject over the period under evaluation? Availability of this information imperative for all DE TOR

7. Who is involved in the implementation?

Highlights relevant issues from past evaluations and

T O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 6 | 16

3. Context and Subject of the Evaluation

If existing, logic model or logical framework should be mentioned. If not available it should also be mentioned

Resources (% funded of total requirements) and key donors. If subject funded from pooled funds, show resource allocated

Key objectives, intended outcomes and activities

Other (from WFP and/or other actors) relevant preceding/ concurrent activities/ interventions/operations

Any changes in planned implementation in terms of coverage, budget, planned beneficiaries and explanations

Gender equality and empowerment of women (GEEW) dimensions as relevant

Maps/graphs for illustration

reviews that are relevant to the evaluation

Gender equality and empowerment of women (GEEW) dimensions explained

Differences between original design and final design are explained if appropriate

T O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 7 | 16

4. Evaluation Approach

Overall: Clarity on the scope of the evaluation, evaluation criteria and key questions, availability and quality of data available to support it, including any significant gaps; as well as the overall approach and aspects of methodology which the evaluation team will incorporate in their response to the TOR and in their proposed methodology.

Expected Content Assessment Criteria Comments/Status

4.1 Scope

The defined scope should narrow down and specify what will be included and not included in the evaluation, in relation to the activities of the subject of the evaluation

Specify the focus of the evaluation including: time frame, limitations, geographic areas, operations and types of activities and specific target groups (including women and girls)

Clarity on what will be covered and what will not in relation to the contents/coverage of the subject of evaluation

Clarity on duration/period that will be covered by the evaluation

Clear justification for the scope and focus, including selection of activities and areas

Clear justification for the target groups covered by the evaluation

4.2. Evaluation Criteria and Questions

Evaluation criteria selected from the list and

Clear explanation of selected evaluation

T O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 8 | 16

justified as appropriate and consistent with evaluation purpose and objectives

Specification of evaluation questions which address the selected criteria

GEEW issues mainstreamed throughout evaluation questions

criteria and justification in case some criteria are not included.

Evaluation questions are clear and relevant to the subject of the evaluation

Evaluation questions sufficiently address the selected evaluation criteria

GEEW is effectively mainstreamed throughout questions

4.3 Data Availability

Identify the main sources of information/data available to the evaluation team

List any gaps in the data and proposed ways for the evaluation team to deal with them

Clear indication of the data available and their sources

Limitations explained and included in the TOR

Suggested mitigation measures

4.4 Methodology

Standard text provided in the template.

Present the overall

Standard text used. Clear delineation of the

overall methodology for T O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 9 | 16

methodology for the evaluation outlining the criteria, impartiality requirement, and the methods that will be employed (quantitative, qualitative or mixed)

Identify key risks and appropriate mitigation/ management measures for the methodology approach proposed for further refinement during inception as appropriate.

Specify how gender issues will be addressed by gender-responsive evaluation methodology, tools and data analysis techniques.

the evaluation outlining the data collection methods and overall approach to ensure impartiality

Key risks and appropriate mitigation/management measures for proposed approach, satisfactorily identified;

Sufficient detail to understand how GEEW issues will be addressed by the methodology

4.5 Quality Assurance

Standard text provided in the template of TOR.

Identify additional measures to assure the quality of the process and product, and thus

Standard text used. Quality standards are

included in the TOR. Quality assurance is built

into the whole process by briefly showing how

T O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 10 | 16

increase the credibility and impartiality of the evaluation.

different products will be quality assured

Requirements are spelled out for the evaluation team to ensure validity and accuracy of data `

5 Phases and Deliverables

Overall: Clear statement of the overall timing and key deliverables for the evaluation.

Expected Content Assessment Criteria Comments/Status

Clarify the timing of the five evaluation phases and the deliverables including milestones

Specify if other products, in addition to the evaluation report, will be required (e.g. Power Point; Free-standing Summary, Video clip or if suggestions for innovative products are requested from the evaluation team).

Suggested table in annex 2 of the ToR Template is used

Reasonable amount of time for each of the phases/steps is provided for

Clearly identified deliverables with stated responsible persons

6 Organization of the EvaluationT O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 11 | 16

Overall: Clear statement on the overall organisation of the evaluation including its management arrangements, the evaluation team and required composition/competencies as well as anticipated lines of communication, timing of key deliverables and key considerations, including security, to be considered when planning to undertake the evaluation.

Expected Content Assessment Criteria Comments/Status

6.1 Evaluation Conduct

Indicate how the evaluation will be conducted and led, including key aspects of communication

Specify ethical considerations for the conduct of the evaluation

Include the evaluation schedule

Specific tasks and outputs are identified for team members in the evaluation schedule.

Specifies team structure and relationship with WFP Evaluation Manager

Consideration of ethical issues that may relate to the subject of the evaluation

Evaluation schedule is logical and provides adequate time for all phases of the evaluation (including consultations on draft documents), and annex 2 in the TOR is used

6.2 Team composition and competencies

Indicate how the evaluation team will be composed (including gender, nationality, and if appropriate cultural background)

The balance of competencies of the evaluation team are consistent with the subject of the evaluation and include, for each

T O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 12 | 16

Key competencies and knowledge of evaluation team, including the appropriateness of the skills combination for the evaluation; language skills; and leadership skills for the team leader

Specify gender expertise on the team and appropriate skills to assess gender dimensions

member, expertise in one or more topics of the evaluation

Inclusion of gender parameters for evaluation team

Address fluency in the language/s spoken

Gender expertise is included

Leadership, analytical and communication competencies are spelt out for the team leader

6.3 Security Considerations

Specify any security considerations that may be relevant

Security considerations are relevant to the context for the evaluation

Security considerations are relevant to the nature of the contracting arrangements with WFP

7 Roles and Responsibilities of Stakeholders

Overall: Clear statement of expectations in terms of involvement of internal and external stakeholders and their respective roles and responsibilities

Expected Content Assessment Criteria Comments/StatusT O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 13 | 16

Describe roles of key stakeholders in the evaluation process (WFP Country/Area/Sub Office; WFP Regional Bureau; WFP Headquarters; National government stakeholders (relevant Ministries, departments, units); Partner UN agencies; Implementing partners); Office of Evaluation, Internal Evaluation Committee, Evaluation Reference Group.

Describe responsibilities of key stakeholders clearly.

All relevant stakeholders are identified, demonstrating impartiality.

Clear understanding of stakeholder interests and concerns.

Indirect stakeholders who may have a role are identified.

Clarity on measures to ensure impartiality and credibility of the evaluation, as they related to each stakeholder

Internal Evaluation Committee and the Evaluation Reference Group have been established and their roles and responsibilities have been set out

8. Communication and Budget

Overall: Clear statement of communication arrangements throughout the process, including how the report will be disseminated and the roles and responsibilities for communication and sharing the report.

T O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 14 | 16

Expected Content Assessment Criteria Comments/Status

8.1 Communication

Specify intentions in relation to the Communication and Learning Plan, and how this will be developed

Determine communication-related roles and responsibilities

Specify the need for translation and the language of each report, if not English

Communication-related roles and responsibilities have been set out

Communication and learning plan is well explained and taken into account in timeline and budget

Requirements for translation clearly set out

8.2 Budget

Overall: Any budgetary guidelines or limitations are clearly set out.

Expected Content Assessment Criteria Comments/Status

Standard text provided in the template of the TOR.

Based on the contracting approach chosen, state how the evaluation budget will be arrived at, and how total cost will be broken down

Clarify whether the budget will include the cost of

Standard text used Clarity on the contracting

method to be used (procurement through open tender, LTA/SLA or HR action)

Clarity on the cost elements that will/should be included in the evaluation budget such as

T O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 15 | 16

workshops or special communication efforts if needed.

of workshops or special communication efforts included, as appropriate.

Annexes Comment Comments/Status

Map(s) Evaluation

Timeline/schedule Evaluation committee

membership Evaluation reference group

membership Acronyms Technical annexes

Assessment criteria: Complete and necessary

T O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 16 | 16