workshop on the development of equity focused and gender … · 2016-09-15 · incorporate outcome...

WORKSHOP ON THE DEVELOPMENT OF EQUITY

FOCUSED AND GENDER RESPONSIVE EVALUATION GUIDELINES FOR CAMBODIA

22-26 AUGUST 2016 KATERINA STOLYARENKO

ANGKOR CENTURY HOTEL, SIEM REAP, CAMBODIA INDEPENDENT CONSULTANT

DAY 122 AUGUST 2016

SESSION 3: REAL-TIME

EQUITY-FOCUSED AND

GENDER RESPOSNIVE

EVALUATIONS

4

REAL-WORLD CHALLENGES TO EFGR EVALUATION

Unavailability of needed secondary data

Scarce local evaluation resources

Limited budgets for evaluations

Institutional and political constraints

Lack of an evaluation culture

Many evaluations are designed by and for external funding

agencies and seldom reflect local and national stakeholder

priorities

REAL-WORLD CHALLENGES IN INTEGRATING GENDER EQUALITY

IN EVALUATION

Conceptual difficulties

Applying the concept of “participation” to evaluation

Methodological difficulties

Applying the concept of “empowerment” to evaluation

Framework difficulties

1. Causal framework

2. Contribution framework

3. Gender analysis framework

8

ADDRESSING BUDGET CONSTRAINTS

A. Clarifying information needs

B. Simplifying the evaluation design

C. Look for reliable secondary data

D. Review sample size

E. Reducing costs of data collection and analysis

9

REDUCING COSTS OF DATA COLLECTION AND

ANALYSIS

Use self-administered questionnaires

Reduce length and complexity of survey instrument

Use direct observation

Obtain estimates from focus groups and community forums

Key informants

Participatory assessment methods

Multi-methods and triangulation

10

RATIONALIZE DATA NEEDS

Identify information needs

Simplify evaluation design (but be prepared to

compensate for ‘missing pieces’)

Review all data collection instruments and cut out any

questions not directly related to the objectives of the

evaluation.

11

LOOK FOR RELIABLE SECONDARY SOURCES

Planning studies, project administrative records, government ministries, other NGOs, universities / research institutes, mass media.

Assess the relevance and reliability of sources for the evaluation with respect to:

o Coverage of the target population

o Time period

o Relevance of the information collected

o Reliability and completeness of the data

o Potential biases

12

SOME WAYS TO SAVE TIME AND MONEY

Depending upon the purpose and level of rigor required, some of the

options might include:

Reducing the number of units studied (communities, families,

schools)

Reducing the number of case studies or the duration and complexity

of the cases

Reducing the duration or frequency of observations

13

ADDRESSING TIME CONSTRAINTS

Reduce time pressures on external consultants

Commission preparatory studies

Video conferences

Hire more consultants/researchers

Incorporate outcome indicators in project monitoring systems and documents

Technology for data inputting/coding

HOW TO OVERCOME CHALLENGES IN INTEGREGATING

GENDER EQUALITY IN EVALUATION?

Strengthen gender sensitive

indicators

Planning evaluations in programme

cycle

Methodological improvement

Evaluation capacity

development

WORKSHOP ON THE DEVELOPMENT OF EQUITY FOCUSED AND GENDER RESPONSIVE EVALUATION GUIDELINES FOR CAMBODIA



DAY 223 AUGUST 2016

SESSION 4: COUNTRY LED M&E SYSTEM

KEY DEFINITIONS

NATIONAL EVALUATION POLICIES NATIONAL EVALUATION SYSTEMS

formal policy defining the purpose, responsibilities, functions and organization of the

public-sector evaluation function

a system put in place to implement NEP

Complement each other

give the normative framework mechanism that operationalize the principles dictated in the NEPs

Sources: Bamberger, et al (2014); Roseinstein (2015)

KEY DEFINITIONS

Formalized NEP - there is an official document, or decree mandating or requiring the use of evaluation

Not-formalized NEP – evaluation is conducted routinely, but there is no written policy, or there is no written policy and there is no evaluation

Well established NEP - functioning at a high level, the system is in place and is operational or evaluation practice is well established, evaluations are conducted and used and an evaluation culture exists

Evolving NEP – is or was in place and revisions are being made

Developing NEP– is being developed through advocacy among evaluators, VOPEs, stakeholders who are pro-evaluation, key players in the government international organizations

Sources: Bamberger, et al (2014); Roseinstein (2015)

THREE PATHWAYS FOR EVOLUTION

OF NATIONAL EVALUATION

POLICIES AND SYSTEMS

FACTORS CONTRIBUTING TO THE DEVELOPMENT OF NEPS

Taking control of the evaluation process

Evaluation capacity development

Integrating evaluation into the budget and financial planning systems

Standardized evaluation systems

Strengthening cross-sectoral planning and integration

Strengthening public accountability

Improving the performance of the public sector

Strengthening the results framework

Decentralization and devolution of power to local communities

HOW NEPS CAN IMPROVE GOVERNMENT PERFORMANCE?

The policy cycle: linking M&E with policy, planning, budgeting and management

FOUR ESSENTIAL BUILDING BLOCKS FOR AN EFFECTIVE NATIONAL M&E SYSTEM

NATIONAL INTEGRATED M&E FRAMEWORK

KEY COMPONENTS OF NEPS INFRASTRUCTURE

ROLES AND RESPONSIBILITIES OF NATIONALSTAKEHOLDERS IN NEPS

ROLE OF DONORS IN ESTABLISHING OF NEPS

CHALLENGES IN BUILDING NEPS–LESSONS FROM INTERNATIONAL EXPERIENCE

M&E drivers

A serious effort to build a national M&E system starts with a substantive government demand for M&E information. It requires political will and sustained commitment. Central leadership and a plan are very important.

M&E information is not an end in itself. A reasonable demand for evaluation is fundamental

Incentives for using M&E information

A strong civil society that demands that public policies and programmes be designed and implemented based on evidence.

The capacity to use M&E information relies on incentives that encourage managers to demand such information and actually use it as part of their normal operations. This could take the form of sanctions for not complying or rewards for meeting requirements.

Internal infrastructure on its own is insufficient to sustain an M&E system. A number of formal requirements associated with its use will oblige managers and senior officials to invest in M&E development.

CHALLENGES IN BUILDING NEPS– LESSONS FROM INTERNATIONAL EXPERIENCE

Stewardship close to the centre

It helps to have the institutional lead of the M&E system close to central government where it can not only lead in the design and roll out of the M&E system across government, but also serve as a M&E ‘champion’ and facilitator.

Since countries’ M&E systems are generally developed incrementally, it is important to monitor progress and make regular adjustments. This ‘oversight’ role can also be part of the central agency’s mandate.

Oversight by the national audit office could be a way of giving broad and public exposure to how well the M&E system is being implemented and to any adjustments that might be needed.

Incentives for using M&E information

There are too few skilled M&E specialists in most national M&E systems. A long-term training strategy needs to include a development component to build the necessary skill sets.

Other (non-technical, but important) audiences for training and M&E orientation include senior officials (who provide the leadership and political support needed to finance and sustain the M&E system); the users of M&E information (budget analysts, poverty analysts, programme managers, etc.); civil society and the private sector.

FORMULATION OF A GENDER-RESPONSIVE NEPS

DAY 223 AUGUST 2016

SUSTAINABLE DEVELOPMENT GOALS (SDGS)

GENDER RELEVANT SDGS12 out of 17 Goals covering gender equality

Millennium Development Goals (MDGs) (2000-2015)

Goal 3: Promote gender equality and empower women

Target 3.A: Eliminate gender disparity in primary and secondary education, preferably by 2005, and in all levels of education no later than 2015

Goal 5: Improve maternal health

Target 5.A: Reduce by three quarters, between 1990 and 2015, the maternal mortality ratio

Target 5.B: Achieve, by 2015, universal access to reproductive health

Sustainable Development Goals (2016-2030)

Goal 5 “Achieve gender equality and empower all women and girls”

9 targets, 14 indicators addressing multiple concerns (none covered in the MDG framework)

Around 34% (58 out of 169) of Targets explicitly or implicitly address GE & EWG

Strong emphasis on disaggregation, including by sex and other relevant characteristics to capture intersecting inequalities (TG 17.18; para 74.g)

FROM MDGS TO SDGS

Marco Segone, UN Women - https://www.youtube.com/watch?v=2n0dgb26i-0

SDG INDICATORS

230 SDG indicators*

32% gender relevant indicators **

21% indicators explicitly to be

disaggregated by sex

* 7 duplicates and 2 triplicates; ** 1 indicator is repeated 3 times under different goals

EVALUATING-SDGS-EQUITY-FOCUSED-AND-GENDER-RESPONSIVE-LENS

1. 2030 Agenda/SDGs

2. EVALAGENDA 2020

3. UN RESOLUTION ON EVALUATION

4. BANGKOK DECLARATION

EVALUATING-SDGS-EQUITY-FOCUSED-AND-GENDER-RESPONSIVE-LENS

EvalGender+

EvalYouth

EvalSDGs

EvalIndigenous

Global Parliamentarian Forum for Development Evaluation

NO ONE LEFT BEHIND

2030 AGENDA/SDGS

EVALAGENDA 2020

The enabling environment for evaluation

Institutional capacities

Individual capacities for evaluation

Inter-linkages among these first three dimensions

CAMBODIA AND SDGS

CMDGs has finished at the end of 2015

CMDGs Assessment

CMDG1: Poverty Met on main indicators

CMDG2: Basic Education Most likely to be met on main indicators

CMDG3: Gender Met on main indicators

CMDG4: Child Health Met

CMDG5: Maternal Health Met

CMDG6: Communicable Diseases Met

CMDG7: Environment Met on main indicators

CMDG8: Partnership Met

CMDG9: Demining Some way to go

INTERNATIONAL EXPERIENCE: NATIONAL EVALUATION POLICIES

STATUS OF NATIONAL EVALUATION POLICIES WORLDWIDE

NEP OR EVALUATION PRACTICE (N=61)

GENDER EQUALITY AND NEPS

GENDER-RESPONSIVE NATIONAL EVALUATION POLICIES AND SYSTEMS

Country where the NEPS integrates gender equality thanks to the existence of the national gender policy

Countries where the NEPS integrates gender equality due to the existence of a national development strategy that integrates gender equality

Countries where the NEPSs do not integrate gender equality, but gender-responsive evaluations are implemented

1

2

3

National gender policy or action plan Reference to gender in the NEPS

Benin Not found No

Bhutan Not found No

Chile National Action Plan on Women, Peace and Security No

Colombia Not found No, but certain categories of “gender” or “wo-

men’s equality” projects are required to include

sex-disaggregated M&E indicators

Costa Rica Not found No

Ethiopia National Action Plan on Gender and Equality Yes

India Not found No

Kenya Plan of Action to Implement the National Policy on

Gender and Development

No

Kyrgyz Republic Not found No

Malaysia Not found No

Morocco Government Plan for Gender Equality 2012-2016 No

Nepal Not found Yes

South Africa Women’s Empowerment and Gender Equality Bill No

Sri Lanka Not found No

Uganda Uganda Gender Policy 2007 No, the Gender Policy includes an M&E strategy

National gender policy and reference to gender in the NEPSs

NEP CASE-STUDIES

South Asia Regional Consultation on National Evaluation Policies 40

Switzerland Colombia Uganda South Africa Canada Malaysia

Population (m) 8.01 47.7 35.6 50.7 34.35 29.76

Area (‘000km2) 41 1141 241 1220 9984 329

GNI per capita ($) 80,970 7,020 510 6,960 50,970 9,820

Life expectancy

(female)

85 78 55 58 83 77

Form of state federalparliamentary

republic

unitary presidentialconstitutional

republic

presidential system, multiparty democratic republic

constitutionalparliamentary republic

federal/constitutional monarchy,

parliamentary system

federal constitutional monarchy

Administrative

divisions

26 cantons 32 department, 1

capital district

111 districts, 1 capital city 9 provinces 10 provinces, 3

territories

13 states, 3 federal

territories

National Evaluation

Association

SEVAL (1996)

GEN (2006)

SIPSE UEA (2001) SAMEA (2005) CES (1981) MES (1995) (initiated),

registered 1999

Legal & Policy

Framework

constitution constitution constitution constitution administrative

policies

administrative directives

Federal Constitution

of the Swiss

Confederation of 18

April 1999

Parliament Act of 13

December 2002

Constitution (1991)

Resolution nr 63 (1994)

Ley organica 152 de

1994

Conpes 3294 de 1994

Ley 1152 de 2007

The Constitution of the Republic of Uganda

(1995)

The Local Governments Act (1997)

The Uganda Bureau of Statistics Act (1998)

The Budget Act (2001)

The National Planning Authority Act (2002)

The Public Finance and Accountability Act

(2003)

The National Audit Act (2003)

The Public Procurement and Disposal of

Public Assets Act (2003)

The Local Governments (Financial and

Accounting) Regulations (2007)

The Education Act (2008)

The NGO Policy (2008)

Constitution (sections 195,

92 and 133) of 1996

Public Service Act of 1999

Public Finance

Management Act of 2003

(sections 27 and 45)

Municipal Finance

Management Act of 2003

Government–wide

Monitoring and Evaluation

System of 2005

Federal Accountability

Act

(2006)

Policy on Evaluation

(2009)

National Action Council

Directive No1 dd. 15

October 2004

Federal Government

Circular No3 “Guideline on

Program Development

Evaluation” dd. 24 August

2005

10th Malaysian Plan 2011-

2015

Annual Budget Circular

http://en.wikipedia.org/wiki/Federal_government

http://en.wikipedia.org/wiki/Parliamentary_republic

http://en.wikipedia.org/wiki/Unitary_state

http://en.wikipedia.org/wiki/Presidential_system

http://en.wikipedia.org/wiki/Constitutional_republic

http://en.wikipedia.org/wiki/Constitutional_democracy

http://en.wikipedia.org/wiki/Parliamentary_republic

http://en.wikipedia.org/wiki/Federation

http://en.wikipedia.org/wiki/Constitutional_monarchy


Country Year Reform Purpose

Canada 1977, revised 1991, 2001, 2009

New Policy on Evaluation Create a comprehensive and reliable base of evaluation evidence for

use in supporting policy and programme improvement, expenditure

management, cabinet decision-making and public reporting. Requires

100% evaluation coverage every 5 years of all programs with Direct

Programme Spending.

Switzerland 1984 Research Program “The

Effectiveness of Public

Measures”

Strengthen the results orientation in the federal administration, promote

transparency and serve public accountability

Colombia 1994, revised 2005

Medium-Term Expenditure

Framework

Improve budget planning

Uganda 2004, revised 2013

National Policy on Public

Sector Monitoring and

Evaluation

Improve government performance in utilization of foreign aid and

demonstrating good governance and accountability to its citizens and

development partners

Malaysia 2005, revised 2013

Integrated Results Based

Management System

Outcome evaluations of government-funded projects and programmes

South Africa 2011 Government-wide M&E

improvement

National Evaluation Policy Framework, management performance

assessment tool, frontline service delivery monitoring, outcomes

monitoring

NEP’s Legislation



Long-term national

plan

Sustainable

development strategy

2015

National

Development

Plan 2010-2014

Vision 2030 2030 National Plan Government

Management

Agenda, Results for

Canadians

Vision 2020

National

ResponsibilityPCA DNP OPM DPME in Presidency CEE ICU

Main Types of

Evaluations

Policy, summative 5 types Policy,

Impact

6 types 5 types Formative

Outcome

M&E Tools SEVAL Evaluation standards SISMEGSISDEVAL

Output budgeting

tool Baraza

Outcomes based approachPlanning process

Monitoring the management performance

of departmentsMonitoring front line service

deliveryNational evaluation system

Monitoring of localgovernment

MRRSMAF

Strategic ReviewsWhole-of-

Government Framework

Government Spending and Performance

Overview

MyResults

Funding for

evaluations

Earmarked funds for

evaluation

External funding

plays an important

role in evaluation

funding,

but the level of

funding from the

national

government is

increasing

External

funding plays

an important

role in

evaluation

funding

Earmarked funds for

evaluation

Earmarked funds for

evaluation

Earmarked funds for

evaluation



Evaluation Programs

Centre for Continuing Education, University of Bern

University of Applied Sciences and Arts, Northwestern Switzerland, College of

education, Institute of further education and consulting, in

cooperation with the University of Bern

N/A Uganda Christian University, Makerere

University, Uganda Management

Institute

CLEAR at the University of the Witwatersrand,

CREST at the University of Stellenbosch

Carleton University, IDEA International Institute,

École Nationaled’Administration Publique,

Queen’s University, Université Laval, Universitéde Montréal, University of

Ottawa, University of Toronto

N/A

Good practices

ARAMISThe ‘External Studies’

Database

Management control meetings with the President

Town Hall MeetingsEvaluation RadarColombia Lider

Community based monitoring systems

Evaluation coordination

The outcomes system

Frontline service delivery

assessments

Results-based Management and

Accountability Frameworks Management Resources

and Results StructureFederal Accountability ActExpenditure Management

SystemStrategic Expenditure

Reviews

ProLLOBBISE

IRBMS

Key Challenges

1. Insufficient level of interest in evaluation

2. Evaluation culture

1) Data quality control

2) Utilization of M&E

1. Incentives framework for encouraging M&E practice

2. Evaluation culture

1. Planning Process

2. National Evaluation

System3. Utilization of

M&E

1. Measuring Evaluation Use

2. Promotion Evaluation Use

1. Evaluation competence

2. Insufficient resources

3. Evaluation discipline in educational institutions

45

NEPS GOAL AND OBJECTIVES

Objective 1

annual formative evaluations by every Ministry and its Programs and Activities

Objective 2

internal evaluations of it Activities and Programs

Objective 3

evaluation of key programmes once in 5 years and usage for strategic plans and budgets/policy or program adjustments

Outcome evaluations of government-

funded projects & programs

Focus

Integrated results-based management system of the public sector

EVALUATION PATH

47

LEGAL AND POLICY FRAMEWORK

REGULATED BY ADMINISTRATIVE DIRECTIVES

Outcome monitoring

• National Action Council Directive No1 dd. 15 October 2004

Outcome evaluation

• Federal Government Circular No3 “Guideline on Program Development Evaluation” dd. 24 August 2005

Outcome based planning

• 10th Malaysian Plan 2011-2015

Funding for Evaluation

• Annual Budget Circulars

• Established in 2005, when the government directed all federal- and state-level ministries and agencies to undertake outcome evaluations of their projects and programmes

• In 2009, the government adopted an outcome-based approach for planning, resource allocation, monitoring and evaluation as part of the 10th Malaysia Plan 2011–2015

The system requires annual formative evaluations for every ministry and its programmes and activities.

The evaluations are incorporated into planning processes and budgetary processes.

The evaluations also serve as supportive information for any proposal for policy or programme adjustments through the Outcome-based Budgeting System.

The approach encourages ministries to conduct internalized self-evaluations.

49

INSTITUTIONAL FRAMEWORK

M&E Champion

Ministry of Finance

ICU under Prime Minister ’s Department

• Prime Minister

• program evaluations under the budget

• project evaluations under Malaysia Plan

Administrative bodies for Evaluation

1

2

54

INTEGRATED APPROACH TO RESULTS-BASED MANAGEMENT (IRBM)

HoVerPrinciple

55

INTEGRATED DEVELOPMENT PLANNINGIntegrated National Development Framework

57

OUTCOMES-BASED BUDGETING (OBB)

58

Use of Program Logic and Programme Theory in Planning, Budgeting & M&E

2010 Rasappan, Arunaselam & Winston, Jerome, CeDRE International

ProLL

Model

59

MIS- MYRESULTS

Public Sector Policy & Program Environment

Doing The Right Things

The Right Way

For The Right People

Financial Accountability

Management Accountability

Program Accountability

MBS Program Agreement:

Verify Client Needs & Objectives

Prioritze Evaluation Questions

Determine Evaluation Design

Information Needed for

Evaluation

Determine Research Design

Design Adjustment

Data Entry

• Stakeholder Analysis of Report

• Stakeholder Feedback

Stakeholder Utilization

Facilitate

Decision-making

Logical Linkages with Substantive Purpose of Program

Data Collection

Data Sources and

Instrumentation

Preliminary Planning &

Assessment Stage

Evaluation Design

Stage

Data Collection &

Analysis Stage

Reporting & Information

Utilization Stage

© 1999 Arunaselam Rasappan/Jerry Winston

Pilot Testing• Data Collection

• Data Tabulation & Analysis

• Data Interpretation

• Validity & Reliability

Testing

Resource Program

Efficiency Improvement Compliance

Determine Evaluation Questions

• Identify pre-determined eval ssues

• Analyse program agreement

• Conduct stakeholder inventory

• Analyse program logic

Findings & Conclusions

Data Validation

Evaluation Reporting

• Multiple reports

• Multiple audiences

• User-oriented reports

• Utilization-focused reports

Data Analysis

Evaluability Assessment

ProLL Evaluation Flow Chart

PROFESSIONAL CAPACITY FOR M&E

Promotion of the evaluation function in Malaysia was stipulated by very productive tri-patriate cooperation established between the public sector (Ministry of Finance), private sector (CeDREInternational) and civil society (Volunteer Organizations of Professional Evaluation –Malaysian Evaluation Society).

Public sector

• Formulates public policies, Design Development strategies, Initiates & promote Development, Implements intervention programs, Uses M&E Services in partnership,

• Manage public resources and Build M&E capacity in Partnership

Private sector

• Provide inputs for public policy design, Advises development/policy planners, Promotes and provides M&E services, Manages private resources, Enhances M&E capabilities via research and development

VOPEs (MES)

• Advocate M&E for development results, Advocates and promotes professionalism in

• M&E, Strengthen M&E community of practitioners, Reaches out to sensitize public/private sector on M&E, Develops M&E capacity, competency and capability, act as honest broker in public-private collaboration, Promotes stronger global network relationships for shared learning

SINERGIA (NATIONAL SYSTEM OF EVALUATION OF THE RESULTS OF PUBLICMANAGEMENT)

NEPS GOAL AND OBJECTIVES

GOAL - to support the government in staying on track and improving performance as a means to achieve its public policy objectives through provision of timely and quality performance information to relevant decision-makers.

OBJECTIVES:

(1) to improve the effectiveness and impact of government policies, programmes and public institutions

(2) to help to improve efficiency and transparency of the planning and allocation of public resources

(3) to stimulate transparency in public management, in particular stimulating oversight by citizens, i.e. social control.

Colombia does not have formalized NEP

LEGAL AND POLICY FRAMEWORK

Evaluation is mandated by the national Constitution in Colombia.

The government also introduced a series of laws, decrees, and regulations to bolster the M&E system

INSTITUTIONAL FRAMEWORK

The Presidency

National Planning

Department

Ministry of Finance

Inter-sector Evaluation Committee

ORIENTING SINERGIA TOWARDS THE SDGS (2014-2030)

Colombia played a pioneering role in the creation of the High Level Inter-Institutional Commission for the Sustainable Development Goals by Decree 280 in 2015.

The commission is a formal monitoring and decision-making body organized around the implementation of the SDGs through public policies, plans, actions and programmes

Task - to direct and coordinate the process of enlisting support for, appropriating and implementing the Agenda 2030 Objectives, in association with stakeholders including territorial entities, civil society, the private sector, the academic world and the international community.


• Pioneer country in having a SDG approach in its 2014-2018National Development Plan and that this constitutes a formalinstrument for outlining the Government’s objectives.

• The plan was drawn up on the basis of three pillars,articulated around five cross-disciplinary strategies and adeveloping strategy of green growth, with the aim ofattaining the national targets for 2018 and making progresstowards the global goals of Agenda 2030.

• 2014-2018 NDP for the first time explicitly acknowledgedterritorial differences and the need to develop them so as toreach a virtuous growth path through six regional chapters,each with its own indicators and targets.

• NDP was drawn up in a participative process includingregional authorities, representatives of civil society and theprivate sector. More than 7,000 people took part in 33regional meetings and 27 sectoral consultations


2014-2018 NDP has taken 92 of the 169 objectives in the SDGs – each with at least one indicator – and they are currently part of SINERGIA’s control panels and monitoring system.

Remaining SDG objectives: i) 17 had already been incorporated into Colombian law or into CONPES documents; ii) 24 others encountered difficulties with the country’s measuring or verification systems, and iii) another 34 were not exclusive to the country

SINERGIA’S STRATEGIC EVALUATIONS IN THE

SDGS FRAMEWORK

DAY 3-523-26 AUGUST 2016

TYPES OF EVALUATION

Project

Programme

Thematic

Strategy/Policy

Joint

Impact

Formative

Summative

Ex-post

• Internal

• Independent

• External

WHAT MAKES EFGRE DIFFERENT FROM TRADITIONAL EVALUATION?

Stage 1: Planning

Stage 2: Preparation

Stage 3: Conduction

Stage 4: Reporting

Stage 5: Use and

Follow Up

focus on engagement, not just reporting

focus on improvement of gender equality & gender equity in a country through usage of evaluation findings

focus on inclusion of stakeholders, duty-bearers and rights-holders in evaluation process

participatory and respectful of all stakeholders (rights holders and duty bearers)

ESSENTIAL ELEMENTS OF EFGRE

1. Assessment of gender equality issues

2. Ensuring evaluation processes equity-focused and gender-responsive

GENDER-RESPONSIVE EVALUATION DESIGN

1. Integration of human rights and gender equality principles in evaluation:

Inclusion (disadvantaged and advantaged groups, duty bears and right holders)

Participation (design, management & conduction of evaluation)

Fair Power Relations (balance power relations between or within advantaged and disadvantaged groups)

2. Focus on:

Utilization

Empowerment

Feminist

STAGE 1: PLANNING

Evaluation Planning

Deciding on Evaluation Types and Timing

Budgeting for Evaluations

EVALUATION PLANNING

1. Identify evaluations to be included in the plan

(define main criteria for selection evaluations)

2. Prepare the evaluation plan

(commission and managing/involved but not managing)

3. Approve the evaluation plan

4. Review and update the evaluation plan

5. Track and report on the evaluation plan implementation

STAGE 2: PREPARATION

Evaluability Assessment

Stakeholders Analysis & Engagement

Development of ToR

Selection of Service Provider

EVALUABILITY ASSESSMENT

PURPOSE

1. To assess the readiness for an evaluation

2. To analyse whether or not an evaluation is worthwhile

3. To identify the needed changes

4. To formulate recommendations

KEY PRINCIPLES

Formative

Learning

Engaging

Need to examine if human rights and gender equality are

integrated into an intervention

MAIN STEPS

Duration

Type (internal/external)

Methodology

Scope

STAKEHOLDERS ANALYSIS

IDENTIFYING STAKEHOLDERS

Who: Stakeholders, disaggregated as appropriate

What: Their role in the intervention

Why: Gains from involvement in the evaluation

Priority: Importance of involvement in the evaluation process

When: Stage of the evaluation to engage them

How: Ways and capacities in which stakeholders will participate

Key QuestionsKey Principles

• Inclusiveness• Participatory and

reflective• Respect• Transparency and

accountability

STAKEHOLDERS ENGAGEMENT

UNEG STANDARD 4.6 ‘STAKEHOLDER ENGAGEMENT AND REFERENCE GROUPS’

Reference Group

Learning Group

Steering Group

Advisory Group

UN WOMEN APPROACH

Management Group Reference Group

TERMS OF REFERENCE

OUTLINE OF THE TORI. Background (programme/project context)

II. Description of the programme/project

III. Purpose (and use of the evaluation)

IV. Objectives (evaluation criteria and key questions)

V. Scope of the evaluation

VI. Evaluation design (process and methods)

VII. Stakeholder participation

VIII. Time frame

IX. Expected deliverables (including standard table of contents for an evaluation report)

X. Management of evaluation

XI. Evaluation team composition, skills and experiences

XII. Ethical code of conduct

EVALUATION CRITERIA

Relevance

Effectiveness

Efficiency

Sustainability

Impact

Gender equality and human rights (e.g. equality and non-discrimination, participation, inclusion, empowerment, accountability and social transformation )

EVALUATION QUESTIONS

To what extent is the intervention relevant to the needs and priorities as defined by beneficiaries?

To what extent is the intervention aligned with relevant normative frameworks for gender equality and women’s empowerment?

To what extent were the expected outcomes achieved?

To what extent was gender equality and women’s empowerment advanced as a result of the intervention?

What were the unintended effects, if any, of the intervention?

To what extent was capacity developed in order to ensure sustainability of efforts and benefits?

How will the benefits of the intervention be secured for rights holders (i.e. what accountability and oversights systems were established)?

GENDER-RESPONSIVE EVALUATION METHODS

Gender analysis frameworks

Feminist theory and methodologies

Appropriateness and relevance to both women and men

Participatory

Disaggregated data by social criteria

Constraints and challenges of informants

Gender roles and power relations

Context and culturally sensitive

Mixed methods (quantitative and qualitative)

ETHICAL CONSIDERATION

• Respect for dignity and diversity

• Right to self-determination

• Fair representation

• Compliance with codes for vulnerable groups

• Redress

• Confidentiality

• Avoidance of harm

UNEG Standards

SELECTION OF SERVICE PROVIDER

1. Consider the expertise required for conduction of evaluation

2. Consult key stakeholders

3. Incorporate in the ToR requirements for the service providers

4. Undertake an open and competitive process

EVALUATION TEAM COMPOSITION, SKILLS/QUALIFICATION

Women and men

Local and/or international perspectives

Evaluation knowledge and experience (quantitative and qualitative methods)

Content/sectoral knowledge and experience

Commitment to human rights and gender equality, and knowledge and experience in evaluating human rights and gender equality interventions

Understanding and application of UN mandates on HR & GE

Experience in and knowledge of participatory approaches and methods

Research and relational skills, including cultural competence

Knowledge of regional/country/local context and language.

UNEG standards

QUALITY ASSURANCE OF TOR

Describe the procedure for undertaking the quality assurance

Make assessment using the checklist

STAGE 3: CONDUCTION

Inception Report

Data Collection

Analysis and Reporting

OUTLINE OF INCEPTION REPORT I. Introduction

• Background and context

• Purpose, objectives and scope of the evaluation

• Theory of change or programme theory

II. Methodology

• Evaluation criteria and elaboration of key questions

• Indicators for measuring results (should be based on programme indicators)

• Evaluation design (method of data collection and analysis)

• Sample and sampling design

• Limitations to the evaluation

III. Evaluation matrix

• Summarizes the key aspects of the evaluation exercise by specifying what will be evaluated and how

IV. Work plan

V. Responsibilities, logistics and support

VI. Annexes

• Documents reviewed

• Draft data collection instruments (questionnaires and interview guides, lists of evaluation team members and contact details).

• Terms of reference

• Evaluation management and reference group members

STAGE 4: REPORTING

Evaluation Report

Stakeholders Involvement in Reviewing of Evaluation Reports

Quality Assurance of Evaluation Reports

STAGE 5: USE AND FOLLOW UP

Using Evaluation Results

Management Response

Communicating Evaluation Results

Follow Up on Implementation of Evaluation Results

MANAGEMENT RESPONSE