1.1.1 presentation
TRANSCRIPT
-
7/28/2019 1.1.1 Presentation
1/50
On-line e-Learning programme on:
EQUITY-FOCUSED EVALUATIONS
Module 1: Evaluation and Equity
Unit: How to design, implement and use Equity-focused evaluations
Marco Segone, UNICEF Evaluation Office
Michael Bamberger, Independent Consultant
1 Disclaimer: The opinions expressed are the personal thinking of the contributors and do not necessarily re ect the policiesor views of UNICEF or any other organization involved and named in
this publication.
-
7/28/2019 1.1.1 Presentation
2/50
How to design and implement
Equity-focused evaluations
Marco Segone,UNICEF Evaluation Office, co-chair UNEG TF on NECD,
Michael Bamberger,Independent consultant
-
7/28/2019 1.1.1 Presentation
3/50
Outline
1. What does equity mean?2. Why does equity matter?
3. What are the determinants of inequity?
4. Equity-based programming: whats thefinal aim?
5. What is an equity-focused evaluation?
6. How to manage Equity-focused
evaluations?
-
7/28/2019 1.1.1 Presentation
4/50
What does equity means?
-
7/28/2019 1.1.1 Presentation
5/50
Defining Equity
Disparities between population groups Some of these disparities may be
unavoidable (e.g. driven by biology)
The disparities between population groupsthat are avoidable and unfair are termed
inequities
Equity is therefore based on notions offairness and social justice
-
7/28/2019 1.1.1 Presentation
6/50
Which one is an Inequity?
1. Children leaving in poorest 20% ofhouseholds in Thailand have nearly doubleChild Mortality rate in comparison to therichest 20%;
2. People aged 60 and above have higherprevalence of coronary heart disease thanpeople aged 25
3. Stunting is 16% higher among girlscompared with boys;
4. Female newborns have lower birth weightcompared with male newborns;
-
7/28/2019 1.1.1 Presentation
7/50
Defining inequity
Inequities are defined as
disparities that are avoidable and
considered unfair and unjust.
-
7/28/2019 1.1.1 Presentation
8/50
Why does equity matter?
-
7/28/2019 1.1.1 Presentation
9/50
Why equity
Equity has a significant positive impact
in reducing poverty
Equity has a positive impact on
economic growth
-
7/28/2019 1.1.1 Presentation
10/50
Why equity
Equity has a positive impact in the construction ofa socially fair and democratic society
Prolonged inequity may lead to the banalization
of inequity
Inequity constitutes a violation of human rightsand hampers the equitable achievements on
Human Development and MDGs
-
7/28/2019 1.1.1 Presentation
11/50
What are the determinants ofinequity?
-
7/28/2019 1.1.1 Presentation
12/50
Determinants: Income
Note: Estimates are based on a subset of 80 countries, covering 60% of the world population (2000-2009). Estimates for the Middle East and North Africa cover 47% of the
population of this region. The graph illustrates differentials by wealth quintiles; these estimates should not be used for comparison with other data sets.
Source: UNICEF global databases, 2010.
Percentage of children under 5 years old who are registered, by household wealth quintile
In some regions, children from the richest households are 2-3times as likely to be registered as those from the poorest
11
-
7/28/2019 1.1.1 Presentation
13/50
Determinants: Income
Nepal stunting trends and equity
Fig-25: Stunting Trend for children (6-23 months) by Wealth
Status
0%
10%
20%
30%
40%
50%
60%
1996 2001 2006
Poorest
Poorer
M iddle
Richer
Richest
Total
Source: DHS data. From NAGA pg. 27
-
7/28/2019 1.1.1 Presentation
14/50
Determinants: locations
District level Data - Nyanza Province
Significant increases in PMTCT coverage: 24% to 73% between 2004-2009
Largely on track to reach Universal Access goal of 80% However, reaching the last 20% will need refocusing strategies and other context
specific analyses by province.
-
7/28/2019 1.1.1 Presentation
15/50
Determinants: locations and Ethnicity
0
5
10
15
20
25
30
35
40
45
RAT MON ST PVH OMC SR KRA KTH PAI PUR KCM KK KEP KCN BMC BTB SHV KSP KAM PV KAN SVR TAK PNH
This increases to 45% (!!) for
17 year olds in Ratanakiri who
dont have Khmer as mother
tongue
In Cambodia, % of 17 years olds who NEVER attended school
-
7/28/2019 1.1.1 Presentation
16/50
Determinants: Gender
The majority of out-of-school children
continue to be girls (53%); achieving
gender parity would mean 3.6 millionmore girls in primary school
-
7/28/2019 1.1.1 Presentation
17/50
Socio-economic: Disabilities
90% of children with disabilities in
developing countries do not attendschool
-
7/28/2019 1.1.1 Presentation
18/50
Multiple Dimensions: Who is not learning? Disparity in
learning
-
7/28/2019 1.1.1 Presentation
19/50
Equity-based programming:whats the final aim?
-
7/28/2019 1.1.1 Presentation
20/50
Final aim
All rights for all children everywhere, by
prioritizing the most deprived
-
7/28/2019 1.1.1 Presentation
21/50
All rights for all children everywhere, by
prioritizing the most deprivedGraph 1: % of children (7-14 years old) out of school, by race/ethnicity
2000 Baseline and goals for national average, black and white children
2010 Goal: reduce by 50% the national average an d the equity ratio between black and white children
5.5
2.8
3.8
2.2
6.9
3.1
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
EquityRatio:1,8
Equityratio:
1,4
Black children
White children
National average
-
7/28/2019 1.1.1 Presentation
22/50
Public policies must be designed
based on the inequity profiles
(% of births attended by medically trained personnel by quintile)
-
7/28/2019 1.1.1 Presentation
23/50
What is
an equity-focusedevaluation?
-
7/28/2019 1.1.1 Presentation
24/50
What is an equity-focused evaluation?
An assessment made of the relevance,
effectiveness, efficiency, impact and sustainability
of interventions on equitable development results.
-
7/28/2019 1.1.1 Presentation
25/50
Purposes of equity-focused evaluation
Accountability
Organizational learning and improvement
Evidence-based policy advocacy
Contribute to Knowledge Management
Empowerment of worst-off groups
National Capacity development for equity-focused M&E systems
-
7/28/2019 1.1.1 Presentation
26/50
HOW to manage
Equity-focusedevaluations?
-
7/28/2019 1.1.1 Presentation
27/50
Key Steps
1. Preparing
for the EFE
1
2. Preparing
the ToR of
an EFE
3. Designing
theevaluation
4.Collecting/
analysing
evidence
432
5. Findings,
conclusion and
reccs
65
6. Utilizing the
evaluation
MANAGE RISKS * MANAGE RISKS * MANAGE RISKS * MANAGE RISKS
-
7/28/2019 1.1.1 Presentation
28/50
Key Steps
1 432 65
MANAGE RISKS * MANAGE RISKS * MANAGE RISKS * MANAGE RISKS
1. Preparing
for the EFE
P i f
-
7/28/2019 1.1.1 Presentation
29/50
. Preparing for
the Equity-focused evaluation
Determining the evaluability of the interventions
equity dimensions
Identifying evaluation stakeholders, including
worst-off Identifying intended use by intended users
Ensuring appropriate budget
Identifying potential challenges in promoting andimplementing Equity-focused evaluations
-
7/28/2019 1.1.1 Presentation
30/50
Key Steps
1 432 65
1. Preparing
for the EFE
2. Preparingthe ToR of
an EFE
-
7/28/2019 1.1.1 Presentation
31/50
Preparing the ToR of an EFE
Defining the scope and purpose of the evaluation
Framing evaluation questions focusing on equity
Selecting technically-strong and culturally-sensitive
evaluation team
-
7/28/2019 1.1.1 Presentation
32/50
Key Steps
1 432 65
1. Preparing
for the EFE
2. Preparingthe ToR of
an EFE
3. Designing
theevaluation
-
7/28/2019 1.1.1 Presentation
33/50
Designing the evaluation
Selecting the appropriate evaluation framework
Selecting the appropriate evaluation design
Selecting the appropriate evaluation methods
-
7/28/2019 1.1.1 Presentation
34/50
Special challenges for evaluating
equity-focused programmes Outcomes difficult to define and measure
Conventional evaluation designs do not work
Innovative evaluation designs are needed
Understanding the complex contexts in which EFE areimplemented
The importance of process
Studying sensitive issues
Vulnerability Political power relations
Culture
Wh i dibl id f
-
7/28/2019 1.1.1 Presentation
35/50
What is credible evidence of
Equity-focused outcomes?
Experimental designs? Can numbers capture the complexity of equity-
focused interventions?
In-depth analysis of lived experience? Can narrative help explain the many factors
affecting outcomes?
Is it possible to generalize from qualitativestudies?
Mixed methods?
-
7/28/2019 1.1.1 Presentation
36/50
The challenge of attribution and
assessing causality Causality and the definition of the
counterfactual The difficulty of finding a comparison group
The danger of positive bias
The human desire to finding causality
The danger of retrospective construction of
causality
Alternatives to the conventional
counterfactual
-
7/28/2019 1.1.1 Presentation
37/50
The importance of mixed method
approaches Combining depth and breadth
Combines narrative reporting of lived-experience and
cultural context, with the ability to generalize
Integrated approach draws on tools andtechniques from a broad range of social science
disciplines
Broadening the range of credible evidence
Triangulation
Selecting the appropriate evaluation methods:
-
7/28/2019 1.1.1 Presentation
38/50
Selecting the appropriate evaluation methods:
simple, complicated and complex programs
Selecting the appropriate evaluation
-
7/28/2019 1.1.1 Presentation
39/50
Selecting the appropriate evaluation
framework
Experimental/ quantitative approaches
Theory-based evaluation
The theory of change
Systems-based approaches
Expenditures and cost-effectiveness
Cost-effectiveness
Benefit incidence analysis Public expenditure tracking
Bottleneck analysis as the integrating framework
B ttl k l d d d f k f t
-
7/28/2019 1.1.1 Presentation
40/50
Bottleneck supply and demand framework: factors
affecting use of services by vulnerable populations
Use of services by
vulnerable population
Supply side
factors
Coverage
Budgets andavailable
resources
Culturally
acceptable
services
Culturally
sensitive staff
Demand side factors
Knowledge,
Attitudes and Practice
of vulnerable groups
Community
ownership
factors affecting
access
distance
cost of travel and
fees
available transport
cultural constraints time poverty
Contextual factors
Economic Political
Institutional
Legal and
administrative
Environmental
-
7/28/2019 1.1.1 Presentation
41/50
Knowledge, attitudes and practice
[KAP studies]
Knowledge about a program and about
the problems it addresses
Attitudes towards the program
Behavioral changes
A h f l ti l it f d
-
7/28/2019 1.1.1 Presentation
42/50
Approaches for evaluating complex equity-focused
policies and programmes
Systems approach to evaluation Unpacking complex policies into components
that can more easily be evaluated
Pipeline designs
Policy gap analysis
Using other countries or sectors as the
comparison group
Concept mapping
Portfolio analysis
-
7/28/2019 1.1.1 Presentation
43/50
Approaches for evaluating simple
projects and programs Experimental and quasi-experimental
designs
Creative ways to define comparison groups
Naturalistic experiments
Theory-based approaches
Non-experimental designs
Qualitative, narrative methods
-
7/28/2019 1.1.1 Presentation
44/50
The potential value of
non-experimental designs Single-case methods Longitudinal designs
Panel studies
Interrupted time series
Case study designs using mixed-method
sample selection
Holistic analysis
-
7/28/2019 1.1.1 Presentation
45/50
Key Steps
1. Preparing
for the EFE
1
2. Preparing
the ToR ofan EFE
3. Designing
the
evaluation
4. Collecting/
analysing
evidence
432 65
Selecting the appropriate
-
7/28/2019 1.1.1 Presentation
46/50
g pp p
data collection methods. Examples from the bottleneck
framework
Use of services by
vulnerable populationObservation
Surveys
Focus groups
Citizen report cards
Supply side
factors
Project
documents
ObservationKey informants
Focus groups
Benefit incidence
analysis
Public
expenditure
tracking
Demand side factors KAP studies
Case studies
Observation
Surveys
Contextual factors
Diagnostic studies
Rapid surveys
Secondary data
Key informants
Mix-Methods and
triangulation
Use for all stages of the
evaluation
-
7/28/2019 1.1.1 Presentation
47/50
Key Steps
1. Preparing
for the EFE
1
2. Preparing
the ToR ofan EFE
3. Designing
the
evaluation
4.Collecting/
analysing
evidence
432
5. Findings,
conclusion and
recommendations
65
MANAGE RISKS * MANAGE RISKS * MANAGE RISKS * MANAGE RISKS
-
7/28/2019 1.1.1 Presentation
48/50
Key Steps
1. Preparing
for the EFE
1
2. Preparing
the ToR ofan EFE
3. Designing
the
evaluation
4.Collecting/
analysing
evidence
432
5. Findings,
conclusion and
reccs
65
6. Utilizing theevaluation
-
7/28/2019 1.1.1 Presentation
49/50
Utilizing the evaluation
The problem of under-utilization
Not addressing clients priority concerns
Poor timing
Not using stakeholders preferred
communication styles
Too many tables/ not enough tables
Too much narrative/ not enough narrative Not available in local languages
Only selected stakeholders receive reports
-
7/28/2019 1.1.1 Presentation
50/50
Utilizing the evaluation
The importance of utilization focused evaluation
designs
Preparing the evaluation report and alternative
forms of reporting Agreeing on an action plan
Disseminating the evaluation and preparing a
Management Response