1.1.1 presentation

Upload: cbjrnmg

Post on 03-Apr-2018

225 views

Category:

Documents


0 download

TRANSCRIPT

  • 7/28/2019 1.1.1 Presentation

    1/50

    On-line e-Learning programme on:

    EQUITY-FOCUSED EVALUATIONS

    Module 1: Evaluation and Equity

    Unit: How to design, implement and use Equity-focused evaluations

    Marco Segone, UNICEF Evaluation Office

    Michael Bamberger, Independent Consultant

    1 Disclaimer: The opinions expressed are the personal thinking of the contributors and do not necessarily re ect the policiesor views of UNICEF or any other organization involved and named in

    this publication.

  • 7/28/2019 1.1.1 Presentation

    2/50

    How to design and implement

    Equity-focused evaluations

    Marco Segone,UNICEF Evaluation Office, co-chair UNEG TF on NECD,

    Michael Bamberger,Independent consultant

  • 7/28/2019 1.1.1 Presentation

    3/50

    Outline

    1. What does equity mean?2. Why does equity matter?

    3. What are the determinants of inequity?

    4. Equity-based programming: whats thefinal aim?

    5. What is an equity-focused evaluation?

    6. How to manage Equity-focused

    evaluations?

  • 7/28/2019 1.1.1 Presentation

    4/50

    What does equity means?

  • 7/28/2019 1.1.1 Presentation

    5/50

    Defining Equity

    Disparities between population groups Some of these disparities may be

    unavoidable (e.g. driven by biology)

    The disparities between population groupsthat are avoidable and unfair are termed

    inequities

    Equity is therefore based on notions offairness and social justice

  • 7/28/2019 1.1.1 Presentation

    6/50

    Which one is an Inequity?

    1. Children leaving in poorest 20% ofhouseholds in Thailand have nearly doubleChild Mortality rate in comparison to therichest 20%;

    2. People aged 60 and above have higherprevalence of coronary heart disease thanpeople aged 25

    3. Stunting is 16% higher among girlscompared with boys;

    4. Female newborns have lower birth weightcompared with male newborns;

  • 7/28/2019 1.1.1 Presentation

    7/50

    Defining inequity

    Inequities are defined as

    disparities that are avoidable and

    considered unfair and unjust.

  • 7/28/2019 1.1.1 Presentation

    8/50

    Why does equity matter?

  • 7/28/2019 1.1.1 Presentation

    9/50

    Why equity

    Equity has a significant positive impact

    in reducing poverty

    Equity has a positive impact on

    economic growth

  • 7/28/2019 1.1.1 Presentation

    10/50

    Why equity

    Equity has a positive impact in the construction ofa socially fair and democratic society

    Prolonged inequity may lead to the banalization

    of inequity

    Inequity constitutes a violation of human rightsand hampers the equitable achievements on

    Human Development and MDGs

  • 7/28/2019 1.1.1 Presentation

    11/50

    What are the determinants ofinequity?

  • 7/28/2019 1.1.1 Presentation

    12/50

    Determinants: Income

    Note: Estimates are based on a subset of 80 countries, covering 60% of the world population (2000-2009). Estimates for the Middle East and North Africa cover 47% of the

    population of this region. The graph illustrates differentials by wealth quintiles; these estimates should not be used for comparison with other data sets.

    Source: UNICEF global databases, 2010.

    Percentage of children under 5 years old who are registered, by household wealth quintile

    In some regions, children from the richest households are 2-3times as likely to be registered as those from the poorest

    11

  • 7/28/2019 1.1.1 Presentation

    13/50

    Determinants: Income

    Nepal stunting trends and equity

    Fig-25: Stunting Trend for children (6-23 months) by Wealth

    Status

    0%

    10%

    20%

    30%

    40%

    50%

    60%

    1996 2001 2006

    Poorest

    Poorer

    M iddle

    Richer

    Richest

    Total

    Source: DHS data. From NAGA pg. 27

  • 7/28/2019 1.1.1 Presentation

    14/50

    Determinants: locations

    District level Data - Nyanza Province

    Significant increases in PMTCT coverage: 24% to 73% between 2004-2009

    Largely on track to reach Universal Access goal of 80% However, reaching the last 20% will need refocusing strategies and other context

    specific analyses by province.

  • 7/28/2019 1.1.1 Presentation

    15/50

    Determinants: locations and Ethnicity

    0

    5

    10

    15

    20

    25

    30

    35

    40

    45

    RAT MON ST PVH OMC SR KRA KTH PAI PUR KCM KK KEP KCN BMC BTB SHV KSP KAM PV KAN SVR TAK PNH

    This increases to 45% (!!) for

    17 year olds in Ratanakiri who

    dont have Khmer as mother

    tongue

    In Cambodia, % of 17 years olds who NEVER attended school

  • 7/28/2019 1.1.1 Presentation

    16/50

    Determinants: Gender

    The majority of out-of-school children

    continue to be girls (53%); achieving

    gender parity would mean 3.6 millionmore girls in primary school

  • 7/28/2019 1.1.1 Presentation

    17/50

    Socio-economic: Disabilities

    90% of children with disabilities in

    developing countries do not attendschool

  • 7/28/2019 1.1.1 Presentation

    18/50

    Multiple Dimensions: Who is not learning? Disparity in

    learning

  • 7/28/2019 1.1.1 Presentation

    19/50

    Equity-based programming:whats the final aim?

  • 7/28/2019 1.1.1 Presentation

    20/50

    Final aim

    All rights for all children everywhere, by

    prioritizing the most deprived

  • 7/28/2019 1.1.1 Presentation

    21/50

    All rights for all children everywhere, by

    prioritizing the most deprivedGraph 1: % of children (7-14 years old) out of school, by race/ethnicity

    2000 Baseline and goals for national average, black and white children

    2010 Goal: reduce by 50% the national average an d the equity ratio between black and white children

    5.5

    2.8

    3.8

    2.2

    6.9

    3.1

    2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

    EquityRatio:1,8

    Equityratio:

    1,4

    Black children

    White children

    National average

  • 7/28/2019 1.1.1 Presentation

    22/50

    Public policies must be designed

    based on the inequity profiles

    (% of births attended by medically trained personnel by quintile)

  • 7/28/2019 1.1.1 Presentation

    23/50

    What is

    an equity-focusedevaluation?

  • 7/28/2019 1.1.1 Presentation

    24/50

    What is an equity-focused evaluation?

    An assessment made of the relevance,

    effectiveness, efficiency, impact and sustainability

    of interventions on equitable development results.

  • 7/28/2019 1.1.1 Presentation

    25/50

    Purposes of equity-focused evaluation

    Accountability

    Organizational learning and improvement

    Evidence-based policy advocacy

    Contribute to Knowledge Management

    Empowerment of worst-off groups

    National Capacity development for equity-focused M&E systems

  • 7/28/2019 1.1.1 Presentation

    26/50

    HOW to manage

    Equity-focusedevaluations?

  • 7/28/2019 1.1.1 Presentation

    27/50

    Key Steps

    1. Preparing

    for the EFE

    1

    2. Preparing

    the ToR of

    an EFE

    3. Designing

    theevaluation

    4.Collecting/

    analysing

    evidence

    432

    5. Findings,

    conclusion and

    reccs

    65

    6. Utilizing the

    evaluation

    MANAGE RISKS * MANAGE RISKS * MANAGE RISKS * MANAGE RISKS

  • 7/28/2019 1.1.1 Presentation

    28/50

    Key Steps

    1 432 65

    MANAGE RISKS * MANAGE RISKS * MANAGE RISKS * MANAGE RISKS

    1. Preparing

    for the EFE

    P i f

  • 7/28/2019 1.1.1 Presentation

    29/50

    . Preparing for

    the Equity-focused evaluation

    Determining the evaluability of the interventions

    equity dimensions

    Identifying evaluation stakeholders, including

    worst-off Identifying intended use by intended users

    Ensuring appropriate budget

    Identifying potential challenges in promoting andimplementing Equity-focused evaluations

  • 7/28/2019 1.1.1 Presentation

    30/50

    Key Steps

    1 432 65

    1. Preparing

    for the EFE

    2. Preparingthe ToR of

    an EFE

  • 7/28/2019 1.1.1 Presentation

    31/50

    Preparing the ToR of an EFE

    Defining the scope and purpose of the evaluation

    Framing evaluation questions focusing on equity

    Selecting technically-strong and culturally-sensitive

    evaluation team

  • 7/28/2019 1.1.1 Presentation

    32/50

    Key Steps

    1 432 65

    1. Preparing

    for the EFE

    2. Preparingthe ToR of

    an EFE

    3. Designing

    theevaluation

  • 7/28/2019 1.1.1 Presentation

    33/50

    Designing the evaluation

    Selecting the appropriate evaluation framework

    Selecting the appropriate evaluation design

    Selecting the appropriate evaluation methods

  • 7/28/2019 1.1.1 Presentation

    34/50

    Special challenges for evaluating

    equity-focused programmes Outcomes difficult to define and measure

    Conventional evaluation designs do not work

    Innovative evaluation designs are needed

    Understanding the complex contexts in which EFE areimplemented

    The importance of process

    Studying sensitive issues

    Vulnerability Political power relations

    Culture

    Wh i dibl id f

  • 7/28/2019 1.1.1 Presentation

    35/50

    What is credible evidence of

    Equity-focused outcomes?

    Experimental designs? Can numbers capture the complexity of equity-

    focused interventions?

    In-depth analysis of lived experience? Can narrative help explain the many factors

    affecting outcomes?

    Is it possible to generalize from qualitativestudies?

    Mixed methods?

  • 7/28/2019 1.1.1 Presentation

    36/50

    The challenge of attribution and

    assessing causality Causality and the definition of the

    counterfactual The difficulty of finding a comparison group

    The danger of positive bias

    The human desire to finding causality

    The danger of retrospective construction of

    causality

    Alternatives to the conventional

    counterfactual

  • 7/28/2019 1.1.1 Presentation

    37/50

    The importance of mixed method

    approaches Combining depth and breadth

    Combines narrative reporting of lived-experience and

    cultural context, with the ability to generalize

    Integrated approach draws on tools andtechniques from a broad range of social science

    disciplines

    Broadening the range of credible evidence

    Triangulation

    Selecting the appropriate evaluation methods:

  • 7/28/2019 1.1.1 Presentation

    38/50

    Selecting the appropriate evaluation methods:

    simple, complicated and complex programs

    Selecting the appropriate evaluation

  • 7/28/2019 1.1.1 Presentation

    39/50

    Selecting the appropriate evaluation

    framework

    Experimental/ quantitative approaches

    Theory-based evaluation

    The theory of change

    Systems-based approaches

    Expenditures and cost-effectiveness

    Cost-effectiveness

    Benefit incidence analysis Public expenditure tracking

    Bottleneck analysis as the integrating framework

    B ttl k l d d d f k f t

  • 7/28/2019 1.1.1 Presentation

    40/50

    Bottleneck supply and demand framework: factors

    affecting use of services by vulnerable populations

    Use of services by

    vulnerable population

    Supply side

    factors

    Coverage

    Budgets andavailable

    resources

    Culturally

    acceptable

    services

    Culturally

    sensitive staff

    Demand side factors

    Knowledge,

    Attitudes and Practice

    of vulnerable groups

    Community

    ownership

    factors affecting

    access

    distance

    cost of travel and

    fees

    available transport

    cultural constraints time poverty

    Contextual factors

    Economic Political

    Institutional

    Legal and

    administrative

    Environmental

  • 7/28/2019 1.1.1 Presentation

    41/50

    Knowledge, attitudes and practice

    [KAP studies]

    Knowledge about a program and about

    the problems it addresses

    Attitudes towards the program

    Behavioral changes

    A h f l ti l it f d

  • 7/28/2019 1.1.1 Presentation

    42/50

    Approaches for evaluating complex equity-focused

    policies and programmes

    Systems approach to evaluation Unpacking complex policies into components

    that can more easily be evaluated

    Pipeline designs

    Policy gap analysis

    Using other countries or sectors as the

    comparison group

    Concept mapping

    Portfolio analysis

  • 7/28/2019 1.1.1 Presentation

    43/50

    Approaches for evaluating simple

    projects and programs Experimental and quasi-experimental

    designs

    Creative ways to define comparison groups

    Naturalistic experiments

    Theory-based approaches

    Non-experimental designs

    Qualitative, narrative methods

  • 7/28/2019 1.1.1 Presentation

    44/50

    The potential value of

    non-experimental designs Single-case methods Longitudinal designs

    Panel studies

    Interrupted time series

    Case study designs using mixed-method

    sample selection

    Holistic analysis

  • 7/28/2019 1.1.1 Presentation

    45/50

    Key Steps

    1. Preparing

    for the EFE

    1

    2. Preparing

    the ToR ofan EFE

    3. Designing

    the

    evaluation

    4. Collecting/

    analysing

    evidence

    432 65

    Selecting the appropriate

  • 7/28/2019 1.1.1 Presentation

    46/50

    g pp p

    data collection methods. Examples from the bottleneck

    framework

    Use of services by

    vulnerable populationObservation

    Surveys

    Focus groups

    Citizen report cards

    Supply side

    factors

    Project

    documents

    ObservationKey informants

    Focus groups

    Benefit incidence

    analysis

    Public

    expenditure

    tracking

    Demand side factors KAP studies

    Case studies

    Observation

    Surveys

    Contextual factors

    Diagnostic studies

    Rapid surveys

    Secondary data

    Key informants

    Mix-Methods and

    triangulation

    Use for all stages of the

    evaluation

  • 7/28/2019 1.1.1 Presentation

    47/50

    Key Steps

    1. Preparing

    for the EFE

    1

    2. Preparing

    the ToR ofan EFE

    3. Designing

    the

    evaluation

    4.Collecting/

    analysing

    evidence

    432

    5. Findings,

    conclusion and

    recommendations

    65

    MANAGE RISKS * MANAGE RISKS * MANAGE RISKS * MANAGE RISKS

  • 7/28/2019 1.1.1 Presentation

    48/50

    Key Steps

    1. Preparing

    for the EFE

    1

    2. Preparing

    the ToR ofan EFE

    3. Designing

    the

    evaluation

    4.Collecting/

    analysing

    evidence

    432

    5. Findings,

    conclusion and

    reccs

    65

    6. Utilizing theevaluation

  • 7/28/2019 1.1.1 Presentation

    49/50

    Utilizing the evaluation

    The problem of under-utilization

    Not addressing clients priority concerns

    Poor timing

    Not using stakeholders preferred

    communication styles

    Too many tables/ not enough tables

    Too much narrative/ not enough narrative Not available in local languages

    Only selected stakeholders receive reports

  • 7/28/2019 1.1.1 Presentation

    50/50

    Utilizing the evaluation

    The importance of utilization focused evaluation

    designs

    Preparing the evaluation report and alternative

    forms of reporting Agreeing on an action plan

    Disseminating the evaluation and preparing a

    Management Response