mixed methods in impact evaluation may 2011

www.3ieimpact.orgHoward White

Mixed Methods in Impact Evaluation

Howard White

International Initiative for Impact Evaluation


Measurement is not evaluation


Why did the Bangladesh Integrated Nutrition Program (BINP) fail?

Why did the Bangladesh Integrated Nutrition Project (BINP) fail?


Comparison of impact estimates


Summary of theory

Target group participate in program (mothers of young children)

Target group for nutritional counselling is the relevant one

Exposure to nutritional counselling results in knowledge acquisition and behaviour change

Behaviour change sufficient to change child nutrition

Improved nutritional outcomes

Children are

correctly identified to be enrolled in the program

Food is delivered to those enrolled

Supplementary feeding is supplemental, i.e. no leakage or substitution

Food is of sufficient

quantity and quality


The theory of change






Children are






Right target group for nutritional counselling








Children are






Knowledge acquired and used








Children are






The right children are enrolled in the programme








Children are






Supplementary feeding is supplementary


Participation rates

0 .0

0 .1

0 .2

0 .3

0 .4

0 .5

0 .6

0 .7

0 .8

0 .9

1 .0

B a se va lu e L iv in g w ith m o th e r-in -la w inR a jn a ga r o r S h a h ra s ti

H igh e r e d u ca tio n N o w a te r o r sa n ita tio n(re m o te lo ca tio n )

Pro

ba

bil

ity

of

pa

rtic

ipa

tio

n i

n g

row

th m

on

ito

rin

ng L iv ing in

R a jnaga r o r S hah ras ti

L iv ing w ith m o the r-in -law

in R o r S


Data used in BINP study

• Project evaluation data (three rounds)

• Save the Children evaluation

• Helen Keller Nutritional Surveillance Survey

• DHS (one round)

• Project reports

• Anthropological studies of village life

• Action research (focus groups, CNP survey)


Examples of ‘atheoretical’ IEs

• School capitation grant studies that don’t ask how the money was used

• BCC intervention studies that don’t ask if behaviour has changed (indeed, almost any study that does not capture behavior change)

• Microfinance studies that don’t look at use of funds and cash flow

• Studies of capacity development that don’t ask if knowledge acquired and used


Different parts of causal chainrequire different analysis

• Factual versus counterfactual

• Examples of factual– Use of funds– Targeting– Participatory processes

• Quantitative and qualitative and the combination of the two


Examples from AP SHGs

Number of SHGs and % penetration

Drop outs & corrupt practices

The angry man

Returns to cows and goats: quantitative ethnography


Why the angry man was angry

Loan allocation is to households not individuals


More examples

• The disconnected in connected villages pretty much everywhere

• The role of the community in social funds in Malawi and Zambia


Poorest don’t connect


People participate in making bricks, not decisions


I have to go now. I have a community in my office


Also mix methods for identification strategy


AP village fund allocation

Fixed funds per community: more households per SHG

Lower membership rates in larger villages


The puzzle of the disconnected households


You can’t carry

electricity on boats


Mixing methods

• Understanding context to– Shape evaluation questions– Design data collection

• Mapping out theory(ies) of change

• Addressing factual questions, leading to…

• … interpretation of counterfactual findings

One example on data collection


General principle: the quality of data deteriorates the more formal the process of data collection


What do questionnaires miss?

• Consumption– Festivals– Labour exchange– Wildfoods

• Net income from household enterprises

• Abuse of nearly all kinds

• Who is a household member?


Protein in Northern Zambia


Who is in the household

roster?

This is my sister, Hana.

Her mummy is my mummy’s

sister


What is to be done?

•Know what questions to ask and how

•Proxy measures

•Enumerator training

•Contrived informality


The challenge of integration

• Parallel studies not integrated studies (multi-disciplinary not inter-disciplinary)

• Why?– At best silo mentality, at worst arrogance (“trust me, I’m an

economist”)– Academic incentives– People just don’t know how to do it

• What to do?– Start with theory of change– Team members who bridge studies– Detailed team discussions around causal chain– Quality of external peer review


Thank you

Visit www.3ieimpact.org

mixed methods in impact evaluation may 2011

Technology