olivier ecker & jef leroy • 2016 ifpri egypt seminar series: what is the role of agriculture...

Improving Nutrition in Egypt

What is the Role of Agriculture and How do we Document Impact?

Jef Leroy & Olivier EckerIFPRI, Washington, DC

IFPRI Egypt Seminar SeriesCairo, 16 Nov 2016

Overview

• Part 1: Nutrition-sensitive agriculture (Ecker)• Part 2: Key challenges and solutions in

evaluating the impact of nutrition-sensitive agricultural programs (Leroy)

Agriculture-nutrition linkages: Theory and evidence

Key message: Agricultural interventions can improve nutrition, but they do not automatically lead to improved nutrition.

Agricultural programs and projects can improve diets and nutritional outcomes of farmers and their families through many different pathways:1. Increasing farm incomes ( productivity gain)2. Improving food consumption from own production ( food

quantity and diet diversity)3. Employment and control over household resources ( e.g.

women empowerment)

Agriculture-nutrition linkages: Theory and evidence

• Growing evidence—from rigorous impact evaluations—suggest that agricultural interventions can improve (child) nutrition, if they …1. Have specific nutrition goals and actions, and2. Focus on women.

• Many factors influence individual nutritional status (incl. diets, health, care). Agriculture interventions are part of the solution, but more is needed. – E.g., in addition to increasing incomes and improving food

availability and access, behavioral change to promote adequate nutrition and health practices is needed.

Pathways linking agriculture to nutrition

Adapted from: Gillespie et al.. 2012. The Agriculture-Nutrition Disconnect in India: What Do We Know? IFPRI Discussion Paper 1187. IFPRI, Washington DC.

Agriculture-nutrition linkages: Knowledge gaps and research needs

• Evidence on the impact of agricultural interventions on nutritional outcomes is still scarce.

• Existing evidence is largely based on small-scale programs in fairly controlled environments (e.g. home gardens).

• Evidence from large-scale, complex agricultural interventions—especially for promotion of agricultural commercialization (“agribusiness”)—is missing so far.

Agriculture and nutrition

In summary (Lancet, 2013):• Agriculture has enormous potential to improve nutrition, but

this potential is yet to be unleashed.• Lack of critical evidence of impact of agriculture on nutrition is

likely due to: – Weaknesses in program design, targeting, implementation,

and lack of nutrition goals and interventions, or– Poor evaluations – design, methods, indicators.

Overview

• Part 1: Nutrition-sensitive agriculture (Ecker)• Part 2: Key challenges and solutions in

evaluating the impact of nutrition-sensitive agricultural programs (Leroy)

Part 2: Key challenges and solutions

1. Challenges in evaluating the impact of nutrition-sensitive agricultural programs

2. Rigorous evaluation approach to evaluate:a) Impactb) Impact pathways, and c) Cost

3. Recommendations on how to address key challenges of carrying out sound evaluations implemented under real life conditions.

What are some key challenges?

• Complexity of nutrition-sensitive agricultural programs

• Long impact pathways and time frames• Trade-off between implementation constraints

and evaluation rigor

Program complexity

Complexity nutrition-sensitive agricultural programs

• Complex in design and implementation:– Multiple goals, multiple inputs, multiple pathways of impact,

multiple outcomes and impacts;– Address both the underlying and direct causes of

undernutrition;– Span across different sectors (e.g. health, agriculture,

education), requiring coordination and integration. • Within each program intervention potential variability in:

– Delivery (quantity and quality);– Utilization;– Adherence to the program protocol.


• Complexity of agricultural programs aimed at improving nutrition



Long impact pathways and time frames

• Long time frames for implementation and evaluation:– Program development and implementation at desired quality – Long pathways from program inputs to effects:

E.g.: installing garden beds, preparing the soil, sowing, planting and harvesting; setting up and implementing the BCC strategy, improving maternal knowledge through repeated BCC sessions, and achieving changes in practices;

– Meaningful effect on biological outcomes such as child anthropometry may require as long as 1,000 days of program exposure.

– Time to design a rigorous impact evaluation:Building the program theory framework, developing the evaluation and sampling design, designing and pre-testing the data collection instruments, training and standardizing enumerators, planning the logistics of the field work, enroll the necessary number of study participants, and seasonality.

• … vs often short time frames imposed by donors


• Complexity of agricultural programs aimed at improving nutrition



Implementation constraints vs. evaluation rigor

Treatment 1Treatment 2Treatment 3ControlRegular program

Cluster randomized controlled design

T24T18TNFPControlRegular program

Part 2: Comprehensive evaluation approach

PROGRAMMaternal

& child health & nutrition

What is the impact of the program?

Impact evaluation

Part 2: Comprehensive evaluation approach: embrace complexity

What is the impact of

the program?

Process + impact evaluation

How and why does the

program (not) have an impact?Cost

study

Cost of the

program?

What is the impact of the program?

• impact=(N|with program) - (N|without program)

• Problem: (N|with program) and (N|without program) never both “observable”

• The key challenge to impact evaluation: what would have happened in the absence of the program = counterfactual

Creating a valid counterfactual

INTERVENTION CONTROL

Experimental designs

• Experimental (or randomized) designs considered gold standard for impact evaluations.

• Randomization: individual or group (cluster) level. • *If* done well, one can assume:

– that both groups are comparable;– that the only difference between the groups is the program;– that the control group provides a valid counterfactual for

the intervention group exposed to the program. differences found in the outcomes of interest attributable to the program.

Experimental designs

• Not always feasible• Often require creative thinking (oversubscription,

public lottery, delayed intervention, stepped wedge design, etc.)

• Alternatively: quasi-experimental designs – use statistical techniques to create a valid comparison

group – E.g.: propensity score matching (PSM), double difference

(or difference-in-difference) approach, regression discontinuity, instrumental variable regressions, etc.

• Stay away from before-and-after or with-and-without designs!

Invalid counterfactuals

68.2

74.8

64.1 64.3

2010 2014

68.2

74.8

64.763.0

2010 2014

68.2

74.8

62.364.1

2010 2014

68.2

74.8

64.6 65.7

2010 2014

-5.5-7.1

-5.2-4.0

-10.0

-5.0

0.0

60.0

65.0

70.0

75.0

80.0

2010-2014 2010-2014 2010-2014 2010-2014

All T24 T18 TNFP

Control Treatment Impact (95% CI)

Impa

ct o

n st

untin

g (p

p)

U

nadj

uste

d st

untin

g (%

)

Wave

68.2

74.8

64.1 64.3

2010 2014

68.2

74.8

64.763.0

2010 2014

68.2

74.8

62.364.1

2010 2014

68.2

74.8

64.6 65.7

2010 2014

-5.5-7.1

-5.2-4.0

-10.0

-5.0

0.0

60.0

65.0

70.0

75.0

80.0

2010-2014 2010-2014 2010-2014 2010-2014

All T24 T18 TNFP


Impa

ct o

n st

untin

g (p

p)

U

nadj

uste

d st

untin

g (%

)

Wave

Stun

ting

(%)

Impa

ct (p

p) 0.0

-5.0

Randomized design

-5.5 pp

68.2

74.8

64.1 64.3

2010 2014

68.2

74.8

64.763.0

2010 2014

68.2

74.8

62.364.1

2010 2014

68.2

74.8

64.6 65.7

2010 2014

-5.5-7.1

-5.2-4.0

-10.0

-5.0

0.0

60.0

65.0

70.0

75.0

80.0

2010-2014 2010-2014 2010-2014 2010-2014

All T24 T18 TNFP


Impa

ct o

n st

untin

g (p

p)

U

nadj

uste

d st

untin

g (%

)

Wave

Before-and-after

+ 0.2 pp

68.2

74.8

64.1 64.3

2010 2014

68.2

74.8

64.763.0

2010 2014

68.2

74.8

62.364.1

2010 2014

68.2

74.8

64.6 65.7

2010 2014

-5.5-7.1

-5.2-4.0

-10.0

-5.0

0.0

60.0

65.0

70.0

75.0

80.0

2010-2014 2010-2014 2010-2014 2010-2014

All T24 T18 TNFP


Impa

ct o

n st

untin

g (p

p)

U

nadj

uste

d st

untin

g (%

)

Wave

With-and-without

- 10.5 pp

Impact evaluation



the program?




study

Cost of the

program?

How and why does the program (not) have an impact?

• Two ways to assess how impact is achieved :1. Measurement of intermediary measures (outcomes) in impact study2. Process evaluation study

• Solid understanding of program theory and program impact pathways is key:– Key program components, – Factors that affect optimal delivery or utilization of each component,– Assumptions associated with each of the components, and – How the components are expected to be linked (pathways of impact)

• Understanding the pathways to impact:– Critical to improving program delivery and effectiveness – Identify what is needed to scale up and to adapt the program in other

settings– Help interpret (lack of) impact

Designing the process evaluation

• Examines the primary inputs, processes, outputs and outcomes along each of the primary program impact pathways

• Addresses five key questions: – Are program services being implemented and provided according to

the program design (inputs and processes)?– Are program services being utilized as intended (outputs)?– What is the quality of the program inputs and services (inputs,

processes and outputs)?– What are the barriers and facilitators to optimal service delivery and

utilization (inputs and processes)?– Is the program on track to have the desired effect on improving

intermediary outputs and outcomes, such as improvements in knowledge?

Impact evaluation



the program?




study

Cost of the

program?

What is the cost of the program?

• Objectives: – Estimate the overall cost of the program, the cost of the main program

components, and the program’s cost-effectiveness. – Estimate savings or cost associated with adding, changing or dropping

program components, adding beneficiaries or scaling up the program. • Method : Activity Based Costing Ingredients (ABC-I) approach.• Steps:

– Using the program impact pathways: detailed description of all program activities.

– Identify the program’s main activities and “ingredients” needed for each.– Identify the different types, quantities and costs of the “ingredients”

necessary for each activity. – Calculate cost of each program activity and of full program.

Two keys to success

• Solid evaluation framework:In-depth understanding of the program, program theory, program impact pathways, rigorous comprehensive evaluation design

• Strong partnership and collaboration between implementers and evaluators

Strong partnership between implementer and evaluator

• Establish collaboration at the program design phase and maintain throughout.

• Align potentially differing priorities, expectations, incentives and time frames.

• Program implementers:Share updates and challenges on program roll out and service delivery

• Evaluators:Provide regular updates on goals, methods and findings from their evaluation activities.

In summary

• Evidence of what works, how and at what cost is extremely limited. • Guidance for future investments requires strong evidence from

rigorous, theory-based comprehensive evaluations of different nutrition-sensitive program models that bring together interventions from a variety of sectors (e.g. health, education, agriculture, social protection, women’s empowerment, water and sanitation, etc.)

• This presentation focused on:– How to design and carry out rigorous impact, process, and cost

evaluations.– How to address some of the perceived insurmountable challenges that

have prevented investments in rigorous evaluations of such programs in the past.

Exciting recent and ongoing studies

• Completed: – Homestead food production:

• Burkina Faso• Improved women’s and child’s health, nutrition• Improved women’s empowerment

– HarvestPlus biofortified orange-fleshed sweet potato (OSP):

• OSP vines were given to 24,000 households in Uganda and Mozambique from 2006 to 2009

• project included agriculture component and nutrition behavior change

• Project increased vitamin A intake of young children

Exciting recent and ongoing studies

• Ongoing:– Improve chicken value chains (Burkina Faso)– Improve quality of milk sold by informal vendors

(Nairobi, Kenya)– Reduce aflatoxin in maize (Kenya)– Improve milk contracts for nomadic farmers

(Senegal)– USAID-funded, large-scale agribusiness program in

horticulture and a basic health care program (Upper Egypt)

olivier ecker & jef leroy • 2016 ifpri egypt seminar series: what is the role of agriculture...

Economy & Finance