1

Results from the impact evaluation of RTIMP in Ghana

Participatory Impact Assessment & Learning Approach (PIALA)

Adinda Van HemelrijckIFAD 26 October 2015 Rome

2

IMI Initiative

$ 90 K(+100K)

$ 230 K(+60 K)

DBRPRTIMP

PIALA Research strategy

DBRP EvaluationVietnam PIALA reflectionsVietnam PIALA debriefs

RTIMP EvaluationGhana PIALA reflections

PIALA learning event

PIALA @ Conferences (AEA‘13, EES’14, UKES’15,

EAS’15, IDEAS’15)

PIALA Practice Paper

PIALA guide for commissioners

3

Project Objectives

Assessing to what extent

impacts occurred (or not)

Debating

how impacts can be

enhanced

Explaining why

impacts occurred (or not)

1. Produce rigorous qualitative and quantitative evidence for global reporting and advocacy

2. Facilitate inclusive analysis and reflection for

collaborative learning

3. Generate a scalable model for strengthening IFAD’s self-evaluation system

PIALA Purposes

4

Design Challenges

• Causal inference in the absence of a clean or credible comparison group• Program effects spill over• Influence of other R&T livelihoods programs• Heterogeneous pattern of program treatment

• Generate solid evidence-based debate about “what has worked for whom, under which conditions and why?”

• Look at systemic impact broader than the program performance to enhance collaborative learning around emerging issues that undermine program theory

5

PIALA features & standards

Focus & frame the evaluation

Collect & link dataManage quality

Synthesise evidenceAnalyse & debate contributions

Rigour

Inclusiveness Feasibility

Systemic ToC approach

Concurrent participatory mixed-methods

Cluster sampling around market-bounded systems

2-stage participatory sensemaking

Configurational analysis

Access to credit for poor HHs

Access to training & extension for poor

HHs

Access to markets

Jobs & livelihood opportunities

Poor people’s voice

District capacity to provide training &

services

Commune capacity in SEDP & poverty reduction planning and management

Provincial capacity in facilitation of local businesses, PPPs and

entrepreneurship

Wealth & wellbeingInstitutional

relationships

Framing the evaluation in Vietnam

MEF

GPCDSF

FFF

Framing the evaluation in Ghana

8

Changes & causes of access to food & income

R&T livelihood changes and causes

Reach and effects of program mechanisms

(DSF, FFF, GPC, MEF)Household Survey (n 840)

on changes in food, assets, income, R&T revenues

Generic Change Analysis (n 439)Ranking & causal flow mapping

of changes in wealth & wellbeing

Livelihood Change Analysis (n 400) Livelihood matrix & causal flow mapping

SenseMaker Lithe (n 393)

Constituent Feedback (n 341)with DSF, FFF & GPC/MEF beneficiaries

Key Informant Interviews (n 100)with officials & service providers

Participatory Sensemaking Workshopswith research participants in 23 districts (n 640) and national (n 106)

Review of secondary data(Ghana Living Standard Survey 2010, RTIMP cost-benefit studies 2014, SPN & MTR 2010, 12, 13 & 14)

Collecting and linking data

PartiFGDs

9

3 Zones - 8 regions 4 commodity chains

(HQCF, PCF, Gari & FYE)

25 random districts

30 random supply chains

840 random HHs

1180 FGD participants (45% women)

152 Parti FGDs(109 gender-specific)

840 HH Surveys

(24% female-headed)

Parti Sensemaking WSs (in 23 districts with 640 ppts;

national with 106 ppts)

100 KIIs (75 districts + 25 national)

10

Configurational causal analysis

1. Per district: – binary coding (0-1)

of program mechanisms – Scoring (0-6)

of causal links and evidence

2. Aggregated: – Unzipping evidence along the ToC:

cluster and compare districts with different configurations of scores & explanations for each causal claim

– zipping up findings along the ToC: draw conclusions about program contributions to impact

Analysing & debating contributions

Strength of causal link

(necessity)

Consistency of causal link

(sufficiency)

1

2

34

5

6

0

11

Scoring of causal links and evidence

DSF: District Stakeholder Forum; FFF: Farmer Field Forum; GPC: Good Practice Centre; MEF: Micro-Enterprise Fund

12

Participatory sensemakingInvolving stakeholders in collective analysis & debate of evidence of impact and areas for future investment (640 pps in districts 23 WSs, 106 in nat WS) • Cross-validate the evidence and the ToC• Obtain extra layer of data• Enable voice (30 % beneficiaries)• Debate “less success” needing “more effort”• Rate program contributions

13

RTIMP Evaluation

Findings

14

Roots & Tubers Improvement andMarketing Program (2007-2014)

PCR 2015:RTIMP operated in 106 districts (out of 216), spent USD 23.6 m (incl. USD 18.83 m IFAD) and reached 15,000 - 20,000 farmers & processors with training, finance and market-linking

Program goal: Enhance household access to food & income through improving R&T livelihoods and strengthen market-based systems generating profitability along R&T commodity chains

Intervention components: 1. Market-linking of supply chains, incl. providing market-oriented information and

training, supporting innovation, and building networks for supply, investment and market linking through District Stakeholder Forums (DSF)

2. Enhance smallholders production, incl. multiplication and distribution of improved R&T planting materials and technologies through Farmer Field Forums (FFFs)

3. Grow smallholder root and tuber processing businesses, incl. upgrading of MSEs to Good Practices Centres (GPC) to serve as demonstration and market hubs, and creating access to finance through Micro-Enterprise Fund (MEF)

15

Livelihoods and poverty status will improve, if smallholders commercialise and become part of strong and inclusive commodity chains

• FFFs will help small R&T farmers commercialise by organising into FBOs and demonstrating improved planting materials and technologies

• R&T smallholders will commercialise and supply the chains, if they have knowledge and capacity to increase production, access markets and develop businesses

• DSFs can help develop inclusive and sustainable R&T commodity chains by linking supply chains to markets

• GPCs will help small R&T processors commercialise by demonstrating improved technologies & equipment, hence helping them access finance through the MEF

• Well-trained R&T smallholders will be able to obtain a loan through the MEF to invest in their businesses and adopt improved equipment and technologies

Assumptions

16

Evaluation questions

1. To what extent did these assumptions hold true (or not) and under which conditions?

2. What were the major barriers for farmers and processors (particularly women & young adults) to commercialize and access old and new markets?

17

• 83 % HHs reported NO food shortage in 2014 (51 % in 2013) • 2009 WFP: 95 % population had access to sufficient food• 2008 RTIMP baseline: av. 85 % HHs living from R&T could feed themselves

• 15 % HH moved up to income categories > USD 2/day due to R&T livelihood improvements

• Increase in total HH income, and HH income from R&T, appear to have reached a ceiling at USD 4-5 /day, indicating improved but limited profitability

Increased access to food & income

18

(sig.ooo)

• Statistical analysis of 837 HH surveys showed a more direct correlation of HH income from R&T with total HH income than with HH access to food

• SenseMaker analysis of 373 micro-narratives from FGDs showed 94 % participants experienced R&T livelihood changes affecting income and education, rather than food

• R&T livelihood changes affected access to food through creating access to income rather than directly

190-500 501-1000 1001-2000 2001-5000 >5001

0

5

10

15

20

25

30

Change in distribution of the range of HH income from R&T in 2009-2014

Change in distribution of the range of total HH income in 2009-2014

0-500 501-1000 1001-2000 2001-5000 > 50010

5

10

15

20

25

30

20

R&T livelihood improvements • R&T livelihood improvements were found relatively strong but inconsistent in

52 % and weak in 48 % supply chains• SenseMaker analysis (n 378) showed that 33 % FGD participants attributed R&T

livelihood changes to RTIMP rather then to other or own efforts• Configurational analysis found livelihood changes attributable to RTIMP in 32 % districts

(strong in 12 % ; weak in 20 %), of which 88 % showing weak market linking

• Very weak/no livelihood improvements were found in districts where RTIMP mechanisms were dysfunctional or not in place

• R&T livelihood improvements occurred country-wide due to new R&T varieties and technologies, causing an influx into R&T farming spilling into processing

• Inadequate business and market linkages + economic downturn turned the tide and caused prices to drop, negatively affecting R&T livelihoods from 2013

• Statistical analysis (n 837) showed overall limited R&T profits and investments, while Configurational analysis found market failure as the main cause

21

Market-linking of R&T supply chains

• Market linking of supply chains through DSFs was found weak in 84 % districts – In 57 % districts, DSFs somewhat contributed to

strengthening supply chains, but failed linking them to markets

– In 43 % districts, DSFs’ contribution to developing supply chains was virtually nil, with no market linking efforts

• In 16 % districts, market linking was found stronger due to stronger DSF & GPC performance, but insufficient to withstand external threats and prevent market saturation

22

R&T supply chain development & smallholder commercialization

• R&T has changed from food to cash crop for the resource-poor, but market saturation has offset initial gains from enhanced production and rendered R&T livelihoods fragile

• Commercialisation of smallholders has remained limited and proven unsustainable in 88 % districts due to limited market opportunities, caused by– Weak market linking combined with overproduction – Poor infrastructure and inappropriate competition regulations, rendering

smallholders more vulnerable to unfair competition (particularly in districts with low/no RTIMP presence)

• Stronger but inconsistent commercialisation of smallholders was found in 12 % districts due to stronger GPC performance

23

Enhanced smallholder R&T production• Enhanced smallholder R&T production and productivity was found strong

in 76 % districts due to adoption of new planting materials & technologies(of which 73 % attributed to RTIMP and 27 % to RTIMP + WAAP/others)

• Weak results in 24 % supply chains due to limited adoption + other factors (incl. yam beetle, weather patterns, land tenure issues, limited extension, input prices)

• 36 % districts had FBOs, but none stemmed from FFFs or were influenced by RTIMP, and in only 2 cases were farmers able to access business finance (Nanumba North and Tano North) – SPN Nov 2014: FBOs successfully developed out of FFFs in 4 locations, but none had yet

been able to obtain credit or bargain better market prices

• At present there is no evidence of whether FBOs could be sufficient for helping farmers bargain better prices, fight unfair competition, obtain business finance, access markets and commercialize – All evidence points to the need for more market opportunities in the first place, requiring

better roads, policies and regulations more supportive of smallholder businessess

24

Access to new R&T planting materials & technologies

• Access to and adoption of new planting materials & technologies was found strong and consistent in 72 % districts, due to the success of FFFs and the visible efficiency and benefits of adoption– PCR 2015: 187,275 farmers received improved R&T planting materials, and

15,154 farmers were trained through 451 FFF

• Women participated and benefitted less since FFFs mostly targeted and reached male farmers (between 40-60 owning 1-2 ha land)– Women generally are more involved in cassava production than men and

traditionally do most of the work – Since R&T changed from food to cash crop, men took a greater interest and FFFs

have encouraged and supported this

• 74 % FFF participants reported they were able to apply what they learned, which helped them expand their businesses

• Young farmers (< 25) and women were less positive about FFF benefits and less confident to express their needs and ask for help

25

Enhanced R&T processing• Enhanced processing as a result of more people growing and processing high

quality cassava was found strong and mostly inconsistent in 50 % cassava supply chains, and weak in the other 50 %.– In 17 % supply chains, this was found strong and consistent due to stronger GPC

performance in terms of market creation, reach, strength and inclusiveness of supply chains, and adoption of improved technologies and equipment

– In 33 % supply chains, this was found strong but inconsistent due to the limited reach of GPC operations and spill-over of excess R&T production to processing using both new and traditional equipment

– In 50 % supply chains, this was found weak due to dysfunctional GPCs (more than half) or a very limited reach and adoption of improved technologies and equipment (nearly nil in more than half of the cases)

• New technologies and equipment have proven cost-efficient and potentially profitable. Yet adoption was limited in 83 % supply chains due to:– limited reach and effectiveness of GPC’s as learning and good practice centres,

and – limited investment capital of GPCs and individual processors (mostly women)

26

• Over half of GPC beneficiaries reported they were able to apply what they learned at the GPC

• Nearly one third found GPCs helped expand their businesses

• Women were more positive than men, although less confident to express their needs and ask for help at GPCs (as were young people < 25 felt less confident)

• Only 9 % GPC leaders were positive about GPC influence on beneficiaries’ businesses and their ability to apply what they learn

Fig 10.1.3.

27

Access to business finance • HH survey and FGDs showed limited access to financial support

to invest in existing or new livelihood activities– Only 15 % HHs obtained finance in 2009-2014 to invest in R&T (of which 55 %

from relatives and friends, 22 % from Susu or local moneylenders, 14 % from RCBs, 4 % from gov programs and 5 % from other sources)

– MEF funding was found “obtained” by processors in 2 districts (Techiman & Mangong) and GPCs in 2 other districts (Assin South & Abura Assebu)

• MEF not available and accessible to most smallholders & GPCs – MEF was found formally unavailable in 50 % districts (RTIMP 2014: only in few)

– Procedure for obtaining and paying off MEF funding was too onerous, making smallholders pre-invest and sustain operations with insufficient capital or immediate returns on investment (RTIMP 2014)

– Reluctance of PFIs to approve applications due to investment risks in present conjuncture (RTIMP 2014)

28

Immerging issuesAnswers to the

evaluation questions

29

Assumptions true (or not)?• Livelihoods & poverty status will improve,

if smallholders commercialise and become part of strong and inclusive commodity chains

PARTLY TRUE• Stronger GPC and DSF performance in 12 % distr• Concerted efforts by partners to develop solid supply

links, build capacities & relations in 12 % distr• Limited R&T profits & investments due to market failure

• R&T smallholders will commercialise, if they have knowledge and capacity

• DSFs can help develop inclusive and sustainable R&T commodity chains

UNTRUE• Weak DSF performance in 84 % distr• Limited reach and capacity of DSFs and GPCs to expand,

innovate and develop markets• Limited market opportunities & unfair competition

• FFFs will help small farmers commercialise by organising into FBOs and demonstrating improved planting materials & technologies

PARTLY TRUE• Large adoption of plant materials & techs in 84 % distr• Commercialization limited by lack of market

opportunities and unfair competition in 88 % distr• Farmer organisation insufficient to overcome these

• GPCs will help small R&T processors commercialise by demonstrating improved technologies & equipment

• Well-trained R&T smallholders will obtain finance through the MEF to invest in R&T

PARTLY UNTRUE• Limited adoption of new techs & equipm in 83 % distr

due to (a) limited GPC reach & effectiveness, and (b) smallholders’ limited investment capital

• MEF largely unavailable and inaccessible in >83 % distr

30

]

Validating the ToC Sufficient HH access

to sufficient food & income

ImprovedR&T

livelihoods

Enhanced market-

linking of smallholders

Strong R&T supply chain linking and smallholder commercialisation

through DSFs + support services

Enhanced smallholder production

Access to improved R&T planting materials & technologies

through FFFs

Enhancedsmallholder processing

Access to improved R&T processing equipment & technologies

and business finance through GPCs and MEF

✗P+T

✔P+T

T✔✗P

31

• Lack of market opportunities and unfair competition due to a failing rural infrastructure and inadequate policy and regulations for smallholder business development

Barriers to smallholder commercialization and market linking?

• Lack of investment capital (particularly for women and young adults <25) due the lack of access to finance caused by (a) investment risks and (b) limited capacity

• Limited capacity of DSFs and GPCs to develop strong supply chains, link these to existing and new markets, and expand their catchments/reach

• Lack of capacity of small farmers and processors to organise into sustainable agri-businesses that are able to create market value

32

Recommendations

• Rethink the DSF mechanism as a forum for inclusive VC linking

• Clarify and monitor the role of GPCs as supply chain leaders

• Develop women- and youth-specific FFFs that are more market/business-oriented

• Develop and pilot an appropriate mechanisms for VC funding

• Advocate for investments, policies & regulations supportive of smallholder commercialization

33

Some final reflections

34

PIALA benefits, costs & requirements• Rigorous evidence of impact of

programs with no/limited counterfactuals and baselines

• ToC of multiple interventions engaging multiple stakeholders in evidence-based reflections on collective impact

• Empowerment-related benefits from engaging beneficiaries and other constituents in data collection and collective sensemaking

• Inclusive analysis and evidence-based debate of emerging issues related to impact and sustainability

• Cost USD 100 - 300 KGhana (full scope – full scale)• 18 K for design, training & ToC• 146 K for method & fieldwork• 70 K for analysis & reporting

(incl. loc & nat sensemaking)

• Committed research team with sufficient capacity and experience in mixed methods impact evaluation

• Evaluation manager (consultant) to support and supervise procurement, design, fieldwork and analysis

Capacity trumps all!

RTIMP Evaluation commissioners: GoG/RTIMP & IFAD Country OfficeRTIMP evaluation team: Participatory Development Associates (PDA)PIALA management team: Edward Heinemann (IFAD)

Adinda Van Hemelrijk (IFAD)

Richard Caldwell (BMGF)

PIALA design support group: IFAD PMD, SKM & IOE; BMGFPIALA core design team: Adinda Van Hemelrijck (team leader)

Irene Guijt, Andre Proctor, Jeremy Holland

PIALA external reference group: Robert Chambers (IDS, UK)Marie Gaarder

(World Bank IEG) Kent Glenzer

(MIIS, USA)