Carlo Azzarri, Beliyou Haile, Cleophelia Roberts, Melanie BacouIFPRI/HarvestChoice

London, July 17, 2014

M&E of Africa RISING

Africa RISING M&E Objectives and Challenges ARBES Project Mapping and Monitoring Tool (PMMT)

− Purpose and scope− Major components

Next Steps

Agenda

2

Africa RISING M&E Objectives and Challenges

3

Support effective project management

Fulfill FtF reporting requirements Generate knowledge on what

works and what doesn’t work Data/analytical components of

the M&E system:− Delineation and characterization

of target farming systems− Inventory of technologies− Ex-ante evaluation* − Attribution assessment*

M&E of Africa RISINGLearning about climbing beans in Linthipe, Dedza 2014.

5

Mega-site stratification by drivers of SI and creation of “development domains”

Identification of representative action research and control sites

Open access M&E data management platform (PMMT)

FtF indicators reporting Household and community

surveys (ARBES) Site visits and focus groups Meetings (survey design, project

review & planning, annual M&E)

M&E ActivitiesClosely watching the fava beans, Sinana, Oct. 2013.

Compiling and verifying data on FtF indicators (suitable level of detail?)

Compiling and verifying data on AR beneficiaries

Sequencing: selection of action sites before setting up specific research activities

Engagement and communication (e.g. lengthy recruitment of local M&E coordinators)

Delays in baseline surveys Sustainability and its

dimensions: indicators and benchmarks?

A few M&E Challenges…

6

Evaluation questions:• Did the program contribute to program goal and objectives? • Did it increase agricultural productivity?• Are the program’s approaches replicable/scalable?• Did farmers adopt new/improved technology(s) and practices?• What are the tradeoffs among different impacts?• How does impact vary across different domains, geographies,

household types, and gender?• Are the program-induced improvements sustainable?• Did the program demonstrate implementation methods that other

projects can use?

Impact Evaluation (IE)

7

Practicality of / approaches to attribution of cause and effect for an AR4D program

(Random) selection of action sites and/or subjects

Conflicting incentives among actors

Attribution with small N (action sites, beneficiary households) – generalizability of results

Impact of a “bundle of interventions”

Confounding factors Moving targets

A few E Challenges…

8

Africa RISING Baseline Evaluation Surveys (ARBES)

9

Malawi 2013; Tanzania, Ghana, Mali, and Ethiopia 2014 ARBES. In 2013 we started an ambitious workplan on conducting household and community surveys in five of the six AR countries in less than a year time (mega-site representative; N=700-1300; 50-page Qx).

AR Evaluation. We would like to assess the overall impact of the program on different outcomes (poverty, nutrition, health,…) using this approach.

Babati (Tanzania) impact evaluation. The team will continue and refine the impact assessment research project in Babati. Comments welcome!

Data and Analysis

Feb 2013

July 2013

Aug 2013 Feb 2014 Feb

2016

Initial planting at demonstration

plots

Follow-up field day: farmers rank preferred seeds

Randomization 996 farmers in 25 villages

205 receive fertilizers and

improved seeds

210 receive AR intervention

End-line survey: measure impacts

105 farmers non-beneficiaries

elsewhere

270 controls

206 Babati farmers non-beneficiaries

Tanzania IE timeline and design

Project Mapping and Monitoring Tool

12

Scope and PurposeMajor Components

http://dev.harvestchoice.org/africarising/

Need for a simple, flexible and engaging tool to report on program performance and capture decisions/adjustments made over 5-year program lifecycle:

− Baseline site characteristics− Site stratification and selection− Intervention details− Bi-annual FtF (and other project-

specific) indicators− Surveys and evaluation results− Links to intermediary and final output

(incl. primary data)13

Evaluation is hard.Monitoring is painful…

Aligning project goals, 2012.

14

MSExcel revisions no more! Grow organically based on

evolving M&E needs and partners’ feedback and capacity – do not overbuild

Flexible, adapted to most M&E designs

Simple to deploy in the field, no strong dependence on high-speed Internet connectivity

Leverage existing CGIAR tools and repositories

Openness (to and from 3rd party applications and databases)

PMMT Design PrinciplesInterviewing a farmer, 2013.

M&E site stratification & selection. Powerful spatial visualization features to provide rich contextual information, and overlays of local biophysical characteristics with socio-economic data in support of action/control site selection process.

15

Powerful Spatial Visualization

Streamlined indicator data entry and reporting. Familiar, secure interface for data entry with pre-set list of indicators help clarify reporting requirements. Data import/export tools for multiple end-uses.

16

Simplified Project Performance Monitoring

17

Series of in-country M&E and PMMT training planned through October 2014.

On-line PMMT User Guide Video tutorial

Additional Training Resources

Interviewing a farmer, 2013.

Next Steps

18

Analysis of ARBES and support to research teams

Add new spatial layers using results from recent LSMS-ISA and ARBES surveys

Provide vertical roll-ups of indicators across megasites, and mini program dashboard

Tighter integration between PMMT and CGIAR report and data catalogs (CGSpace, AgTrials, and ILRI CKAN)

Allow 3rd-party spatial data

sources to be read into and visualized alongside Africa RISING layers

Build support for off-line data entry (possibly using mobile devices)

Embed HarvestChoice MAPPR tools (point/polygon/domain summaries) for advanced spatial analyses

New hiring (Data Manager, ARF in ESA, SRA in WA and ETH?)

Plan for AREES/ARMES in 2016

M&E Roadmap

19

The End

Thanks!

20

ARBES Qx household summary contents

Location info, GPS Roster Education Labor Health Women and child anthropometrics Agriculture -general- Crop inputs (Conservation Agr.) Crop production Crop inputs (costs) Crop inputs (labor time use) Crop inputs (seeds) Crop sales Crop storage Livestock ownership and income Livestock feed

SECOND VISIT Problems and coping strategies Agricultural extension and AR program Other income Credit Housing, utilities, assets, distance to

services Subject welfare and food security Food Expenditures / Consumption Non-Food Expenditures Shocks Re-contact info

FIRST VISITHead Individual Best Informed

ARBES Qx community summary contents (5 to 8 informants)

Location info, GPS

Informants’ roster

Access to basic services

Agricultural labor, extension services, agricultural problems

Land use

Demographics, cooperatives, migration Water access, shocks, food consumption Market prices Conversion of non-standard units

IE Design

Beneficiary HHs Non-beneficiary HHs Control HHs

Action Sites Control Sites

Spillover effects

Program impact