Standing Panel on Impact AssessmentDoug Gollin, SPIA Chair, 15 Sept 2016

Contenthttp://impact.cgiar.org/

• Work since last ISPC meeting (Lima, May 2016)• First SIAC synthesis report: The “rigor revolution” in impact

assessment for agricultural researchCausal identificationMeasurementRepresentativeness of sampling

• Towards a second phase for the SIAC program

Recent calendar http://impact.cgiar.org/

June• 3ie / IFAD workshop “Designing and implementing high-quality, policy-

relevant impact evaluations” (SPIA contribution: DNA fingerprinting into IEs)• FERDI / SPIA workshop “Agricultural innovation: Learning for adopting”

(Portfolio of 4 SIAC RCTs, linking with other RCTs outside, and thinking broadly about social sciences across CGIAR + Science of innovation / impact)

• MEL CoP “Taskforce on selection of harmonized indicators” (advice / caution)

July• Impact Assessment Focal Point Meeting (our main CoP – identify CAPI need)• SIAC long-term / large-scale impact assessment studies: Mid-term meeting

(Portfolio of 7 studies, updates, course-correction)August• SIAC workshop on innovative methods for measuring adoption of

agricultural technologies (Fingerprinting, remote sensing, SMS and CAPI surveys - establishing proof of concept and thinking about scaling up)

Previous approaches to IA in the CGIARhttp://impact.cgiar.org/

• SPIA core business for a long time was to generate aggregate estimates of rates of return to investments in the CGIAR

• Methodologically simple but crude• Appropriate for the era:

• Technologies easily identified• Impacts largely unidimensional

• Impact modelled as an increase in production multiplied by a price.

• Consumers and/or producers benefit depending on assumptions• Extent of broad pool of benefits is a function of:

% adoptionAverage productivity gain per unit adoption

The ‘rigor revolution’ in IAhttp://impact.cgiar.org/

• SPIA commissioned paper to Alain de Janvry and Elisabeth Sadoulet in 2010

• Their paper “Recent Advances in Impact Analysis Methods for Ex-post Impact Assessments of Agricultural Technology: Options for the CGIAR”, published in 2011 laid foundations for big changes

• Emphasized the need to take seriously that comparisons of adopters and non-adopters do not account for all the differences between them, especially in relation to “unobservables.”

• Recommended a portfolio of RCTs

1. Causal Identificationhttp://impact.cgiar.org/

• SPIA took on board these recommendations.• Opted not to focus entirely on RCTs.• Methodologically agnostic – we want careful and appropriate

combinations of methods• More complicated question is being clear about which non-

experimental research designs are appropriate, and in what contexts

Institutionalizing new methodshttp://impact.cgiar.org/

• Quality-rating system for impact studies / claims in CGIAR launched earlier this year – no voluntary take-up

Future:• Shift into regular audits / reviews of impact claims (retrospective)

as well as continuing a forward looking advice function reviewing research designs for future impact studies (prospective)

• Periodic, predictable synthesis reports on state of knowledge of impacts

2. Measurement mattershttp://impact.cgiar.org/

Adoption• Often the critical missing data for understanding impacts from

different streams of research• Definition and measurement of adoption far from simple

Outcome variables• Productivity (Plot area measurement? Crop-cuts?)• Remote sensing for environmental benefit streams?• Data on diets? Anthropometry?

Genotype Farmer-elicited name

Maize in Uganda: SPIA / LSMS-ISA / UBoS / Diversity Arrays

• Data from 540 HHs in 45 enumeration areas

• Enumerators from UBoS trained for 1 month

• CAPI-based survey + grain-based highly-quantitative DArTSeqgenotyping

• 2% of farmers were correct about the variety they were growing

Line transects: Reference value

Enumerator with visual-aid

Self-report with visual-aid

Enumerator estimate

Self-report

Remote-sensing

Drone

On the ground or from the air? Measuring crop residue retention, Ethiopia (teff, maize, wheat, barley)

3. Statistical representativenesshttp://impact.cgiar.org

• World Bank Living Standards Measurement Study –Integrated Surveys of Agriculture (LSMS-ISA)

• 8 countries in SSA – all important to CGIAR• Average of 5,000 HHs / country, nationally

representative• Panel – visited every 2 years

SPIA role:• Surveys lack modules / questions on agricultural

technologies (varieties, NRM practices)• SPIA’s comparative advantage to work to improve

this for benefit of CGIAR as a whole

Future: • Help bring about a geographic focusing of CGIAR

Towards SIAC phase 2: (2018-2022)http://impact.cgiar.org/

1. Country baselines and monitoring in key geographies

2. Database of claims of policy influence resulting from CGIAR research

3. Maintain focused competitively-commissioned portfolio of ex-post impact assessments

4. Synthesis reports on predictable and regular production cycle

5. Audits of impact claims

6. Improving the prediction of technology success in farmers’ fields

7. Capacity-building of economics / social science function

8. More work on methods as a CGIAR-wide public good

Country baselines and monitoringhttp://impact.cgiar.org/

What?• Proposing 6 high-priority countries + possible spillover countries• Phased approach between now and 2018/19 for first waves• Partnerships with national government agencies and local implementing

bodies with expertise in survey management• Collaboration with CGIAR partners on identifying priorities and piloting

methods

Why SPIA?• CGIAR-wide public goods• Established network of potential partners (World Bank LSMS-ISA; IFAD;

Excellence in Breeding - ICRISAT/CIMMYT; Big Data – CIAT/IFPRI; Diversity Arrays)

• Independence• Long history of documenting technology adoption

SIAC PHASE 1 (piloting now) SIAC PHASE 2? 6 country CRP total (M HHs)

Overall CRP totalEthiopia Tanzania Uganda India B’desh Vietnam

% share

A4NH 0.50 0.56 1.80 2.50 3.10 8.5 20.5 41CCAFS 0.80 0.40 3.00 0.50 1.00 5.7 10.9 53DCL 8.00 10.00 18.0 25 72Fish 0.11 1.80 1.9 4.9 39FTA 4.20 1.20 1.50 6.20 2.60 15.7 41.3 38Livestock 2.05 1.44 0.31 0.48 0.16 0.11 4.6 6.5 70Maize 2.50 1.10 0.30 3.70 0.80 8.4 15.0 56PIM 1.00 5.00 2.00 1.50 9.5 14.5 66Rice 0.09 4.27 1.19 0.76 6.3 16.5 38RTB 0.30 0.80 0.30 1.4 8.0 18Wheat 2.00 0.01 8.00 0.34 10.4 17.2 60WLE 1.00 0.55 12.50 2.75 16.8 21.0 80Country total (M HHs) 22.05 10.76 4.71 52.95 12.14 4.47 107.1 201.2 53

% share of all rural HHs 127 141 72 29 52 29

N.B. - CRP data from March full proposal versions (not yet checked against revisions)