ETHIOPIAThe impact of client oriented services:

The Rural Capacity Building Project (RCBP)Ministry of Agriculture and Rural

Development

CROSS-COUNTRY WORKSHOP FOR IMPACT EVALUATIONS IN AGRICULTURE AND COMMUNITY DRIVEN DEVELOPMENT

Addis Ababa, April 13-16, 2009

Team Members

Mulugeta Hiluf, Sorssa Natea, Tigist Redda, Anteneh Girma, Ibrahim Worku, Feiruz Yimer,

Arianna Legovini

Title

The Development Objective of the RCBP is to strengthen agricultural services and systems to make them more responsive to client needs. Major Components(i)ATVET, (ii)Extension, (iii)Agricultural Research and (iv)Marketing

Three areas of intervention are being considered for evaluation

PROJECT COMPONENTS

Title

The evaluation will have three priorities:

1. Gaining an understanding of overall project impact,

2. Measuring the effectiveness of the Farmer Innovation Fund (FIF) and investigating alternative implementation modalities to secure high levels of innovation, and

3. Measuring the effectiveness of Farmer Research and Extension Groups (FREGs)

PRIORITIES FOR LEARNING: INTERVENTIONS

Title• Farmer services’ satisfaction (3-6 months from start of service)

• Household adoption of new technologies (12 months from start of service)

• Woman-headed household adoption of new technologies (12 months from start)

• New extension methods developed (12 months from start of service)

• Male-headed household productivity of main commodity (24 months from start)

• Female-headed household productivity of main commodity (24 months from start)

• Male-headed household income (24 months from start of service)

• Female-headed household income (24 months from start of service)

OUTCOME INDICATORS

Title1. What is the impact of the RCBP’s ATVET,

extension, agricultural research and marketing components on female and male farmers, pastoralists and agro-pastoralists?

2. a. How does demand-driven innovative service delivery compare with traditional supply-driven services?

2. b. Does the quality, creativity and intensity of the facilitation process affect the amount of innovation proposed by farmers’ groups, and the impact on project outcomes?

3. Does the Farmer Research and Extension Groups (FREGs) (participatory approach) develop and transfer technologies responsive to farmer needs?

EVALUATION QUESTIONS

TitleEvaluation constraint: The intervention units have already been assigned to treatment. Extension component:

Treatment: 2502 FTC/kebeles in 138 woredas assigned based on criteriaComparison: Propensity Score Matching (PSM) based on same criteria

(using 2008 baseline data) Estimator: difference-in-difference

ATVET component: PROBLEM: selection bias and small universe of treatment colleges. Treatment: 6 ATVET colleges hand picked from a universe of 25 ATVET collegesComparison: PSM weighing all remainingEstimator: Before and After/ difference-in-difference ???

Research component: This component will be monitored to ensure that grants are awarded, the diversity of grantees, and whether the grants accomplish their task.

EVALUATION DESIGNQuestion 1: Impact of the RCBP

components

TitleEVALUATION CONSTRAINT: Intervention units already assigned to treatment.

TREATMENT: 100 farmers’ training centers (FTCs) in 100 kebeles in 20 woredas.

THE IDENTIFICATION STRATEGY will use two types of comparison groups:

i.Within acceptable proposals, random assignment to treatment and control. Use facilitation process to elicit excess demand to increase the number of good innovative proposals submitted in each FTC. The control group will be affected by information spillovers. To measure these spillovers,

ii.The strategy will adopt a PSM approach to identify comparison FTCs in the same woredas as the assigned FTCs. PSM will be done on the basis of the criteria used for assignment of beneficiary FTCs.

Estimator: Difference in Difference

EVALUATION DESIGNQuestion 2.a Impact of the Farmer

Innovation Fund

TitleThe 100 FTCs will be randomly assigned to two separate treatments that will vary the intensity of innovation promotion within each FTC. Relative treatment effect will be measured as the difference in mean outcomes between treatment one and two.

Treatment 1.

Facilitation provided by extension worker, trained in innovative approaches. A handful of farmers will travel to locations in Ethiopia were innovative practices have been identified.

Treatment 2.

Facilitation provided by a small team of innovation experts together with the extension worker, trained in innovative approaches. Farmers in the FTC presented with multimedia of innovative approaches, and receive motivational and practical support for developing ideas.

Estimator: Single difference

EVALUATION DESIGNQuestion 2.b Impact of alternative mechanisms for facilitating

adoption of innovation

TitleEvaluation strategy: Encouragement design

Select 200 kebeles that are potential beneficiaries of the FREG support.

Randomly assign 100 kebeles and encourage the zonal REFLACs to submit proposals for these kebeles.

Proposals from encouraged and non-encouraged kebeles will be reviewed and accepted following the FREG criteria.

Treatment:

Encouragement of 100 kebeles, and approval of all quality proposals.

Control:

Non-encouragement of 100 kebeles, and approval of all quality proposals.

Estimator:

Instrumental variable estimator to measure the impact of FREG on the treated

EVALUATION DESIGNQuestion 3. Impact of FREG on development and transfer

technologies responsive to farmer needs

Title• Extensive RCBP baseline data was collected in 2008:

ATVET component: Survey of 25 TVET colleges (full universe)

Extension component: Survey of households (10,000 observations in assigned and non assigned kebeles); Surveys of FTCs, kebeles, woredas, zones and regional offices

Research component: Surveys of regional and federal research institutes, research centers, agricultural universities, agribusinesses and households

• Two rounds of follow up surveys will be scheduled. Oversampling in FIF and FREG areas required.

SAMPLING AND DATA

Title

Baseline surveys Jan-Feb 2008Baseline analysis 2009Follow up surveys I Jan-Feb 2010Mid-term impact evaluation analysis 2010Follow up surveys II Jan-Feb 2011Final impact evaluation analysis 2011

TIMELINE FOR IMPACT EVALUATION

Title

MOARD

RCBP

EIAR

AAU

World Bank

CIDA

IMPACT EVALUATION TEAM: STAFFING

TitleData collection: TBD

Sources:

• RCBP

• Gender Action Plan?

• 3IE?

• CIDA?

• WB?

BUDGET

Thank you!!!