issues and challenges in m&e of rural livelihoods programs: designing feedback loops rohini...

Issues and Challenges in M&E of Rural Livelihoods Programs: Designing Feedback Loops

Rohini PandeMohammad Kamal Professor of

Public Policy12 August, 2013

1

Evidence for Policy Design Initiative@CID, Harvard University

2

The NRLM FrameworkProposal:Increase household income of rural poor via sustainable livelihood enhancement and improved access to financial and non-financial services

Mission components: Social, financial, and economic inclusion

Mission features:• Decentralized implementation• Prospective design of M&E framework

3

NRLM M&E: What we want to know?

1. Why do we expect NRLM to achieve its goals?

2. How do we know that, in practice, it is achieving these goals?

3. If it isn’t, how can we modify it so that it does achieve the goals?

4

Talk Outline

A Holistic view of M&E: Smart Policy Design Policy development Monitoring and Feedback loops

Working with smart policy design and feedback loops• Testing design features for microfinance• MIS systems as feedback mechanisms: Data

visualization in MNREGA

• Final summing up

5

Talk Outline

A Holistic view of M&E: Smart Policy Design Policy development Monitoring and Feedback loops

Working with smart policy design and feedback loops• Testing design features for microfinance• MIS systems as feedback mechanisms: Data

visualization in MNREGA

• Final summing up

6

Smart Policy Design: Policy Development

What is the policy goal

Is existing evidence consistent with the

theory?

What is your theory for how to achieve

the goal?

Refine objectives, theory Policy design

7

Smart Policy Design: Monitoring and Feedback Loops

Policy design and pilot rollout

Is policy achieving its goals

Refine objectives, theory

Large scale rollout but

keep monitoring

monitor

8

Talk Outline A Holistic view of M&E: Smart Policy Design

Policy development Monitoring and Feedback loops

Working with smart policy design and feedback loops• Testing design features for microfinance• MIS systems as feedback mechanisms: Data

visualization in MNREGA

• Final summing up

9

Objective:Promotion and growth of micro-enterprises

Smart Policy Design: Testing Design Features in Microfinance

10

From Objective to Policy

Objective:Support growth of micro-enterprises.

Hypothesis:Micro-enterprises have high growth potential but are limited by lack of financing. Policy: Access to Microfinance

11

Smart Policy Design: Policy Development

What is the policy goal

Is existing evidence consistent with the

theory?

What is your theory for how to achieve

the goal?

Refine objectives, theory Policy design

12

Is existing evidence consistent with theory?The typical evidence provided to examine microfinance impacts:1. High returns to providing enterprises cash• 4-5% monthly returns

2. High rate of growth of microfinance use through India3. High repayment rates• With most MFIs posting over 90% repayment rates

Concludes: Microloans are a sustainable way of providing the poor finance

13

Is existing evidence consistent with hypothesis?But is that the correct evidence support the hypothesis?

– Directly test impact of microfinance on enterprize growth– Rigorous impact evaluations show small average impact

Triangulate evidence to reformulate design1. Why could this be given that micro-entrepreneurs have

high growth potential and site credit constraints as a predominant obstacle to the growth of their businesses?

2. How can we design an evaluation to test where the answer lies?

For redesign: Pilot new policies

14

How should policy be redesigned?or what feedback is needed? One possibility: Replace private sector providers by community organizationsBut why will this solve the problem?

Another possibility (and one we test)Maybe the microfinance contract is too rigid and this implies entrepreneurs don’t undertake high risk high return activities.

15

Refine policy: What limits returns to microfinance?

Policy design

Theory: Entrepreneurs

limited by lack of credit

Empirics: High returns to capital, but

impact of microfinance

limited

Conc

eptu

alfr

amew

ork

Policy goal : Encourage entrepreneurship

From Objective to Policy Goal

16

Smart Policy Design: Monitoring and Feedback Loops

Does the evidence enable better policy design?

How will we monitor?

What is the pilot policy

17

Identify pilot policyWhat do we know about microenterprises?•Subject to frequent household and demand shocks•High rate of business closings•Lack of insurance to overcome shocks•Irregular income streams with businesses highly subject to seasonality

Conclude: •Microentrepreneurs unable to make profitable investments that require a longer return horizon•Need flexibility in loan contract to make such investments

18

What do we know about microfinance contracts?•Standard Grameen microcredit loan contract– Immediate repayment obligation: One week between

loan disbursement and first repayment– Frequent loan repayment: Weekly repayment

•Rigid contract

Conclude: •Test impact of increasing repayment flexibility to microentrepreneurs.– Provide a grace period between loan disbursal and

repayment

19

Competing theories!1. Our theory: Presented above2. Competing theory: Contractual flexibility can have adverse effects

Recall need for evaluation

20

Smart Policy Design Method

What is the pilot policy?

How to monitor

21

Isevidence consistent with the theory?

The question: Does a grace period before loan repayment starts enable enterprise growth?

Method to test the question1: •Work with large MFI provider in Kolkata•By lottery select some MFI groups that receivethe grace period contract•Other groups remain on standard contract

1 Field, Pande, Papp, and Rigol (2013)

22

Pilot design

• Half of study sample receives standard contract‒ Immediate repayment obligation:

Two weeks between loan disbursement and first repayment

‒ Loan repayment:Fortnightly repayment

• Half of study sample receives contract with a grace period

‒ Immediate repayment obligation:Two months between loan disbursement and first repayment

‒ Loan repayment: Fortnightly repayment

23

Results

24

Smart Policy Design: Monitoring and Feedback Loops

Policy design and pilot rollout

Is policy achieving its goals

Refine objectives, theory

Large scale rollout but

keep monitoring

monitor

25

Using this evidence to enable smart policy design for NRLM

1. Under NRLMs SHGschoose loan repayment terms for group members.

2. IS Trade-offs faced by SHG similar to MFI providers? • If SHG has incentives to ensure high repayment but the

group doesn’t benefit from single member’s success then they may go for conservative rigid repayment schedule

• Policy question: How can SHG incentives be aligned with those of individual member? • What is the hypothesis? • What evaluation mechanism exists for feedback?

26

Talk Outline

A Holistic view of M&E: Smart Policy Design Policy development Monitoring and Feedback loops

Working with smart policy design and feedback loops• Testing design features for microfinance• MIS systems as feedback mechanisms: Data

visualization in MNREGA

• Final summing up

27

MIS and Smart Policy Design: Example of NREGA

Smart policy design is about building feedback loops into the policy process.

The NREGA data visualization project seeks to do this by:• Identifying key program performance indicators• Presenting descriptive statistics with an eye toward

digestibility and aesthetics• Facilitating basic hypothesis testing – investigate

correlations between key outcomes and administrative inputs

• Making program data easier to access for the public and academics (encourage external feedback)

28

Smart Policy Design Framework:First causal hypotheses

•Use simple descriptive statistics to identify problem areas and confirm/reject initial hypotheses

•Develop and investigate explanations for observed phenomena

• Start with simple correlations between policy inputs and outcomes; move on to more complex regression analysis, field evaluations; update hypotheses at each iteration

• Idea to build increasingly defensible causal explanations

29

NREGA implementation: Need for focus

• The NREGA MIS collects dozens of indicators at multiple levels of granularity

– Even highly motivated implementers may not have capacity to sift through information and get a clear picture of program performance

• These need to be distilled into a few key measures of program performance such as female and SC/ST participation or average time to payment.

30

Example: female participation in NREGA

Getting data on female participation currently takes 4+ clicks (if you know where to look).

District-level data only available state-wise, block-level data only available district wise – hard to get a broad view of performance without resorting to data scraping

Increasing data accessibility is low-hanging fruit

31

Reusable charts can increase data accessibility

Web technology exists to pull data from MIS and display it as interactive graphics

Charts are reusable – we can use the same chart format to display many user-selected indicators from the MIS

Technology (JavaScript and the d3.js library) is open source with a large support community

32

Reusable charts can increase data accessibility

33

Reusable charts can increase data accessibility

34

Ongoing work: Data visualization as a feedback loopHow should administrators use MIS/data visualization to identify potential policy levers?

Possible hypotheses:1. Demand Side: Absolute supply of work affects female

participation through male-first work allocation2. Supply side: Higher female empowerment implies greater

female participation

First step (work in progress!): Develop software that combines different data to generate user-defined scatterSuch initial diagnostics can help kickstart smart policy design.

35

Demand side: Examining correlations

36

Supply side: Examining correlations

37

Talk Outline

A Holistic view of M&E: Smart Policy Design Policy development Monitoring and Feedback loops

Working with smart policy design and feedback loops• Testing design features for microfinance• MIS systems as feedback mechanisms: Data

visualization in MNREGA

• Final summing up

Smart Policy Design: The process

38

• Identify your objective• Develop a theory that explains why a specific policy

will achieve the objective– What is the mechanism by which the policy

causesthe objective to be achieved?– What is evidence that these causal links will work?• How similar is problem to other contexts from

which we have evidence on similar policies?• What more do you need to know?

• Use monitoring and feedback loops to generate this evidence

Takeaways for Designing NRLM M&E

39

M&E should not stand apart from policy design and implementation process.Rather it’s design should ensure feedback loops with policy process.•Shouldhavelittle to no distinction between how to think about monitoring versus evaluation.•Monitoring mechanisms should enable concurrent evaluation.•In all cases, iterate between theory, existing evidence and pilots.