Lessons from the meta-evaluation of MGNREGA

Ranjani.K.MurthyEngendering Policy through

Evaluation Project- ISST

Objectives of the meta-evaluation

Sensitive to gender and social

equity?

Preparation

Methodology

Reports

Findings on impact/design

Framework for meta-evaluation

Adapted UN System Wide

Approach Evaluation

Performance Indicator

Substantive equality

Empowerment

Parameters: gender and equity sensitivity

Evaluation preparation

• 1. Evaluability• 2. Stakeholder

identification• 3.Team

expertise and balance

Evaluation methodology

• 4 Evaluation criteria

• 5.Evaluation questions

• 6 Evaluation approach

• 7.Evaluation indicators

• 8.Evaluation methods

• 9 Data analysis

Evaluation report and use

• 10. Evaluation Report

• 11. Validation process

• 12 Government response

• 13 Dissemination

Evaluation findings

• 14 Findings on design, implementation impact and institutional arrangement

Rating system each evaluation

Not available N/A

Missing0

Partially met1

Fully met 2

Meta scores

0-0.5 • Missing

0-.51 to 1.25

1.26-1.75

• Meets

>1.76 • Exceeds

Approaches

Scope

• Professional Institute Network Studies: 21

• Concurrent Evaluation of MGNREGA: 1

• Total: 22

MGNREGA Act, 2005

Provide for the enhancement of the livelihood security of the households

One hundred days of wage employment in every year to every household whose adult member volunteers for unskilled work

Minimum and equal wages; at-least 33% women; <less than 5 km; conditional creches

Subsequent orders/recommendations gender and social equity

Pregnant/differently

abled

Trolley

Toilets

ICDS

50% mates

Single women

Bank account

33% for SC/ST/OBCs

Land and irrigation

development

Fourth Edition of Guidelines, 2013

Empowerment of the socially disadvantaged, especially women, Scheduled Castes (SCs) and Schedules Tribes (STs), through the processes of a rights-based legislation(one of 7 goals)

Evaluation preparation – Score 1

Evaluability1

86% studies - 3 yrs

Design-partial integration

Data- partially available

Team0.8

Women 28% of team40% of team leaders

Muslim TL none

One third-social developmentorganisations

Stakeholder identification

1.33

WorkersWomen workers

PRIs, MatesDifferent castes

Local governmentGram Rozgar Sevaks

Government

Less, Muslims, single women, disabled,

pregnant and lactating women, transgender,

Evaluation methodology (1.22)

Evaluation approach (1.32)

CE: Quantitative; sampling good and poor performance. “Beneficiary and mandays”

Quantitative & qualitative methods used (62%). Household: 34% Purpose sampling of women, SCs 22%

Evaluation questions (1.18)CE: Implementation (work, wages, amenities) and impact, PIN: Implementation (100%), impact (57%), context,

design, planning (24%) and institutional (14%)

Evaluation criteria (1)CE: 9 parameters -nothing gender/social specific PIN studies 17 parameters, 1 refers to women’s

empowerment and SC/ST inclusion

Evaluation methodology

Data analysis (1.38)

CE: Wide variation in women’s participation 24% adopt gender analysis- construction of ‘worker’, visibility, strategic or practical, identity, and GEWI index, empowering

Methods and tools (1.23)CE: Questionnaire with workers

Pin: 81% Questonnaire, 19% with non worker-othersFGDs: 52%, case studies: 29%, PRA: 14%. Ethics?

Evaluation indicators (1.23)CE: % workers women/WHH, work created for women &

SC/STs, and access to crechesPin: Same as CE (80%), nutrition and LFPR, appropriateness of

tools, worker’s committee, child labor and VAW (<20%),

Evaluation report and use

Report (1.27)•CE: GSE integrated into findings. No section on recommendations and conclusions •Pin: GSE Integrated into findings, followed by conclusion, recommendations and method

Validation (1.02)•PIN: 29% with district/state government, 10% PRIs and NGOs, and 5% taking it back to workers

Management Response (0.67), Dissemination (1.05) •One PIN study reports that a government official agreed to look into a recommendation-recording of BPL status in muster roll

•All reports on govt. webpage, 33% ppt, on web. Copy right government, no budget for dissemination to women

Assessment of impact on women's empowerment and gender equality by 12 PIN studies (1.08)

33%

42%

25%

Empowered

Mixed

Not empowered

ReasonsPositive Mixed/negative

31-46% of employment to women?

Expansion of women’s mobility, and increase in income

Formation of women only gangs in some places

Migration of women reduced

Enabled women experiencing domestic violence to stand up (north east)

Worker assumed to be a male: Productivity norms, amenities, timings (Bihar exception)

Elderly women, pregnant and lactating women, differently abled women need different norms; single women less represented in most states

Invisibility of women’s work:Pair work, payment to men/joint accounts, undervaluation of lifting/throwing, women proxy workers, over working of water vendors

Receipt of equal wages varied- 40% to 99%; men perceive it below dignity to work on equal wages; demanded money for consumption of alcohol

Little sharing of work; women already undernourished

No provision for development of land of WHHs or women’s control over CPRs created; Caste discrimination- tools, land development, well digging wages, water, work sites

Mates 0-33% women- Sexual harassment in work place noted in 2/3 studies

Women less found in social audits, gram sabha meetings and VMCs

Meta-score: 1.15 (prep 1.22, method: 1.22, report & use: 1.02, impact, 1.08)

Appr

oach

ing • Management response to

report (0.67)• Evaluability (1)• Evaluation criteria (1)• Dissemination of report (1.05)• Performance on GE/SE (1.08)• Validation of findings (1.09)• Evaluation questions (1.18)• Evaluation indicators (1.23)• Evaluation methods (1.23)

Mee

ting • Evaluation report (1.27)

• Evaluation team (1.29)• Evaluation approach (1.32)• Stakeholder analysis (1.33)• Data analysis (1.38)

Recommendations: Substantive Equality

Preparation Methodology Report /use Impact- Strengthen design from a GSE lens- Ensure disaggregated data is available - Ensure all evaluation organisations (EOs) have GSE expertise.-At-least 50% of the EO heads and team may be women- preferably from marginalised groups

-TOR may spell out that GSE is a cross cutting theme- - GSE questions and indicators to cover design, implementation, impact and institutional arrangements-Changes in gender relations in various institutions should be captured-The present questionnaire may be revised from a GSE lends - Qualitative methods for GSE assessment may be elaborated

- Validate with marginalised workers-Government response on web-site-Budget for dissemination to all-in particular marginalised groups

- Individual JC/guar- Transgender- Different norms- Not male timings- Anemia tablet- Pair work-

individual payment- Individual account- Women’s gang- Maternity benefit- Toilet/sanitary nap- Land transfer- Collective mgt- Care- Mech for

harassment- Spouse

sensitisation- Mates and staff- Unions

Put substantive equality and marginalised women’s empowerment at the center

Combine MGNREGS with other progs. e.g women’s land/resource rights