A Critical Analysis of Current M&E Frameworks

Insights from AWID’s research on M&E frameworksBy Srilatha Batliwala, Scholar Associate, AWID

SBatliwala, AWID

December, 2009

Draft for Discussion

• AWID’s research on M&E is being broadened, deepened and sharpened, and a newer version of our overview paper will be available in the second half of 2010

• This PowerPoint is thus a reflection of our work-in-progress, and intended for discussion, rather than a final viewpoint.

SBatliwala, AWID

First, let’s remember that:

“When you work for women’s interests, it’s two steps forward - and at least one step back. And those steps back are… often evidence of your effectiveness; they represent the threat you have posed to the power structure, and its attempt to push you back. Sadly, even our ‘success stories’ are sometimes nothing more than ways the power structure is trying to accommodate and contain the threat of more fundamental change by making small concessions to us.”

Sheela Patel, 1987SBatliwala, AWID

Assumptions in M&E

1. Everything should be measured2. Everything can be measured – even

complex processes of social change 3. Measurement will enhance our ability

to accelerate, deepen, replicate positive change, or improve achievement of desired goals

4. Change is predictable – we know what it will look like, where it will occur, and how to assess it

SBatliwala, AWID

5. The macro-political environment is stable, democratic, upholds basic rights, and protects change agents and change processes – i.e., law and order, an impartial judiciary and police, due process, rights of association, civil liberties, an independent media, etc. are inevitably present in every context

6. The organizational environment is stable and unchanging – staff don’t leave, core operational costs are secure, and you haven’t been shut down or harassed

Assumptions in M&E

SBatliwala, AWID

Why M&E?

In theory, to:•Learn how change happens, make it happen faster, better, etc.•Analyze and sharpen our role in the change process•Empower our constituencies•Practice accountability – to donors, constituencies, peers, public, etc.•To advance our advocacy agenda

SBatliwala, AWID

Why M&E?

In practice, to:•Satisfy donor requirements•Prove we’re doing what they funded us to do•Leverage more funding•Protect ourselves - from backlash, attacks, slander campaigns, etc.

SBatliwala, AWID

What isn’t working

• Very few M&E frameworks actually enable us to understand whether change has happened, or gender power altered – they measure performance (program inputs and outputs), not change

• What they measure may not be indicative of the change they’re measuring at all – (e.g., GDP!)

SBatliwala, AWID

What isn’t working

• Many current frameworks are very linear, and look for simplistic cause-effect relationships (x intervention = y effect = z change)

• One such, the logical framework, is often focused on performance - “[log frames] focus on the expected achievements laid out in the matrix – rather than the work itself.” ,

SBatliwala, AWID

What isn’t working

• Most frameworks do not provide for tracking negative change, reversals, backlash, unexpected events, etc., that push back or shift the direction of the change process. In women’s rights work, this is vital, because the most effective work seriously challenges patriarchal / other social power structures, creating negative reactions.

• Reactions / backlash / negative change is often evidence of positive impact!

SBatliwala, AWID

What isn’t working

• Most current frameworks are completely inappropriate for certain kinds of social change organizations and strategies: E.g., advocacy, training / capacity building, knowledge production, challenging discourse, etc.

• Consequently, these organizations are forced to measure their processes, outreach and outputs (number programs held, number of participants, publications, attendance at rallies, etc.), rather than their impact.

SBatliwala, AWID

What isn’t working

• Several false binaries and dichotomies are embedded within or underlie many M&E approaches – e.g., “quantitative-qualitative”, “subjective-objective”, “macro-micro”, “success-failure”, and so forth.

• These create problematic hierarchies rather than approaches that can integrate and transcend such dualities.

SBatliwala, AWID

What isn’t working

• Disjuncture between change measures and our time frames. The changes we are trying to track may not be visible within the time frame in which we are seeking them

• Most current M&E frameworks are neither gendered nor feminist in their principles or methodology.

“It takes ten years to build an organization, twenty years to build

a movement, and thirty years before you

see lasting impact.”Ela Bhat

SBatliwala, AWID

Measuring Social Abstractions – the key questions

1. What is it? Definition2. Where is it? Location (geographic,

social, political, institutional, sectoral)3. What are its boundaries? Spatial,

demographic, and conceptual4. What does it look like? Characteristics

& situational analysis5. What can be measured? Measurable

dimensions, sub-units, indicators6. When do we measure? Baseline +

frequency of assessmentSBatliwala, AWID

Elements of Feminist M&E• Feminist answers to the social

abstraction questions• Informed by / embedded in feminist

values and principles• Based on complexity• Use the best available tools (both

quantitative and qualitative)• Appropriate and differentiated time

frames• Track backlash / reversals• Designed for the level, nature of work

and strategies of each organizationSBatliwala, AWID

Principles of Feminist M&E

1. The right of our constituency to inform, co-design and participate in the monitoring and evaluation of change processes

2. Respect for the voice and perspective of all key stakeholders

3. Prioritizing learning in our M&E goals4. Positioning M&E as a political activity5. Integrating political and social forces into

our analysis & frameworks

SBatliwala, AWID

Principles of Feminist M&E

6. Avoiding attribution, assessing contribution

7. Eschewing false binaries / dichotomies8. Not using M&E for punitive purposes9. Capturing, analyzing and addressing

negative changes, reversals10. Willingness to abandon, revise, recast

our frameworks

SBatliwala, AWID

First steps in doing it differently:

1. Problem definition & situational analysis2. Enunciation of our M&E principles3. Development of customized M&E

framework & indicators (see forthcoming AWID guideline)

4. Establish a baseline5. Track and map

• Our processes• Our performance and • External change – positive and

negative SBatliwala, AWID