Monitoring & Evaluation

Agenda

• Introduction• Basic concepts

Definitions of M&EWhy do M&E?

• Project PlanningTheory of changeResult chainAssumptions and risks

• M&E– Introduction to M&E

systems– How to measure

• What indicators to use

• How to collect data• What targets to set

– Putting it all together

Definitions

• MonitoringRoutine, systematic collection of data relating to project

performanceProvides regular feedback and early indications of

progress (or lack of progress)

• EvaluationEpisodic and objective assessment of projectOften done at mid-point and at end of projectCan determine whether objectives were achieved, how

efficient the project was, and what the project’s impact was

Why do M&E?

• Knowledge of impactM&E tells you whether your projects are meeting

objectives and having results• Improved decision-making

Information generated through M&E helps you make better decisions about projects

• Greater accountability and transparencyM&E data allows donors and others to assess how well and

efficiently you are using resources

Theory of Change

• What is the theory of change for your project?• A chain of so thats that lead to the change you hope

for• Example: we teach children how to wash their hands

So that they will understand the importance of washing their hands AND will have the skills to wash their hands properly

So that they will wash their hands regularly So that they will be less susceptible to certain illnessesSo that the incidence of certain illnesses will decrease So that child mortality will decrease

Result Chain

Def: A picture of the chain of “so that” relationships that lead to the desired

outcomes

Source: University of Wisconsin Extension, Developing a Logic Model

Result Chain – Simplest Form

Inputs Activities Outputs

Purpose/ Short-term

objective

Goal/ Long-term

objective

Result Chain – More Detailed

Inputs

•What you invest

•Staff•Volunteers•Time•Money•Research base

•Materials•Equipment•Technology•Partners

Activities

•What you do•Specific actions you undertake to achieve particular outputs

Outputs

•What you deliver•Products and/or services you deliver (~1 yr)•You should be able to guarantee these•Examples:•Conduct workshops•Deliver services•Develop products/curriculum/resources•Train•Provide counseling•Assess•Facilitate•Partner

Purpose/ Objectives

•What the short- and medium-term results are

•Situations, conditions, behaviour changed as a consequence of the project (~3-5 yrs)

•Outside your control

•Learning:•Changes in awareness, knowledge, attitudes, skills, opinions, aspirations, motivations

•Action:•Changes in behaviour, practice, policies, decision-making,

Goal / Long-Term Objective

•What the ultimate impact is

•Long-term impact of the project (+ 5 years)

•Project contributes to goal, but likely doesn’t achieve it

•Conditions:•Changes in social, economic, civic, environmental conditions

Result Chain Example: HIV Prevention for Sex Workers

Inputs

•Staff - 3 trainers

•Volunteers – national network of 8 volunteers

•Time – 20 hours/month

•Money – space rental, snacks purchase, transportation reimbursement

•Research base – knowledge of STDs, knowledge of training techniques

•Materials – workshop materials

Activities

•Make schedule of workshops

•Coordinate with local volunteers to recruit workshop participants

•Plan workshop curriculum

•Assemble workshop materials

•Book workshop facilities

Outputs

•Conduct 1 workshop every 2 months

•Target Cambodian sex workers

Purpose/Objectives

•Learning:•Participants are aware of the importance of HIV prevention

•Participants have knowledge of means of transmission, methods of prevention

•Participants have negotiation skills

•Action:•Participants practice safer sex

Goal/Long-Term Objective

•HIV prevalence among Cambodian sex workers declines

Assumptions and Risks

• How do you plan for events, conditions, and decisions beyond your direct control?

• Assumptions positive statements about what will go right

• Risksnegative statements about what might go wrong

Source: AusAid, AusGuideline 3.3 The Logical Framework Approach, 2005; NORAD, The Logical Framework Approach (LFA): Handbook for objectives oriented planning (4th ed.), 1999.

Assumptions and Risks

Inputs Activities Outputs

Purpose/Short-term

objective

Goal/Long-term

objective

Decreasing controlIncreasing significance of risks

Assumptions Assumptions Assumptions Assumptions

Decreasing controlIncreasing significance of risks

Models and M&E

• A result chain says what results you expect to achieve• M&E tells you if you are achieving the results you expected

to achieve

Good M&E Systems

• Dynamicencourage `learning by doing’

• Participative and gender sensitiveseek to overcome barriers of gender, age, power, and

culture• Reflective

encourage creating regular space and time for analysing information and reflecting on the theories of change

• Evolvingadapting and changing in order to keep them as light and

simple as possible while providing `real time’ information

Source: AusAid, Guidance on M&E for Civil Society Programs, December 2008

How to Measure: Indicators

• Indicators tell you how you will recognize success• Indicators are a unit of measurement

Quantitative Indicators

• Need a balance between quantitative and qualitative indicators

• Quantitative: Can be measured in numbers

number of water pumps installedamount of rice harvestedpercentage of children immunizedHIV prevalence rate student-teacher ratiofrequency of attending class

Qualitative Indicators

• Qualitative: reflect people’s judgments, opinions, perceptions, and attitudes

perception of well-beingappropriateness of interventionquality of engagementlevel of commitmentsense of empowerment

Indicators to Use

1) What questions do you want to answer? (i.e. your Key Performance Questions)?2) What data can provide the evidence you need to answer your KPQs? Is the data high-quality? Is it participative?3) What data can you collect? Do the benefits justify the costs?

Indicators to Use

1) What questions do you want to answer? (i.e. your Key Performance Questions)?• This depends on

Your audience– Management? Donors? Beneficiaries?

The purpose of measurement– To assess impact? Efficiency? Equity?

Conflicts can arise between M&E for accountability and M&E for learning

Indicators to Use

Level in result chain

Key Performance Questions - examples

Inputs • What interventions and resources are needed?• Are resources being used efficiently?

Activities • What are we doing? Are we doing it right?• Have planned activities been completed on time and within the budget?

Outputs • Are we implementing the project as planned?• What direct tangible products or services has the project/programme delivered?

Purpose/ objectives

• Are interventions working or making a difference?• What changes have occurred as a result of the outputs? Are these changes contributing towards the

project purpose and desired impact? • Has the project achieved the changes for which it can realistically be held accountable?

Goal/ long-term objective

• Are we intervening on a large enough scale?• To what extent has the project contributed towards its longer term goals? Why or why not? • What unanticipated positive or negative consequences did the project have? Why did they arise?

Sources: International Federation of Red Cross and Red Crescent Societies, Monitoring and Evaluation in a Nutshell, 2007; Global Fund, Monitoring and Evaluation Toolkit, 2009

Indicators to Use

2) What data can provide the evidence you need to answer your KPQs? Is the data high-quality? Is it participative?

Indicators to Use

Level in result chain

What to monitor & evaluate/ Key Performance Questions

Key Performance Indicators - examples

Inputs • Are resources being used efficiently? • Number of staff hours spent• Money spent

Activities • Have planned activities been completed on time and within the budget?

• What unplanned activities have been completed?

• Percent of milestones achieved on time

• Variance between budget and actual

Outputs • What direct tangible products or services has the project delivered?

• Number of workshops held• Number of sex workers trained

Purpose/ objectives

• What changes have occurred as a result of the outputs? To what extent are these likely to contribute towards the project purpose and desired impact?

• Has the project achieved the changes for which it can realistically be held accountable?

• Percent of workshop participants demonstrating a high level of knowledge of HIV transmission and prevention

• Percent of workshop participants reporting using condoms “frequently” or “always”

Goal/ long-term objective

• To what extent has the project contributed towards its longer term goals? Why or why not?

• What unanticiplated positive or negative consequences did the project have? Why did they arise?

• HIV prevalence rate among Cambodian sex workers

Indicators to Use

Ensure that high-quality data will be collected. It must be• Accurate• Reliable• Precise• Complete• Timely• With Integrity• Respectful of confidentiality

Source: Global Fund, Monitoring and Evaluation Toolkit, 2009

Indicators to Use

3) What data can you reasonably collect?

Indicators to Use

Inputs Activities Outputs

Purpose/Short-term

objective

Goal/Long-term

objective

Increasing difficulty of measurementDecreasing frequency of measurement

Result Chain

Indicators to Use

Increasing difficulty of measurement

Hierarchy of Effects

Participation

•Number and frequency of people reached

•Intensity of contact

Reactions

•Degree of satisfaction with programme

•Level of interest

•Feelings toward activities, educational methods

Learning

•Changes in knowledge, attitudes, skills, aspirations

Actions

•Changes in behaviours and practices

Social-economic-environmental Improvements

•Changes in society

Source: Bennett and Rockwell, 1995, Targeting Outcomes of Programs

Indicators to Use

• Do the benefits justify the costs?• Look for a balance between rigour and realism

Source: Results-Based Management Tools at CIDA: A How-to Guide, http://www.acdi-cida.gc.ca

How to Collect Data

Identify for each Performance Indicator:

Definition Examples

Data source(s) People, organisations, documents providing data about the indicator

Beneficiaries, partners, government documents

Data collection methods/tools

How data will be collected Analysis of documents, surveys, interviews, focus groups, observation, case study, tests, testimonials, expert panel

Formula/ scale / assessment method

How the data will be captured

Different scales

Frequency and duration

How often data will be collected and for how long

Single time, continuously, weekly/monthly/annually, baseline/mid-term/final

Responsible person(s)

Who will collect the data Staff, outside evaluators

Indicators to Use

• Indicators ≠ Targets• Indicators are units of measurement

E.g. HIV prevalence rate among Cambodian sex workers• Targets specify a particular value for an indicator to

be accomplished by a specific dateE.g. HIV prevalence rate among Cambodian sex workers is

reduced by 50% by 2020• Targets are essential: they are a concrete expression

of your goals and objectives

Targets to Set

• You can only set targets once you have baseline data• Baseline data show the situation before you start the

project

Targets to Set

• Targets can be set as Absolute targets (e.g. increase by 5)Proportional or percentage targets (e.g. increase by 5%)Relative to benchmarks (e.g. be within the top three

schools in our area)Relative to costs or budgets (e.g. increase or reduce by 5%

same level of budget).

Sources: Advanced Performance Institute, How to Design Key Performance Indicators, 2010; Results-Based Management Tools at CIDA: A How-to Guide, http://www.acdi-cida.gc.ca

Targets to Set

• Targets should be SMARTSpecific, Measurable, Attainable, Relevant, Time boundAmbitious but realistic

Targets to Set

• Targets should account forTrends and past performancePredictable variation in performance (e.g. seasonal cycles)National targets, best practice benchmarks, etcResult chain (i.e. do not set outcome targets before you

have set input targets)Time lag between intervention and effectDependence on others (e.g. partners, government actors)

Targets to Set

• Indicator - # of workshops held• Add target group (for whom?) - # of workshops for sex

workers• Add quality (what? how well?) - # of workshops for sex

workers addressing HIV prevention• Add quantity (how much?) - # of workshops for sex workers

addressing HIV prevention increased from 10 to 12• Add time (by when?) - # of workshops for sex workers

addressing HIV prevention increased from 10 to 12 in 2012• Add place (where?) -# of workshops for sex workers

addressing HIV prevention increased to 6 in Phnom Penh and 6 in the provinces in 2012

Source: NORAD, The Logical Framework Approach (LFA): Handbook for objectives oriented planning (4th ed.), 1999.

Putting it All Together…

Key Performance Questions

Key Performance Indicators

Baseline Target Means of Verification

Input Are resources being used efficiently?

# staff hours/workshop 90 in 2011 50 in 2012

Staff time sheets

Activities Have planned activities been completed on time and within the budget?

% variance between budgeted and actual spending

2012 budget

2012 budget

Records kept by accountant

Outputs What direct tangible products or services has the project/programme delivered?

# workshops held

# sex workers reached

10/year in 2011

300/year in 2011

12/year in 2012

480/year in 2012

Participant lists

Purpose/Objectives

What changes have occurred as a result of the outputs and to what extent are these likely to contribute towards the project/programme purpose and desired impact?

Has the project/programme achieved the changes for which it can realistically be held accountable?

% of workshop participants demonstrating a high level of knowledge of HIV transmission and prevention

% of workshop participants reporting using condoms “frequently” or “always”

50% before the workshop

5% before the workshop

90% after the workshop

20% after 1 month10% after 6 months

Questionnaires before and after workshop

Surveys before workshop, after 1 month, and after 6 months

Goal/Long-term objective

To what extent has the project/programme contributed towards its longer term goals?

HIV prevalence rate among Cambodian sex workers

X% n 2011 Y% 2020 UNAIDS statistics

Want to Keep Going?

Put Pari’s M&E Services to work for your organization!

• We can help you set up an M&E system from scratch or assess your current system and help you improve it

• We’ll asses your team’s capacity and make an inventory of current monitoring, evaluation and reporting activities

• We’ll work with your team to map out your programmes/projects, select indicators, and create tools and schedules for tracking data and engaging in evaluation

• We’ll help you pilot the new system, gather baseline data and set targets

• We’ll work alongside your team to maintain the system

Email us at allie@thepariproject.com to learn more