MONITORING & EVALUATION TOWARDS TRACKING HIV/AIDS RESPONSE

PROGRESS

Workshop organized by the IAS at ICASA 2008

By Laetitia Lienart

IAS Evaluation Coordinator

A FEW WORDS ON M&E

Why M&E is important? It is an essential part of our daily work, helping us to identify any unwanted developments and to continually improve our work.

What does stand for Evaluation?It is the objective analysis of information in order to answer specific

questions and to make assessments based on previously determined

success indicators. Potential assessment criteria for projects are: Relevance: Do the project goals correspond with the needs of the target

group(s)? Efficiency: Are the resources used for the intervention adequate? Effectiveness: Have the project goals been met? Impact: Which positive and negative, direct and indirect, intended and

unintended effects have occurred as a result of the project? Sustainability: Can the benefits of the project endure?

A FEW WORDS ON M&E (cont.)

What does stand for Monitoring?

Unlike evaluation, it is a continual analysis of information, which

gives indications as to whether or not a project is running

according to plan. On this basis a project can be adapted and

processes optimized, as needed.

What is essential in the M&E process?

Although it is not the only requirement, it is essential to have a

well structured project guided by a logical framework.

What is a LF?

A table giving a clear and synthetic picture of the project goal,

purpose, outputs and activities, their respective Key Performance

Indicators and sources of verification as well as risks which could

affect the project implementation.

Why is a LF useful ?

essential in the Monitoring & Evaluation process but also

reference tool during the whole project cycle towards Results-Based

Management (RBM), i.e. a management strategy focusing on

performance and achievement of objectives

reference tool for on-going reporting and communication between

partners

 

LOGFRAME INTRODUCTION

The LF process helps guide the planning of a journey from where we are now, HERE, to where we want to go, THERE. 1 - Who are ‘we’? Who has an interest and is a main stakeholder to be involved? 2 - Where are we now? What are the problems and related needs? 3 - Where do we want to be? What are the options? What are our objectives & expected results? 4 - How will we get there? What is the resulting strategy? What are activities do we have to undertake? 5 - What may stop us getting there? What are the risks and how can we manage them? What assumptions are we making? 6 - How will we know if we’ve got there? What are our indicators and targets? What evidence do we need? 7 – What do we need to get there? What detailed resources and related budget are needed?

HERE

THERE

LOGFRAME INTRODUCTION (cont.)

The Goal is the higher order objective, the longer term impact,

that the project will contribute to.

Use only one Goal statement.

Some progress towards the Goal should be measurable during the

lifetime of the project.

The Goal defines the overall “big picture” need or problem being

addressed; it expresses the justification of what is planned.

Formulate the goal as a verb: e.g. Accelerate the response to

HIV/AIDS in the Nbiya region.

STEP 1 - DEFINE THE ‘DEVELOPMENT OBJECTIVE’ OR GOAL

The Purpose describes the specific and immediate results of the

project.

The Purpose should be SMART (Specific/Simple, Measurable,

Achievable, Relevant & Time-based).

Have only one Purpose. If you think you have more, then you may

need more than one logframe; or your multiple purposes are in fact

indicators of a single purpose or lower outputs.

STEP 2 - DEFINE THE ‘IMMEDIATE OBJECTIVE’ OR PURPOSE

The Purpose should not be entirely deliverable, i.e. fully within the

project manager’s control. If it is deliverable, then it should be an

Output. Ex: ‘You can take a patient to the hospital, but you can’t make sure

he/she will follow doctor’s prescriptions’. The project may be ‘delivering’ the

access to hospital, but it cannot control the behaviour of the patient.

The project manager can best exert influence over Purpose achievement by

maximising the completeness of delivery of the Outputs and mitigating

risks.

Formulate the purpose as a verb: e.g. Improve access to HIV/AIDS

treatment in the Nbiya region.

STEP 2 - DEFINE THE ‘IMMEDIATE OBJECTIVE’ OR PURPOSE (cont.)

The Outputs describe what the project will deliver in order to

achieve the Purpose. They are the results that the project must

deliver in the control of the project manager.

Outputs are nouns + adjectives. E.g. a) Medical infrastructures

rehabilitated/reinforced in the Nbiya region; b) HIV/AIDS awareness

of Nbiya region’s inhabitants raised; c) Health care workers of the

Nbiya region adequatly trained on HIV/AIDS; d) Partnerships

between Nbiya’s government and drug supplier(s) initiated; etc.

Typically there are between 2 – 8 Outputs; any more than that and

the logframe will become over-complicated.

STEP 3 - DESCRIBE THE ‘RESULTS/PRODUCTS’ OR OUTPUTS

The Activities describe what actions will be undertaken to achieve

each output.

Activities are usually actions (use nouns). E.g. needs assessment,

recruitment of experts/consultants, design of materials, development

of training programme, selection of participants/trainers/suppliers,

organization of coordination meetings, distribution of awareness

materials, implementation of works, procurement of equipment &

supplies, etc.

STEP 4 – DEFINE THE ACTIVITIES

When the four rows of column 1 have been drafted, the logic needs

to be tested: use the IF/THEN test to check cause and effect,

reading from the bottom up:

If we do these activities, then this output will be delivered.

If we deliver these outputs, then this purpose will be achieved.

If the purpose is achieved, then this will contribute to the Goal.

STEP 5 – TEST THE LOGIC FROM THE BOTTOM TO THE TOP

Taking all the activities needed for Output 1, ask the question: ‘if we

complete these Activities successfully, then what can stop us

delivering Output 1?’ . Repeat for all the other Outputs taking each

Output and its associated activities in turn.

Taking all the Outputs together, ask the question: ‘if we deliver all

these Outputs successfully, then what can stop us achieving our

Purpose?’

Now ask the question: ‘if we are achieve our Purpose successfully,

then what can stop us contributing to the Goal?’

IDENTIFY THE RISKS

STEP 6 – UNDERTAKE A RISK ANALYSIS

What is its likely impact ? high, medium or low.

What is its likely probability? high, medium or low. You may at this

point decide to hereafter disregard insignificant risks.

Discuss & agree possible mitigation measures; transfer them into

Column 1 (i.e. extra activities) of your LF.

Example:

risk: decrease in the # of health care workers affected to the Nbiya region

mitigation measure/new activity: advocacy/lobbying to the relevant authorities

& stakeholders involved in health care workers management

STEP 6 – UNDERTAKE A RISK ANALYSIS (cont.)

ANALYSE AND MANAGE EACH RISK

Assumptions are what remains after the mitigation measures have

been put in place (even if mitigation measures are successful, it is

unlikely you can remove the risk completely).

Assumptions are external factors which could affect the success of the

project but over which the project manager has no direct control.

Example: ‘’Strong commitment and continuous support of the Nbiya’s

authorities towards combatting HIV/AIDS‘’

Logic test: once Activities have been carried out, and if the

Assumptions at this level hold true, Outputs will be delivered. Test

the same logic for the upper levels.

STEP 7 – FORMULATE THE ASSUMPTIONS

Step 8 – IDENTIFY KEY PERFORMANCE INDICATORS (KPI)

One of the key strengths of the logframe approach is that it forces

the planning team to build into the design how the project will be

monitored and evaluated.

Indicators and verification are needed to show what data we

intend to use to measure progress, and how that data will be

collected.

In most circumstances there is not enough baseline data available at

the design stage against which progress can be measured; in which

case the logframe helps to pinpoint the gaps and determine

what needs to be done.

Use the maxim Q Q T - Quantity, Quality, Time.

Step 1: Set the basic indicator.

Step 2: Add Quantity - an amount or percentage that will be achieved.

Step 3: Add Quality – a built-in quality measure to specify the indicator.

Step 4: Add Time – when this should be accomplished by. Avoid using

general phrases such as ‘As soon as possible.’ It is important to include a

balance of terminal/end of project indicators (TARGETS) and some mid-term

indicators (MILESTONES).

Step 8 – FORMULATE KPIs

Step 1: Increase in access of HIV/AIDS treatment

Step 2 (Quantity): [%] Increase over the baseline in number of HIV/AIDS

infected people (of the Nbiya region) receiving drugs or At least [X] % of

HIV/AIDS infected people (of the Nbiya region) receive drugs

Step 3 (Quality): [%] Increase over the baseline in number of HIV/AIDS

infected people (of the Nbiya region) receiving drugs on a sustainable way

Step 4 (Time): [%] Increase over the baseline in number of HIV/AIDS

infected people (of the Nbiya region) receiving drugs on a sustainable way

by [year/month] or within [duration]

Step 8 – FORMULATE A KPI (example)

Indicators are a means by which change will be measured while

milestones & targets are respectively mid-term and end “goals”.

As in the early stages of a project, there may not be the baseline

information needed to set targets, there will be mostly indicators.

Further stakeholder meetings and data gathering will lead to targets.

Example (for a project lasting 4 years ):

[%] Increase over the baseline in number of HIV/AIDS infected people receiving drugs on a sustainable way within [X] years is an indicator

10% Increase over the baseline in number of HIV/AIDS infected people receiving drugs on a sustainable way within 2 years is a milestone

25% Increase over the baseline in number of HIV/AIDS infected people receiving drugs on a sustainable way within 4 years is a target

Step 8 – INDICATOR EVOLVING INTO MILESTONE/TARGET

The final element of developing the logframe is to decide how KPIs

will be measured – what evidence we will use. This is a vital stage

of the initial planning that is often overlooked. Building in evidence

sources at this stage will make the monitoring and evaluating of the

project easier.

The evidence will almost invariably be documents (or sometimes

interviews, films, DVDs, videos or audiotapes.)

It should be considered as you formulate your indicators; so

complete columns 2 and 3 of the LF at the same time.

Step 9 – IDENTIFY THE EVIDENCE

Some typical sources of verification

Minutes of meetings and attendance lists

Stakeholder feedback & focus groups

Surveys and reports

Newspapers, radio and TV recordings, photographs, satellite imagery

National and international statistics

Project records, reviews and reports

External evaluation reports, training evaluation questionnaires, etc.

Step 9 – IDENTIFY THE EVIDENCE (cont.)

Step 10 - AND FINALLY, A FIFTH ROW?

For simplicity we say a logframe is a 4x4 grid but often a further 4

boxes are attached under the 16 x box logframe to indicate the

costs and resources needed to fund the project activities

together with pre-condition assumptions that need to be agreed.

Inputs Budget summary

Staff months

Accounts

summary

Pre-conditions;

Assumptions

made to carry out

the activities

EXAMPLE OF A COMPLETE LF

Always date your LF!

Intervention logic Indicators / Milestones / Targets Verification Assumptions

Goal : Accelerate the response

to HIV/AIDS in the Nbiya region.

Proportion of HIV/AIDS

infections decreased by [X]% by

Year 4 (I)

Reports, statistics,

surveys, etc.

Purpose: Improve access to

HIV/AIDS treatment in the Nbiya

region.

25% Increase over the baseline

in number of HIV/AIDS infected

people receiving drugs on a

sustainable way within Year 4 (T)

Reports, statistics,

surveys, hospital

patient/attendance

list, drug delivery

receipts, etc.

Sustainable

commitment from

authorities &

partners

Outputs:

O1. Medical infrastructures

rehabilitated/reinforced in the

Nbiya region

O2. HIV/AIDS awareness of

Nbiya region’s inhabitants

raised

O3. Health care workers of the

Nbiya region adequately trained

on HIV/AIDS

O4. Partnerships between

Nbiya’s government and drug

supplier(s) initiated

1. At least one hospital is running

independently in the Nbiya

region

by Year 2 (M)

2. # of awareness campaigns

organised by Year 2 (I)

3. At least 50 health care

workers of the Nbiya region

trained by Year 4 (T)

4.1 # of partnerships initiated by

Year 2 (I)

4.2 At least 2 partnership

agreements signed by Year 4 (T)

1. Reports of regular

control visits to the

hospital

2. Posters/leaflets &

other awareness

materials

3. Training reports &

post-training skills

evaluation stats

4. MoU/agreement,

minutes of meetings

Medical

infrastructures

regularly

maintained.

No security

restriction that

may prevent

reaching pops.

All trainees fully

affected to

HIV/AIDS & on a

sustainable way

Activities: to be developed with the same method

THE LF AS A COMMUNICATION TOOL

The LF can help report to project partners, sponsors & other

stakeholders what we are doing and why. This can be achieved by

taking a Step-by-step presentation approach

1. Goal: "The overall goal is to ............."

2. Purpose: "In order to contribute to this goal we in this project will............"

3. Outputs: "We will achieve this objective by taking direct responsibility for............"

4. Activities: "Let me describe our strategy in more detail. We believe that if we .............."

5. Activity level Assumptions: "and if .........."

6. Output level Indicators: "we will achieve our targets of ............."

7. Purpose: "In addition to reaching these targets, several other things must happen if we are to achieve our major objective of ............"

8. Output level Assumptions: "These other factors, outside our direct control, include ........."

9. Purpose level Assumptions: "We believe that if we can achieve our major objective, we will contribute to our overall goal. This contribution is, however, affected by factors outside of this project. These include ........ All of these factors taken together will be sufficient to realise this goal. The strategy we propose is an important and cost effective step towards that end."

10. Verification: "We propose that our performance be monitored and assessed in the following way..........."

THE LF AS A COMMUNICATION TOOL (cont.)

REPORTING USING THE LF

Complete the table for each different logic intervention level and at different times during the project cycle.

PROGRESS/MONITORING REPORT COUNTRY…………PROJECT TITLE……………… PERIOD COVERED………… CODE…………… DATE …………….. PREPARED BY…………………………

Project

Structure

Indicators of

achievement

Progress Comments &

recommendations

Rating

Insert

activities

and inputs

from the LF

Insert

indicators from

the LF

Provide a

report

against each

activity and

input

Provide comments

against each activity

and input (e.g. on the

extent to which the

assumptions are

being met) &

recommendations

where appropriate

1. Likely to be

completely achieved

2. Likely to be partially

achieved

3. Unlikely to be

achieved

x Too early to judge the

extent of achievement

WHERE TO GO FOR FURTHER INFORMATION

http://www.parcinfo.org http://ec.europa.eu/europeaid/evaluation/methodology/

egeval/methods/mth_cyc_en.htm http://www.worldbank.org/evaluation/ http://www.oecd.org/pages/

0,2966,en_35038640_35039563_1_1_1_1_1,00.html http://www.undp.org/eo/ http://www.uneval.org/ http://www.unaids.org/en/PolicyAndPractice/

MonitoringAndEvaluation/default.asp http://www.ideas-int.org/

TASK TO PRACTICE THE LFA

Subject: the Mbiya region is severely affected by HIV/AIDS, with

children and women among the most infected populations. You wish

to apply for a tender launched by a major donor in order to address

health problems in that region.

Methodology: each working group is expected to follow all the

above-presented steps to eventually design and present a complete

LF with at least 3 outputs and 3 activities per output. Use the fact

sheet as main background.

EVALUATION

As your feedback is of most importance to assess the success of

ICASA 2008, you will be invited shortly after the conference to

complete an online survey (available in both English & French) -

all data will be kept confidential.

In order to participate in the evaluation process, please share your

contact details by filling out the form under circulation.