participant’s guide en/participants... · - exercise 5 in small groups: simulation of a...

PARTICIPANT’S GUIDE

WORKSHOP

ON MONITORING & EVALUATION

FOR PROJECTS SUPPORTED BY THE CCAA PROGRAM

Gorée, December 7-11, 2009

Prepared by Adama A. Ndiaye and Nathalie Beaulieu,

with contributions from Abdou Fall and Florence Etta

Workshop on Monitoring and Evaluation for Projects supported by the CCAA program. Gorée, 7-

11 December 2009

2

1. WORKSHOP CONCEPT NOTE .............................................................................................................................. 3

2. M&E PRACTICE AND CONSTRAINTS WITHIN PROJECTS ....................................................................... 7

3. SOME M&E KEY CONCEPTS ................................................................................................................................. 8

3.1 Monitoring ................................................................................................................................................................ 8

3.2 Evaluation .......................................................................................................................................................... 8

3.3 Monitoring & Evaluation ............................................................................................................................. 9

3.4 participatory action research (par) ........................................................................................................... 11

4. OUTLINE OF TOOLS TO BE SEEN DURING THE WEEK ......................................................................... 13

4.1 the results chain .................................................................................................................................................. 13

4.2 Outcome mapping............................................................................................................................................... 13

4.3 Outcome Journals................................................................................................................................................ 13

4.4 Interim technical reports ................................................................................................................................. 14

4.5 Visions-actions-partnerships (VAP) ........................................................................................................... 14

4.6 participatory analysis of components of risk .......................................................................................... 14

4.7. Collecting of Testimonial narratives .......................................................................................................... 15

4.8 The Most Significant Changes approach (MSC) ...................................................................................... 15

5. WORKSHOP EVALUATION TOOLS ................................................................................................................. 15

5.1 Process Documentation ................................................................................................................................... 16

5.2 Workshop Outcome journal ........................................................................................................................... 17

5.3 Situation Barometer and Spider Web as graph representation tool ............................................. 18

6. The Results Chain, a Result Based Management (RBM)tool ............................................................... 19

7. OUTCOME MAPPING (OM) ............................................................................................................................... 22

7.1 The Outcome Mapping Philosophy .............................................................................................................. 22

7.3 Use of OM to Enrich the Results chain ....................................................................................................... 26

8. Outcome journals .................................................................................................................................................. 32

9. INTERIM TECHNICAL REPORTS ..................................................................................................................... 41

10. Visions-Actions-Partnerships ...................................................................................................................... 43

11. Participatory analysis of the components of risk ................................................................................ 45

12. Collecting testimonial narratives ............................................................................................................... 50

13. The most significant change approach (MSC) ........................................................................................ 52

BIBLIOGRAPHY ................................................................................................................................................................ 54


11 December 2009

3

1. WORKSHOP CONCEPT NOTE

Introduction

The Climate Change Adaptation in Africa (CCAA) is a joint initiative between the

International Development Research Centre (IDRC) and the Department for International

Development (DfID) or the United Kingdom. The program recognizes that ongoing

monitoring and evaluation (M&E) are essential components of adaptive management within

development and adaptation initiatives. They can help persons and organizations to

strengthen their capacity to adapt to climate change by allowing them to reflect on the

outcomes of the adaptation options that they are implementing or testing. The program

therefore strongly encourages project teams to engage in M&E at the project level as well as

with their partners in participatory processes to draw lessons from their research and

observations and to adjust their activities to reach their objectives. M&E is part of the

participatory action research approach that the program promotes.

M&E is also important for project management and accountability. Project teams are

required to submit interim technical reports to their program officer every six months.

These reports should describe the projects’ progress in terms of activities, outputs,

outcomes, lessons learned and research findings for the reporting period. The program does

not prescribe a specific method to monitor and evaluate this progress but rather encourages

teams to use tools from different approaches as they see fit, taking advantage of their

comfort with certain tools that they have used in the past. This workshop aims at helping

teams to enrich their current M&E approach with tools that could help them better

address specific challenges of participatory action research and climate change adaptation.

Teams can learn by interacting with other teams and can take advantage of the program’s

and trainers’ experience in supporting previous projects.

Objectives of the workshop

for team representatives to exchange on experiences in their current M&E practice and reflect on how these can be improved

to introduce team representatives to M&E tools which help address specific challenges of evaluating adaptation to climate change and participatory action research

to provide team representatives with training materials which they can use to relay workshop findings to their colleagues

Methodology of the workshop


11 December 2009

4

The approach of the workshop will be interactive. One representative of each of up to 25

project components will be invited to participate in the workshop. The project will include

Francophone and Anglophone participants. There will be lectures in plenary, with

simultaneous interpretation, on basic principles of M&E as well as on the basic principles of

the tools to be practiced. During about 60% of the time, participants will be split in language

groups (one Francophone, the other Anglophone) to work on the application of the tools to

their projects, for peer consultations or to work on group exercises with facilitators speaking

their language. Some tools such as process documentation, outcome journals and situation

evaluation will be used by project participants to monitor and evaluate each day of the

workshop. Participants using these tools to evaluate the workshop will rotate in such a way

that all participants will have used all tools by the end of the week.

Participants will receive, at the beginning of the workshop, a printed version of the bilingual

workshop guide including outlines of the presentations and instructions for exercises. A

CDROM with workshop results and training materials in English and French will be provided

to participants at the end of so they can relay the lessons learned to their teams. It will

include powerpoint presentations, examples of tool use in existing projects, exercise

instructions, and complementary documentation.

About the tools to be discussed and used in the workshop

Experience with ongoing CCAA projects shows that teams have no difficulty describing their

activities and outputs in their interim technical reports. However, they find it much more

difficult to describe the outcomes of their activities, i.e. the effects that they are having on

the partners with whom they are working. Adaptation often unfolds as changes of practices

or of ways of doing things, and these changes are often difficult to document using more

classical tools, such as the ones included in widely used results based management

approaches.

This difficulty has motivated the program to use Outcome Mapping to help teams anticipate

the types of outcomes that they expect and/or desire and, when possible, to plan their

activities to reach these outcomes more effectively. However, Outcome Mapping cannot be

used in isolation for all the M&E tasks of a participatory action research project on climate

change adaptation. One of the functions that it lacks is the definition of state indicators

related to biophysical and socioeconomic conditions to use in diagnoses, baselines and

subsequent evaluations.

Some tools of outcome mapping can be combined with other tools of results-based

management to develop a framework for project-level monitoring. For example, one can


11 December 2009

5

develop a results chain where the outcomes are enriched using principles of outcome

mapping. Some other tools exist for participatory diagnosis, and that can serve as an

introduction to participatory action planning as well as participatory M&E in a way that is

coherent with methods used at the project level. These approaches are used at the stage of

conceptualizing change, for defining aims, understanding the current situation and what

needs to be done by whom in order for the project to reach its goal.

The actual monitoring and evaluation during project implementation can be done through a

number of tools such as process documentation and outcome journals. The documentation

of outcomes during the life of a project can greatly benefit from story based approaches,

including the Most Significant Change approach, and by the acquisition of testimonial

narratives. In summary, the program would like to introduce the team representatives to

the following tools:

Tools for conceptualizing change (defining aims, understanding the current situation and

what should be done by whom to reach the goal)

At the project level

The results chain (a tool of results based management)

Identifying boundary partners and progress markers (tools of the Outcome Mapping approach, that can help enrich the results chain)

Using visioning related to the project goal, to define indicators of state that could be used for diagnosis and subsequent evaluations.

At the level of participatory action research groups

Visions-actions-partnerships, integrating

participatory analysis of the components of risk

Tools for monitoring

Process documentation (a tool from the participatory action research field guide)

Outcome journals (a tool from outcome mapping, adjusted by CCAA)

Observation or measurement of indicators of state (the situation barometer)

Testimonial narratives

Most significant change

Preliminary program

Monday December 7th


11 December 2009

6

Boarding bus for Gorée boat terminal at 9:15

Boat for Gorée at 10:00

Start of activities: 11:00

- Welcome address - Presentation of participants and their expectations - Presentation by CCAA about M&E in the program - Presentation of results of questionnaires sent to participants - Exercise 1 in small groups. Peer learning about problems encountered until now and

useful practices found by the teams - Presentation of exercise results in plenary - Lecture : M&E basic concepts. Overview of the tools to be used during the week. - Review of M&E tools that will be used to monitor and evaluate the workshop on a

day to day basis End of activities, 18:00

Tuesday December 8th: Fleshing out outcomes with outcome mapping


- Presentation by participants of the evaluation of the preceding day - Lecture: The results chain - Exercise 2: Constructing a results chain from the elements of your proposal:

activities, outputs, specific objectives/outcomes, general objective - Lecture: basic concepts of outcome mapping; example from Nigeria of how they can

be used to enrich a results chain - Exercise 3: defining boundary partners, key outcomes and progress markers;

integrating them in your results chain End of activities, 17:00

Wednesday December 9th How to tie this together in your project


- Presentation by participants of the evaluation of the preceding day - Using outcome journals in your project. Presentation of an example from Benin - How to include your M&E information in interim technical reports - Exercise 4 in small groups: tell colleagues about your project’s progress in terms of

activities, outputs and outcomes - Lunch at 13:00

Free afternoon (possibility of taking the boat to Dakar at 14:00 and 15:00)


11 December 2009

7

Thursday, December 10th: Participatory approaches


- Presentation by participants of the evaluation of the preceding day - Lecture on the Visions-Actions-Partnership approach and participatory analysis of

components of risk - Exercise 5 in small groups: Simulation of a Visions-Actions-Partnerships exercise by a

local participatory action research group in your projects, integrating a participatory analysis of climate risks.

- Exercise 6: Indicators of state or of the situation for your project. End of activities, 17:00

Friday December 11th: Testimonial narratives and story-based approaches


- Presentation by participants of the evaluation of the preceding day - Lecture on how to acquire testimonial narratives to support the documentation of

progress markers and on the most significant change approach - Exercise 7 in small groups: Using the Most significant change approach about our

experiences during the workshop - Evaluation of the workshop - Closure at 15:30

Return to Dakar with boat leaving at 16:30, minibus to Novotel.

2. M&E PRACTICE AND CONSTRAINTS WITHIN

PROJECTS

An evaluation questionnaire was sent to all participants in an effort to better understand

their monitoring and evaluation practices and the constraints faced in their project activities.

The results of the questionnaire will be compiled by the facilitators of the workshop and

exhibited in plenary at the beginning of the workshop. Discussions will follow resulting in

M&E practice related conclusions. The issues raised during the discussions of practices could

be dealt with in the following exercise to facilitate peer learning.

Exercise 1: Peer Learning;

In groups of 4 or 5, choose one or several M&E topics or functions some group members feel

uncomfortable with, in terms of their implementation and for which other members have


11 December 2009

8

solutions to propose. Discuss them and propose solutions depending on each participant’s

experience. Fill in the following table (1 hour).

Topic or Function Constraints faced Solutions Proposed

Feedback in Plenary:

The group reporters present in plenary the topic they have discussed and the conclusions and

recommendations they made. (1 hour)

3. SOME M&E KEY CONCEPTS

3.1 MONITORING

It is a continuous process of systematic data collection to provide managers and stakeholders

about of development initiatives with information about the progress made, the objectives

achieved, and the funds allocated. The monitoring data are used as the basis for evaluation

and learning from experience capitalization. It includes two distinct types:

- Administrative and financial monitoring in terms of activity implementation; and

- Technical monitoring dealing with progress in terms of field technical activities.

3.2 EVALUATION

It is a systematic and objective assessment of a project, programme, or policy in progress or

concluded, from design, implementation, to its results. The aim is to determine the

relevance and achievement of objectives, development efficiency, effectiveness, impact and

sustainability. An evaluation process should provide reliable and useful information

conducive to the integration of the lessons learned in the decision making processes of the

beneficiaries and donors. The word “evaluation” also designates a process as systematic and


11 December 2009

9

objective as possible in which the value and scope of a planned, ongoing or concluded

initiative.

3.3 MONITORING & EVALUATION

Monitoring & Evaluation (M&E) is a continuous monitoring process in which periodic

reflections are incorporated in an effort to evaluate progress, effectiveness, and relevance of

the activities being implemented. M&E is more effective if combined with planning

mechanisms in place in the group, project, or organization. In participatory action research,

participatory M&E helps collect research data. It will be often used to test adaptation

options or adaptation support mechanisms. M&E is most often managed by the group or

the project itself. It is most often a self evaluation and reflection process. Evaluations, on

the other hand, are most often done by external persons.

3.3.1 Result Based M&E

Change is monitored at three levels:

Within “us” the activities or processes, their direct outputs, and the new knowledge

generated;

Within the partners “we” want to influence : the project’s immediate or intermediate

outcomes;

In the environment or community: the ultimate results sometimes called impacts.

They can be monitored using indicators of state.

Results, targets, and indicators

It is important to distinguish results, targets, and indicators. Results can be

expressed in relatively general terms. (i.e.: knowledge improvement of slum

dwellers about emergency measures to take in case of flooding);

Targets are quantifiable levels of indicators that a project or group wants to reach at a specific time. When it is possible to determine them, they must be more precise, and “SMART”. The targets can even be graduated. The development of control data gives a starting point that can be compared with the performance target;

Indicators are quantitative or qualitative factors or variables that provide a

simple and reliable basis for assessing achievement, change or performance. (Ref. IFAD M&E Guide).


11 December 2009

10

In the evaluation process, one compares the value or quality of an indicator with the

corresponding target.

Example:

Result Indicators Control Data Targets

Improvement of country children access to pre-school curricula

Proportion of children eligible in urban areas enrolled in pre-school curricula

In 2008, 75 % of 3-5 year old children in urban areas are enrolled in pre-school curricula 2.

By 2013, 85 % of 3-5 year old children in urban areas are enrolled in pre-school curricula

Milestones, Progress markers, Indicators

While the word ‘Indicator’ is commonly used for all types of results,

It is suggested to use it specifically for ultimate results. They are status indicators ;

For outcomes, it is suggested to use the expression “progress markers”;

For activities and outputs, it is suggested to use the word “milestones”.

The Concept of SMART Targets

Specific, Measurable, Achievable, Realistic, and Time-bound

An example of a SMART target is « the bilharzias infection rate within the population of our

community is below 5% in one year to come”. The relevant indicator is the bilharzias

infection rate of the population in the community.

Contribution versus attribution: two ways of defining the word “Impact”

The word impact refers to positive or negative changes whether they are voluntary or not,

on individuals and their conditions of living, institutions, and the environment…

a) Which the project has contributed to;

b) Which are caused by the project

In the first case, changes that are produced and the project’s contribution in that respect can

simply be documented. This is called contribution analysis.


11 December 2009

11

In the second case, the portion of observed change caused by the project or by a particular

intervention is to be determined. It is called an attribution analysis.

The CCAA programme expects its projects to conduct contribution analyses. Moreover, the

research methodology of several of them requires an attribution analysis, for example to

define the effectiveness of a particular adaptive measure.

Attribution Analysis for Impact Evaluation

An attribution analysis requires a comparison between BEFORE and AFTER, and between

WITH and WITHOUT.

Therefore, data should be acquired on the reference situation before the intervention (this

dataset is usually called a baseline) as well as in groups who have not been subjected to the

intervention (these are called the control groups).

The following requirements are necessary for an attribution analysis to be possible:

Having a control group with comparable conditions (the intervention should be the

only thing distinguishing them);

The causal link between the intervention and the measured variables is simple and

direct;

There should be no external factor which visibly interferes with the variables being

observed.

The International Institute for Impact Studies and the World Bank encourage the use of

experimental methodologies whenever it is possible, where the “treatment” is given to

random subjects in an attempt to have an identical control group. This is called Random

Design. In development matters, ethical and organizational issues are raised about this type

of experimentation. There are quasi experimental methodologies in order to find relatively

comparable groups among those who do or do not undergo the intervention. However, in

development projects, causal links are usually quite complex, making attribution analysis

very difficult.

3.4 PARTICIPATORY ACTION RESEARCH (PAR)

3.4.1 Definition and Principles

PAR is based on the assumption that the participants should not only be partners in the

research process, but they should also feel it belongs to them. Participants have control over


11 December 2009

12

the research activities as their own programme of action. They regard the research activities

as a process which is to be incorporated in the daily solution of problems rather than

something needing external initiation. Although the external researcher often has a role to

play, this role often remains peripheral. Participatory action research includes three items

which are useful to the process and the results:

- Participatory methodologies;

- Researchers who are also stakeholders ; and

- The praxis which associates reflection and action.

The strength of participatory action research lies within its faculty to positively influence practice, while systematically collecting data. It is devoted to encouraging voluntary changes. Stakeholders in the group carry out research and interact amongst themselves. Systematic feedback can help evaluate data accuracy and adjust the process over the time.

3.4.2 Participatory Action Research in the Climate Change Adaptation Programme

There are two parallel processes:

Participatory action research (PAR) as such, carried out by PAR groups with the

support of researchers, designed to respond to questions posed by the group

members, and which are relevant to the community represented by the group;

Research at the project level aiming at summarizing the knowledge generated in

the PAR groups and disseminating it to a larger audience including the scientific

community

The researchers involved in the project can also contribute to the PAR by bringing a

synthesis of the state of the art in this area, a support in terms of experimental design

and data analysis, if needed.

The Key stages of the PAR Cycle

Situation analysis ;

Conceptualization of Change ;

Planning ;

Management of Change (including monitoring, evaluation, and re-planning)

Cross-Cutting Tools (to be used throughout the stages)

Facilitation

Process Documentation.


11 December 2009

13

4. OUTLINE OF TOOLS TO BE SEEN DURING THE

WEEK

The interim technical report is the only mandatory tool discussed in this workshop. All other

tools are presented to facilitate reflection and research. Their use is optional; they can be

adapted and modified or teams can use other tools that have the same purpose. The writing

of the interim report requires describing activities, outputs and outcomes. The following

tools will help understand the relationship between these, to document outcomes and to

collect data for their research.

4.1 THE RESULTS CHAIN

The results chain is one of the tools used in Results Based Management (RBM). It describes

the causal and logical links between inputs, activities, outputs, outcomes and final outcomes

in a project. It indicates the route from a current situation to a vision of change to be

achieved. The approach requires reflection about the best ways and strategies to adopt in

order to achieve those targeted changes.

4.2 OUTCOME MAPPING It is an integrated planning and M&E approach. It conceptualises outcomes as changes in

behaviours, relationships and practices of key partners that the project interacts with

directly and aspires to influence. These partners are called boundary partners. This

methodology characterizes and evaluates the contribution of a project in the achievement of

these outcomes. It takes into account the larger context of development but concentrates its

assessments within the project’s sphere of influence. It is useful in climate change

adaptation research because adaptation often unfolds as changes in practices and

behaviour. It complements Results Based Management by facilitating the definition of

outcomes and their related progress markers. It also allows the identification of desired and

ideal outcomes that the project cannot hold itself accountable to but that could indicate that

tested options or mechanisms are effectively contributing to adaptation.

4.3 OUTCOME JOURNALS


11 December 2009

14

This is one of the tools included in the Outcome Mapping. The journals monitor changes based on the previously identified progress markers. They aim at describing the changes produced, how the boundary partners achieved them, the contributing factors, the lessons learned, all this in an effort to document the outcomes for future analysis or evaluation.

4.4 INTERIM TECHNICAL REPORTS

The interim technical reports are the main mechanism with which the CCAA programme can

compile research findings, assess products and outcomes, summarize the lessons learned,

and share them with the larger community of people interested in climate change

adaptation. Most of CCAA supported projects are requested to submit a progress report

every six months.

4.5 VISIONS-ACTIONS-PARTNERSHIPS (VAP)

The VAP approach aims at:

Providing a simple and intuitive exercise that CCAA teams can facilitate with the participatory action research groups with whom they work;

Stimulating ownership of the process by the participants through the expression of individual actions by each of them;

Helping define roles and responsibilities ;

The approach involves participants each expressing their vision, actions and desired

partnerships and then defining a common set of vision, actions and partnerships for the

group. The word Vision describes the desired conditions if the project or intervention was

successful. Actions indicate what each participant can do to contribute to the vision.

Partnerships address what participants need from other persons or organizations for their

vision to be achieved (the partner and what is expected from him/her are identified).

4.6 PARTICIPATORY ANALYSIS OF COMPONENTS OF RISK

With the participatory analysis of the components of risk, the participants in a participatory

action research can debate, for each type of given climate hazards, about observed or

potential consequences, factors that increase their exposure, those that increase their


11 December 2009

15

sensitivity, those that can help them cope with the hazard, on actions that can be taken, and

on potential contributions from external stakeholders in an effort to mitigate risks. They can

use those results in the VAP methodology for a more effective definition of planned actions

and expected partnerships. They can also conduct this analysis at various stages of the

research process in order to see if there has been an evolution in the factors described or

the consequences observed.

4.7. COLLECTING OF TESTIMONIAL NARRATIVES

Testimonial narratives can provide evidence about the outcomes of a project or activity.

Testimonial narratives are qualitative data which help understand the significance of the

documented outcomes from the interviewee’s point of view. They are personal and

subjective. The person tells his/her experience in his/her value system. Testimonial

narratives can then be used for communication and knowledge sharing purposes.

4.8 THE MOST SIGNIFICANT CHANGES APPROACH (MSC)

The MSC is a methodology aims at identifying the most significant changes produced in a

community, following interventions, and which are documented in the form of narratives. It

is appropriate with organizations desiring to learn from their experiences. It is an approach

which can supplement other M&E approaches but does not replace them. A subjective

approach which informs on the values of those involved and indicates what makes a

difference for them. It also helps grasp uncommon changes and experiences which did or did

not work.

5. WORKSHOP EVALUATION TOOLS

For the workshop participants to become comfortable with some monitoring tools,

three of them will be used to monitor the workshop. They will be used rotatively by the

five teams whose composition will be decided during the workshop. The following table

shows the teams who will use the tools on each day of the workshop.

Tool Monday Tuesday Wednesday Thursday Friday

Process

Documentation

1 3 2 4 5


11 December 2009

16

Workshop

Outcome

journal

2 4 1 5 3

The Situation

Barometer

3 5 4 2 1

5.1 PROCESS DOCUMENTATION

Process documentation is one of the tools proposed in the Participatory Action Research

Field guide for CCAA programme (German et al., in development). It aims at generating

reflection about the process before and after intervention with the participatory action

research groups. To monitor the workshop, the team of participants responsible for this tool

for the day will interview the trainers and fill in the following table.

Before Objective: What do you want to achieve in general, and at

this stage of the process, in particular?

Approach: What will you do to reach this objective? What

measures will you take and why? Who will be involved at

each stage and why?

Process M&E Aspects: What will be observed, monitored,

and documented as the process moves on? Which

indictors will be used to evaluate progress?

After Approach: What did you do to reach that objective? Did you

modify your approach? If yes, how and why?

Success: What worked and why?

Challenges: What did not work? What were the obstacles,

and why did they occur?

Appreciation: What did you learn during the activity and

that you did not know before?

Resolutions: What decisions were taken by the

participants?

Lessons: What lessons or observations did you gain from

the experiences and which can be shared with those who

want to solve similar problems? What surprised you when

you discovered it (about the approach and the lessons

learned)?


11 December 2009

17

By next

time

Recommendations: What will you do in the same way or

differently next time? What can be done to overcome

obstacles encountered during implementation?

5.2 WORKSHOP OUTCOME JOURNAL

The Outcome journal monitors changes in terms of capacity, knowledge, perceptions of

trainers’ boundary partners, meaning the workshop participants. The teams in charge of

using the Journal are expected to define relevant progress markers for the day during which

monitoring is being carried out, and collect evidence to support their observations. The

evidence can be testimonial narratives by the participants, or results of a survey designed by

the team in charge and administered with their colleagues.

Workshop Specific Objective

Progress

Markers

What happened? Date Lessons

Learned/Correctiv

e measures

Evidence

1-For team representatives to exchange on experiences in their current M&E practice and reflect on how these can be improved

2-To introduce team representatives to M&E tools which help address specific challenges of evaluating adaptation to climate change and participatory action research

3- To provide team representatives with training materials which they can use to relay workshop findings to their colleagues


11 December 2009

18

5.3 SITUATION BAROMETER AND SPIDER WEB AS GRAPH REPRESENTATION TOOL

The team in charge of this tool will define parameters (Indicators of State) to be measured

during the day. To collect data, they can use surveys filled by a sample or the whole

population of participants. For each parameter, they will be marking from 1 to 5. They will

represent the data in a Spider Web graph.

Parameter

Level

1

2

3

4

5

The data can be represented in a spider web diagram (or « Radar » in Microsoft Excel) for a

better interpretation. For example, the following diagram would correspond with the

following table:

Day Level

Quality of Presentations

Participants’ Involvement

Understanding of M&E Inclusion in Technical reports

Comfort with M&E in PAR groups Fatigue

Tuesday 5 5 1 1 1

Wednesday 5 5 5 1 3

Thursday 5 5 5 4 1


11 December 2009

19

6. THE RESULTS CHAIN, A RESULT BASED MANAGEMENT

(RBM)TOOL

Result Based Management

RBM is primarily a philosophy and a management approach which stresses the achievement

of results in planning, implementation, and monitoring & evaluation. It is also an approach

suitable for clear description of the changes an organization or project wants to produce. Its

logic emphasizes learning and accountability through the initiative process.

RBM is justified by the requirements of a current context where:

- Project teams want to produce and deliver more effectively and more efficiently;

- Target groups, the public, and the civil society want improved services, better

transparency in management, and reports on project funds; and

- Donors want a more transparent, effective, and efficient use of aid resources.

Principles and Tools

0

1

2

3

4

5

Quality ofpresentations

Involvement ofparticipants

Und. aboutinclusion of M&E in

technical reports

Comfort with doingM&E with PAR

groups

FatigueTuesday

Wednesday

Thursday


11 December 2009

20

RBM assumes the principle that involvement of all partners throughout the

project/programme (design, planning, implementation, M&E of results), can help improve

relevance, effectiveness, and sustainability of development initiatives. It also expects the

commitment to results to guide the definition of project/programme activities and

management strategy. Other principles also characterize the approach

Continuous learning: project/programme implementation should follow an iterative and

gradual learning process ;

Partnership: perfect collaboration with all partners resulting in an agreement on results

to be achieved and on ways and means to do so;

Accountability: ensure a workplace where there is accountability for clearly defined

results shared by all partners.

The main RBM tool for result planning is the Results chain. Another widely used tool (that

we will not use in this workshop) is the logical framework matrix which describes, for each

stage of the results chain, some objectively verifiable indicators, means of verification,

assumptions and risks.

The results chain

The Results chain helps develop a critical reflection on moving from X to Y. It illustrates the

types of links between its various elements: causal link between inputs, activities, and

outputs; logical links between outputs and immediate outcomes; contribution links between

immediate, intermediate, and final outcomes over time.

Definitions:

Inputs: financial, material, human resources used to produce outputs from the activities.

Activities: action taken or work done in an effort to produce outputs

Outputs: direct products or services derived from a project activities

Development Results

Inputs Activities Outputs Immediate

Outcomes

Intermediate

Outcomes

Final

Outcomes


11 December 2009

21

Outcome: descriptive or measurable change produced by a causal relationship

Operational Results: immediate results of an activity (i.e. number of training sessions

conducted). They are produced on a short term basis;

Development Results: real changes in terms of human development produced by a

project. They are produced on a long term basis.

Immediate Outcomes: changes directly attributed to a project outputs. They are short

term results.

Intermediate Outcomes: changes that are logically expected once the immediate

results are reached. They are medium term results. Other words: outcomes, indirect

effects.

Final Outcomes: a higher level of change which can be expected following several

intermediate results. They are long term results. Other words: impacts (to be used with

caution, as seen earlier).

Exercise 2: Prepare a results chain from the elements of your proposal

From the elements of your proposal, define an initial version of your results chain. You will

later elaborate on the chain and integrate elements of other tools. Start from the specific

objectives of your project and try to translate them into immediate outcomes. Afterwards,

describe the intermediate outcomes that may derive from the immediate outcomes. Then,

you can use the general objective to define the final outcomes. Finally, define the inputs,

activities, and outputs which produced the immediate results. Try to fill in the following table

for two specific objectives of your project.


11 December 2009

22

Inputs Activities Outputs Immediate

Outcomes

Intermediate

Outcomes

“Effects,

outcomes”

Final

Outcomes

“Impacts”

Financial,

material,

and human

resources

used to

produce

outputs

from the

activities

Action

taken or

work done

in an effort

to produce

outputs

Direct

products or

services

from a

project

activities

Changes

directly

attributed to a

project

outputs

(Specific

Objectives)

Changes which

are logically

expected once the

immediate

outcomes are

reached

Higher level of

change that

may be

expected or

derived from

several

intermediate

results

7. OUTCOME MAPPING (OM)

7.1 THE OUTCOME MAPPING PHILOSOPHY

It is an integrated planning and M&E approach. It conceptualises outcomes as changes in

behaviours, relationships, and practices of key partners that the project interacts with

directly and aspires to influence. These partners are called “boundary partners”. This

methodology characterizes and evaluates the contribution of a project to the achievement

of the results. It takes into account the larger context of development but concentrates

assessments within the project’s sphere of influence.

OM is based on a specific type of results: outcomes as changes in behaviours. Outcomes are

defined as modifications of behaviours, relationships, activities, or actions of individuals,

groups, and organizations the project are in direct contact with. There is a logical link

between those outcomes and the project activities, but there is not necessarily a causal link

between the two of them. With these outcomes, one desires to help produce some aspects

of human well-being and liveable environment by providing partners with skills, resources,

and new tools helpful for their development process.


11 December 2009

23

- The project influence is critical from the establishment of inputs to the achievement

of outputs ;

- Afterwards, the role of the partners is crucial in the achievement of the targeted

change;

- Impacts are outcomes further to these changes;

- The role of the initiative or project is to clearly know how to define the desired

changes within the partners and determine their contribution to their achievement;

- Behaviour change is not a linear process as somme people could think;

- It is not produced through a direct causal link with an activity ;

- Rather, it derives from the interactions with the partners and their environment, who

achieve the conditions suitable for change

Relative Influences Along the Results

Chain

Low

High

Project Endogenous Stakeholders

Infl

uen

ce

Inputs

activities

Outputs

Outcomes Impacts


11 December 2009

24

The OM approach includes a series of tools. The use of the comprehensive approach implies

the following stages: definition of a vision, definition of the project mission, identification of

the boundary partners, definition of the outcome challenges, identification of progress

markers, definition of intervention strategies for each boundary partner, and definition of

the strategies to increase organizational performance. In the following paragraphs, we will

discuss only some of these stages which can complement the results chain and the project

conceptualisation included in CCAA research proposals. We will discuss concepts of vision,

boundary partners, outcome challenges and progress markers.

7.2 DEFINITIONS

7.2.1 Vision

The vision describes economic, political, social, or environmental changes that the program hopes to help bring about, as well as broad behavioural changes in key boundary partners. It represents the ideal that the program wants to support and should be sufficiently broad and inspirational to remain relevant over time, despite changing circumstances.

The vision is related to the program's objectives but goes deeper, is broader in scope, and is longer-term. The ultimate achievement of the vision lies beyond the program's capability; however, its activities should contribute to and facilitate that end. It is the program's contribution toward the vision (through its boundary partners) that will be measured in an evaluation — not whether the vision was achieved.

To have the team members describe the vision, the following questions can help : "what are

your dreams of success? What changes do you want to try to help bring about? Imagine the

context in three to five years when the program has been very successful: what would be

realizations should be achieved?"1

7.2.2 Boundary Partners

They are the individuals, groups, or organizations where the project desires to promote

change in an effort to help achieve its vision. They are called boundary (partners) because,

even if the project team has to work with them to produce the changes, they do not have

any control over those partners. The project is at the boundaries of their area and the

project team tries to facilitate the process by allowing them some access to resources, new

ideas or opportunities.

1 Earl S, Carden F and Smutylo (2001): Outcome Mapping, Building Learning and Reflection into Development

Programs. IDRC


11 December 2009

25

OM requires from the projects the appointment of identifiable partners whose behaviours

are expected to be influenced by the project or programme activities. Given the complexity

of the stakeholders within the project environment, the notion of boundary partners helps

differentiate those who are key for the achievement of essential results targeted by the

project from others. When the programme cannot directly influence a stakeholder, the

group has to know who they can influence who in turn will influence that stakeholder, who

will be included in the boundary partners. This will help the project team focus on the

project’s sphere of influence and adopt a larger vision.

To identify those partners, the project members should at first list the stakeholders they

think the project should work with in order to realize the vision. Then, the following

questions will help see who the boundary partners are:

Who does the project’s success mostly depend on?

Who does the project want to encourage change with in order to help realize the vision?

7.2.3 Outcome challenges

The outcome challenges or targeted outcomes describe the way behaviours, relationships,

activities or actions of individuals, groups or institutions will change if the programme is

successful. They are worded in order to show behaviour change. Those changes, even if they

are idealistic, should be somewhat realistic, for two reasons: the wording stresses the fact

that development is achieved by and for people, and that if the programme can influence

the realization of outcomes, it cannot have control over them. The project helps produce

changes, but at the end of the day, the boundary partners have the responsibility and power

to bring about those changes.

The outcome challenges are expected to show what the behaviours and relationships of the

stakeholder with other will be if the programme potential as a tool for change was fully

exploited. They help clarify, specify, and describe those changes. As behaviour changes

within groups and organizations cannot be considered isolated, many of them are presented

in a single wording of the targeted outcomes instead of writing distinct wordings.

To have the team members define the outcome challenges, they have to answer the

following questions: “Ideally, how different should the boundary partners’ actions be in

order to help realize the vision? What new relationships should have been built? How will

the existing relationships change?”


11 December 2009

26

7.2.4 Progress markers

They are a graduated series of descriptions illustrating the gradual levels of change a

boundary partner has to go through for the realization of the targeted vision. They indicate a

progression, from the minimum expected, as an ultimate reaction of the boundary partner

to the programme basic activities, to what one desires the partners would do if the

programme had a deep influence.

They are prioritized into three levels:

- Expected or “expect to see”: The progress expected from the partners, which could

be a reactive involvement of the boundary partner or the progress easily achievable;

- Desired, or “love to see” : The progress that could be expected from the partners in if

they had a more active involvement than what is expected in the project

- Ideal or “love to see” progress that could be the partners and which represents a real

transformation.

The differentiation of the three levels of markers helps differentiate and specify the

immediate and intermediate results in the results chain.

7.3 USE OF OM TO ENRICH THE RESULTS CHAIN

The results chain can be enriched by regarding the immediate outcomes as expected

changes for boundary partners. Intermediate outcomes can be identified as the desired or

ideal changes for the same partners. In this case, it is about changes derived from an

additional involvement of those partners with regards to what is expected within the

framework of the project. The intermediate results can also include changes in the boundary

partners of our boundary partners.

It is worth mentioning there is a key difference between the philosophy based on result

based management and the OM philosophy. The results identified in result based

management are all expected results; they need to be realistic. OM allows to complement

this by identifying desired or ideal results.and identifying them as such. Participatory action

research in climate change adapting often aims at testing the adaptation support

mechanisms; the fulfillment (or not) of the desired or ideal progress markers helps verify if

the support mechanisms are effective and eventually to identify the obstacles that will allow

to make adjustments to the tested mechanisms.

The following example was developed in October 2009 by NEST NGO in a project titled:

“Encouraging interactions between urban and rural populations in an effort to face climate


11 December 2009

27

change: an adapting experience in the town of Aba and its regions, South-East of Nigeria” led

by NEST NGO.

It is pointed out that the participatory action research groups the project is working with

have been identified as mechanisms of interaction with boundary partners of the project.

The project coordination team directly interacts with the participants of the groups, and

those groups interact with other boundary partners. It can be said that the participatory

action research groups are boundary partners of the project coordination team. The

participatory action research groups are part of the project, which is why they are identified

as interaction mechanisms rather than boundary partners of the project.

Workshop on Monitoring and Evaluation for Projects supported by the CCAA program. Gorée, 7-11 December 2009

28

Results chain of the project INCITING RURAL-URBAN INTERACTIONS TO COPE WITH CLIMATE CHANGE: AN ADAPTATION EXPERIMENT IN ABA AND

ITS REGION, SOUTHEASTERN NIGERIA.

Specific objective

Activity

(or stage)

Outputs

(deliverables by the project team)

Immediate or Direct Outcomes

“Expect-to-see” progress markers that the project is accountable for. These are related to the interaction mechanisms Critical assumptions: the interaction

mechanisms work Risks: lack of involvement of key stakeholders, political instability

Intermediate outcomes

“Like to see” and “love to see” progress markers

The project is not accountable to these

Critical assumptions: the identified stakeholders make

a significant contribution Risks: Lack of involvement of the identified stakeholders,

political instability

General objective, goal, or final outcomes (relating to the

project vision) Ideal situation. Indicators should be measured at the

beginning, during the project and at the end in involved

households, communities and the municipality

Objective 1: To assist the Committee of Supervisory Councilors of the municipal council of Aba to identify and evaluate cooperation strategies that would reduce rural – urban migration and urban flooding.

Stage 1.1: Participatory situation analysis of the rural-urban system

Output 1.1: Report of the situation analysis (including visioning, analysis of vulnerability and coping capacity, institutional analysis).

Outcome 1.1. Participants in Municipal/State multi-stakeholder reflection group aware of

existing and potential policy, institutional and material mechanisms that could reduce vulnerability of the urban-rural system.

City Council-

TBD during meetings State

-Agricultural extension services disseminates options and weather info in other communities -Use findings to develop policies and strengthen institutions -Include research findings in state plans

Municipal and State governments as well as community level associations in all of Nigeria are working together and are able to reduce the vulnerability of the rural – urban systems and of their components (communities and individual men and women) to climate change related threats. Possible indicators (measurable at household, village, local govt and higher levels): Exposure / sensitivity

-% of people living in flood prone areas -% of people living in eroded areas -% of income that can be lost in the eventuality of the

Stage 1.2. Participatory action planning and exploring of possible collaboration mechanisms

Output 1.2 Report on the participatory action planning process and on the analysis of different options considered

Outcome 1.2. Participating stakeholders collectively

agree on collaborative mechanisms to strengthen and to evaluate

Stage 1.3 Participatory evaluation of collaboration mechanisms

Output 1.3 : Report on the results of the experimentation process

Outcome 1.3 Participating stakeholders have

improved skills in implementing, monitoring and evaluation of changes and performance related to at last one implemented mechanism

Objective 2: To enable existing associations in selected

Stage2.1 Inception

Output 2.1 Compiled report of visits in the six communities and on the composition of the multi-stakeholder

Individuals and CSO

-Apply findings in their own activities -Replicate the options in other communities


29

rural and urban communities in and around the city of Aba to explore and evaluate strategies and policies to reduce their vulnerability and increase adaptive capacity to climate change

reflection groups -Conduct advocacy activities Local Governments

-Engage with communities in identifying solutions to climate related hazards -Provide institutional and infrastructural support to the communities adaptation initiatives -Include project results in their plans

considered climate hazard Lack of resilience

-Time that it takes for specific aspects (housing, production, etc) to get back to normal after a hazard has happened Migration (disaggregated by

gender) -% of work-capable people who have left the household for work -%of income from remittances Poverty

-% of people not having access to water at less than 15 minute walk -%of youth that has not completed primary school (disaggregated by gender) -% of people undernourished -%of people malnourished Coping mechanisms

-Level of early warning mechanisms -Level of compensation or palliative mechanisms (shelter, insurance, etc)

Stage 2.2: Participatory situation analysis,

Output 2.2 Compiled report on participatory situation analysis in the six communities,

Outcome 2.2 Participants in Reflection groups in six rural and urban communities are aware

of factors that make them more vulnerable to climate-related hazards and of existing and potential coping mechanisms

Stage 2.3 Participatory planning and exploring of adaptive options

Output 2.3 Compiled report on planning meetings in the six communities

Outcome 2.3. -Participating stakeholders have

improved skills in evaluating the feasibility and cost-benefit of options. -They link with resource persons who could help them to implement options.

Stage 2.4 Participatory testing of adaptive options

Output 2.4: Report on the process and results of the experimentation

Outcome 2.4. Participating stakeholders have

improved skills in implementing, monitoring and evaluation of changes and performance related to at last one implemented mechanism

Objective 3 Encourage sustainability and scaling up of the strategies through a wide dissemination of the outcomes and lessons learnt from action research.

Stage 3.1 Develop plan for knowledge sharing and

Output 3.1 An outline of the plan for lesson sharing and the actual mechanism

Scientific community

-Cite project results Municipal and State government , NGOs and Networks to whom the materials were sent are

implementing or promoting options promoted by project

Stage 3.2. Synthesis of observations, analyses of action research and lessons learnt

Output 3.2. detailed report on observations and synthesis of lessons learnt

Stage 3.3 Produce dissemination materials on lessons learnt, including scientific articles

Output 3.3 Dissemination materials (training modules, video documentary, radio scripts, scientific publications) produced

Outcome 3.3 Journals and book editors

accept publications


30

Stage 3.4 Disseminate materials and lessons learned , including with BNRCC stakeholder forum, and evaluate their potential uptake

Output 3.4: Report on the survey of persons who have received the dissemination materials

BNRCC National Stakeholder group

-Integrate lessons learned in the development of national strategy -Use findings of the project as a tool for their own activities Municipal and State government , NGOs and Networks to whom the materials were sent are aware of options promoted

by project


31

Diagram of boundary partners (BPs)

Community-

level

discussion

BNRCC National

stakeholder

forum (1)

Scientific journals and

book editors

Convene PAR

process

Submit articles

and book chapters

Send

dissemination

materials

Participate

Local govts Individuals and CSOs Municipal

council State

Scientific

community

State and

municipal

governmen

ts, NGOs,


11 December 2009

32

Exercise 3: For your project, define boundary partners, key immediate and intermediate

outcomes and relevant progress markers.

Instructions:

Define who are the boundary partners involved in the project. For two of them, develop an

outcome challenge, Define expected markers which correspond with the immediate results

and the desired and ideal markers corresponding with the intermediate results.

Boundary Partner

Outcome challenge

Expected Markers (immediate outcomes)

Desired Markers (intermediate outcomes)

Ideal Markers (intermediate outcomes)

Then, review your results chain and include the immediate or intermediate outcomes you

have defined if they are not already included.

8. OUTCOME JOURNALS

For progress monitoring, the outcome mapping encourages the use of an Outcomes journal for each boundary partner identified as key by project. It is also possible to have a journal for all boundary partners. The Journal includes the progress markers, a description of the level of change, and an indication about the boundary partners who achieved the changes. It helps describe the rationale of the change, the persons and circumstances, the evidences of change, why changes were not planned, and specify the lessons learned and develop a file of the context, having in view a future analysis or evaluation.

The progress markers that the journals are based on are not expected to describe in a fixed manner the way the process of change is developed, but rather the main steps of the progress towards the objective, meaning the realization of the outcomes. If the programme project team or the boundary partners think they reveal behavior change, activities, actions, or relationships described by the progress markers, this information should be recorded. The data collected about all progress markers should reflect the complexity of change in individuals, groups, or organizations. They should always be situated in their context and explained, for them to be useful.


11 December 2009

33

Progress markers help monitor trends in boundary partner behaviours and the discussions in that respect. Even if there is no causal link between the measures taken by the project team and the changes in the boundary partners, when compiling the information in the outcome journal, one can understand the reason why the measures do or do not influence the boundary partners. As a result of this information, the project team could improve the project’s performance and encourage its boundary partners to achieve extended changes. Moreover, a file of observed changes is created. The files can be combined on a regular basis so as to give the history of the influence and change, with regards to areas of interest or achievement.

The progress markers should not be regarded as untouchable during the monitoring period. If the programme officers do not observe changes in the boundary partners after sometime, they should wonder if the problem is in the progress markers or in the strategies used to promote change. Are the progress markers suitable (in other words, has the context of the boundary partners evolved into progress markers which no longer reflect the desired or needed change?). Should the programme do otherwise to encourage change? Have the boundary partners changed its orientation? If the progress markers are no longer appropriate indicators of change, they should be reviewed so as to reflect the new conditions.

Example: An extract from a project Outcomes journal (Source: IDID-ONG, Projet PARBCC ; Renforcement des capacités des acteurs ruraux béninois face aux changements climatiques. Octobre 2009). This extract is for the boundary partner (“Producers”)


34

OUTCOME JOURNAL

PL PL

Function

Progress

Markers

Statement (Who, How, What, Where) Date (When) Lessons learned/corrective

measures

Supporting

document

(name of file)

PRODU

CERS

Outcome Challenge : Producers plan their agricultural activities based on their weather agro information, apply conclusive

adaptive options, and share experiences within CCPAs.

Expect to see :

• Producers effectively compile the weather agro data

• Producers develop new agricultural calendars

• Producers apply conclusive options in their lands.

2nd four-month period of year 1 (August through December 2007)

Producers resort to project officers or to CeCPAs

for information about CC following the first radio

programmes and publication of the first 2

newsletters

December 2007

up to now

The benefiiaries call upon

us at local level through

isolated action (sensitization

of rainmakers, integration of

climate debates during

CCPA meetings). Request

for budgets by CCPAs to

increase and carry out other

sensitization activities.

Progress

Reports

3rd four-month period of Year 1 (January through May 2008)

Populations solicit sensitization sessions on

climate change (Aîfa and Avamè villages in the

"Communes" of Zè and Tori-Bossito)

March 2008 Progress

Reports


35

Producers accepted the phenologic observations

of crops on their land every 10 days.

Since April 2008 Weather Data

Collection

Sheets

following the programme about ASECNA

information about risks of crop flooding in the

south and in the north, producers carried out an

early harvest

Since May 2008 Progress

Report

Producers respond to facilitators' call for

information sharing sessions (ie: feedback on

asessment study on CC and endogenous adaptive

options)

Since May 2008 Progress

Reports

1st four-month period of Year 2 (May through August 2008)

Producers are interested in information included

in newsletters, particularly the practical advice

and apply them (ie: almost all farmers of the

Commune of Allada chose this year the short

cycle maize. Mature harvesting was precocious

for producers in the southern area upon

recommendation of the CCPA in order to avoid

the harmful effects of sudden flooding).

June 2008 Request for more

newsletters and extracts of

practical advice.

Progress

Reports

Producers are interested in farming experiments

and allocate plots of land no less than 400

sq/meter to PARBCC for school fields ("Champs

Ecoles Paysans" (05 of them for each Commune)

Since July 2008

up to now

Producers rely on farming

experiments in order to

develop their technical

capacities

Progress

Reports

Producers are eager to start the experiements. Since July 2008 Progress

Reports

The producers are in groups of 5 for the

facilitation of school fields

August 2008 Progress

Reports


36

2nd four-month period of Year 2(September through December 2008)

Producers enriched the contents of Newsletters

N°3 and 4 based on specific realities experienced

prior to their publication (at Sakété, the CCPA

pointed that apart from the maize, there is no

more compliance with the technical processes,

and the Council readapted it to the maize). Two

CCPAs in the Département des Collines rejected

the practical advice in Newsletter n°4 with regard

to maize and groundnuts early harvest and

pointed the fact that the first rains heralding the

harmattan have already fallen, and the advent of

the harmattan should be enjoyed for effective

drying of crops

September

through

November 2008

Stakeholders have accurate

knowledge about the

climate evolution, and

which need to be enhanced

Progress

Reports

Producers observe on a weekly basis the CEP

plots, they fill the simplified data collection

sheets on the status of crops and share the

vegetative behaviours of crops based on various

vegetative stages

September

through

December 2008

Experience sharing between

producers are conducive to

adaptive development

Progress

Reports,

current

capacities of

farmers

Farmers identify with the CEPs capacity building

needs: role and more effective management of

organic matter on the land

November/2008

Farmers improve the gathering shared analysis

capacity about an issue in order to identify a

relevant solution

November through December 2008

3rd four-month period of Year 2 (December 08 through April 2009)


37

Farmers in Ouidah (Fonkounmè) and

Klouékanmè (Tokanmè Aliho) adopted the soil

mulching option in their own lands as early as

this agricultural campaign

April 2009 The producers followed

with interest the first

farming experiments

Report on

first farming

experiments

1st half of Year 3 (May 09 to October 09)

The producers followed with interest all the

phases of implementation of the second

experiment year of climate hazard management

practices

Avril à Octobre

09

Search for CC adaptive

practices is an ongoing

concern for producers

Year 2

farming

experimentati

on reports

The producers implement and monitor the

experiemented activity options in the 12 target

communes; 3 options are implemented in each

commune.

August - Sept. 09

The producers analyze, evaluate and note every

15 days the performance of the options

implemented about maize farming

April through September 09

The producers carry out land mulching or

dumping of harvest scraps which used to be burnt

Throughout the

agricultural

campaign

Producers are aware of the

importance of

mineralization, on the spot,

of the scarps and weeds

more useful to the soils than

to the burnt-over area

Field

findings,

sharing

sessions


38

Producers in Kouzounmè and Gbeffadji in the

Commune of Kpomassè have chosen to practice

angola pea based headgrow farming during the

low season

October 2009 Conclusive options are

being adopted

August 2009

participatory

evaluation

report

Some producers in Dogbo have taken into

account the advice in Newsletter n°9 and save

their crops with early harvest while the other

were surprised by flooding following the Mono

river overflow

August 2009 The producers recognize the

relevance of the practical

advice in newsletters

Field findings

Like to see:

• Producers develop ownership of the technologic packages

• producers imporve the agricultural skills

3rd four-month period of Year 1 (January through May 2008)

Some producers have already changed their

agricultural practices such as the adoption of

ridging (a producer in Klouékanmè started this

type of ridging following the advice of PARBCC

facilitators).

Since March

2008

Field findings



39

Farmers in Ahozon in the Commune of Ouidah

introduced a modification of the Zay option

practice: instead of holes between the one way

ploughing hills, they dig the holes between the

hills on the lines but dig longitudinal trenches

between the lines and fill them with organic

matter. At the following season, the lines replace

the trenches and vice versa. An added value for

them: the trenches filled with organic matter

mobilize water, facilitate its infiltration and

gradual use in plants. Moreover, at the following

season, crops benefitted from the organic matter

in the trenches before receiving the seeds.

avr-09 The farmers develop

ownership of the conclusive

options et somewhat adopt

them

Field findings

Sharing with

farmers

involved

Love to see :

• Producers in search for agro weather forecasts

• Producers disseminate the conclusive options


The (Toviklin CCPA member) producers plan to visit the various arrondissements of the Commune to update their peers about the practical advice. Mr ….., who is member of Toviklin Comune Union is now disseminating the advice to his colleagues.

February 2009 Since February 2009 and is in progress

It is observed that producers are gradually interested in practical advice

CCPA meeting minutes (April 2009)


40

Producer in the Commune of Gogounou ask the projet if they can disseminate the practical advice and other sensitization messages initiated by CCPA in the municipal radio programmes of FM NONSINA in Bembèrèkè (but it is up to PARBCC to accommodate the transportation fees and premium to the communicators

It is observed that producers

are gradually interested in

practical advice

Minutes of

session with

CCPA in

Gogounou.

Area report

In the Plateau and Atlantic plateaux, producers desired to have for the low season the calendar of seed sowing for each culture

July 2009 Producers are in search for

advice about seed planting

dates

Monthly

report of

Plateau and

Atlantic

facilitators

(July 09)

The Pandri pilot producer has disseminated and trained his group members on the option maize headgerow farming of gliricidia and asked the projet where they can get gliricidia seed plants

September 2009 The producers have strarted to disseminate relevant adaptive options

Mission report, September 2009


11 December 2009

41

9. INTERIM TECHNICAL REPORTS

Below is a note for project leaders. It summarizes the guidelines for progress technical

reports and explains how the M&E results can be included:

Dear project leader,

Below you will find important information regarding reporting requirements for projects

funded by DFID and IDRC under the CCAA programme. It is important that you read this

information carefully.

1. Interim technical reports

These reports are the main mechanism through which the program can compile research

findings, outputs and outcomes, make syntheses of lessons learned, and share interim

lessons with the wider climate change community. Most CCAA projects are required to

provide an interim report every six months, with a few on shorter or longer reporting cycles.

In order to make this as straightforward a task as possible, we wish to remind you of the

guidelines and requirements for these reports.

Guidelines for preparing interim technical reports can be found in annex 1 of the document

“Grants to Institutions: A Guide to Administrative Procedures”. The full document can be

found at http://www.idrc.ca/en/ev-57093-201-1-DO_TOPIC.html, or you can go straight to

the Annex at: http://www.idrc.ca/en/ev-57097-201-1-DO_TOPIC.html

Reports should be quite short, to a maximum of 20 pages. They should contain the following

items:

Title page and table of contents Synthesis (half-page to one-page) Research problem (short reminder, flag any changes made since the proposal) Summary of research findings Project implementation and management issues (including progress relative to

milestones and/or specific objectives) Project outputs and dissemination Outcomes and impact Recommendations

2. Reporting on activities and outputs

http://www.idrc.ca/en/ev-57093-201-1-DO_TOPIC.html



11 December 2009

42

Regarding activities, your interim report should summarise the following information (as

relevant to the reporting period):

which methodologies were used to evaluate vulnerability and coping or adaptive capacity with your partners

how different adaptation options were considered, developed and/or tested.

how the project team has engaged with vulnerable groups and/or policy makers

how different forms of information and knowledge were shared

If you have produced any research outputs, their full reference should be given (authors,

date, title, place, number of pages, etc) and you should send copies of them with your

technical report. If they are available to the public at a website, please indicate the link so

that we can further disseminate it. Please note that workshop reports are considered to be

research outputs. You can find a template for the title page and abstract for research

outputs at the following address, or in Annex 2 of the previously mentioned guide to

administrative procedures, http://www.idrc.ca/en/ev-134015-201-1-DO_TOPIC.html.

3. Reporting on outcomes

Wee want to share with you the following advice, which we hope will help you in the process

of compiling information on outcomes for these reports.

You can describe your project outcomes as accomplishments, changes of practices,

relationships or behaviours of partners or stakeholders with whom your project interacts.

The program management team is specifically interested in reading about:

stakeholders being better able to evaluate the factors that make them more vulnerable and others that help them cope with climate change;

active exchange and learning between researchers, vulnerable groups and policymakers; at-risk groups or individuals developing, testing and/or adopting adaptation options; and policy-makers requesting and/or using information about climate change or

vulnerability, taking climate into account into their decisions.

Project outcomes can also include changes in biophysical or socio-economic conditions.

Reductions in vulnerability can sometimes be evidenced by consequences of a threat being

less severe than they were before, or by the range of conditions that a system can endure

being wider. They can also sometimes be related to certain assets such as food reserves,

shelters, accessibility, etc. We encourage you to provide evidence of these changes, when

relevant, through measurements, surveys, testimonials and reports of meetings with

stakeholders. This evidence does not need to be in your interim reports but your report

should indicate where it can be found.



11 December 2009

43

In addition to this, anecdotal stories can be extremely useful to convey the meaning of these

changes for your partners. We encourage you to report on the most significant changes

observed by your project in the reporting period. In turn, to obtain material to report, you

could ask the groups that you work with to describe the most significant changes that they

observed.

Many thanks for your attention to these matters!

Best regards,

The CCAA program management team

Exercise 4: Simulating a technical report through conversations. In small groups, tell your

colleagues about the progress made in your project in terms of activities, outputs, outcomes,

new knowledge generated, and lessons learned. If your project is recent, give the expected

results. A member of the group is expected to take notes and be the reporter.

10. VISIONS-ACTIONS-PARTNERSHIPS

The Nested Visions-Actions-Partnerships

This approach is presented as an introductory exercise to M&E

Why this approach?

To provide a simple and intuitive exercise that CCAA project teams can conduct with

participatory action research groups

To facilitate ownership in participants through the expression of possible individual

actions

To help determine roles and responsibilities

It can be used

To feed into other M&E approaches (eg. OM or Results Based Management)


11 December 2009

44

To develop a simple participatory M&E system without using these methods

To help conceptualise change, a step in Participatory Action Research (PAR)

Who can conduct this exercise?

Members of PAR groups that the project works with

Members of the project team, taking into account the visions, actions and

partnerships expressed by the groups

The project can use it to consult all its stakeholders to learn about their expectations

and potential roles

The terms:

Vision: description of the desired situation if the project were extremely successful.

Not a statement, can be a list.

Actions: what each participant can do to contribute to that vision

Partnerships: what each participant needs from other persons or organisations in

order for the vision to fulfill

How to implement it

Flexible, can be done with cards, on paper or orally

Participants write their individual Visions, Actions and sought Partnerships

The group develops a common Vision, Actions, Partnerships from the individual

contributions, but now thinking of the role of the group

From the individual to the group and from the group to the project

Process of harmonising of visions, and having the actions of some actors correspond

to the expectations of others

Definition of roles

o Of individuals within a group

o Of groups, researchers and other actors within a project

Identification of partners that we hope to influence (partners of the PAR group can

include the research team)


11 December 2009

45

Variants in the implementation (in function of time available, etc…)

The common vision can be discussed in group before the members define their

actions and partnerships

A diagnosis can be embedded, where present conditions are described in function of

the vision. Risk analysis can be done in the scope of this diagnosis.

How to use results in M&E

The vision can be used to define the principal objective of the project and indicators

of state

Actions can be used to define activities or strategies, as well as milestones to

measure progress of execution

Partnerships can be used to define boundary partners and outcomes.

11. PARTICIPATORY ANALYSIS OF THE COMPONENTS OF

RISK

It is important to describe the factors affecting risk, in order to be able to track changes in

them. Risk, the probability that a negative consequence will occur as a result of a change, is

a function of the hazard, the probability that the change itself with take place, and of

vulnerability. Vulnerability, in itself, is a function of exposure, sensitivity and the capacity to

cope with the change or the hazard considered.

In other words:

Risk = probability of a given undesired outcome following a given hazard

Risk = Hazard x Vulnerability

Where Hazard is the probability of occurrence of a given hazard

Vulnerability = f(Exposure, Sensitivity, 1/Capacity)

So

Risk = f(Hazard, Exposure, Sensitivity, 1/Capacity)

Capacity to what and of whom?


11 December 2009

46

IPCC, in its definition of vulnerability, speaks of “adaptive capacity”. However, it

could be more useful to speak of the “capacity to cope with the specified hazard”

Depending on the risk considered, social institutions can contribute to the capacity of

individuals.

Why speak of risk rather than vulnerability?

The word vulnerability has different meanings in different circles

Analysis of risk has the advantage of being more specific because users need to

specify the hazard and the risk they want to avoid

The adaptation process

Diminishes risk by

o Reducing vulnerability

o Increasing coping capacity

We assume that we cannot influence the hazard

Adaptive capacity = capacity to reduce exposure, sensitivity or to increase capacity to

cope with the hazard

Application in a participatory diagnosis

The objective is not to quantify risk

How to describe components of risk that will allow us to evaluate if there has been a

change over time

Can allow the identification of possible actions and external contributions

This analysis can be done for different hazards that are obstacles to attaining the

vision

How do we know if there has been adaptation?

When a given threat occurs, the consequences are less severe than they were before

“we” have reduced our exposure to the hazard

we have reduced the factors that make us more sensitive

We have increased the factors that allow us to cope with the hazards


11 December 2009

47

External contributions

Higher up administrative levels or actors external to the group can help it reduce

exposure/sensitivity and increase coping capacity

Reflecting on external contributions can help the group to identify the partners it

wishes to influence

This can allow the project team to identify research topics or requests to convey to

policy-makers

The following table presents, for illustration purposes, a fictitious example for a disfavoured

urban community

Haz

ard

s

Risks,

observed

consequence

s

Exposure Sensitivity Capacity To be done

by us

External

contributions

Flo

od

s (e

very

yea

r d

uri

ng

rain

y se

aso

n)

Damaging of

houses by

water

Houses in

low-lying

areas are

more

exposed

Houses made

of dirt bricks

are more

sensitive

Houses on

stilts are less

affected.

Those living

in two-story

houses can

move their

furniture

upstairs

Prevent new

people from

settling in

low-lying

areas

The city could

build

drainages and

execute their

relocation

program

Increase of

malaria

because of

increase of

number of

mosquitoes

that bree in

standing

water (30

cases this

year, 7 fatal)

Everybody is

exposed

because

mosquitoes

move around

freely

-Children,

pregnant

women and

elderly are

more

sensitive

-Some people

are not aware

of the illness

and do not

consult the

clinic

-Some people

have

mosquito

nets

-Those who

consult the

clinic are

treated

-Use

mosquito

nets (for

those who

can afford

them

-Organise

sensitization

meetings

-Drainage by

the city

-Distribution

of mosquito

nets

-Distribution

of medication


11 December 2009

48

Vio

len

t w

ind

s d

uri

ng

har

mat

an Damage or

destruction

of slum

houses (each

year, one in

10 houses)

Houses

closest to the

coast are

most exposed

Huts made of

metal sheets

or cardboard

are most

sensitive

The

inhabitants of

these huts

reconstruct

them quickly

but their

personal

effects are

damaged

Improve

constructions

Social

housing

programs

Exercise 5: Role play with Visions-Actions-Partnerships, integrating a participatory analysis of

components of risk

Scenario:

A project team works with a multi-stakeholder participatory-action research in a locality

During the participatory diagnosis stage, and to explore possible actions, the team facilitates

an exercise of visions-Actions Partnerships as well as an analysis of components of risk

around one hazard considered as the most important in this locality.

Instructions:

Form small groups of the same language around a table

Agree on the type environment of the locality and the hazard to be considered

Each member of the group represents a different stakeholder, including a facilitator.

Ensure that at-risk individuals and decision-makers are represented.

Each person writes on a piece of paper

Vision: His/her individual description of ideal conditions

Actions: What he or she can do to contribute to the vision

Partnerships: What he or she needs from other persons or organizations for the

vision to be possible (identify the partners and what is expected from him/her)

Then…

Each person reads his/her answers to the rest of the small group

The group tries to identify complementarities, within the group, between the actions

of some members and the needs of others


11 December 2009

49

Develop the following table for the participatory action-research group, identifying

the partnerships and expectations towards actors outside the group

Vision Actions Partnerships

Integrating participatory analysis of the components of risk into the exercise:

Fill the following table for at least one hazard identified by the group, identifying actions by

the group and possible contributions from external actors

Haz

ard

s

Risks,

observed

consequen

ces

Exposure Sensitivity Capacity To be done

by the group

External

contributions

Review the Visions-Actions-Partnerships table, in particular the columns relative to actions

and partnerships, to integrate new elements that might have come out during the analysis of

components of risk.

Reflect on the tools:

Have you found both tools to be useful? Could one of them have been enough on its

own?

Have you found it useful to make all participants talk systematically?

Have you been able to identify roles and responsibilities within the group?


11 December 2009

50

Has the diagnosis of the components of risk allowed you to identify actions or

partnerships that you had not thought about previously?

Can the elements written in the columns on the components of risk help you to

identify indicators to track the adaptation process?

Exercise 6: Develop indicators of state or of the situation for your project.

Use a vision of the project goal to define indicators of state that could be used for an initial

assessment and subsequent evaluations.

Refine the vision related to your general objective incorporating elements of the risk analysis,

as you imagine it (or have seen it) done by the participatory action research groups you work

with. If you aim to help some actors develop their capacity to adapt or to cope with certain

hazards, or to help them reduce their vulnerability, how would you see that changes have

occurred?

12. COLLECTING TESTIMONIAL NARRATIVES

Acquiring testimonial narratives is one way of collecting evidence of the outcomes of a

project or activity. Surveys are another way of doing it. If open-ended questions are

included, surveys can also help collect testimonies. However, if a survey is conducted with a

confidentiality mention, the interviewees’ identities should not be revealed when quoting

their answers.

Testimonial narratives are qualitative information which help understand the documented

outcomes from the point of view of the interviewee. They are personal and subjective. The

person tells his/her experience in his/her value system.

The narratives can then be used for communication and knowledge sharing purposes. For

example, a testimony of a female farmer who speaks about how she managed to implement

an adaptive option could help transmit knowledge to another female farmer, better than by

any other means.

Distinct M&E and Communication objectives

There should be a distinction between the acquisition of testimonial narratives for M&E

purposes and the use of testimonial narratives for communication purposes. The

communication objectives, particularly those of public relations, should not bias the

collection of testimonies. These should help collect evidence about all sorts of effects,


11 December 2009

51

successes, and failures, on the favourable and unfavourable opinions in a way to inform the

adjustments in planning or future planning and to contribute to the research process . They

are raw data which are not interpreted, and which will be transcribed word for word.

When developing communication products, the testimonial narratives that will help transmit

the desired message will be selected. Extracts can be taken, context elements added, and a

third person narration can be used to paraphrase what the person said, without altering the

meaning. Photographs of the interviewee can be added.

Under which form and where are testimonial narratives recorded?

The most manageable type of testimony is the written one. It can be recorded in trip

reports, meeting minutes, or workshop reports. These are formal and verifiable project

documents. It may be useful to take photographs of interviewees that can be used to

develop communication products in which the narratives can be used.

Testimonial narratives can also be in the forms of video or audio records. This format can be

challenging in terms of indexing and storing. It can be useful to transcribe them and

incorporate them in a report so that they can easily be consulted. Their added value is that

they can be incorporated in documentaries or radio programmes.

How are questions asked?

There are various ways of asking questions, depending on the type of information that is

needed. If a particular outcome is to be documented, questions should be asked in this

respect without manipulating the answers. Openness in encouraged in case the outcome is

not shown by the interviewee.

a) The narrative: questions can be asked on what triggered the action, its process, the

obstacles encountered, the supportive individuals, the lessons learned, and to

anticipate what may happen in the future.

An example (drawn from an experience of the project team working on : Vulnerability and

Adaptation to Climate Change in Agricultural Systems in Madagascar, led by the “Ecole

Supérieure des Sciences Agronomiques” which found that pepper growing is one of the

potential adapting options for clove producers. Indeed, clove trees are extremely sensitive

to cyclones.

What prompted you to grow pepper? (If he/she does not respond, ask him/her:

What was the situation like before?);


11 December 2009

52

How did you get the seed plants?

Are there people who helped you?

What obstacles did you encounter?

What lessons learned from your experience can be useful to other clove producers?

What has pepper growing changed in your everyday life?

How does it affect your hope for the future?

What are the potential challenges, constraints, and obstacles?

b) Appreciative Inquiry: one can ask what worked, and in which circumstances he/she

felt most satisfied in the documented activities;

c) The Most Significant Change: For example, what capacities have you developed

during the workshop that will be most useful to you?

13. THE MOST SIGNIFICANT CHANGE APPROACH (MSC)

MSC is a methodology that aims at identifying the most significant changes which occurred

in a community following interventions, and to document them in the form of narratives or

stories told by the beneficiaries. It is appropriate for organizations willing to learn from their

experiences. It can can complement other M&E approaches and without replacing them. It is

a subjective approach which provides information about the values of those involved,

indicating what is important for them. It also helps identify uncommon changes and

experiences which did or did not work well.

The methodology was developed by Rick Davis; it deals with M&E challenge in complex

participatory programmes in rural development. It comes from his PhD dissertation on

organizational learning within NGOs. MSC uses a selective sampling of the rich cases of

information in an effort to promote learning.

It is an emerging methodology, and various modifications were made during its application.

The MSC methodology suggested here includes ten steps.

A Guidebook was developed by Rick Davis and Jesse Dart (Davis an Dart, 2001). It proposes

the following steps:

Step 1: How to start and generate interest

Step 2: Define the areas of change


11 December 2009

53

Step 3: Define the reporting period

Step 4: Collect narratives of significant changes

Step 5: Choose narratives of the most significant changes

Step 6: Get feedback of the selection results

Step 7: Verify narratives

Step 8: Quantify

Step 9: Second order analysis and meta-evaluation

Step 10: Review the system

In CCAA projects, the approach can be used in participatory action research groups. Group

members are individually invited to submit a narrative about the most significant change

they have noticed in relation to their involvement in the project. The change can be about

them or about the partners the group wants to influence, if relevant. Then, the group can

select the narrative representing the most significant change for the group. The project

coordination team can compile the narratives selected by each of the groups and share them

with the other groups.

Exercise 7: Applying the Most Significant Change approach to your experience in the

workshop

In small groups of 4 or 5 (to obtain three small groups from the larger language group):

Each person presents a narrative of what has been, in his/her opinion, the most

significant change in terms of his/her perceptions of M&E, the understanding of

project, or capacities;

The group chooses, either by consensus or by vote if necessary, the most significant

change for the small group;

The reporter of the small group will present, in a plenary within the larger language

group, the selected narrative;

The larger language group will vote for the selection of the winning narrative;

The two winning narratives (French speaking and English speaking) are presented in

plenary with simultaneous translation.

Each participant is asked to submit his/her narrative to the workshop reporter for him/her to

compile and print them in order to be shown in the room for M&E purposes of the workshop.


11 December 2009

54

BIBLIOGRAPHY

ACDI (2008) : définitions révisées des principaux termes de la gestion axée sur les résultats.

Pour l’énoncé de principe 2008 sur la gestion axée sur les résultats. Direction générale des

politiques stratégiques et du rendement. ACDI

http://www.acdi-cida.gc.ca/inet/images.nsf/vLUImages/Results-

basedManagement/$file/GAR_ENONCE_PRINCIPE_2008_COMPANION.pdf

Beaulieu, N., Jaramillo, J., Leclerc, G. (2002). The vision-action-requests approach across

administrative levels: a methodological proposal for the strategic planning of rural development.

Internal report, CIAT/MTD, Cali/Montpellier, 30 p.

http://ciat-

library.ciat.cgiar.org/documentos_electronicos_ciat/Articulos_Ciat/report_manual_var_2002.pdf

Binnendijk, A.(200). Results Based Management in the Development Co-operation Agencies: A

Review of Experience, DAC Working Party on Aid Evaluation.

http://www.oecd.org/dataoecd/secure/14/29/31950852.pdf

Davies, R., and Dart, J. (2005). The ‘Most Significant Change’ (MSC) Technique: A Guide to its Use.

www.mande.co.uk/docs/MSCGuide.pdf

Earl S., Carden, F et Smutylo, T (2001) La Cartographie des Incidences. Intégrer l’apprentissage

et la réflexion dans les programmes de développement

http://web.idrc.ca/openebooks/962-3/

GIEC (2007). Contribution du Groupe de travail II au quatrième Rapport d’évaluation du Groupe

d’experts intergouvernemental sur l’évolution du climat. Résumé à l’intention des décideurs.

www.ipcc.ch/pdf/assessment-report/ar4/wg2/ar4-wg2-spm-fr.pdf

Projet Quartiers du Monde. Histoires urbaines (2003) Documents sur la recherche action

participative. http://www.quartiersdumonde.org/fra/qdm/info.php?id=67

http://www.acdi-cida.gc.ca/inet/images.nsf/vLUImages/Results-basedManagement/$file/GAR_ENONCE_PRINCIPE_2008_COMPANION.pdf

http://www.acdi-cida.gc.ca/inet/images.nsf/vLUImages/Results-basedManagement/$file/GAR_ENONCE_PRINCIPE_2008_COMPANION.pdf

http://ciat-library.ciat.cgiar.org/documentos_electronicos_ciat/Articulos_Ciat/report_manual_var_2002.pdf

http://ciat-library.ciat.cgiar.org/documentos_electronicos_ciat/Articulos_Ciat/report_manual_var_2002.pdf

http://www.oecd.org/dataoecd/secure/14/29/31950852.pdf

http://web.idrc.ca/openebooks/962-3/

http://www.quartiersdumonde.org/fra/qdm/info.php?id=67


11 December 2009

55

Secrétariat du Conseil du Trésor du Gouvernement du Canada (2001). Guide d’élaboration des

cadres de gestion et de responsabilisation axés sur les résultats. http://www.tbs-

sct.gc.ca/eval/pubs/RMAF-CGRR/RMAF_Guide_f.pdf

UNDHA (1992). Internationally agreed glossary of basic terms related to disaster management.

United Nations Department of Humanitarian Affairs, Geneva.

http://www.tbs-sct.gc.ca/eval/pubs/RMAF-CGRR/RMAF_Guide_f.pdf

