Designing Influential EvaluationsSession 3Evaluation Design

Uganda Evaluation Week - Pre-Conference Workshop19th and 20th May 2014

2

Training ObjectivesBy the end of the session,

participants will:Understand the use of

programme logic models and theory of change

Understand how to develop evaluation questions‘If you don't

know where you are going,

any road will get you there. ’

Lewis Caroll, author of ‘Alice in Wonderland’

3

LOGFRAME AND THEORY OF CHANGE

Logic models

4

Simple results chain

InputsActivitie

s/Process

Outputs Purpose

Goal

Physical & financialresources

Processes which turn Inputs into Outputs

Operational changes (new skills or abilities, the availability of new products and services).

Behavioural or institutional/changes in response to the outputs

Progress towards thematic priorities

Results

External perspective

Internal perspective

Management control diminishes

External factors become more important

5

A logframe is an ‘abstract’ of a project

Inputs Costs

The resources required

How much the project will cost

What the project will do

& deliver

Outputs

Activities

Goal(s)

Purpose

Objectives

What the project seeks

to achieve

Factors that may affect progress & success

Pre-conditions

Assumptions

How progress & success will be measured

What

will be

How

will it be

measured? measured?

Indicators Verification

6

The missing middle

Activities Outputs Purpose Goal

7

Response to outputs – the ‘miracle’!

Goal

Purpose

Outputs

Activities Inputs Costs

Objectives Indicators Verification Assumptions

Response to project servicesBehaviour or system change!

Critical conditions for

success

8

DiscussionTurn to your neighbour or work in

a small groupDiscuss for 5 minutes:

◦What are the common weaknesses in development and use of logframes?

Summarise 2-3 pointsBrief plenary discussion on

common themes

9

Analysis of DFID logframesDrew & Albone, 2008; Agulhas, 2007Problems: Only 18% of performance

indicators measured outcomes

64% measured ‘inputs’ and ‘processes’

Only 8% disaggregated data by gender

Indicators not specific and time bound

Measurement and the evidence base not well planned

Assumptions/risks not monitorable

Clear targets support better performance◦ Only 24% of projects

with a partially clear Purpose scored 1 or 2 at OPR

◦ 73% of projects with unclear Purpose scored <3

Statement

Fully clear

Partially clear Unclear

Goal 79% 14% 6%

Purpose 54% 35% 10%

Output 64% 29% 6%

10

Limitations in logframes

Simple staged structure

Linear logic with no scope for recursive change and

multiple pathways

Hard to cope with emergent

outcomes

Theory of change more

suited to complex

processes

11

What is ‘Theory of change’?

A theory of change is a set of statements about how change happens, and/or an attempt to describe underlying logic and

assumptions

12

Key elements of a theory of change

• Outline planned programme inputs, activities, outputs, expected outcomes and contribution to impact

• Set out the underlying logic of the

intervention specifying all key steps

• Identify assumptions and risks

• Explicitly set out potential contextual and external factors which might affect the outcomes

Potentially richer analysis that deals with logframe weaknesses

But no agreed

format or convention

13

Funds to Min of Ed.

School needs assessment

Curriculum development

BuildingsAttendance

Teachers

Decision to fund

Select teachers Mobilise PTA for building

Teachers attend training

Build/ rehabilitate schools

Teachers return to schools

Teachers apply new skills

Improved facilitiessupplemented by materials

& curriculum

Parents respond to improvementsand permit boys & girls to enrol

Boys & girlsrespond to

improved quality

Increase in teaching quality

Theory of change for a primary education project

Enrolmentimproves

Attendance improves

Retention improves

Boys & girls literacy rates

increase

Assume teachers can travel to training

Risk of teachersmoving schools

Assume trainingeffective

Assume Min of Ed funds books & materials

Assume children can be released from other tasks

Assume building is of satisfactory

quality

Assume selection on merit

Risk of children leaving for employment

or family duties

Assume no barriers to

children enrolling

Other contextual factors such as

roads & transport

Start

End

14

Discussion exercise

Working in small groups, consider how a Theory of Change approach might affect the way you plan your evaluation?

Prepare your ideas for presentation and discussion in plenary.

15

EVALUATION QUESTIONS

Theory of change & evaluation criteria

16

Formulating questions – from simple logframe for education

Funding & staffing

School building, teacher training & curriculum development

Improved access to & quality of schooling

Increased net enrolment

Increased literacy

Inputs Process Outputs

Outcomes Impact

1 2 3 4 5

1. What was the distribution of funding? How many staff were appointed?

2. How many schools received building projects? How many teachers were trained? How was the curriculum developed?

3. How much did access change? What aspects of quality were improved?

4. Did a change in net enrolment take place?

5. Did literacy rates improve?

17

Funds to Min of Ed.

School needs assessment

Curriculum development

BuildingsAttendance

Teachers

Decision to fund

Select teachers Mobilise PTA for building

Teachers attend training

Build/ rehabilitate schools

Teachers return to schools

Teachers apply new skills

Improved facilitiessupplemented by materials

& curriculum

Parents respond to improvementsand permit boys & girls to enrol

Boys & girlsrespond to

improved quality

Increase in teaching quality

Theory of change for a primary education project

Enrolmentimproves

Attendance improves

Retention improves

Boys & girls literacy rates

increase

Assume teachers can travel to training

Risk of teachersmoving schools

Assume trainingeffective

Assume Min of Ed funds books & materials

Assume children can be released from other tasks

Assume building is of satisfactory

quality

Assume selection on merit

Risk of children leaving for employment

or family duties

Assume no barriers to

children enrolling

Other contextual factors such as

roads & transport

Start

End

18

If … then … - the intervention logic

If the school needs assessment is effectiveThen the schools with the greatest need and potential will be chosen

If teachers are selected on merit Then those with potential will be put forward for training

If teachers are able to travelThen selected teachers will attend training

If teachers gain new skillsAnd do not leave for jobs in other locationsAnd facilities are improved by the building programmeAnd improved facilities are complemented by teaching materials and

the new curriculumThen teachers will apply new skills

If teachers will apply new skillsThen quality of teaching will improve

If parents respond to improvementsAnd children can be released from other tasksAnd there are no socio-cultural barriers to boys and girls enrolling

Then enrolment of boys and girls will increaseIf boys and girls respond to improved quality

Then attendance will increase

19

Hypotheses about education improvement

1• The needs assessment accurately identifies the

schools with the greatest need and potential

2 • Teacher training is effective at building skills

3• Trained teachers are motivated to remain at

their schools

4• Improvement in teaching quality is clearly

recognised by the local community

5• Parents prioritise education above other social

and economic factors

20

Formulating questions

Funding & staffing

School building, teacher training & curriculum development

Improved access to & quality of schooling

Increased net enrolment

Increased literacy

Inputs Process Outputs

Outcomes Impact

1 2 3 4 5

1. How were schools selected for the project; what was known about their enrolment and literacy status in advance?

2. Who was selected for teacher training; how effective was it; how many teachers were retained afterwards?

3. How much did access change and for who; what aspects of quality were improved and against what standard?

4. Did a change in net enrolment take place? Was it the result of the outputs delivered under this project or did other factors such as improved roads and transport, or demographic changes affect the outcome?

5. Did literacy rates improve? For both boys and girls? How widespread or consistent have these changes been. Do they reflect specific locations or a general trend?

21

OECD-DAC Evaluation criteria

Relevance

• The extent to which the objectives of a development intervention are consistent with beneficiaries’ requirements, country needs, global priorities and partners’ and donors’ policies.

Efficiency• A measure of how economically

resources/inputs (funds, expertise, time, etc.) are converted to results.

Effectiveness

• The extent to which the development intervention’s objectives were achieved, or are expected to be achieved, taking into account their relative importance.

Impact

• Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended.

Sustainability

• The continuation of benefits from a development intervention after major development assistance has been completed. The probability of continued long-term benefits. The resilience to risk of the net benefit flows over time.

22

Relate to evaluation criteria

Inputs Process Outputs Outcomes Impact

Funding & staffing

School building, teacher training & curriculum development

Improved access to & quality of schooling

Increased net enrolment

Increased literacy

Economy Efficiency Effectiveness Impact

23

Developing questionsFollow the

project logic

Explore the theory of change –

people’s response; validity of assumptions

Did results correspond with policy intentions?

Structure according to evaluation

criteria

Consider specific

questions such as VfM

24

Discussion exercise

Working in small groups, consider at what stage evaluation questions should be developed and whose responsibility that should be?

Prepare your ideas for presentation and discussion in plenary.

SummaryAnalyse the intervention logic

using a logic model such as the logframe or (better) theory of change

Develop evaluation questions from the details of the intervention logic

These provide the basis for considering an approach and methods

An evaluation framework can help link questions to approach and methods

END