betterevaluation - webinar number 2: define

Part 2 of 8 AEA Coffee Break Webinars 2013

Simon Hearn, ODI, London

DEFINE what is to be evaluated

The Rainbow Framework

Why do we need to start with a clear definition?

Source: Hobbies on a Budget / Flickr

Three evaluation tasks in DEFINING

Develop initial description

Develop program theory or

logic model

Develop initial description

1.

Thumbnail description

Vision & Mission

Peak Experience

Aims & Objectives

Develop program theory or logic model

2.

Options for representing logic models

Pipeline / results chain

Logical framework

Outcomes hierarchy / theory of change

Realist Matrix

Options for representing logic models

Pipeline / results chain

Logical framework

Outcomes hierarchy / theory of change

Realist Matrix

INPUTS• Apples • People at risk of

poor health

IMMEDIATE RESULTS• Apples eaten

SHORT-TERM RESULTS• Improved

nutritional status

LONGER-TERM RESULTS• Improved health

Options for representing logic models

Pipeline / results chain

Logical framework

Outcomes hierarchy / theory of change

Realist Matrix

RESULTS AREA INDICATOR MEANS OF VERIFICATION

ASSUMPTIONS/ RISKS

GOAL:Improved health

Decreased sick days

School and work records

Good status largely due to nutrition

PURPOSE:Improved nutritional

status

Adequate levels of

Vitamin CSample blood tests Apples retain

Vitamin C

OUTPUT: Apples eaten

Apples replaced by apple cores

Visual inspection Apples eaten on location

ACTIVITIES: Deliver apples

Required numbers delivered

Delivery receipts

ACTIVITIES: Buy apples

Required numbers bought

Purchase receiptsVisual inspection

Options for representing logic models

Pipeline / results chain

Logical framework

Outcomes hierarchy / theory of change

Realist Matrix

Outcomes chain showing possible alternative causal paths

Improved health

Improved vitamin C levels

Apples eaten (whole or juice)

Apples delivered to schools

Improved quercetin levels

Red apple skin eaten

Decreased Body Mass Index

Apples eaten instead of junk

food snacks

Options for representing logic models

Pipeline / results chain

Logical framework

Outcomes hierarchy / theory of change

Realist Matrix

Context Mechanism Outcome

Healthy people at high risk of gastric cancer

Quercetin from red apple skin (or red onion) protect cells from free radicals

Reduced risk of cancer

Smokers or people with regular exposure to radiation

Chemicals in cigarette smoke or radiation damages DNA in healthy cells

Continued increased risk of cancer

Individuals with vitamin deficiency through malnutrition

Sufficient vitamin C absorbed when juice from apples (or oranges) consumed

Stronger immune system

Individuals with vitamin deficiency through excessive alcohol

Alcohol destroys vitamin C No change in vitamin C level

Individuals with obesity due to excessive snacking on high fat, high calorie foods

Decreased calorie intake due to substitution of apples (or carrot sticks)

Reduced obesity and related conditions

Individuals with obesity due to genetic condition Decrease sugar intake through

substitution of applesNo change in obesity but lower risk of diabetes

Context Mechanism Outcome

Healthy people at high risk of gastric cancer

Quercetin from red apple skin (or red onion) protect cells from free radicals

Reduced risk of cancer

Smokers or people with regular exposure to radiation

Chemicals in cigarette smoke or radiation damages DNA in healthy cells

Continued increased risk of cancer

Individuals with vitamin deficiency through malnutrition

Sufficient vitamin C absorbed when juice from apples (or oranges) consumed

Stronger immune system

Individuals with vitamin deficiency through excessive alcohol

Alcohol destroys vitamin C No change in vitamin C level

Individuals with obesity due to excessive snacking on high fat, high calorie foods

Decreased calorie intake due to substitution of apples (or carrot sticks)

Reduced obesity and related conditions

Individuals with obesity due to genetic condition Decrease sugar intake through

substitution of applesNo change in obesity but lower risk of diabetes

http://betterevaluation.org/resource/example/rubrics-oakden

Identify potential unintended or negative results

3.

Source: www.smarttoolkit.net

Negative program theory

Risk assessment

Key informant interviews

Six Hats Thinking

Defining – Framing iterationInitial

description

Discuss evaluation purpose

Refine description

Define evaluation questions

Develop logic model

Agree scope of

evaluation

DEFINE

DEFINEDEFINE

FRAME

FRAME

FRAME

Simon HearnResearch Fellow, ODIs.hearn@odi.org.uk

http://betterevaluation.org/plan/definefacebook.com/betterevaluation@bettereval

SWOT Analysis

Five Why’s

Backcasting

Options for developing logic models

Previous research / evaluation

Group facilitation