ego - an introduction to research and impact evaluation

8/8/2019 EGO - An Introduction to Research and Impact Evaluation

1/24

An introduction to

Research and Impact Evaluation

October 2010

Maor Bar-Ziv


2/24

Landscaping Your evaluations so far

Introduction to research The research process

Introduction to evaluation The evaluation framework

Assessing needs

3 Specific tools The Project Profile Tool

The Outcomes Star

Social Return On Investment

Basic Research Methods and Impact Assessment


3/24

Over to you! Harvesting

Your evaluations ahead

Gathering evidence

Shaping your questions The Funnel Effect

Summary

Practical tips

Impact Assessment Planning


4/24

Your evaluations so far

1. What evidence do you currently gather on impact?

2. What story do you tell about this evidence?

To whom?

How often?

3. If this evidence was for the sole purpose of learning, would

you do things differently?

How?

Landscaping


5/24

Research is a fumble Basic research

Applied research Evaluations

Research process:

1. Environmental analysis2. Research design

3. Sampling

4. Data gathering

5. Data analysis

6. Synthesis7. Write up

8. Dissemination

Introduction to research


6/24

What should we be measuring?

To state the obvious?

Change?

For ourselves?

For an external audience?

= Framework

Introduction to evaluation

1. Landscape

2. Objectives

3. Project Story

- Inputs

- Outputs4. Stakeholders

5. Impact

- Outcomes

6. Decisions

7. Action8. Dissemination


7/24

How deep and how wide should we go?The data

Quantitative

Qualitative

FinancialWho should be gathering it, and when?

Bottom-up

Qualitative evidence Top-down

Quantitative evidence

Introduction to evaluation continued


8/24

What is a need?

Recognising need

Necessities/entitlements

Interests

Aspirations, opportunities, choices

Assessing needs


9/24

Calendar of events

National statistics

Case studies

# visitors on a website Online data

Chat rooms, networking sites...

Service provider beneficiary map Target group vs. existing service providers

Photographs, camcorder footage, audio records

Diaries

Examples of needs/impact data


10/24

Menu of possibilities Needs - you vs. stakeholders

Resources

Expertise

Surveys, interviews, focus groups

The Project Profile Tool

The Outcomes Star

Social Return On Investment

The tools


11/24

Participatory self-evaluation

The project profile

Project landscape

Project narrative Project family

roles, experiences, challenges, suggestions

Project innovations

Project impact Wish list

The Project Profile Tool


12/24

Background

Longitudinal data

The process

Who are your stakeholders? What outcomes matter to them?

Collaborate on designing the star

Gather the data

Generate distance travelled Generate average distance travelled for each outcome

Relationship between different outcomes

The Outcomes Star


13/24

The Outcomes Star continued


14/24

The Outcomes Star continued


15/24

History

Based on a set of principles:1. A tool

2. Stakeholders

3. Theory of change

4. Value

5. Material

6. Do not over claim

7. Transparency8. The story

9. Verify the result

Social Return On Investment (SROI)


16/24

Scope Past, present or future?

All or part of organisation, activity, period?

Forecast or evaluation?

Who youre not going to consider and why

Activities

Inputs and outputs

Outcomes

(appropriate) Indicators

Sensible to measure

Relevant to scope

Produce significant numbers

SROI continued


17/24

Six Marys Place (Jul 07)

Stakeholder Trainees

Activity Work placement

Outcome Improved recovery & mental well being

Indicator Fewer hospital admissions

Quantity 108

Cost 2,336 per admission

Source NHS cost book 2007

http://www.isdscotland.org/isd/360.htmlImpact 108 x 2,336 = 252,288

SROI example
http://www.isdscotland.org/isd/360.htmlhttp://www.isdscotland.org/isd/360.html


18/24

How do we put a financial figure on our added value? Price is a proxy for value

Types of financial proxies

Stakeholder value

Changes to cost Potential changes to cost

Changes to income

Sources of financial proxies

Sources from existing research

Comparisons

Calculations with estimations

SROI Financial Proxies


19/24

Attribution

Deadweight

Impact calculation

Impact = Outcomes Attribution and Deadweight

SROI calculation

Total Impact

----------------- = SROI

Total Input

SROI Calculation


20/24

Your evaluations ahead

1. Think through which tools you would use and why

2. Lets discuss these ideas

3. Are there any differences in the tools you would use on your

projects? What accounts for these?

Harvesting


21/24

Start early

Before

During

After

A while after follow up

Gather needs data again, and again, and again

Gathering evidence


22/24

The Funnel Effect Sequencing

What, how... why

Types of questions

1. Main/intro questions2. Follow up questions

3. Probing questions

4. Operationalising

5. Direct/leading questions for verification

6. Structuring questions

7. Interpreting questions

Shaping your questions


23/24

Multiple sources of information Multiple points of view

Unanticipated change

The ripple effect amongst beneficiaries

Balance of qualitative and quantitative

Case studies Demonstrate needs changed

Keep a project journal (blog)

Who else is out there?

Disseminate wisely

Participation brings popularity

The data is under your nose!

Practical tips


24/24

Maor Bar-Ziv

t +61 (0)430 228 555

e [email protected]

Stay in touch!

ego - an introduction to research and impact evaluation

Documents