Constructing your impact model –running the exercise

Introduction

A group exercise to construct a model foryour project, initiative or service lies at theheart of the impact evaluation model (IEM).This guidance outlines how to run such anexercise in one hour and create a basicmodel. An hour is enough time to createthe key elements of a model, althoughthere may still be a need to make additionalrefinements afterwards.

Materials• Post-it notes and pens• Prompt questions from the left-hand side of the

model (see additional tools section) • Space on flip chart/wall/brown paper to

separate inputs, outputs, intermediate outcomes and final outcomes

The diagram below illustrates how this should be laidout and the additional tools section illustrates otherways of organising the model for the exercise.

Inputs- What are the key design features of our project,

initiative or service?- Who is this project, initiative or service targeted at?- What quantity of the service, activity or

intervention do we aim to deliver?

Outputs- What are the levels of quality in our project,

initiative or service that we want to achieve?- What do we want users and providers to think about this project, initiative or service?- How do we want our target audience to respond?

Intermediate outcomes- What aspects of users’ or participants’

a) knowledge or skills b) behaviour or c) attitudes do we want to change through their interaction with the service?

Final outcomes- Which final outcomes or community outcomes do

we hope to impact upon?

Evidencesources

ModelcomponentsStage

Inputs

Outputs

Intermediateoutcomes

Finaloutcomes

Model layout

Constructing your impact model 2

PrecursorsThis guidance assumes that the group conducting theexercise has already agreed and understood the languageused in the IEM or agreed some alternative language(see the additional tools section if this is not the case).To run the exercise, you must have an exercise leader –the person leading the group’s discussion – and aproblem owner. The problem owner could be the personwho needs the data or the one responsible for follow-upactivities. They will be the final decision-maker if thereare any points the group cannot agree on, otherwiseeveryone participates equally.

ProcessStep one: set ground rules and decide on the order in which to complete the model (five minutes)

It is a good idea to set some ground rules at the start ofthe exercise – these could include:

• no suggestion is a bad suggestion – everyone has theirown perspective on the project, initiative or service

• nive everyone ‘airtime’ – listen to one another• there is no ‘right’ answer – only what the group

judges to be sensible• if something gets excluded from the model, it

doesn’t mean it’s not important.

The boxes in the model can be completed in any order,although we suggest the decision is based on where theproject, initiative or service is in terms of its delivery(see diagram above).

Step two: get people talking about the model (10 minutes)

It is critical that everyone participates in the exerciseright from the start. To do this, begin with a brainstormabout what people feel should go into the differentboxes (the additional tools section contains someprompt questions to get you started). The exerciseleader can front the discussion and write down on Post-its what people say, placing these on the modelthemselves, or each individual can complete and attachtheir own Post-its.

At this stage there are no wrong or right answers and noone’s contribution, or where they put the Post-its,should be challenged. However, it can be useful to guidepeople in how to write their Post-its. Post-its should:

Describe a positive end state and not justrestate the targetNot: A 0.5 per cent drop in NEET rateRather: All 16- to 18-year-olds engaging in learning

Have a clear ‘who’

Not: Classes well attendedRather: Non-English-speaking parents attend classes

Generally (but not always) include a verb andshould avoid vague adjectives

Not: Parent’s confidentRather: Parent’s engaging confidently in their children’smaths learning

A useful test of any Post-it is whether there is a clearlyidentifiable and undesirable opposite – if not, then thePost-it may not be specific enough.

Plan

Review Do

If the project, initiative orservice is still at the planning stage, the team may wish to start with final outcomes and work back to inputs.

If the project, initiative orservice is already being delivered, then the team may want to look at final outcomesand inputs and then fill in outputs and intermediate outcomes from there.

If the team is looking back at a completed project, initiative or service, then it may wish to work downward from inputs to final outcomes.

Using the IEM at different delivery stages

Constructing your impact model 3

Step three: populate the model (20 minutes)

There is now a collection of Post-its on the model. Theexercise leader takes the group through each of themodel’s middle column of boxes in the order that wasagreed initially. For each box, the exercise leader islooking to answer the following questions:

• Is everyone clear on what each Post-it means?• Do the Post-its in the box answer the prompt

questions to the left of the box? Are any others needed?

• Are some of the Post-its in the wrong box – shouldsome be moved up or down?

• Do some Post-its refer to important goals orelements that are not the focus of this project,initiative or service? Do they need to be removed?

It is important to emphasise that there often isn’t aright or wrong answer for where a Post-it should beplaced – it is a judgment. The exercise leader’s job is totry to achieve a group consensus around this decision (and use the problem owner if this is not possible).

Step four: prioritise the elements (10 minutes)

The resource available to collect evidence after themodel has been completed should help to determinethe complexity of the model, but, in general, it is wiseto have no more than four things at each stage. Theexercise leader must re-emphasise the need forsimplicity and that if something is removed from themodel it does not mean it is not important.

The exercise leader can either hold a group discussionabout what should remain in the model or they can askparticipants to dot vote. In the latter, each participanthas six votes for the things they think must staysomewhere in the model – and they mark these votesby drawing a dot on each relevant Post-it. The Post-itswith the greatest number of votes are then retained andthe others are moved to one side.

The exercise leader needs to help the group ensure themodel can work. For example, for each intermediateoutcome it should be possible to identify an equivalentinput or output that will help achieve it. So there shouldbe nothing in any of the boxes that does not havesomething that refers to it, or is relevant to it, in theboxes above and below.

Step five: identify potential evidence sources (10 minutes)

The exercise leader should give participants a couple ofminutes to write down potential evidence sources thatalready exist and to stick these in the right-hand boxesof the model.

They should lead a group discussion about any potentialnew evidence sources or data collection needed.Participants should be encouraged to think creativelyabout information gathering, particularly in terms of:

• who the data is collected from – try to avoid the“usual suspects” who are always asked forinformation or those who may be biased in their response, and

• When and how the data is collected – particularlyavoid jumping to paper-based surveys if there maybe more natural gatherings or activities wheninformation could be collected.

Step six: agree the indicators and success levels(time dependent)

Indicators are the specific questions or responses thatthe project leader will gather from the evidence sources.The following example illustrates this:

Evidence source: Interviews with parents

Indicator: parents’ level of agreement with thestatement – “I feel engaged in my child’s learning”

Success level: 75 per cent of parents strongly agreethat they feel engaged in their children’s learning

These indicators and success levels are likely to featurein the discussion of evidence sources above so the timeneeded may depend on this earlier discussion. Theindicators and levels needed for success may need to be completed by the project manager afterwards andcirculated to the group.

Step seven: sense check the model (five minutes)

The final step is for the exercise leader to getparticipants to step back from the model and check thatits contents make sense. There are some questions inthe additional tools section to help you do this, but akey test is to ask participants whether, using the modelin front of them, they think they could present a two-minute synopsis on what they are doing (the project,initiative or service), how they think it will work andwhat they hope it will achieve.

Constructing your impact model 4

Additional tools

1. Layouts for impact models

Page one of this guidance illustrates how the boxes in the model can be arranged. However, it is quite possible to layout similar boxes in different ways on flip charts or brown paper. The following photos illustrate ways to do this.

Alternative model layout for flip charts

Alternative model layout for brown paper

Constructing your impact model 5

2. Language used in the model

The following diagram outlines the meanings that the Training and Development Agency for Schools (TDA) suggestsfor the terms used in the IEM.

Outline of model language

If teams do not use the language above, they do need to agree on a common set of terms to use when constructingtheir models. The diagram on the following page illustrates a number of terms that other models may use, which areoften equivalent to the terms used in the TDA model.

Inputs

Meaning Rule of thumb

Outputs

Intermediate outcomes

Final outcomes

A brief description of a project’s, initiative’s or service’s key components, its resources, what the planned activities are and who is it for

Inputs are usually something withinyour control that you planned to do, designed for, or put into a project, initiative or service

Outputs are usually something that looks at the project, initiative or service through the eyes of the user or the intended audience. This could still be some sortof planned activity – for example, how tailored a service was – but if so this should be focused on user-centred activities

Final outcomes are likely to be at a levelwhere they would appear as an indicator among the 200 National Indicators for Local Area Agreements (LAAs)

A direct product of what is delivered at the input stage. It must be related to the actual quality of delivery and have a particular emphasis on how users have engaged with and immediately responded to the project, initiative or service

Intermediate outcomes are generally about the population who have comeinto contact with a project, initiative or service

How you want your service users to beinfluenced by the project, initiative or service, particularly in terms of the changes in knowledge/skills, attitudes or behaviour you hope to see

The social phenomena that you are looking to address within the communityas a whole

Related terms used and other frameworks

A way to help the group understand the terms being used in the model and come to a consensus on their use, isto run a quick exercise to construct a model based on a daily experience. We suggest that the exercise leaderstarts with a model in mind and completes a number of Post-its for its main components. When these are shownto the group, they should not be organised in any particular order.

The group exercise is to organise the components into inputs, outputs, intermediate outcomes and finaloutcomes. When constructing the main model, the exercise leader should push the participants to use the rulesof thumb that feature in this guidance, to help them come to a common position. However, as with a real model,there may be some components that require a judgement about where they should be positioned. The importantaspect is that the group can come to a working consensus on where different components should sit in themodel and take this consensus into the creation of their real model.

The diagram on the following page illustrates how some unsorted components of a model for a dinner partycould be sorted into the impact model categories.

Constructing your impact model 6

Inputs

Outputs

Intermediate outcomes

Final outcomes

Cost, resources, activities, process, efficiency, overhead, unit costs, staffing ration,participation

Efficiency, staff turnover, staff morale, waiting time, waiting list, accident rate,customer satisfaction, service quality, product, total quality management (TQM),staffing ratio, participation, results

Staff turnover, staff morale, cost/benefit, client results, client outcomes, valueadded, service outcome, direct impact

Community outcome, impact, value-added, social return on investment (SROI), well-being, “so what”, social capital

Constructing your impact model 7

‘Dinner party’ exercise

Recipes for threecourses

A nice bottle of red wine2.5kg leg of lambSeating planInputs

My guests’ hunger is satisfied

My guests get on with each other

Guests enjoy the food

Six good friends attend my dinnerOutputs

My guests invite me back to dinner

Some of my guestscontact one

another

I see my guests more frequently

Intermediate outcomes

I have friends when I need them

My friends value my friendshipI have more friendsFinal outcomes

Suggested alignment to the model

Recipes for threecourses

My guests’ hunger is satisfied

My guests get on with each other

Guests enjoy the food

I have friends when I need them

Six good friends attend my dinner

A nice bottle of red wine

2.5kg leg of lamb

My guests invite me back to dinner

Some of my guestscontact one

another

I see my guests more frequently

My friends value my friendship

I have more friends

Seating plan

Suggested Post-its

3. Prompt questions to get people talking

The following are some questions you may want to use to generate conversation at the beginning of the exercise:

• Why are we doing this project, initiative or service?• Who is the customer for this piece of work?• What will people think, feel, say or do as a result of this work?• Does anyone have any strong views about ‘what works’ with this kind of project, initiative or service?• Why are we delivering the project, initiative or service in this way?• What will the future look like if we are outrageously successful?

Training and Development Agency for Schools151 Buckingham Palace Road, London, SW1W 9SZTDA switchboard: t 0870 4960 123

Publications: t 0845 6060 323 e publications@tda.gov.uk

www.tda.gov.uk© TDA 2009