“measuring what matters”

Deb James, Early Years Evaluation Team

• Share some of what we learnt from our evaluation – from areas

with plans in place

• Introduce an approach to “measuring what matters” and some

materials (all optional!) that you might find helpful

• Do a group exercise looking at how you could start to measure

the difference you’re making and tell a stronger story about your

successes

In today’s workshop we will …

Using information to make a judgement about how you’re doing;

assessing the value or worth of what you’re doing.

What IS evaluation?

• Accountability

• Learning and improvement

• Motivation and engagement

Why it isn’t always a priority …

“It comes at the end”

“It’s done to you by an

expert”

“It’s paperwork … actually

doing stuff is more important

than form filling”

“Not sure where to start”

… and why it matters?

What did we find in our

own recent evaluation?

Ten messages from our work with

areas that have plans in place.

Let’s see if they ring true for you?

1. Plans vary but many areas share similar goals and priorities –

People-Place-Community. Local context is key to understanding

what areas plan to do – how you tackle isolation in one area can be

very different to how you might do it in another.

2. Some great stories about changes happening – some small, some

very significant – areas themselves are best placed to say what’s

small and what’s significant. The distance travelled matters.

3. Partnerships worry about things being slower than anticipated

(though there is no national pressure on deadlines). Causes include

sometimes having over-ambitious goals or being a bit let down by

partners.

Learning about plans and progress

and reviewing …

4. For many areas in the first year delivering on their plan, “behind the

scenes” work had taken more energy and time than anticipated and

a lot of achievements were in setting up/maintaining partnerships

rather than “out front” (making changes in the community).

5. Following review activities, a common realisation has been that the

Big Local journey is up and down, even round and round in circles,

rather than in a straight line of growth and improvement – and that’s

OK.

6. Some anxiety about evaluation … It’s too soon? Where do we

start? Don’t we need special skills and knowledge? We need to

focus on doing the work not measuring the results?

…

7. Some things seem too hard to measure – eg, ambitious or long-

term changes, or “fluffy” changes.

8. Areas are realising that not everything works – it isn’t always

comfortable, but it’s helpful to acknowledge this and Local Trust is

not expecting everything to work or run smoothly all the time.

9. A pause to reflect and review was helpful … it can be hard to find

the time but worth it

10. Because things sometimes feel slow, there’s a danger that

reviewing can leave areas feeling demoralised. It’s good to think

more broadly about success and to value more than just how

many things have been ticked off a ‘to do’ list.

…

*

PLAN - Planning and identifying what matters most

- what you need to assess or measure and how you

might do it

DO - Collecting the information you need and

analysing it – thinking about what it means

USE - Using your findings – to make judgements,

to learn/make changes/improve, to report to others

What does evaluation involve?

1. PLAN

2. DO

3. USE

A common view is that evaluation is simply about collecting

information (eg, doing a survey) and then writing a report about what

you find. In fact it involves three main stages:

Stage 1:

Evaluation planning

Five tips to help you plan how you’ll assess your

progress – help you be clearer about what you want to

measure and how you might measure it.

1. PLAN

2. DO

3. USE

1. Define success

If you don’t have a clear, shared

view of what success looks like,

how will you know you’ve

achieved it?

1. Define success

Example – an area’s success criteria

Our success criteria …

When considering how successful we have been, we’ll see if we’ve done what we

said we would, we’ll see if we’ve made a difference, and we will also consider how

true we have been to these values:

• we will have worked in an inclusive and respectful way

• we will have kept talking and listening (ongoing community engagement)

• we will have been bold and imaginative, not afraid to try new things

• we will have linked projects and activities together where it makes good sense

• we will have done the best we can

• we will leave a legacy behind

*f

Residents will report an increased confidence in accessing job

readiness and employment support services and a more positive

outlook on improving their financial situation through work, having

been assisted through a new local high-street service that is friendly,

informal and community-led

What would you measure? It helps to get to the heart of the

change, and to express changes as single, simple statements.

2. Describe clear changes

Members of the community

will help us run the job club,

sharing their time and skills

with others

Out-of-work residents who

use the club will feel more

confident about seeking

support to find

training/employment

Out-of-work residents who

use the club will feel more

positive about their chances

of finding work.

We will set up an informal,

friendly job club in our Big

Local Hub to support long-

term unemployed residents

Activity Activity Change Change

We will offer youth nights at the community

centre

Young people will be more employable

We will offer group work

sessions during youth nights at the community

centre

Young people will improve

their social and communication

skills

Young people will be more employable

3. Have stories that work

Would a story map help?

The things we do or

fund to make a

difference (activities)

The differences we want

to make

(changes/results)

End result

1. Activities that preserve or

improve the appearance

of our green spaces and

open spaces

2. Activities that promote

community access/

enjoyment of our open

spaces e.g. growing

projects, allotments

Our open spaces are (and

look) better cared for

More people are involved in

activities that protect or

improve our local environment

Our open spaces are more

accessible and welcoming

More use of our parks and

open spaces (that is, they are

better used by all parts of the

community)

An

improved

environment

that is cared

for and

enjoyed by

our

community

Pause

Suggestions so far. You might want to take time out to …

(1) consider what success means to your partnership. What else, what

other criteria - besides what you’ve done and the difference you make -

would you like to assess yourself by …

(2) make sure you are clear about the changes you want to make and

how you think you’re going to make them, is it clear what the changes

are and who they are for …

(3) check you are clear about how what you do will help bring about the

difference you want to make. Check your ‘story’ works … is it worth

trying to map/summarise your ‘change stories’? This can have a

number of benefits – making things seem more measurable, helping

you focus on what kinds of activities to try/fund, helping you tell your

story.

4. Be specific

(break ‘big’ changes down)

• Breaking big changes down and being more specific about them

can make them easier to measure – that is, spend some time just

clarifying what you mean by ‘big’ changes like “community

cohesion” or “social inclusion”, “quality of life”, or “improved

environment”

• Some changes are long-term and lots of other factors come into

play … try to focus on what’s realistically in your control

Example – break down big changes

Our big change

Our specific changes

(what an improved local environment

means to us)

Improved local

environment for

the whole

community

Cleaner open spaces

Safer open spaces

Our open spaces are better used by local

people

The community take better care of the space

What does the change look

like?

How will we know? What

signs or indicators would be

good evidence?

(things you can count,

things people tell you, things

you can see are all valid)

5. Ask ‘how will we know’ questions

Improved local environment for the whole community

Our specific changes

(what an improved local environment

means to us)

What we will count, measure or

observe

(signs of the change)

Cleaner open spaces Level of littering

Level of graffiti

Level of complaints

Level of use of dog litter bins

How clean people feel the space is

Safer open spaces ?

Spaces better used by local people ?

The community take better care of the

space ?

*

Exercise - how will you know? (what evidence?)

*

Our specific changes

(what an improved local

environment means to us)

What we will count, measure or observe

(signs of the change)

Cleaner open spaces • Level of littering

• Level of graffiti

• Level of complaints

• Level of use of dog litter bins

• How clean people feel space is

Safer open spaces • Level of reported crime

• Number of accidents in play area

• People’s perceptions of crime

• How safe people feel spaces are

More visited/used by local

people

• Level of use by families

• Proportions/profile of those using (eg, age, families)

• Level of bookings for community events and attendance

at these

The community take better

care of the space

• Number of volunteers helping with litter picks and events

• Level of littering/graffiti (see above)

Our specific

changes

(what an improved

local environment

means to us)

What we will count, measure

or observe

(signs of the change)

How we will

count it

(methods)

Cleaner open spaces • Level of littering

• Level of graffiti

• Level of complaints

• Level of use of dog litter bins

• How clean people feel space is

Annual litter pick

Council records

Survey

Safer open spaces • Level of reported crime

• Number of accidents in play area

• People’s perceptions of crime

• How safe people feel spaces are

Police records

Council records

Survey

More visited/used by

local people

• Level of use by families

• Level of bookings for community

events and attendance at these

Question in schools

parent survey

Council records

The community take

better care of the space

• Number of volunteers helping with

litter picks and events

• Level of littering/graffiti (see above)

Monitoring records of

volunteers

(as above)

The last part of the plan (collecting evidence)

How will you know exercise

Choose a change that you’d like to measure progress with –

eg, older people will have a better quality of life, young people

will be reaching their full potential, or one of your “fuzzy” ones!

(1) How will you know you’re making a difference? In

your group come up with three or more signs you could look

for or measure to help you know if the change is happening.

(2) What methods could you use to collect evidence?

(3) Who could help and how?

Stage 2:

collecting and analysing

information

1. PLAN

2. DO

3. USE

Some of the options open to you and some

good practice tips

Depending on your questions and the type of information you need (eg, numbers

or stories), you could use:

1. Survey questionnaires and feedback forms (your own)

2. Standardised questionnaires or scales (others)

3. Available data (info that others have already collected)

4. Interviews or focus groups (what people think, how things are perceived)

5. Participatory, creative and arts methods (great for young people)

6. Case studies (often based on interviews)

7. Documents and records - your own (minutes, event reports) or others’

(news articles, correspondence)

8. Visual records (photographs, video)

Other? Is anyone using any other methods? (Check out the Local Trust

“checking your progress toolkit” for ideas and tips on what to use and when.)

Options for collecting information

1. Prioritise – you can’t measure everything or collect

information from everyone. What steps or changes matter

most to telling your story?

2. Be realistic – and proportionate – what you do to measure

your success shouldn’t get in the way of actually doing stuff.

3. Be lawyers not scientists? - think about ‘the best

possible evidence’ not ‘scientific proof’

4. Be a recycler! – what else is already being

measured/counted, what info is already out there that you

could use for your purposes?

5. Be flexible and open - this kind of work isn’t ‘neat’ and we

are all trying something new, taking risks and trying to learn

more about what works. Be open to using different

approaches and methods – counting numbers and capturing

stories are both valid.

Top tips for collecting information

6. Consider different perspectives – it can help to

include different groups and different viewpoints

7. Try to capture learning as well as success – try

and create the kind of environment where you can learn

from what doesn’t work as well as what does!

8. Think about how you’ll use information BEFORE

you collect it

9. Don’t collect information you won’t really use

10. ALWAYS do a test run first of any questionnaire or form

you use

…

Stage 3:

Using your findings

Reflect. Learn. Improve. Share. Celebrate.

1. PLAN

2. DO

3. USE

• Remember that “the purpose of evaluation is to improve, not

prove.” (J Stufflebeam)

• There are lots of good reasons for reviewing regularly,

building in space/time to review as you go along, not least

because it helps you see if something’s not working and

means you can change direction more quickly if you need to.

• Your report to Local Trust will ask you to reflect on your

learning … should be a group activity, try to avoid one person

writing on behalf of the group as you can lose useful insights

and assumptions can be made.

• Aim for honesty! “All my successes have been built on my

failures.” (Benjamin Disraeli)

• Ask the right questions … including what worked and what

didn’t and what did we learn?

Reflect, learn, improve

Asking the right questions is key

• The achievement we’re most

proud of? What’s gone well?

• What do we feel more

confident about? What have

we got better at doing?

• What difference have we

made or contributed to in our

area? How do we know

we’ve made this difference?

• What’s not gone well? What

are the main lessons we’ve

learnt from things that

haven’t gone so well? What

could we do better?

Don’t stop with your report to Local Trust – consider using:

• Meetings, events or community forums

• Websites, facebook timelines, blogs or twitter

• Newsletter articles and case studies

• Photo galleries and posters

Here’s some examples of sharing and celebrating that we liked

using visuals, facebook, posters, and using story telling and

posting impact stories online …

Share and celebrate