Applied Policy and Practice Research Unit

Chris FoxDan Ellingworth

‘Modern policy making’ Forward looking: Defining policy outcomes and taking a long term view Outward looking: Take account of national, European and international situation;

learning from experience of other countries; recognising regional variations. Innovative, flexible: Questioning established ways of dealing with things, encouraging

new and creative ideas, identifying and managing risk. Joining up: Joining up the work of different government departments; ensuring that

implementation is part of the policy process. Inclusive: Consulting those responsible for implementation and those affected by the

policy; carrying out an impact assessment Evidence based: Basing policy decisions and advice upon the best available evidence;

ensuring evidence is available in an accessible and meaningful form. Evaluated: Systematic evaluation of the effectiveness of policy. Reviews: Policy constantly reviewed to ensure it is dealing with problems it was

designed to solve. Lessons learned: Learning from experience of what works and what does not

Cabinet Office (1999) Professional policy making for the twenty-first century London: Cabinet Office

What is evidence-based policy?• Evidence-based policy (EBP). An approach that:

– “helps people make well informed decisions about policies, programmes and projects by putting the best available evidence from research at the heart of policy development and implementation” (Davies 1999 quoted in Davies 2004)

• Contrasted with opinion-based policy which:– “relies heavily on either the selective use of evidence (e.g.

on single studies irrespective of quality) or on the untested views of individuals or groups, often inspired by ideological standpoints, prejudices, or speculative conjecture.” (Davies 2004)

Opinion-based policy

Taken from Chalmers, I. (2003) ‘Campbell and Cochrane: the need for generosity of spirit and mutual support’ Jerry Lee Lecture: 3rd Annual Campbell Colloquiem

Taken from Chalmers, I. (2003) ‘Campbell and Cochrane: the need for generosity of spirit and mutual support’ Jerry Lee Lecture: 3 rd Annual Campbell Colloquiem

What factors influence policy?

Taken from Davies, P. (2004) Is evidence-based government possible Jerry Lee Lecture 2004

An evaluation frameworkShould it work? (Theory of change)

What is the underlying ‘theory of change’ which explains why the intervention will make an impact?

Can it work? (Implementation evaluation) Has the project been properly implemented? What were the

challenges to implementation and how were they overcome? Does it work? (Impact evaluation)

What is the impact of the intervention?Is it worth it? (Economic evaluation)

What are the resource implications of implementing the intervention and what benefits will it deliver?

Adapted from Haynes, B., 1999. BMJ; 319:652-653 ( 11 September )

Should it work? Theories of changeSometimes programmes and projects:

don’t have clear aims or change their aims over timeare ‘shoe horned’ into a set of funding criteria to

access that fundingdon’t have clearly defined processes and structuresare implemented in a multi-agency context where

different agencies will have different ideas about what the project is trying to achieve

Theories of change“What is the conceptual link from an

intervention's inputs to the production of its outputs and, subsequently, to its impacts on society in terms of results and outcomes?”

(United Kingdom Evaluation Society Glossary of evaluation terms http://www.evaluation.org.uk/Pub_library/Glossary.htm)

Scared straightStarted as US programme in 1970sTarget group: At risk or delinquent young peoplePrison visits including tour and confrontational meeting

with serving prisonersGraphic depiction of life in prisonTV documentary in US extols its virtuesAlso used in US

Visit to a Prison by juveniles

First Hand Experience of Prison Life

Exposure to Prison Life and Prisoners as Positive Role Models

Stimulates or Attracts Juveniles Towards Crime

Increases Crime and Offending

Visit to a Prison by juveniles

First Hand Experience of Prison Life

Exposure to Prison Life and Prisoners as Negative Role Models

Frightens or Scares Juveniles Away from Crime

Reduces Crime and Offending

or . . .

Can it work? Process evaluationIdentify whether the intervention has been

implemented as intendedExplore how a service or policy is delivered

and experienced in practiceIdentify the mechanisms by which it can

produce the desired effects, potential barriers and facilitators

Identify circumstances under which successful operation might be replicated

ScurvyJames Lind (1716 – 1794): Ship’s SurgeonLind selected 12 men, all suffering from scurvyDivided them into six pairs, giving each group different additions to their

basic diet. CiderSeawaterA mixture of garlic, mustard and horseradishSpoonfuls of vinegarOrangesLemons.

Those fed citrus fruits experienced a remarkable recoveryIn 1753, he published 'A Treatise of the Scurvy‘http://www.bbc.co.uk/history/historic_figures/lind_james.shtml

Scale Impact evaluation methodology Necessary conditions

5 Random Control Trial: Random assignment and

analysis of comparable units to program and

comparison groups.

Ability to randomly assign participants/non-

participants to projects

4 Matched pairs: A comparison between multiple units

with and without the intervention; or using comparison

units that evidence only minor differences.

Close match between group of programme

participants and non- participants

3 Multivariate model: A comparison between two or

more comparable units of analysis, one with and one

without the intervention, where there are differences in

the relevant characteristics of the units.

Existence of comparable group of non-participations

2 Before/after or time series analysis: temporal

sequence between the intervention and the measure.

Ability to measure before and after intervention

1 Correlation: Correlation between an intervention and a

measure at a single pointing time.

Availability of data to show correlation

Sherman et al. (1998) Preventing Crime. What works, what doesn’t, what’s promising. National Institute of Justice.

Measuring impactChoice of impact measures is often limited by

availability of data or resources to collect primary data.In the UK, re-offending data can be accessed via

the Police National Computer (PNC) or the Offender Index

For less tangible outcomes such as family reconciliation or increased self-confidence the only option is often expensive face-to-face interviews/surveys.

Finding a comparatorA comparator group or area may not be

readily available.While a programme or pilot area may be willing to

cooperate with an evaluation an organisation that is not part of the evaluation but is being asked to provide data for a comparator area may be hard to persuade to cooperate.

A regional or national roll-out may limit the number of potential comparator areas

ThroughputsProgrammes and projects often struggle to

generate sufficient throughputs to allow for robust statistical analysis.Set-up times are often under-estimatedThe need for inter-agency cooperation is

often under-estimated

TimescalesOften evaluation timescales preclude

appropriate follow-up periods to identify the long-term impact of interventions.

The Home Office recommends that reconviction studies use a 2 year follow-up period.

By the time a cohort to study has been generated this will often require an evaluation period of 3 – 4 years.

Systematic ReviewsSystematic reviews are overviews of the existing

research literature on a topicA comprehensive search of print, electronic, and

unpublished sources is made.Sources identified are screened to see if they are

relevant.The quality of the sources (the strength of the

evidence) is then assessed.Sometimes the findings from individual studies are

pooled in a process called meta-analysis

Scared straight: systematic reviewThe systematic review shows that:

“The program increases the percentage of the treatment group committing new offences anywere from 1% to 30%.” (365) (emphasis added)

Petrosino, A., Turpin-Petrosino, C., and Finckenauer, J. (2000) ‘Well-Meaning Programs Can Have Harmful Effects! Lessons from Experiments of Programs Such as Scared Straight’ Crime Delinquency 46; 354

If so, Chris and Dan run a Third Year Module called ‘What works in social and criminal policy’ that looks in more detail at how policy is made and the role of evaluation in supporting the policy-making process.