using propensity score matching to understand what works in reducing re-offending gss methodology...

Using propensity score matching to understand what works in reducing re-offending

GSS Methodology Symposium

Sarah French & Aidan Mews, Ministry of Justice

1st July2015

What will be covered

• What is propensity score matching (PSM)?

• History of using PSM in MoJ

• Justice Data Lab aims, history and how it works

• Methodology

• Key outcomes

• Developments

3

• We find individuals who were not treated, but who were very similar at the point treatment started.

TREATMENT

CONTROL

?

Observational Studies

4

What Propensity Scores Are, and Why They Are Important

Propensity Score = Pr (Treated | background info)

• Mimics an RCT in that the treatment is purely random for individuals with similar values of the background variables

• Groups of subjects with similar propensity scores can be expected to have similar values of all of the background information, in the aggregate.

Propensity Score Matching

Scale of propensity score

0 1

Control offenders

Treatment offenders

Key

Matched offenders

Control offenders are matched to treatment offenders if the control offender’s propensity score are within a specified range away from the treatment offender’s propensity score.

History of propensity score matching in MoJ

PSM first used in 2010 Compendium of re-offending to compare the effectiveness of two sentences

Since then used:

.. for further comparisons of sentences & sentence requirements .. to look at the impact of various interventions to reduce re-offending

.. to evaluate the relationship between employment and re-offending

Aim of the Justice Data Lab

Launched in April 2013

..to improve the evidence base on successful rehabilitation..

..by giving organisations working with offenders secure and legal access to aggregate re-offending data

..enabling them to better assess the impact of their work on re-offending

Why do we have the Justice Data Lab?

In 2012 we identified that charitable organisations in particular found it difficult to access re-offending data on their clients…

… this meant that they could not understand how effective their services were at rehabilitating offenders…

… and they were therefore unable to understand how their services could be improved, or have the evidence for further funding

It soon became clear that there was intense interest in this initiative from both public and private sector organisations too

How does the Justice Data Lab work?

Individual level data sent securely to MoJ

Provider organisation

MoJ

Analysis and Matching

Aggregate data return

Process overview

• Data upload template (60 minimum)

• Match to MoJ/DWP data to get treatment group– find sentences for offenders from PNC

– match to DWP/HMRC data

• Matched control group via PSM (30 minimum)– Many to one matching, radius matching with replacement

• Assess quality of matched control group by analysing standardised mean differences

• Significance testing on re-offending measures compared between treatment and control groups

Key characteristics of ‘research reports’

• Binary re-offending rate (overall, resulting in custody, severe, 1 year, 2 year)

• Frequency of re-offending

• Survival (or death) analysis

• Representativeness of matched treatment group

• Quality of matching

• Sensitivity analysis

• Caveats (e.g. unobserved characteristics not included in matching process)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0 30 60 90 120 150 180 210 240 270 300 330 365

Days into follow-up period

Immediate Custody (1-4 years)

Immediate Custody (less than 12 months)

‘Death’ analysis

Re-

offe

nd

ing

rate

What is provided to Justice Data Lab users?

• One year re-offending rate

• Frequency of re-offending

• Time to re-offending

• Information on characteristics of both the treatment and control groups

0%

10%

20%

30%

40%

50%

60%

WYJS Participants (82offender records)

Matched ControlGroup (41,403 offender

records)

On

e ye

ar p

rove

n r

e-o

ffen

din

g r

ate

The best estimates for the one year proven re-offending rate for offenders who received an intervention from WYJS, and a matched

control group.

Key Justice Data Lab outcomes

Of the 125 reports published so far:

• 29 reports indicated statistically significant reductions in re-offending on the one year proven re-offending rate

• 89 reports indicated insufficient evidence to draw a conclusion about the effect on the one year proven re-offending rate

• Of these 89, 11 reports detail statistically significant reductions in the frequency of re-offending

• 7 reports indicated a statistically significant increase in re-offending on the one year proven re-offending rate

Developments in PSM use within MoJ

• Move away from 1-1 to 1-many matching

• Use of more information in matching process

• More thinking about methodological improvements to reduce bias

• Wider range of outcome measures

• Post matching regression analysis

Justice Data Lab Developments in Progress

• Providing additional information on the re-offending outcomes, such as severity of re-offending

• Enhancing understanding of the criminogenic needs of individuals – through the use of Offender Assessment (OASys) data

• Understanding more about individuals that are not matched in an analysis

• Official Statistics methodology review

• Supporting other government departments on potential Data Labs

Contact Details

Email: justice.datalab@justice.gsi.gov.uk

Accessing the Justice Data Lab service:https://www.gov.uk/government/publications/justice-data-lab

Published reports:www.gov.uk/government/collections/justice-data-lab-pilot-statistics