An introduction to Evidence & Insight

Two random control trials in search of context

Prof. Betsy Stanko Dr. Paul Dawson

Evidence & InsightMayor’s Office for Policing And CrimePaul.dawson2@met.police.ukBetsy.stanko@mopac.london.gov.uk

http://www.amazon.co.uk/Police-Research-Evidence-Recommendations-SpringerBriefs/dp/3319206478

Our experience captured in print

• E&I is the largest dedicated civilian research unit in UK policing & Crime and justice in London.

• Approximately (20) specialist staff, such as researchers, analysts, criminologists & psychologists.

• Based in the MPS for previous 9 years. Moved to MOPAC April 2014 with a broader scope.

• Conduct wide variety of performance reports, problem profiles and research across crime and justice in London (Large scale surveys e.g.,14,000 Londoners) to experimental evaluations).

• Working to drive change and improvements that benefit London (within policing, crime and justice…) from the inside (really is unique).

• Evidence Based decision making – becoming a requirement – not a luxury – taking hold of policing at present. A Fantastic opportunity to push through improvements and promote evidence based working…

• How much of policing is based on evidence of ‘what works’ as opposed to police instinct or experience? How much of EBP is embedded into the heart of the organisation or at periphery?

An Introduction to Evidence and Insight

• A quasi-experimental evaluation of a 11 million pound initiative to reduce crime that found no impact. Importance of programme integrity, implementation, learning cultures and responding to negative findings. Thick skin…

• Large scale public perception surveys – e.g., public confidence and victim satisfaction. Developed statistical models and data in public domain.

• Process evaluation around tagging alcohol offenders (performance, staff and offender views).

• Longitudinal surveys of Police graduate recruits to track their journey through the organisation…

• Large scale problem profiles on issues combining performance, external and internal research.

• The impact of a predictive burglary mapping system? Is it even being used on the ground?

• A Randomised Control Trial of providing Body Worn Video to response team officers… will this lead to better outcomes (e.g., less complaints, Stop and Search outcomes)?

• A test to see whether writing to offenders can influence their offending behaviour (e.g., arrest).

An Introduction to our research – variety of research data

http://www.college.police.uk/News/College-news/Documents/BWV%20report%20Nov%202015.pdf

Body Worn Video (BWV) EvaluationAims of the Evaluation

• Commenced April 2014 - MOPAC, the MPS and the College of Policing (CoP) developed the largest urban Randomised Control Trial (RCT) of Body Worn Video (BWV) to date.

• Innovative, high profile and ambitious. Potential wide ranging impacts for policing – professionalism, reputation & productivity.

• MOPAC – have a key role in enabling innovation. • Wanted to explore 3 broad aspects:

- to examine the MPS process of implementing the technology, capturing learning/challenges- to assess the impact(s) of the technology- to examine the cost-benefit of the technology

¹The Rialto police Department’s Body Worn Video Camera Experiment: Operation Candid Camera (2013) Owens, C., Mann, D., Mckenna, R. (2014). The Essex Body Worn Video Trial: The impact of Body Worn Video on criminal justice outcomes of domestic abuse incidents.

Previous Evaluations• Landmark U.S. study in Rialto (California) demonstrated benefits in terms of reduction

in complaints (from 24% to 3%) and officer use of force (from 61% to 25%). But… - Small numbers- US based- Contamination- Not a full range of outcomes

• Others trials have include gathering footage for training material, CJ outcomes and domestic abuse (DA) responses.

• Not all pilots have been evaluated or evaluations have been limited by the design or size of the sample. MOPAC research will go a long way to help address an evidence gap on the holistic impact of BWV on numerous outcomes.

Body Worn Video (BWV) Evaluation

Evaluation Design: RCT• A Randomised Control Trial (RCT): gold standard in

understanding if ‘something’ has worked. • A RCT - greater certainty any change in outcome is a direct

result of the use of BWVs; and makes measurement of outcomes at the beginning unnecessary (as the two groups should be the same).

• Random allocation is crucial to ensure the groups are identical - randomly selecting if an intervention is received or not, (in this case the usage of BWVs) and monitoring a set of predetermined theoretically linked outcomes.

Maryland scaleBoroughs: Brent, Barnet, Bexley, Bromley, Ealing, Camden, Croydon, Hillingdon, Havering, Lewisham

Evaluation Design: Borough Selection

• 10 boroughs - selection was analytically driven, based on some general assumptions e.g. 500 available cameras.

• Further exclusion and inclusion criteria:

Exclusion criteria: those currently use BWV cameras; boroughs with a consistently low complaint rate; boroughs whose inclusion would otherwise adversely affect the analysis due to the relatively large shift team sizes or number of officers on personal shift patterns.

Inclusion criteria: boroughs with a relatively high rate of stop and search; boroughs with a relatively high crime rate.

Evaluation Design: Officer Type & Team selectionA within borough design chosen – as variation within one borough was less than between two different boroughs. • As contamination between the treatment (those who get cameras) and control (those who do not get cameras)

needed to be prevented/minimised entire teams were selected.

• Emergency response teams - chosen as they deployed to a wide variety of situations; will make the admin and back office functions more standardised/cheaper; and will ensure a level of control over usage.

• Teams were randomly selected - 2 per borough as treatment and 3 in the control group (not individual officers as per many prior BWVs).

Evaluation Design: Mixed MethodologyA range of social research methods used to attempt to understand the holistic picture….

Officer surveys – process x3

Officer surveys – attitudinal x2

Officer Interviews

Court Ethnography

12,000 thousand Londoners (PAS)

14,000 victims of crime (USS) DPS complaints database

CRIS records (e.g., arrests and charges)

Stops database

Statistical modelling

Usage of BWV

Body Worn Video has been linked to many different outcomes. The pilot was to take a holistic overview:- What impact will it have on Criminal Justice outcomes? (e.g. arrests and convictions) - What impact will it have on officer interactions with the public? (e.g. stop and search and complaints)- Will it change officer attitudes and self reported behaviour?- Will it save any time/money?- Are there any unintended consequences?

• A range of data sources (CRIS, Stops database, DPS complaints, PAS, USS, survey data), many at an individual officer level have been combined to provide an understanding of what has happened.

• Each outcome statistically tested at an individual borough and then a combined level to assess impact.

Outcomes: What does success look like?

• Analysis further complicated – by the fact that some officers in the treatment group never received cameras AND some officers received them, but chose not to use them!!

Officers are given guidance any time they are likely to make 'pocket book notes', they should think about switching the camera on. • There are incidents considered 'mandatory' for recording e.g. stop and search & incidents of domestic abuse. • Anything recorded is kept on the cloud for 31 days before an automatic data cull. Anything marked as

'evidential' will be stored in line with MOPI guidelines. • When the 'on' button is pushed, the footage will record (video only, no audio) from 30 seconds before it was

pressed.

Video Recordings

BWV implementing innovation

• Implementing anything is difficult – implementing innovation more so…

• Found numerous teething problems in the set-up and implementation

• Issues in the training (i.e., lack on ongoing support, lack of refresher training)

• Delays in installing infrastructure

• Shortfalls in camera numbers

• Frequent movement of staff

• Usage: 42% of officers with a camera recorded 10 or more clips per active month,

26% recorded less than 5 clips per active month, and 6% uploaded nothing.

TOTAL POLICING

Nudging offenders...

A test of personalisation and offending behaviours

Background to Nudge and criminal behaviour

TOTAL POLICING

• The rise of behavioral insights – aka, Nudge

• Many diverse topics (from male urination to organ donation)

• Recent interest aimed at crime - although no clear evidence on offenders

• Personalisation a potential area from wider nudge research

• Bumblebee 2015 an opportunity (albeit small)

• A scene set for innovation...

• PS – our first attempt did not work. In 2012 there was organisational support, yet

Borough Commanders were not keen as many had their ‘own’ letters that were sent

to offenders (something we called parochial stubborness - many different designs,

questionable quality).

The letters......

TOTAL POLICING

Letters and enhanced personalisation

Same text, names, help-line – half with photo/ half without

Total of 176 letters (80 Photo; 96 no photo)

Letters delivered Nov-Dec 2014 (pre bumblebee peak)

Outcome a focus on offending

20 Boroughs: Camden, Westminster, Hackney,

Waltham forest, Tower Hamlets,

Havering, Newham, Barking and Dagenham,

Lambeth, Bromley, Harrow, Bexley,

Richmond, Hounslow, Merton,

Wandsworth, Ealing, Hillingdon, Haringey,

Croydon

Selection of offenders – key criteria

Challenges in implementation

Letter 1

Letter 2

A comparison to similar offenders...

TOTAL POLICING

Age

Gender

Avg offences 18 month before

Any Offence 18 month

before

Avg offences 6 month before

Any Offence 6 month before

Letter Offender (n=176)

Avg. 32

Range 17-

61

91.5%

Male3.10 67.6% 1.15 41.5%

Comparison (n=183)

Avg. 31

Range 17-

59

91.3%

Male3.34 68.3% 1.03 43.7%

• Boroughs randomly allocated to 'letter' conditions (e.g., letter 1 or 2).

• Controls (no letter) were drawn from eligible nominal list within corresponding boroughs.

• Boroughs sought to identify eligible offenders to receive a letter. Criteria included recent criminality. However boroughs also identified other individuals to send the letter to...

• We then developed an experimental group matched to these letter offenders on a number of relevant variables: age, gender, offending in previous 6 and 18 months.

• A strong match overall between groups....

Is there evidence that receiving a letter impacted upon committing any offence

Any offence in the six months subsequent to the letter.

7 percentage point difference between receiving any letter versus no letter (39.8% vs. 46.4%) and any offence*.

(*p=0.16, x2=1.92)

Committed ANY offence 6 months subsequent

Total offences 6m pre post

Letter: -12% decreaseControl: -16% decrease

Avg offences 6 months post

Letter: 1.03Control: 0.86(p= .42)

-9 -8 -7 -6 -5 -4 -3 -2 -1 1 2 3 4 5 60

10

20

30

40

50

60

70

Any LetterNo Letter

Of those that reoffended, our letter offenders were quicker to commit any offence

• In terms of the binary 'any offence' measure – no difference in proportion that offended in the first month subsequent to the letter (Control 35%, Any letter 34%, L1 32%, L2 36%)

• Subsequently we see our letter groups are quicker to commit any offence (more so Letter 2)

…But a short term impact on the average number of offences?

38% of all control group offending took part in that first month (bumblebee peak?)

Far less within our letter offenders (12% for letter one and 19% for letter 2).

In terms of average offences – for those to reoffend, again control group offenders committed an average of 2

offences in the first month, compared to 1.3 for our Letter group.

Average number of offences subsequent to letter….

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

1 2 3 4 5 6

Any letterLetter 1Letter 2No Letter

In month 1

Letter offenders fewer average offences

(Any letter .16, Controls .32, p=.07)

Letter offenders significantly more in later months

Wider influence on offending – burglary and home boroughs

Any letter group less likely to commit any burglary offence in the six months subsequent (15% Vs 21%) (X2= 2.51, p=.113, phi=.084). Again letter groups appeared to offend less in month 1. Letter group more average burglary’s (.16 v .3) (p=0.55).Both comparable reduction in Burglary: Letter (75 – 54) (-28%) v Control (61 – 45) (-26%.)

Home and Away

Letter groups offended more than controls within delivery borough in general, but offending here decreased by 5% compared to a 3% increase in controls.

Controls appeared to offend more outside their delivery borough (based on home borough data), remaining steady pre and post intervention.

Whereas letter groups offending outside delivery borough increased by around 8%.

Any Letter Letter 1 Letter 2 No letter0

5

10

15

20

25

13.2911.25

17.5

21.8620.81 19.7917.7

21.86

Offended outside delivery borough PrePost

Prop

ortio

n of

gro

up (%

)

Any Letter Letter 1 Letter 2 No Letter0

5

10

15

20

25

30

35

40

33.5330

35.42

22.4

28.9 27.531.25

25.14

Offended within delivery borough PrePost

Prop

ortio

n of

gro

up (%

)

Implications

• Small scale test of innovation

• Implementation issues

• Rolled out again in a improved manner (e.g., on all prison releases – timely)

• Mixed picture in terms of results

• Weak evidence that receiving any letter can impact upon committing any offence

• Of those that do reoffend, letter offenders appear to do this quicker

• Some change in that first month regarding average offences… albeit later peaks

• Home and away

• Differential impact – badge of honour? Delivery method? Frequency?

Wrap up: final thoughts

• The largest PCC research team in the country – decade of experience in the MPS

• Blending performance and social research into Evidence Based Policing

• Wide range of methodologies – steering a PCC from within to be evidence based

• Continued learning about the need for strong implementation

• Better use of police data for non police researchers - the oblique challenge

• How much of EBP is embedded into the heart of the organisation or at periphery

• Receptivity challenges (how to embed evidence, is there a willing culture)

• What does it really mean to be evidence based? Central movement and vision

http://www.amazon.co.uk/Police-Research-Evidence-Recommendations-SpringerBriefs/dp/3319206478

Our experience captured in print