1 do the claims for spending billions on crime reduction initiatives stand up do the claims for...
TRANSCRIPT
11
Do the Claims for Spending Billions Do the Claims for Spending Billions on Crime Reduction Initiatives Stand on Crime Reduction Initiatives Stand
upupRadical Statistics Conference 2006Radical Statistics Conference 2006
Paul MarchantPaul [email protected]@leedsmet.ac.uk
22
AimsAims A look at knowing ’What Works’.A look at knowing ’What Works’.
Things worth encouraging:Things worth encouraging: ‘‘Good Statistics’ Good Statistics’
(as many arguments are statistical in (as many arguments are statistical in nature)nature)
Transparency in design and reportingTransparency in design and reporting ‘‘Investigative Statistics’Investigative Statistics’ Scientific scepticismScientific scepticism Research which is sufficiently sound in order Research which is sufficiently sound in order
to properly justify spending on major to properly justify spending on major programmesprogrammes
33
Science Science Acquiring knowledge using data from Acquiring knowledge using data from
observation and experiment.observation and experiment. An inherently uncertain matter……Statistics!An inherently uncertain matter……Statistics! Not all results given, are the same. Therefore Not all results given, are the same. Therefore
there is the need to synthesise. there is the need to synthesise. Science is a public matter: not just because of Science is a public matter: not just because of
the impact of the products of science, but the impact of the products of science, but also because of the need to check work. also because of the need to check work.
Need open access, 'to pretty much Need open access, 'to pretty much everything', so that the work can be everything', so that the work can be replicated and checked (data, methods, clear replicated and checked (data, methods, clear complete reports).complete reports).(Need protocols to be published in advance.)(Need protocols to be published in advance.)
44
A scientific answer.…A scientific answer.…
is never something like x = 1.23is never something like x = 1.23 nor is it just x = 1.23 nor is it just x = 1.23 ± 0.45± 0.45 but rather also how it was derived, what but rather also how it was derived, what
assumptions and approximations are assumptions and approximations are involved, so that outsiders can scrutinize.involved, so that outsiders can scrutinize.
Just because assumptions are not Just because assumptions are not mentioned does not mean they are not mentioned does not mean they are not being made!being made!
55
Some quotations Some quotations ““The combination of some data and an The combination of some data and an
aching desire for an answer does not aching desire for an answer does not ensure that a reasonable answer can be ensure that a reasonable answer can be extracted from a body of data.” extracted from a body of data.” John Tukey John Tukey
““While every data set contains noise, some While every data set contains noise, some data sets may contain signals. Therefore data sets may contain signals. Therefore before you can detect a signal within any before you can detect a signal within any given data set you must first filter out the given data set you must first filter out the noise.”noise.”
Donald J. Wheeler in Understanding Variation: Donald J. Wheeler in Understanding Variation: the key to managing chaos. Pub SPCthe key to managing chaos. Pub SPC
66
Time Variation in CrimeTime Variation in Crime
It appears that little is known about It appears that little is known about how crime varies on the small scale. how crime varies on the small scale. Therefore it is difficult to be clear if Therefore it is difficult to be clear if any changes are due to a crime any changes are due to a crime reduction intervention. reduction intervention.
77
The Randomised Controlled The Randomised Controlled TrialTrial
(A truly marvellous scientific (A truly marvellous scientific invention)invention) Note to avoid ‘bias’:Note to avoid ‘bias’:
Allocation is best Allocation is best made tamper-proof. made tamper-proof. (e.g. use (e.g. use
‘concealment’)‘concealment’) Use multiple Use multiple
blinding of:blinding of: patients, patients, physicians, physicians, assessors, assessors, analysts …analysts …
Population
Take Sample
Randomise to 2 groups
Old Treatment
Compare outcomes (averages) recognising that
these are sample results and subject to sampling variation when applying back to the population
New Treatment
88
Counts of those cured and not Counts of those cured and not cured under the two cured under the two
treatmentstreatmentsCured Not
CuredNew Treatment a b
Control(Standard treatment)
c d
99
The Odds Ratio The Odds Ratio
OR = Ratio of the odds of cure under the two OR = Ratio of the odds of cure under the two treatmentstreatments
== Odds of cure under new treatmentOdds of cure under new treatment = = a/ba/b Odds of cure under old treatment c/d Odds of cure under old treatment c/d
== adadbcbc
If If OR > 1 the new treatment is betterOR > 1 the new treatment is betterOR < 1 the old treatment is betterOR < 1 the old treatment is better
Cure Not
New a b
Old c d
1010
But …But …
… … there is there is sampling variabilitysampling variability..Consider a table: Consider a table:
With OR = With OR = 60604545 = 1.23 = 1.23 404055 55 is not good evidence for a difference in is not good evidence for a difference in
treatment effectiveness. treatment effectiveness. The numbers are small and the sample OR = The numbers are small and the sample OR =
1.23 could be due to chance, when in fact 1.23 could be due to chance, when in fact the population OR=1.0 the population OR=1.0
CureCure NotNot
6060 4040
5555 4545
1111
Sampling VariationSampling Variation
Sampling Variation is given by the (asymptotic) Sampling Variation is given by the (asymptotic) standard error of ln(OR)standard error of ln(OR)
(S.E (ln(OR)) )(S.E (ln(OR)) )22= Var(ln(OR))= Var(ln(OR))== 11 + + 11 + + 11 + + 1 1
a b c d a b c d If the events are If the events are statistically independentstatistically independent..The sample ln(OR) is distributed about the The sample ln(OR) is distributed about the
population ln(OR) in a ‘Normal’/Gaussian population ln(OR) in a ‘Normal’/Gaussian fashion with its standard deviation given by fashion with its standard deviation given by that calculated from the s.e. (5% resides that calculated from the s.e. (5% resides outside 1.96 s.e.)outside 1.96 s.e.)
1212
Crime counts before and afterCrime counts before and afterin the two areasin the two areas
Examine the Cross Product Ratio (CPR) = Examine the Cross Product Ratio (CPR) = a/ba/b c/d c/d
If it is convincingly >1 then lighting works against crime.If it is convincingly >1 then lighting works against crime.
Before After
Treatment Area(lighting increased)
a b
Comparison Area(lighting stays the same)
c d
1313
Lighting and crime Lighting and crime
Much justification for exterior lighting Much justification for exterior lighting is made on the basis of crime is made on the basis of crime reduction.reduction.
(e.g. The Institution of Lighting (e.g. The Institution of Lighting Engineers)Engineers)
There seem to be many ‘theoretical There seem to be many ‘theoretical suggestions’ why lighting might suggestions’ why lighting might increase or decrease crime. increase or decrease crime.
1414
Light has a good press!Light has a good press!
‘…‘…God said, let there be light and God said, let there be light and there was light. And God saw the there was light. And God saw the light, that it was good….’ Genesis light, that it was good….’ Genesis Chapter 1 Verses3/4.Chapter 1 Verses3/4.
It seems ‘wicked’ to question the It seems ‘wicked’ to question the benefit of lighting.benefit of lighting.
However there is a ‘dark side’; However there is a ‘dark side’; lighting’s environmental impacts, lighting’s environmental impacts, possible health impacts.possible health impacts.
1515
My ‘Interest’My ‘Interest’ “…“…Paul Marchant, statistician at Leeds Paul Marchant, statistician at Leeds
Metropolitan University who argues that statistics Metropolitan University who argues that statistics used in the Home Office Study 251 could equally used in the Home Office Study 251 could equally be used to show that street lighting actually be used to show that street lighting actually increases levels of crime. This is an argument increases levels of crime. This is an argument which the APPLG, alongside the ILE, would hope to which the APPLG, alongside the ILE, would hope to show as utterly absurd. Of course it is worth show as utterly absurd. Of course it is worth noting that Paul Marchant is also an astronomer noting that Paul Marchant is also an astronomer as well as being a statistician, and that this may as well as being a statistician, and that this may lead to some bias in his interpretation of the lead to some bias in his interpretation of the statistics he refers to.”statistics he refers to.”
P56 of the March/April 2004 issue of the Lighting P56 of the March/April 2004 issue of the Lighting JournalJournal,, the magazine of the Institution of Lighting the magazine of the Institution of Lighting Engineers.Engineers.
APPLG= The All-Party Parliamentary Lighting GroupAPPLG= The All-Party Parliamentary Lighting GroupILE= The Institution of Lighting EngineersILE= The Institution of Lighting Engineers
1616
Odds ratio.1 1 10
Study % Weight Odds ratio (95% CI)
3.82 (2.28,6.40) Birmingham 0.7
1.72 (1.17,2.52) Stoke-on-Trent 1.7
1.44 (1.17,1.77) Dudley 6.3
1.39 (1.04,1.86) Atlanta 3.3
1.38 (0.97,1.97) Fort Worth 2.2
1.37 (1.06,1.77) Milwaukee 4.2
1.35 (1.23,1.48) Bristol 36.3
1.24 (0.95,1.64) Kansas City 4.0
1.14 (0.62,2.08) Dover 0.9
1.02 (0.75,1.40) Harrisburg 3.4
0.98 (0.88,1.11) New Orleans 24.4
0.94 (0.79,1.12) Portland 10.9
0.75 (0.48,1.19) Indianapolis 1.8
1.22 (1.15,1.29) Overall (95% CI)
Odds ratio.1 1 10
Study % Weight Odds ratio (95% CI)
3.82 (2.28,6.40) Birmingham 0.7
1.72 (1.17,2.52) Stoke-on-Trent 1.7
1.44 (1.17,1.77) Dudley 6.3
1.39 (1.04,1.86) Atlanta 3.3
1.38 (0.97,1.97) Fort Worth 2.2
1.37 (1.06,1.77) Milwaukee 4.2
1.35 (1.23,1.48) Bristol 36.3
1.24 (0.95,1.64) Kansas City 4.0
1.14 (0.62,2.08) Dover 0.9
1.02 (0.75,1.40) Harrisburg 3.4
0.98 (0.88,1.11) New Orleans 24.4
0.94 (0.79,1.12) Portland 10.9
0.75 (0.48,1.19) Indianapolis 1.8
1.22 (1.15,1.29) Overall (95% CI)
Odds ratio.1 1 10
Study % Weight Odds ratio (95% CI)
3.82 (2.28,6.40) Birmingham 0.7
1.72 (1.17,2.52) Stoke-on-Trent 1.7
1.44 (1.17,1.77) Dudley 6.3
1.39 (1.04,1.86) Atlanta 3.3
1.38 (0.97,1.97) Fort Worth 2.2
1.37 (1.06,1.77) Milwaukee 4.2
1.35 (1.23,1.48) Bristol 36.3
1.24 (0.95,1.64) Kansas City 4.0
1.14 (0.62,2.08) Dover 0.9
1.02 (0.75,1.40) Harrisburg 3.4
0.98 (0.88,1.11) New Orleans 24.4
0.94 (0.79,1.12) Portland 10.9
0.75 (0.48,1.19) Indianapolis 1.8
1.22 (1.15,1.29) Overall (95% CI)
Odds ratio.1 1 10
Study % Weight Odds ratio (95% CI)
3.82 (2.28,6.40) Birmingham 0.7
1.72 (1.17,2.52) Stoke-on-Trent 1.7
1.44 (1.17,1.77) Dudley 6.3
1.39 (1.04,1.86) Atlanta 3.3
1.38 (0.97,1.97) Fort Worth 2.2
1.37 (1.06,1.77) Milwaukee 4.2
1.35 (1.23,1.48) Bristol 36.3
1.24 (0.95,1.64) Kansas City 4.0
1.14 (0.62,2.08) Dover 0.9
1.02 (0.75,1.40) Harrisburg 3.4
0.98 (0.88,1.11) New Orleans 24.4
0.94 (0.79,1.12) Portland 10.9
0.75 (0.48,1.19) Indianapolis 1.8
1.22 (1.15,1.29) Overall (95% CI)
HORS251HORS251 StudyStudy STATA metanSTATA metanOdds RatioOdds Ratio(95% CI)(95% CI) % Weight% Weight
Forest Plot from Meta-Forest Plot from Meta-analysisanalysis
1717
The Confidence IntervalsThe Confidence IntervalsThe confidence intervals of individual The confidence intervals of individual
studies, which go to give the combined studies, which go to give the combined result, are calculated as though the events result, are calculated as though the events come from independent random samples. come from independent random samples.
So that:So that:
Var (ln(CPR))Var (ln(CPR)) = = 11 + + 11 + + 11 + + 1 1 a b c da b c d
But this can not be!But this can not be!
1818
Correlations Within Correlations Within Crime is committed by criminals!Crime is committed by criminals! Look at the data!Look at the data! Most studies are just Most studies are just
counts of crime in the 2 areas, before counts of crime in the 2 areas, before and after. But one that is not is Bristol.and after. But one that is not is Bristol.
If the independent random samples If the independent random samples assumption were correct, the variance assumption were correct, the variance of the count would be expected to be of the count would be expected to be approximately equal to the mean approximately equal to the mean count. But it is not. It is an order of count. But it is not. It is an order of magnitude higher.magnitude higher.
1919
The Bristol Study (Shaftoe The Bristol Study (Shaftoe 1994)1994)
1990.01989.01988.01987.01986.0
1500
1400
1300
1200
1100
1000
900
800
700
600
Year
No.
Cri
mes
1311
1464
1160
1237
14241374
142915171414
840840846
727
648
833838
685630
Number of Crimes Reported(in half-year periods)
Brighter Lighting
Control
New lighting introduced from July 87 to March 89 as marked onthe time axis.
Shaftoe said ‘no discernable lighting benefit’ but HORS251 says Shaftoe said ‘no discernable lighting benefit’ but HORS251 says z=6.6 ! Note: had the data for the year immediately prior to the z=6.6 ! Note: had the data for the year immediately prior to the introduction of the relighting, i.e. periods 2 and 3, been used introduction of the relighting, i.e. periods 2 and 3, been used rather than unnaturally using periods 1 and 2 which leaves a rather than unnaturally using periods 1 and 2 which leaves a gap of ½ year, the effect found would have been half of that gap of ½ year, the effect found would have been half of that claimed. (Shows large variability.)claimed. (Shows large variability.)
2020
Overdispersion (1)Overdispersion (1)
The Bristol (and Birmingham) studies show The Bristol (and Birmingham) studies show large variation over time when the light level large variation over time when the light level is constant. The variance is many times the is constant. The variance is many times the mean; mean; ’’OverdispersionOverdispersion’. ’.
(The(The large ≈ 60 heterogeneity statistic, Q, given by the meta-analysis of the 13 studies, also suggests this. A large Q shows that there is an inconsistency between, within study variation and between study variation)
2121
Overdispersion (2)Overdispersion (2)
The problem for HORS251 is that the The problem for HORS251 is that the confidence intervals around the effect size confidence intervals around the effect size must therefore be substantially increased. must therefore be substantially increased.
Also because the underlying overdispersion Also because the underlying overdispersion is not properly known for individual is not properly known for individual studies, we can not say what weights we studies, we can not say what weights we must use, as these will must use, as these will notnot be the same as be the same as used in the original incorrect HORS251 used in the original incorrect HORS251 meta-analysis.meta-analysis.
2222
Examine Overdispersion in Comparison Examine Overdispersion in Comparison AreasAreas
Study NameStudy Name DDobsobs = = ss22//xx AtlantaAtlanta 58.359458.3594
MilwaukeeMilwaukee 13.754713.7547
PortlandPortland 7.70177.7017
Kansas CityKansas City 9.74499.7449
HarrisburgHarrisburg 1.57481.5748
New OrleansNew Orleans 46.681046.6810
Fort WorthFort Worth 0.29340.2934
IndianapolisIndianapolis 0.04000.0400
DoverDover 4.76474.7647
BristolBristol 44.711644.7116
BirminghamBirmingham 1.53061.5306
DudleyDudley 4.44204.4420
Stoke-on-TrentStoke-on-Trent 0.00830.0083
DDobsobs are extremely are extremely variable and right variable and right skewed. The skewed. The arithmeticarithmetic mean is mean is 15 for these 15 for these comparison areas. comparison areas. (Larger still if the mean (Larger still if the mean includes weighting by includes weighting by number of crimes.)number of crimes.)
Calculated from the Calculated from the before and after before and after counts in the counts in the comparison areascomparison areas
2323
What observed overdispersions What observed overdispersions are expected if repeated samples are expected if repeated samples
taken?taken?It can be shown:It can be shown: for a Normal(for a Normal(μμ, , σσ22) such that ) such that σσ22 = = κμκμ, (i.e. the , (i.e. the
variance = the overdispersion factor (k) variance = the overdispersion factor (k) the the mean (mean (μμ) )) )
that the sampling distribution of the observed that the sampling distribution of the observed overdispersion koverdispersion kobsobs (= s (= s22//xx ) is (provided k<< ) is (provided k<<μμ) ) approximatelyapproximately
Chi-squared 1df, scaled by k.Chi-squared 1df, scaled by k.This can be written equivalently asThis can be written equivalently asGamma (scale =2k, shape=1/2) Gamma (scale =2k, shape=1/2) This is a right skewed distribution with arithmetic This is a right skewed distribution with arithmetic
mean = k. mean = k.
[If the before-after correlation = ρ, then k is replaced by k(1-ρ). [If the before-after correlation = ρ, then k is replaced by k(1-ρ). This is the effective overdispersion, relevant for a before-after This is the effective overdispersion, relevant for a before-after study, i.e. the quantity which is of interest. Indeed sstudy, i.e. the quantity which is of interest. Indeed s2 2
estimates estimates σσ22(1-ρ).](1-ρ).]
2424
The CDF of the HORS251 The CDF of the HORS251 overdispersionsoverdispersions
Sample overdispersion for each study's comparison area
Perc
ent
100806040200
100
80
60
40
20
0
Shape 0.5Scale 29.78N 13
HORS251 Empirical CDF and Gamma with shape=1/ 2 and scale=fitted
k=15k=15
2525
Other crime data for Other crime data for confirmationconfirmation
I used a data set of burglary count I used a data set of burglary count data from 124 anonymised small data from 124 anonymised small areas. The data was from a project areas. The data was from a project described in Tilley et al. 1999. described in Tilley et al. 1999.
Has counts of similar size to those in Has counts of similar size to those in HORS251.HORS251.
2626
Overdispersion
Perc
ent
120100806040200
100
80
60
40
20
0
Shape Scale N0.5 20.24 1180.5 21.26 116
VariableODB23thouODB12thou
CDF Overdispersion of burglary data (both available intervals: 1-2, 2-3)
Interval 1 to 2
Interval 2 to 3
Burglary data from 124 areas
k=10 both
2727
What to conclude about What to conclude about overdispersionoverdispersion
Both the HORS251 and the burglary data show Both the HORS251 and the burglary data show great overdispersion of 10 or more.great overdispersion of 10 or more.
In the case of HORS251 and the CI is (7.9, 38.7).In the case of HORS251 and the CI is (7.9, 38.7).(Similar results are obtained with using the quasi-Poisson (Similar results are obtained with using the quasi-Poisson
Generalised Linear Model in R)Generalised Linear Model in R)
A big problem in HORS251 is essentially confusion about A big problem in HORS251 is essentially confusion about ‘Unit ‘Unit of Analysis’of Analysis’. (It is Area, not Crime-event). (It is Area, not Crime-event)
2828
The Dudley Study (1)The Dudley Study (1) Used a household crime survey. Painter and Used a household crime survey. Painter and
Farrington 1997. in Situational Crime Prevention: Farrington 1997. in Situational Crime Prevention: Successful case studies.Successful case studies. (Also there’s the Stoke on (Also there’s the Stoke on Trent Study) Trent Study)
Question “Did you experience crime in the past year Question “Did you experience crime in the past year if so how many?”if so how many?”
Two areas supposedly matched; one has lighting Two areas supposedly matched; one has lighting increased, the other stays the same. The household increased, the other stays the same. The household survey was carried out before and after the new survey was carried out before and after the new lighting introduced. Households were planned to be lighting introduced. Households were planned to be linked before and after, but did linked before and after, but did notnot happen (nor in happen (nor in Stoke)! Stoke)!
The reports of the studies make great claims of The reports of the studies make great claims of success of the effectiveness of lighting. Claimed by success of the effectiveness of lighting. Claimed by the Institution of Lighting Engineers as 'Proof' that the Institution of Lighting Engineers as 'Proof' that lighting is effective against crime. lighting is effective against crime.
2929
The Dudley Study (2):The Dudley Study (2):Some Problems uncoveredSome Problems uncovered
Eventually I was given some of the data. Eventually I was given some of the data. (With a limited number of background (With a limited number of background variables however.)variables however.)
Markedly different crime rates at the start Markedly different crime rates at the start between the 2 areas.between the 2 areas.
One tailed testing used to claim a One tailed testing used to claim a statistically significant effect.statistically significant effect.
Overdispersion: Variance of the number of Overdispersion: Variance of the number of crimes per household is much larger than crimes per household is much larger than the mean, therefore Poisson methods are the mean, therefore Poisson methods are inappropriate. inappropriate.
3030
The Poisson Model is Inappropriate.The Poisson Model is Inappropriate.(See below, the distribution of crime counts for households.)(See below, the distribution of crime counts for households.)
0.2
.4.6
Pre
dic
ted P
r(y=
k)
from
pois
son/O
bserv
ed P
r(y=
k)
from
pois
son
0 2 4 6 8 10Count
Predicted Pr(y=k) from poisson Observed Pr(y=k) from poisson
Fitted and observed for Poisson Model
3131
The Dudley Study (3):The Dudley Study (3):Some problems uncovered cont.Some problems uncovered cont.
Differential loss to follow up.Differential loss to follow up. Old people are much less prone to experience crime Old people are much less prone to experience crime
and their number is much reduced due to loss to and their number is much reduced due to loss to follow-up in the comparison area. So the relative follow-up in the comparison area. So the relative composition changes during the experiment. composition changes during the experiment.
Results are very sensitive to the loss or addition of Results are very sensitive to the loss or addition of just one personjust one person
But importantly there is But importantly there is correlation between correlation between householdshouseholds, giving extra overdispersion , giving extra overdispersion (variability).(variability).
Essentially it’s a Essentially it’s a non-randomised two-cluster non-randomised two-cluster trialtrial..
3232
Spatial Correlation (1)Spatial Correlation (1)
An expression can be derived for the variance of An expression can be derived for the variance of
ln(CPR) for a household survey, before and ln(CPR) for a household survey, before and after, intervention-comparison study, i.e. of after, intervention-comparison study, i.e. of the Dudley type. This includes, in addition to the Dudley type. This includes, in addition to the variability between households, the variability between households, bothboth: :
(1)(1) correlations within households between times. correlations within households between times.
(2)(2) correlations correlations betweenbetween households at any one households at any one time. time.
3333
Spatial Correlation (2)Spatial Correlation (2)
What you get basically is the expression you What you get basically is the expression you would get if you ignored the correlation would get if you ignored the correlation between households at one time, i.e. between households at one time, i.e. ignored the spatial correlation, multiplied ignored the spatial correlation, multiplied by the ‘Design Effect’, Deff. (Just as in by the ‘Design Effect’, Deff. (Just as in clustered surveys / trials)clustered surveys / trials)
Deff=(1+(n-1) ρDeff=(1+(n-1) ρss))
ρρss = the spatial correlation = the spatial correlation
n = the number in a cluster, i.e. arean = the number in a cluster, i.e. area
3434
Spatial Correlation (3)Spatial Correlation (3) The spatial correlation was not taken into The spatial correlation was not taken into
account in the Dudley and Stoke analyses thus account in the Dudley and Stoke analyses thus ignoring the fact that neighbours ‘share risk’.ignoring the fact that neighbours ‘share risk’.
An expression for the variance of the logarithm An expression for the variance of the logarithm of the Cross Product Ratio CPR is:of the Cross Product Ratio CPR is:
1 1 1 1 1 1ln 2 tVar CPR λDeff r
a b c d ab cd
3636
The response given to my The response given to my pointing out that pointing out that
overdispersion exists (1)overdispersion exists (1)The expression for the household survey type The expression for the household survey type
study, that I give above, study, that I give above, but without Deffbut without Deff was used on the original Dudley Poisson was used on the original Dudley Poisson result to give a variance adjustment of only result to give a variance adjustment of only about 3 about 3 (i.e. just (i.e. just λ estimate)λ estimate). This . This overdispersion adjustment was then applied overdispersion adjustment was then applied in the meta-analysis for in the meta-analysis for allall 13 studies. 13 studies.
See Addendum to HORS251 (added in Sept. See Addendum to HORS251 (added in Sept. 2003), with which I most profoundly disagree, 2003), with which I most profoundly disagree, ….even though my name is mentioned!….even though my name is mentioned!
3737
The response given to my The response given to my pointing out that pointing out that
overdispersion exists (2)overdispersion exists (2)Additionally, Farrington and Welsh (2004), Additionally, Farrington and Welsh (2004),
following my own short article in the BJC, following my own short article in the BJC, cite the cite the geometricgeometric mean of the s mean of the s22//xx of of the 13 studies of HORS251 to justify a the 13 studies of HORS251 to justify a small value. small value.
However, as I have indicated here, it is the However, as I have indicated here, it is the arithmeticarithmetic mean which is appropriate, mean which is appropriate, showing that the overdispersion is much showing that the overdispersion is much bigger, with a value of something like 15.bigger, with a value of something like 15.
3838
The response given to my The response given to my pointing out that pointing out that
overdispersion exists (3)overdispersion exists (3) Farrington and Welsh justify their Farrington and Welsh justify their
overdispersion estimate because if one overdispersion estimate because if one divides the original heterogeneity statistic, divides the original heterogeneity statistic, Q=60, by their favoured estimate an Q=60, by their favoured estimate an ‘acceptable’ heterogeneity statistic results! ‘acceptable’ heterogeneity statistic results! Note it is usual to use Q to uncover anomalies Note it is usual to use Q to uncover anomalies in the data rather than remove them! The in the data rather than remove them! The larger value of overdispersion, 15, would larger value of overdispersion, 15, would indicate excessive homogeneity Q indicate excessive homogeneity Q revisedrevised = = 60/15 as might result from publication bias. 60/15 as might result from publication bias. (There is no register for study protocols, which (There is no register for study protocols, which would guard against publication bias).would guard against publication bias).
3939
Lack of Equivalence between Lack of Equivalence between AreasAreas
Invariably it is the most crime-ridden Invariably it is the most crime-ridden area that gets the lighting, whereas the area that gets the lighting, whereas the relatively crime-free ‘relatively crime-free ‘control control ’ is not re-’ is not re-lit. So there is lack of equivalence at lit. So there is lack of equivalence at the start. One effect of this is to allow the start. One effect of this is to allow ‘‘regression towards the meanregression towards the mean’’ to to operate. (see later) operate. (see later)
The name The name ‘Control‘Control Area’ is a misnomer. Area’ is a misnomer. ‘‘ComparisonComparison Area’ is a better name. Area’ is a better name.
4040X The before measurement
Y T
he a
fter
mea
sure
men
t
Cloud ofDataPoints
Line of Equality
0 10050
0
50
100
Line of mean of Y for a given X
4141
The response given to the lack of The response given to the lack of equivalence between the 2 areas. equivalence between the 2 areas. (RTM)(RTM) ‘‘Regression towards the mean’ (RTM) has Regression towards the mean’ (RTM) has
not been acknowledged to be a problem, not been acknowledged to be a problem, after I pointed it out.after I pointed it out.
The burglary data shows RTM nicely. The burglary data shows RTM nicely. Splitting the data for the 124 areas into 2, Splitting the data for the 124 areas into 2, above or below the mean burglary rate in above or below the mean burglary rate in the first year, exhibits a tendency in the the first year, exhibits a tendency in the following year for the high burglary rate following year for the high burglary rate group to show a fall and the low burglary group to show a fall and the low burglary rate group to show a rise.rate group to show a rise.
4242
RTM Example from the Burglary RTM Example from the Burglary Data Data
Rate1 Above Its Mean or Not (1=Above the Mean)Incr
ease
in B
urg
lary
Count
goin
g fro
m P
eriod 1
to 2
*10
200
100
0
-100
-200
-300
-400
-71.0455
5.875
Increase in Burglary Count period 1 to 2 Depending on whether the rate was above the mean at period 1, or not
(Seen in period 2 to 3 also. And using rate, rather than (Seen in period 2 to 3 also. And using rate, rather than count)count)
4343
Regression Towards the Mean RTM: Regression Towards the Mean RTM: Seeing effects which aren’t thereSeeing effects which aren’t there
A statistical novice might interpret the fact A statistical novice might interpret the fact that the high burglary rate group shows that the high burglary rate group shows a reduction in burglaries (-71), as a reduction in burglaries (-71), as opposed to the low rate group (+6), as opposed to the low rate group (+6), as evidence of ‘something important going evidence of ‘something important going on’ rather than just what is expected on’ rather than just what is expected when you have correlated data. RTM when you have correlated data. RTM follows from correlation. (As Francis follows from correlation. (As Francis Galton discovered more than a century Galton discovered more than a century ago, in the 1880s.)ago, in the 1880s.)
4444
The response given to the lack of The response given to the lack of equivalence between the 2 areas. equivalence between the 2 areas.
(RTM)(RTM) Farrington and Welsh (2006) claim that RTM is a Farrington and Welsh (2006) claim that RTM is a
not problem because the effect in crimes counted not problem because the effect in crimes counted in 250 Police ‘Basic Command Units’ going from in 250 Police ‘Basic Command Units’ going from 2002/3 to 2003/4 showed only small effect. This is 2002/3 to 2003/4 showed only small effect. This is hardly surprising as the areas and hence the hardly surprising as the areas and hence the number of crimes counted are an order of number of crimes counted are an order of magnitude larger than in HORS251 so the year to magnitude larger than in HORS251 so the year to year correlation is high. Note Wrigley (1995) “This year correlation is high. Note Wrigley (1995) “This tendency for correlation coefficients to increase in tendency for correlation coefficients to increase in magnitude as the size of the areal unit involved magnitude as the size of the areal unit involved increases has been known since the work of increases has been known since the work of Gehlke and Biehl (1934)”.Gehlke and Biehl (1934)”.
4545
Bristol study revisitedBristol study revisitedA lighting benefit effect with p=0.011 is claimed in A lighting benefit effect with p=0.011 is claimed in
reply to me. But this depends on an arbitrary, reply to me. But this depends on an arbitrary, specific regression model that requires the specific regression model that requires the variance to be the same in both areas and to variance to be the same in both areas and to include a linear time trend, include a linear time trend, identicalidentical in both areas, in both areas, but which is not ‘statistically significant’.but which is not ‘statistically significant’.
On the other hand, a model which just uses the crime On the other hand, a model which just uses the crime count in the comparison area as predictor of crime count in the comparison area as predictor of crime in the re-lit area (in the spirit of HORS251) shows in the re-lit area (in the spirit of HORS251) shows no stat. sig. effect.no stat. sig. effect.
Does the data really look as if there is such a clear Does the data really look as if there is such a clear effect, i.e. one which would only occur 1 time in effect, i.e. one which would only occur 1 time in 100, when there is in fact no lighting benefit? 100, when there is in fact no lighting benefit?
4646
The Bristol data againThe Bristol data again
1990.01989.01988.01987.01986.0
1500
1400
1300
1200
1100
1000
900
800
700
600
Year
No.
Crim
es
1311
1464
1160
1237
14241374
142915171414
840840846
727
648
833838
685630
Number of Crimes Reported(in half-year periods)
Brighter Lighting
Control
New lighting introduced from July 87 to March 89 as marked onthe time axis.
It seems to me that this hardly presents clear evidence for lighting benefit! There’s a big problem of model uncertainty.
4747
Cost benefit analysesCost benefit analyses
Cost benefit analysis has been done based on very Cost benefit analysis has been done based on very few studies by lighting and crime researchers (and few studies by lighting and crime researchers (and gives a highly favourable result for lighting). gives a highly favourable result for lighting). However doing this only compounds the problem. As However doing this only compounds the problem. As an unknown, unproven benefit/harm is being an unknown, unproven benefit/harm is being compared with uncertain costs.compared with uncertain costs.
We need to get much better information to do such We need to get much better information to do such an exercise properly otherwise it tends to look an exercise properly otherwise it tends to look ‘scientific’ to the eye of a novice, when in fact it ‘scientific’ to the eye of a novice, when in fact it isn’t, because of flimsy data and method. isn’t, because of flimsy data and method.
4848
‘‘Researchvertising’Researchvertising’
Unsurprisingly HORS251 and the Unsurprisingly HORS251 and the Dudley Study are used by the Dudley Study are used by the lighting industry to promote its lighting industry to promote its wares. Also ‘responsibilities under wares. Also ‘responsibilities under the crime and disorder act’ are the crime and disorder act’ are invokedinvoked
4949
My take on lighting and My take on lighting and crimecrime
It may be that lighting reduces crime, or may be it It may be that lighting reduces crime, or may be it increases crime. We have to look at the evidence as increases crime. We have to look at the evidence as given. The conclusion, at present, is: We do not given. The conclusion, at present, is: We do not know....yet we ought to know!know....yet we ought to know!
Note, I know of no scientific trials of exterior Note, I know of no scientific trials of exterior 'Security' lighting. So no one knows if this works. 'Security' lighting. So no one knows if this works.
We ought to take a ‘Popperian’ view and entertain We ought to take a ‘Popperian’ view and entertain the possibility of light being ineffective or worse, the possibility of light being ineffective or worse, against crime.against crime.
Of course we all need light at night, to see by. Of course we all need light at night, to see by. (Those concerned about light pollution are basically (Those concerned about light pollution are basically talking ‘lamp-shades’). However there is no sound talking ‘lamp-shades’). However there is no sound evidence we need light to protect us from crime, in evidence we need light to protect us from crime, in spite of claims.spite of claims.
5050
Car AlarmsCar Alarms
It seems that there is little evidence It seems that there is little evidence that car alarms prevent cars being that car alarms prevent cars being broken into etc. broken into etc.
But they do disturb people’s sleep!But they do disturb people’s sleep! Attempt in New York to get them Attempt in New York to get them
banned, (rely on passive methods of banned, (rely on passive methods of risk reduction instead.)risk reduction instead.)
5151
Wider problems of inappropriate Wider problems of inappropriate
methodsmethods The costs of crime and attempts at its reduction The costs of crime and attempts at its reduction
are large.are large. Similar problems probably exist for the Similar problems probably exist for the
evaluation of other area-based crime reduction evaluation of other area-based crime reduction interventions, too. (They certainly do for interventions, too. (They certainly do for HORS252 on CCTV where the same methods as HORS252 on CCTV where the same methods as HORS251 are used on 18 studies, Q=270. HORS251 are used on 18 studies, Q=270. However no effect of CCTV is claimed). However no effect of CCTV is claimed). Problems seem to be encouraged by the Problems seem to be encouraged by the ‘Maryland Scientific Methods Scale’ which ‘Maryland Scientific Methods Scale’ which seems to suggest that weaker designs, than seems to suggest that weaker designs, than RCTs, might suffice. RCTs, might suffice.
We do need to have proper evidence to decide We do need to have proper evidence to decide ‘what works’ in crime and in all spheres . ‘what works’ in crime and in all spheres .
5252
Crime Prevention.….an Crime Prevention.….an ‘Art’ ?‘Art’ ?
Or is it more like 17Or is it more like 17thth Century Century medicine?!medicine?!
5353
Some considerations for Some considerations for evaluation of anythingevaluation of anything
Interventions are expensive, as are the Interventions are expensive, as are the consequences, so their effects need to be consequences, so their effects need to be researched to a very high standard…e.g. the researched to a very high standard…e.g. the necessity of using randomisation. necessity of using randomisation.
Also the target population could be depleted Also the target population could be depleted through poorly conducted studies.through poorly conducted studies.
Evaluation needs to be done right.Evaluation needs to be done right. Caution is needed with systematic Caution is needed with systematic
reviewing/meta-analysis as it involves moving reviewing/meta-analysis as it involves moving away from the primary sources.away from the primary sources.
It is possible to do ‘roll out’ of a programme in a It is possible to do ‘roll out’ of a programme in a way that is amenable to proper scientific way that is amenable to proper scientific evaluation.evaluation.
5454
Much can be borrowed from Much can be borrowed from
methods in health researchmethods in health research For example in area-based crime reduction For example in area-based crime reduction The Methods of Cluster Randomised Trials a appropriate.The Methods of Cluster Randomised Trials a appropriate.See e.g. See e.g. Ukoumunne et al (1999) Ukoumunne et al (1999) andandCampbell et al (2004)….CONSORT extension for Cluster Campbell et al (2004)….CONSORT extension for Cluster Randomised Trials.Randomised Trials.
Also, post-implementation surveillance is highly desirable for Also, post-implementation surveillance is highly desirable for any programme.any programme.
(Note; it is problematic enough to determine What Works in (Note; it is problematic enough to determine What Works in healthcare where the ‘unit’ is ‘person’, through e.g. healthcare where the ‘unit’ is ‘person’, through e.g. dissemination bias.)dissemination bias.)
5555
High standards are needed for High standards are needed for
evidenceevidence NeedNeed Adequate funding to provide quality research, as the Adequate funding to provide quality research, as the
costs of rolling out programmes on a national scale costs of rolling out programmes on a national scale are huge. Need to be aware of the costs of are huge. Need to be aware of the costs of implementing ineffective or counterproductive implementing ineffective or counterproductive programmes.programmes.
Effective random allocation for experiments.Effective random allocation for experiments. Blinding where possible, e.g. of assessors.Blinding where possible, e.g. of assessors. Pre-publication of protocols.Pre-publication of protocols. Open-access in order to check any work. Open-access in order to check any work.
(As, for example, reports/papers may confuse (As, for example, reports/papers may confuse standard deviations and standard errors or, as above, standard deviations and standard errors or, as above, not recognise correlated data.) not recognise correlated data.)
Let’s ultimately have the raw data!Let’s ultimately have the raw data!
5656
ConclusionConclusion There is no good scientific evidence to show that There is no good scientific evidence to show that
night time lighting reduces crime. (In spite of the night time lighting reduces crime. (In spite of the claims).claims).
The effect of some other crime reduction methods is The effect of some other crime reduction methods is also questionable, because of inherent weakness of also questionable, because of inherent weakness of research methods used, and needs examination. research methods used, and needs examination. (Look at the variation in comparison areas.). (Look at the variation in comparison areas.). Variation in crime is poorly understood.Variation in crime is poorly understood.
High quality studies, sound in design through to High quality studies, sound in design through to conclusion, are necessary in important and costly conclusion, are necessary in important and costly matters matters
Statisticians have a vital role in finding ‘what works’.Statisticians have a vital role in finding ‘what works’. Eminence in any subject does not guarantee the Eminence in any subject does not guarantee the
correctness of statistical pronouncements.correctness of statistical pronouncements.
5757
Final points Final points Do be investigative. Don't just mind your own Do be investigative. Don't just mind your own
business.business.--------------------------------------------------------------------------------------------------------------------Stephen Senn quote “Trust nobody, check everything” Stephen Senn quote “Trust nobody, check everything”
from ‘Fear and Loathing in Pharmaceutical Statistics’, from ‘Fear and Loathing in Pharmaceutical Statistics’, RSS2002RSS2002
HG Wells quote “Statistical thinking will one day be as HG Wells quote “Statistical thinking will one day be as necessary for efficient citizenship as the ability to necessary for efficient citizenship as the ability to read and write”. read and write”.
The RSS magazine Significance which aims to The RSS magazine Significance which aims to encourage ‘statistical thinking’ June 2005 p62 has an encourage ‘statistical thinking’ June 2005 p62 has an article by me ‘Evaluating area-wide crime reduction article by me ‘Evaluating area-wide crime reduction measures’.measures’.
5858
ReferencesReferencesCampbell MK, Elbourne DR, Altman DG for the CONSORT
Group (2004) CONSORT statement: extension to cluster randomised trials. BMJ 328 702-708. http://bmj.bmjjournals.com/cgi/reprint/328/7441/702
Farrington D.P. and Welsh B.C. (2002) The Effects of Improved Street Lighting on Crime: A Systematic Review, Home Office Research Study 251, http://www.homeoffice.gov.uk/rds/pdfs2/hors251.pdf
Farrington D.P. and Welsh B.C. (2004) Measuring the Effects of Improved Street Lighting on Crime: A reply to Dr. Marchant The British Journal of Criminology 44 448-467 http://bjc.oupjournals.org/cgi/content/abstract/44/3/448
Farrington D.P. and Welsh B.C. (2006) How Important is Regression to the Mean in Area-Based Crime Prevention Research?, Crime Prevention and Community Safety 8
Marchant P.R. (2004) A Demonstration that the Claim that Brighter Lighting Reduces Crime is Unfounded The British Journal of Criminology 44 441-447 http://bjc.oupjournals.org/cgi/content/abstract/44/3/441
5959
References continuedReferences continuedMarchant P.R. (2005) What Works? A Critical Note on the Evaluation of Crime
Reduction Initiatives,Crime Prevention and Community Safety 7 7-13 www.extenza-eps.com/extenza/loadHTML?objectIDValue=63645&type=abstract
Painter, K. and Farrington, D. P. (1997) The Crime Reducing Effect of Improved Street Lighting: The Dudley Project, in R.V. Clarke ed., Situational Crime Prevention: Successful case studies 209-226 Harrow and Heston, Guilderland NY.
Shaftoe, H (1994) Easton/Ashley, Bristol: Lighting Improvements, in S. Osborn (ed.) Housing Safe Communities: An Evaluation of Recent Initiatives 72-77, Safe Neighbourhoods Unit, London
Tilley N., Pease K., Hough M. and Brown R. (1999) Burglary Prevention: Early Lessons from the Crime Reduction Programme, Crime Reduction Research series Paper1 London Home Office
Ukoumunne, O. C., Gulliford, M. C., Chinn, S., Sterne, J. A. C., Burney, P. G. J., and Donner, A. (1999). Evaluation of Health Interventions at Area and Organisation level, British Medical Journal, 319 376-379 http://bmj.bmjjournals.com/cgi/content/full/319/7206/376
Wrigley N., Revisiting the Modifiable Areal Unit Problem and Ecological Fallacy Wrigley N., Revisiting the Modifiable Areal Unit Problem and Ecological Fallacy pp49-71 in Gould PR, Hoare AG and Cliff AD Eds Diffusing Geography: pp49-71 in Gould PR, Hoare AG and Cliff AD Eds Diffusing Geography: Essays for Peter Haggett Essays for Peter Haggett
6060
Further material written in this area Further material written in this area
I wrote ‘Failing to measure any effect of increased I wrote ‘Failing to measure any effect of increased lighting on crime: A reply to Profs. Farrington and lighting on crime: A reply to Profs. Farrington and Welsh’ It was sent to the Home Office in Dec. 2004Welsh’ It was sent to the Home Office in Dec. 2004
www.imresearch.org/praxiscentre/Papers/RevReplyToFW1www.imresearch.org/praxiscentre/Papers/RevReplyToFW1B.pdfB.pdf
Dr. Barry Clark of the Astronomical Society of Victoria, Dr. Barry Clark of the Astronomical Society of Victoria, Australia has written much on the oft repeated but Australia has written much on the oft repeated but seemingly dubious claim that lighting reduces crime.seemingly dubious claim that lighting reduces crime.
http://www.asv.org.au/index.php?http://www.asv.org.au/index.php?option=com_content&task=view&id=33&Itemid=76option=com_content&task=view&id=33&Itemid=76
6161
Appendix: Other Useful Resources to Appendix: Other Useful Resources to assist in evidence based workassist in evidence based work
1.1. RCTs designing and writing upRCTs designing and writing upThe CONSORT statement (CONSORT = The CONSORT statement (CONSORT = Consolidated Reporting Of Randomised Trials) is Consolidated Reporting Of Randomised Trials) is very useful for reporting results and also thinking very useful for reporting results and also thinking about design. about design.
www.consort-statement.orgwww.consort-statement.org 2.2. The The QUORUMQUORUM statement: Quality Reporting of statement: Quality Reporting of
Meta-analyses.Meta-analyses.3.3. The The American Association for Public Opinion American Association for Public Opinion
ResearchResearch www.aapor.orgwww.aapor.org has useful information has useful information about Standards and Best Practices in survey about Standards and Best Practices in survey methodsmethods