From sifting to synthesis

Evidence Base Camp: February 2014

Levin WhellerPractice Development TeamResearch Analysis and Information Unit

This session…1. Recap on REAs2. While you were away…

• Peer review of sifting and the task ahead

3. Mapping research evidence• Systematic Mapping – the what and the why• An example: Mapping Evidence on Public Order Tactics

4. Appraising research evidence• The Maryland Scale of Scientific Methods• Some hypothetical examples• Beyond Maryland: non-impact questions • Qualitative methods

5. Completing a pro forma• Practical group work

Recap

Systematic reviews and “what works”

• Explicit objective/ question

• Studies are selected on explicit criteria • Thorough search for studies • Each study is screened according to uniform criteria • Reasons for excluding studies are clearly documented

Rapid Evidence Assessments

• Use systematic principles and the same process as a systematic review…

• …but make compromises given available time and resources

• Pragmatic and transparent approach

Some key principles

• Demonstrate consistency in searching/ sifting

• Document search and sift process– Process should be transparent and repeatable

• Specify required quality of evidence – Systematic reviews only?– Pre-post studies only?– All ‘empirical’ papers?

• Be explicit/ transparent about the limitations of the approach

The process (in a nutshell)

1. Draft search terms

2. Draft sift criteria

3. Sift received abstracts

4. Request full papers

5. Read and ‘grade’ papers

6. Write it up (‘synthesis’)

SiftingIdentifying truly relevant literature

SearchingIdentifying potentially relevant literature

SynthesisSummarising what relevant literature tells us

SearchingIdentifying potentially relevant literature

• Boolean is a powerful tool and can be used to create quite complex searches. The standard Boolean operators are:

ANDORNOT (or AND NOT) (advanced users only!)

• Boolean is most effective when used in combination with:

PARENTHESES (round brackets)TRUNCATION e.g. politic*WILDCARDS e.g. randomi?eDOUBLE QUOTES for specific phrases

The Basics Boolean operators

Domestic violence searchSearch terms1. ((((Domestic OR Familial OR

Marital OR Spousal OR Partner) Near/2 (Abus* OR violen* OR control* OR coerc*)) OR "domestic homicide" OR "marital rape" OR "intimate partner violence" OR (coercive NEAR/2 (control OR behavio*r*)))

2. AND (risk AND (predict* OR indicat* OR factor* OR management OR identif* OR assess* OR prevent* OR likelihood OR probability))

3. AND ("systematic review" OR "literature review*" OR trial* OR "RCT" OR experiment* OR evaluat* OR "best practice*" OR "good practice*" OR effective* OR assess* OR "What Works" OR impact* OR success*))

Riskc 1m papers

‘What w

orks’

c 2m papers

DA/ DV

c 75k papers

Initial search identified 1,531 abstracts to sift

Searching as a ‘blunt tool’Research Question: What interventions have been shown to be effective in preventing or reducing theft from the person?

Theft from person synonyms include: Robbery, Mugging, Dipping, Pick pocketing, Assault with intent to rob, Snatching, Theft from person, Bag theft, Bag snatch, Personal theft…

Profile on circadian blood pressure and the influencing factors in essential hypertensive patients after treatment

So how do we end up with an abstract titled…

Returned by theft search…Yuan Gang Qiu et al. Profile on circadian blood pressure and the influencing factors in essential hypertensive patients after treatmentNon-dippers were defined as those whose nocturnal decrease in mean systolic BP and/or mean diastolic BP was < 10% of the daytime BP. Binary logistic regression analysis was used to evaluate the correlation between circadian blood pressure profile and factors as gender, age, height, body mass index (BMI), family history of premature cardiovascular disease, women under age 65 or men under age 55, smoking habits, grade of hypertension, and strategy of antihypertensive drugs.

Similarly, the incidence of non-dippers in patients of overweight (24 </= BMI < 28) and obesity (BMI >/= 28) were 3.0 and 4.8 times of those in subjects of normal weight (P = 0.003 and 0.009, respectively). Compared with patients treated with long-acting calcium channel blockers (CCBs), patients treated with angiotensin-converting enzyme (ACE) inhibitors or angiotensin II receptor blockers (ARBs) alone had less prevalence of nondippers (OR = 0.139, P = 0.010).

SiftingIdentifying truly relevant literature

Sift criteria

• The same set of questions are applied to all abstracts to attempt to remove bias/ overt ‘judgement calls’.

• Typically, a lot of papers (up to 95%!) identified by the search are excluded at this stage of the process.

Question Answer ActionQ1. Does the paper consider theft

from the person or relevant synonyms?

No Exclude

Yes Go to Q2

Unclear Go to Q2

Q2. Does the paper consider interventions that aim to reduce/prevent theft from the person or relevant synonyms?

No Exclude

Yes Go to Q3

Unclear Exclude

Q3. Does the paper include empirical data or methods?

No Exclude

Yes Include

Unclear Can’t exclude

While you were away…

Peer Review: Second sift…

‘Red’ abstracts – ‘Dip sample’ taken, e.g. one in 10 ‘red’

abstracts were peer reviewed– Check for ‘false-exclusions’

‘Green’ and ‘Amber’ abstracts– All resifted to review against the inclusion

criteria– Check for ‘false-positives’– Decide where possible on ‘Ambers’

Final list of included studies sent to library…

Second sift…REA Abstracts First sift Second sift

Domestic Abuse 2,678 622 411

Public Order 572 182 38

• Always expect to reduce the number of studies substantially through second sifting

• Standard process for all REAs and SRs

• Dual coding and often used and disagreements discussed between coders to make a final decision

• Clarify queries/ refine criteria as a group

• Speed of EBC does not enable all of this!!!

While you were away…Area Abstracts

returnedIncluded after 1st

sift

Included after 2nd

sift

Available for

synthesis

Acute Mental Health Crisis

1,408 173 55 38

Barriers to career progression

1,205 153 82 46

Reducing theft from the person

844 192 82 58

Supporting mental well-being

1,229 162 93 72

Responding to prostitution

979 195 87 48

SynthesisSummarising what relevant literature tells us

Systematic Mapping…• Methodology developed by EPPI-Centre• Useful first step of synthesis process

Mapping can:1. Describe the nature and coverage of

research in the topic area2. Identify gaps in the research literature3. Describe the design of studies and direct

further review work

Systematic Mapping…What does it involve:• Collection of key information about each study into a pro forma/ template

Systematic Mapping…Collate pro forma results into a searchable bibliographic database

What is the evidence on Public Order Tactics?Country of focus

0

5

10

15

20

25

UK

USA

Canada

Europe

Multiple

What is the evidence on Public Order Tactics?Research Method

0

5

10

15

20

Observations Interviews Secondary/ DocumentSurvey Scientific Newspaper analysisOther

Public Order Tactics - MappedOverall: There is only limited research available on public order policing tactics. Only 38 papers were found to be relevant and based on empirical methods.

Age: Most of the research (32/38 papers) has been published since 2000.

Methods: Most papers used qualitative research methods, typically involving observations (n=19) or interviews (n=18). Only one study used experimental methods.

Public Order Tactics - MappedUnit of study: Only five studies focussed directly on public order tactics. Instead, studies primarily focussed on an event or series of events (n=17) or the police more generally (n=10).

Tactics: Of the 30 approved public order tactics, only nine were mentioned in the research evidence identified for this map. The most frequently mentioned tactics were containment (n=5), Police Liaison Teams (n=3) and CS smoke (n=3).

Appraising research evidence

Maryland Scientific Methods Scale (SMS).(AKA Maryland Scale)

• Developed by Sherman et al (1997)• Established method for judging the quality of

impact studies• Useful for judging how much weight to give

different studies identified in a ‘what works’/ ‘what’s effective’ REA

• Limited use for judging qualitative studies and questions asking ‘how’ things work

What does ‘good’ or ‘robust’ ‘what works’ evidence look like?

Systematic Reviews(Based on level 3-5 studies)

5 Randomised controlled trials

4Before/after measures

Multiple site comparisons

3Before/after measuresTwo site comparisons

2Before/after measures

No comparison site

1One-off measure

No comparison site

Study designs increasingly rule out potential alternative causes

Statements about ‘what works’

Statements about ‘what’s promising’

Study designs cannot rule out potential alternative causes

Statements about possible impact

‘Level 1’ Evidence“Two months after the introduction of mobile information, officers were spending 55% of their time out on patrol”

Problems?

1. Can’t rule out other explanations2. Time spent on patrol before mobile info? 3. Is 55% good/ bad/ average?4. No before/ after measures5. No comparison group

Can’t say if mobile information makes a difference/ is worthwhile

‘Level 3’ evidence

Problems?

1. Still can’t rule out other explanations for change2. Can’t control for differences between the two areas –

different culture, different case load, management, etc.

% of time on patrol Pre Post

Western District(receives intervention)

43% 50%

Eastern District(no intervention)

46% 50%

We still can’t be sure that improvements are down to the intervention rather than other factors

“Two months after the introduction of mobile information, officers in the Western District had increased their time on the street by 7 percentage points, while officers in the Eastern District had seen an increase of 4 percentage points…”

2,000 officers in force

RCTs - ‘gold standard’ evaluation design

A randomsample of

600 officers

300 officers in treatment

group

Outcomesmeasured

300 officers in control

group

Get mobile data

Outcomesmeasured

Businessas

usual

‘Level 5’ Evidence - RCTs

1. Participants are randomly selected from the population and randomly assigned to each group to minimise the chance of bias

“Two months after the start of the trial, officers with access to mobile data were spending 55% of their time out on patrol, whilst officers in the control group were spending 43% of their time out on patrol”

2. Randomisation helps us rule out other potential explanations for differences in outcome measures

3. A control group means we can compare differences in outcomes between groups where the only difference is the intervention (e.g. mobile data

4. RCTs enable us to make strong statements about cause and effect

We can say that officers with access to mobile data spend more time on the street… in this context.

Systematic reviews

Taken from: Mazerolle, L., Bennett, S., Davis, J., Sargeant, E. and Manning, M. (2013) Procedural Justice and Police Legitimacy: A Systematic Review of the Research Evidence. Campbell Systematic Reviews 2013:1.

Overall, legitimacy interventions resulted in a large, significant increase in positive perceptions of police.

Systematic Reviews(Based on level 3-5 studies)

5 Randomised controlled trials

4Before/after measures

Multiple site comparisons

3Before/after measuresTwo site comparisons

2Before/after measures

No comparison site

1One-off measure

No comparison site

Study designs increasingly rule out potential alternative causes

Statements about ‘what works’

Statements about ‘what’s promising’

Study designs cannot rule out potential alternative causes

Statements about possible impact

What does ‘good’ or ‘robust’ ‘what works’ evidence look like?

Beyond Maryland…• The Maryland scale is focussed on impact questions… and

studies might answer a range of other questions…

• Qualitative studies don’t fit into the Maryland dichotomies– You can have different quality qualitative work as well

Question type Example

Needs What do people want or need?

Process Why/how does it work?

Implementation What is required to make it work?

Correlation What relationships are seen between phenomena?

Attitudinal What do people think? What are their experiences?

Economic How much does it cost and with what benefit/harm?

• How ‘good’ a study is will depend upon how appropriate the methods used are for answering the research question…

Quality of Qualitative Research

• Frameworks to assess/ appraise the quality of qualitative research use ‘principles’ rather than hard-and-fast rules

• Cabinet Office e.g. in 2003 sets key principles being that qualitative research should be: • contributory in advancing

wider knowledge • defensible in design • rigorous in conduct • credible in claim

http://www.civilservice.gov.uk/wp-content/uploads/2011/09/a_quality_framework_tcm6-38740.pdf

What we are doing at EBC

• Mapping the evidence in our 5 areas…• Presenting key info in charts• Drawing initial conclusions from studies in

each area• Some studies will not be impact studies• Presentation of findings to NPLs and senior

staff across the service…

Completing a pro forma

Info required Try looking first in…

Aims Background/ Introduction section

Methods Methods section

Quality of evidence Methods section

Findings Discussion/ Results section

Conclusions Results/ Conclusions section

1. Basic info should be recorded• Reference (title/ author)• Date of publication• Country

2. Check if the study should be included• Check the full paper against the inclusion criteria• If the study meets the inclusion criteria, complete the rest of the pro forma• If not, exclude the study and note your reasons why

Synthesis in practice• Complete a pro forma for 2 articles– 1. GMP study summary – 15 mins– 2. An included article for your area – 15 mins

• Look at the methods of the study and try to ‘score’ it on the Maryland Scale– Ignore the section on inclusion criteria– Cover findings/ conclusions if you have time but

the methods are your focus– Some examples might not fit the scale…

Synthesis in practiceGMP study summary• 10 mins to individually complete as much of

the pro forma as possible• 5 mins to discuss with neighbour/ compare

findings• Group feedback

GMP study summary (1/2)The GMP procedural justice training experiment: Wheller et al

UK (2013) Policing

Test the impact of training on officer attitudes, officer behaviour and victim satisfaction

Training

576 Random

GMP study summary (2/2)

Training had a positive impact on officer attitudes, officer behaviour and victim satisfaction

Training works (in this context!)

Synthesis in practiceIncluded study from your area• 10 mins to individually complete as much of

the pro forma as possible• 5 mins to discuss with neighbour/ compare

findings• Group feedback

Theft study summary (1/2)Does Heroin Prescription Reduce Crime?: Killias & Rabasa

Switzerland 1997 Policing/ Health

Study of the impact of heroin prescription programme on crime rates in Switzerland

Drug prescription/ treatment

319 Non-random

Theft study summary (2/2)

Number of outcomes – comparison of rates of (self reported) crime pre and post intervention.

Heroin prescription reduces (self reported) criminality. Police statistics also show reductions in crime.

Wellbeing study summary (1/2)Mental, physical, and behavioural outcomes associated with perceived work stress in police officers: Gershon et al

USA 2009 Policing

Study links between perceived stress and impact of ‘coping’ on work stress and health

None – non-impact study

1,072 Non- random

Wellbeing study summary (2/2)

Outlines a number of statistical relationships drawn from the survey, e.g. between job stressors and work stress

“results underscore the need to re-evaluate police training of recruits… to ensure they get training necessary to meet the daily challenges and demands of police work”

Career progression study summary (1/2)Encounters in social cyberspace: e-mentoring for professional women: Headlam-Wells et al

UK 2006 Training

Look at barriers for women in career progression and evaluate the effects of an e-mentoring programme

E-mentoring programme

122 (89) Non-random

Career progression study summary (2/2)

Mentoring had a positive impact on mentee ‘employability skills’. Mentoring helped with mentee development.

Suggestion that mentoring may be particularly important to women (but this study does not compare to a similar group of men…)

Prostitution study summary (1/2)Attitudes regarding criminal justice responses to sex trafficking: Muftic

Bosnia and Herzegovina

? Policing

Examine law enforcement attitudes to sanctioning people involved in the sex trade.

None – non-impact study

363 Self reporting

Prostitution study summary (2/2)

A number of factors are associated with officer support for arrest, etc of trafficking victims, traffickers and customers – including their degree of adherence to prostitution myths

Training for officers “need[s] to include the provision of factual information regarding prostitution and sex trafficking”

MH crisis study summary (1/2)Studying Chicago Police Dept’s Crisis Intervention Team: Watson

USA 2010 Policing

Test the impact of training on officer attitudes, officer behaviour and victim satisfaction

Crisis Intervention Team

216 Matched sites

MH crisis study summary (2/2)

CIT officers directed a significantly greater proportion of subjects to services than non-CIT officers. CIT did not have an impact on arrest decisions.

Challenging to conduct studies of CIT, but possible. Study suggests CIT may improve outcomes of police encounters with persons with mental illness.

What’s next…Area Available for synthesis

Acute Mental Health Crisis 38

Barriers to career progression 46

Reducing theft from the person 58

Supporting mental well-being 72

Responding to prostitution 48

Get synthesising!

In the breakout rooms you’ll have everything you need to turn the available papers for your group into a resplendent and wonderful presentation to wow our senior guests tomorrow…

Good luck!