heads you’re in; tails you’re out: how rcts have evolved in dwp
DESCRIPTION
Heads you’re in; tails you’re out: How RCTs have evolved in DWP. Jane Hall [email protected]. Background. Increasing use of RCTs Emphasis on evidence-based policy Limited UK experience in social policy arena Practical lessons not theoretical debate. Overview. - PowerPoint PPT PresentationTRANSCRIPT
![Page 2: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/2.jpg)
Background
Increasing use of RCTs
Emphasis on evidence-based policy
Limited UK experience in social policy arena
Practical lessons not theoretical debate
![Page 3: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/3.jpg)
Overview
Chronology of RCTs in DWP Site selection and preparation Identifying the eligible population Dealing with resistance Performing the random assignment Monitoring take-up
![Page 4: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/4.jpg)
Chronology of RCTs
Restart Various New Deals Employment Zones JRRP ERA JSA Intervention Pilots ND50+ Mandatory IAP
![Page 5: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/5.jpg)
Site Selection and Preparation
Need commitment from the top
All parties need to buy-in
Set-up is resource intensive
Personal visits
Pilot the approach
![Page 6: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/6.jpg)
Identifying and recruiting the eligible population
Can they be easily identified Self-selection Suitability of the population Monitor P & C Group characteristics Selling techniques Sample sizes: Sub-group analysis
![Page 7: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/7.jpg)
Dealing with resistance
Busting the myth
Significant investment in training at all levels
Aides and FAQs
Scripts
![Page 8: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/8.jpg)
Performing the Random Allocation
Needs to be sophisticated Not open to sabotage/gaming Block allocation: Maintain P:C ratio Different techniques
– NINO– Call Centre – On-line algorithm– Random numbers
![Page 9: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/9.jpg)
Monitoring take-up
Keep track of P & C Group
Ensure only P Group receive the treatment
Monitor key characteristics of P & C Group
Be prepared to redesign the random allocation
![Page 10: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/10.jpg)
Expect the Unexpected
Results may not be what you anticipate
A fair allocation of resources?
Participation rates can be disappointing
![Page 11: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/11.jpg)
Operational ChallengesThe ERA Experience
Jenny Carrino
![Page 12: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/12.jpg)
Overview
The ERA Policy Key Challenges
– Random Assignment (RA) Process– Customer Understanding of RA– Creation of ‘Informal’ Refusers– Jobcentre Plus Target Structure– Technical Assistance
![Page 13: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/13.jpg)
The ERA Policy
To test interventions to improve retention and advancement– Adviser support– Funding for training– Financial Incentives
6 Jobcentre Plus districts Three customer groups
NDLP ND25+ WTC To test the effectiveness of using RA to evaluate social
policy in the UK
![Page 14: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/14.jpg)
Random Assignment 1
Issue: The random allocation process Lessons Learnt
– Importance of transparency in the allocation process– Avoiding contamination
Outcomes– Most customers and staff viewed random assignment
as fair and justified
![Page 15: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/15.jpg)
Random Assignment 2
Issue: The Informed Consent Process Lessons Learnt
– Standardisation - adviser scripts and leaflets Outcomes
– Not everyone fully understood what they had signed up for
– Too much information at initial interviews – conduct RA as a stand alone interview
![Page 16: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/16.jpg)
Random Assignment 3
Issue: Creation of a group of ‘informal’ refusers
![Page 17: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/17.jpg)
What do we mean by Informal Refusers?
ELIGIBLE POPULATION
CUSTOMERS RA’D OR ON SYSTEM
FORMAL REFUSERS
INFORMAL REFUSERS
SIGNIFICANTLY DIFFERENT
![Page 18: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/18.jpg)
Informal Refusers
Why this happened– The decision to use RA – RA to ERA was voluntary– Influences from both advisers and customers
Outcomes– Creation of a ‘third’ group– Analysis to identify whether this group are different to
the ERA population Lesson Learnt
– If possible monitor intake closely against eligible population
![Page 19: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/19.jpg)
Ensuring a Treatment - Targets
Issue: The Jobcentre Plus Target Structure– Some adviser behaviour negatively affected – Senior management buy-in affected
Lessons Learnt– Policies need to reflect the organisations reward
system– Need to be able to monitor and feedback to
implementation managers
![Page 20: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/20.jpg)
Technical Assistance
Issue: Ensuring the effective delivery of RA Lessons Learnt
– US model of on-site RA assistance– Avoiding contamination– Monitoring Performance
Outcomes– Advisers felt supported during the RA period – Initial confusion over the role of TAs– Some districts deferred responsibility of ERA
implementation
![Page 21: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/21.jpg)
Summary of Key Challenges
RA Process Informed Consent Creation of informal refusers Ensuring a Treatment Providing effective support to delivery agents
![Page 22: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/22.jpg)
Jobseekers Allowance (JSA)Intervention Pilots
Jayne Middlemas
![Page 23: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/23.jpg)
JSA Intervention Pilots
JSA Intervention Regime The Pilots Evaluation Random assignment The data Did Random Assignment work? Results
![Page 24: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/24.jpg)
JSA Intervention Regime
First Contact New Jobseeker Interview (NJI) Financial Assessor Interview Fortnightly Jobsearch Review (FJR) 13 week review
![Page 25: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/25.jpg)
The Pilots
Introduced in January 2005 108 Jobcentres in 10 Districts took part Each Jobcentre took part in a single pilot Aim to deliver resource savings on the FJR without
reducing unemployment off-flow rates.
![Page 26: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/26.jpg)
The pilots (cont)
Five different approaches: Excusal of signing for first 13 wks of claim Excusal of signing for first 7 wks of claim Telephone signing Shortened FJR Group signing
![Page 27: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/27.jpg)
The pilots (cont)
Some groups excluded: Part-time workers 16 and 17 year olds People with no fixed abode People known to have had a fraudulent claim in the past
![Page 28: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/28.jpg)
Evaluation
Customers randomly allocated Work study to record resources used Comparison of off-flows Qualitative evaluation
![Page 29: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/29.jpg)
Random Assignment
50% programme, 50% control
ORC International Call Centre
Two methods: Adviser calls immediately prior to each NJI Jobcentre calls at start of day with details of all clients
due to attend an NJI that day
ORC also provided random call-in date
![Page 30: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/30.jpg)
Data
Data collected during random assignment JUVOS data – derived from the Jobseekers Allowance
Payment System (JSAPS) HMRC Employment Data
![Page 31: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/31.jpg)
Did Random Assignment Work?
66,600 randomly assigned 33,100 programme & 33,400 control All pilots and Districts close to 50/50 split
Was everyone assigned? Difficult to answer precisely Number randomly assigned around 90% of total new
claims. Excluded groups likely to account for 8 to 12%
![Page 32: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/32.jpg)
Were People Wrongly Assigned?
19% had no new JSA claim during the pilot Incorrect NI numbers may mean we can’t find some
claims Jobcentres didn’t always inform us of those who failed to
attend Can’t identify excluded groups in the data
![Page 33: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/33.jpg)
Internal Validity
Compared characteristics for programme and control groups
Very little difference was found by gender, age or ethnic origin
Concluded that the control group is well suited to providing a counterfactual for the programme group
![Page 34: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/34.jpg)
External Validity
Pilot Jobcentres account for small proportion of all new JSA claims across the country
Gender, age & ethnicity of new claimants in pilot areas different to country as a whole
Some difference in local unemployment rates Weighted results to take account of differences
![Page 35: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/35.jpg)
Results
13 week excusal pilot
![Page 36: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/36.jpg)
% still claiming after x weeks
0
20
40
60
80
100
0 2 4 6 8 10 12 14 16 18 20 22 24 26
Weeks
Per
cen
tag
e
Control Programme
![Page 37: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/37.jpg)
Length of Claim
0
0.2
0.4
0.6
0.8
1
1.2
0 28 56 84 112 140 168
Length of claim (days)
Per
cen
tag
e
Control Programme
![Page 38: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/38.jpg)
Results
Average length of claim is 5.9 days greater in programme group than in control group
Weighting the results to be representative nationally suggests an increase in 6.1 days in average length of claim
No difference in the proportions who moved into work
![Page 39: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/39.jpg)
Results (cont)
4 reasons for difference in length of claim: Some people take longer to find work Some people take longer to tell us they have found work Some control group customers fail to attend and have to
start a new claim Some people fail to sign off for other reasons
![Page 40: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/40.jpg)
Results (cont)
Work study provided estimates of savings Extra benefits paid as a result of increase in average
length of claim exceed savings Qualitative evaluation suggested that the pilot was
implemented well Customers were happy not to have to attend every
fortnight
![Page 41: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/41.jpg)
More information
DWP Research Report 300: The Qualitative Evaluation of the JSA Intervention Regime Pilots
DWP Research Report 382: Jobseeker’s Allowance Intervention Pilots Quantitative Evaluation
Available on DWP Website: http://www.dwp.gov.uk/asd/asd5/rrs-index.asp
![Page 42: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/42.jpg)
Job Retention and Rehabilitation Pilot
Lessons learnt in running an RCT
James Holland
![Page 43: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/43.jpg)
Structure
1. Background to JRRP
2. Results of the trial
3. Hypotheses
4. Conclusions: Importance of complementary methodologies
![Page 44: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/44.jpg)
Design
4 - way trial To test the effectiveness of a person centred case
management approach and increased range of treatments in helping people retain work– Health care focused– Workplace focused– Combined health care and/or workplace focused– Control group
Four service providers in six parts of the country Participants were people off work sick and unlikely to
return to work without help
![Page 45: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/45.jpg)
Routes through the trial
Project Marketing
Approach Contact Centre
Contact Centre
Explanation
Eligibility
Screening
Decline
Ineligible
Screened out
Screened in: Randomisation
Health 25%
Work 25% Combined 25%
Control 25%
Providers make contact
Written Consent
Assessment and Intervention
Return to work
Out of work
Surveys
![Page 46: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/46.jpg)
Evaluation Design
Impact and process evaluation Cost benefit Analysis Components of the evaluation
– Survey of those screened out and the control group– Outcome survey– Panel study – Focus studies– Database of contacts and treatments– Costs exercise
![Page 47: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/47.jpg)
Impact Measures
Primary impact measure – 13 week return to work
Secondary impact measures– Health
– Household income Costs and benefits Operation of JRRP as a RCT
![Page 48: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/48.jpg)
Results
13 Week Return to Work
Intervention group %
Health 43.5
Workplace 45.1
Combined 44.4
All interventions 44.4
Control 44.7
![Page 49: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/49.jpg)
Results
6 Week Return to Work
Intervention group %
Health 55.7
Workplace 56.4
Combined 56.5
All interventions 56.2
Control 53.0
![Page 50: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/50.jpg)
Results
2 Week Return to Work
Intervention group %
Health 61.5
Workplace 61.4
Combined 62.1
All interventions 61.7
Control 59.3
![Page 51: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/51.jpg)
Results
Positive impact among those off work because of an injury
Negative impact among those off work because of a mental health condition
Unaffected return to work rates for those with other health conditions
Positive impact on health, particularly mild depression
A RCT can work in a voluntary labour market setting
![Page 52: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/52.jpg)
Hypotheses for findings
1. The interventions were too weak
2. The interventions were delivered in an unhelpful way
3. Too many external barriers
4. Withdrawal rate too high
5. The self-selecting participants were the ‘wrong group’
![Page 53: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/53.jpg)
Missing evidence
Evidence of problems mostly drawn from qual research
This generates hypotheses/explanations but does not allow for quantification
Biggest gap is (quant) understanding of behaviour of control group
In retrospect, needed data on self-motivation and better understanding of participant/provider interaction
![Page 54: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/54.jpg)
Thoughts on how to do it better
Set out possible scenarios at start Early qual research on behaviours of participants
and control group Early impact estimates so that later research can
be adapted
![Page 55: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/55.jpg)
Conclusions
RCTs are the gold standard programme evaluation
But a number of problems, practically and methodologically
Need supplementing with a good quality process evaluation
![Page 56: Heads you’re in; tails you’re out: How RCTs have evolved in DWP](https://reader035.vdocument.in/reader035/viewer/2022062323/5681584c550346895dc5a3f0/html5/thumbnails/56.jpg)
Contact Details
James Holland
Disability and Work Division, DWP
0114 209 8280
Reports available via www.dwp.gsi.gov.uk/asd/asd5