Download - The Research Team
RESULTS FROM AN EXPERIMENT TO PREVENT REFUSALS IN A CONTINUING LONGITUDINAL STUDY
Presentation to PHSRN Workshop on Attrition in Cohort Studies, Royal Statistical Society, October 2009
The Research Team
Principal Investigator:
IAN PLEWIS, CCSR/SOCIAL STATISTICS, UNIVERSITY OF MANCHESTER.
Co-Investigators:
LISA CALDERWOOD, CLS, INSTITUTE OF EDUCATION, LONDON
REBECCA TAYLOR, NATCEN, LONDON
Research Officer:
SOS KETENDE, CLS
Preventing Refusals
• Seeking to prevent/convert refusals in an ongoing cohort study – the Millennium Cohort Study – with two interventions applied in an experimental framework.
• Testing the hypothesis that devoting extra field resources to the problem of increasing cooperation will bring benefits in the forms of increased precision and less attrition bias.
• Guided by the view that failure to cooperate in later waves of a longitudinal study, conditional on initial cooperation, will be largely circumstantial.
• Intervention 1: providing extra information in the form of a letter/leaflet that addresses previously reported concerns and reasons for not continuing to participate.
• Control condition: no leaflet
Piloted in wave 4 dress rehearsal
Reasons for refusal, wave 3
1= “Too busy” (36%)1= “Nothing has changed” (36%)1= “Don’t see public benefit” (36%) 4 “Don’t want to bother” (23%)5 “Stressful family situation” (15%)6 “Survey too long” (10%)7 “Looking after children” (8.9%)8 “Don’t see personal benefit” (5.5%)9 “Questions too personal” (4.8%)10 “Survey not important” (4.3%)
• Intervention 2: reissue all refusals to the interviewer (except ‘hard’ refusals), usually to a different interviewer.
• Control condition: standard NatCen reissue strategy (MCS, Wave 3 – just 4.4%).
Crossed designI1, I2. All initial and potentially convertible refusals reallocated to a different interviewer; leaflet sent in advance of the visit from the new interviewer.
C, I2. All initial and potentially convertible refusals reallocated to a different interviewer; no prior leaflet.
I1, C. Reissues of initial refusals follows NatCen standard practice of reissuing a proportion of all initial refusals; leaflet sent to all initial refusals prior to visit from new interviewer and to all non-issued and potentially convertible refusals with freephone number for those choosing to participate after all.
C, C. Reissues of initial refusals follows NatCen standard practice of reissuing a proportion of all initial refusals; no leaflet. This group was likely to contain very few reissued cases.
Issued sample at wave 4 of MCS: n = 15350 (GB only) – this was allocated to each of the four experimental groups within the seven GB strata.
‘Intention to treat’ sample – all refusals at first issue: n = 1660 (11%).
Experimental Group by Stratum
England Wales Scotland
GROUP Ad. Disad. Ethnic Ad. Disad. Ad. Disad. Total
Control (n) 91 133 70 14 43 25 44 420
Leaflet (n) 93 129 77 17 57 24 40 437
Reissue only (n)
79 120 63 24 48 22 33 389
Reissue + leaflet (n)
88 119 70 22 61 24 30 414
Total (%) 21 30 17 5 13 6 9 100
Issued sample (%)
28 28 15 5 11 6 7 100
Outcome numbers by Intention to Treat Group
Control Leaflet Reissue only
Reissue + leaflet
Fully productive 2 3 44 54
Partially productive 0 0 24 17
Unproductive 418 434 321 343
No evidence to support any effect of the leaflet.
Reissuing appears to be effective.
Exclusions by Experimental Group
Note * - p < 0.01
Control Leaflet Reissue only
Reissue + leaflet
‘Hard’ refusals * 67 (16%)
61 (14%)
80 (21%)
100 (24%)
Other exclusions 0 9 5 8
Productive % (eligible) 0.56 0.82 22 23
Conversion attempt rate: 76% (for the two reissue groups). Burton et al. (2006), BHPS, waves 4 to 6: 40%. Productive rate (MCS) : 23% Productive rate (BHPS): 34% but only 13% for F2F interview.
Intensive reissuing reduced refusal rate from 11% to 10%. However, ratio of full to partially productive interviews = 2.4 compared with 8.0 for cases not refusing initially.
Conclusions
Intensive reissuing in an ongoing birth cohort study is effective, possibly more so for main respondents than for partners.
Leaflet addressing respondents’ reasons for
refusing is not effective.
Further analyses
1. What was the previous response pattern for the converted refusers?
2. What are the characteristics of the converted
cases? 3. Is refusal conversion cost effective?