roundtable on research design, data ...as an industry we are keen to better understand our...

© 2016 Ipsos. All rights reserved. Contains Ipsos' Confidential and Proprietary information and may

not be disclosed or reproduced without the prior written consent of Ipsos.

1

18th May, 2017

RESEARCH DESIGN,

ROUNDTABLE ON

DATA COLLECTION

& INNOVATION



2

18th May, 2017

UK RANDOM PROBABILITY

RESPONSE RATES ON

FACE TO FACE SOCIAL SURVEYS

Keith Bolling (Kantar Public) & Kirby Swales (NatCen)

3

EVIDENCE AND CONTRIBUTIONS SUPPLIED BY…

4

CONTENTS

1

2

3

Background: what has been happening to response rates?

Looking beneath the bonnet – levels of effort

Implications and conclusions

5

BACKGROUND : WHAT HAS BEEN HAPPENING TO RESPONSE RATES

01

6

SURVEY RESPONSE RATES BY YEAR: AVERAGE

British Social Attitudes Average

30

35

40

45

50

55

60

65

70

75

80

2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015

Resp

on

se R

ate

ch

an

ge (

%)

7

SURVEY RESPONSE RATES BY YEAR: DETAIL

BARB

British Social Attitudes

Crime Survey for England and Wales

Family Resources Survey

Health Survey for England

Labour Force Survey

Living Costs and Food Survey

National Travel Survey

Average

30

35

40

45

50

55

60

65

70

75

80

2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015

Resp

on

se R

ate

ch

an

ge (

%)

8

SURVEY RESPONSE RATES: CHANGE OVER TIME

-30

-25

-20

-15

-10

-5

0

5

Resp

on

se R

ate

ch

an

ge (

%)

1998/2001* - 2015 2003 - 2015

9

RESPONSE RATES FOR THE HOUSEHOLD BUDGET SURVEY

Finland

France

Lithuania

Poland

Slovakia

Slovenia

Sweden

20

30

40

50

60

70

80

90

100

1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015

Resp

on

se R

ate

ch

an

ge (

%)

10

LONGITUDINAL STUDIES ATTRITION RATES

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11 12 13

Resp

on

se R

ate

(%

)

Number of sweeps or waves

Growing Up in Scotland BHPS English Longitudinal Study of Ageing

Millenium Cohort Study National Child Development Study 1970 British Cohort Study

11

POSSIBLE EXPLANATION

Less trust in government, brands and professions

Survey fatigue

Availability

Accessibility

12

LESS TRUST IN GOVERNMENT AND POLITICIANS

0

5

10

15

20

25

30

35

40

45

1986

198…

1987

1988

1989

1990

1991

1992

1993

1994

1995

1996

1997

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

2012

%

Almost never

13

MAJOR FACTORS AFFECTING RESPONSE

Response

rate

Messaging

Channels

Volume

Timing

PARTICIPANT

COMMUNICATIONS

Volumes &

allocation

Management

Reissues

FIELDWORK

SURVEY DESIGN

Sample type

Geography

Clustering

Sample data

Selection method

Interview length

Topic

Mode Training

Motivation

Management

INTERVIEWER/NURSE

Type

Value

INCENTIVESocietal

change

14

LOOKING BENEATH THE BONNET : LEVELS OF EFFORT

02

15

DETAILS OF THREE CASE STUDIES

BARB BSA CSEW

First conducted 1991 1983 1981

Sample design Tightly clustered Clustered Unclustered

PSUs Based on OAs Based on postcode

sectors

Bespoke but based

on LSOAs

Selection Household level

interview

One adult (18+)

randomly selected

One adult (16+)

randomly selected

Assignment size

(issued sample)

c. 17-18 addresses c. 26 addresses c. 32 addresses

Incentives None Conditional voucher

(£10 from 2014)

Stamps sent with

advance letter

Interview length c. 15 mins c. 65 mins c. 50 mins

Fieldwork length Up to 6 weeks Around 4 months Max. 6 months

16

CASE STUDY RESPONSE RATES

51%

46%

75%

72%

69%

63%

30%

35%

40%

45%

50%

55%

60%

65%

70%

75%

80%

2007 2008 2009 2010 2011 2012 2013 2014 2015 2016

Resp

on

se r

ate

Survey year

BSA CSEW BARB

17

CUMULATIVE RESPONSE RATE AFTER X VISITS NON-CONTACT RATES HAVE BEEN KEPT AT LOW LEVELS

7% 6% 6% 6% 6% 6% 5% 7% 6% 8%

0%

20%

40%

60%

80%

100%

2007 2008 2009 2010 2011 2012 2013 2014 2015 2016

4% 4% 4% 3% 3%4% 3% 3% 4% 4%

0%

20%

40%

60%

80%

100%

2007 2008 2009 2010 2011 2012 2013 2014 2015 2016

16% 16%13% 10% 10% 9% 9% 9% 9% 6%

0%

20%

40%

60%

80%

100%

2007 2008 2009 2010 2011 2012 2013 2014 2015 2016

BSA

BARB

CSEW

18

CUMULATIVE RESPONSE RATEREFUSAL RATES HAVE INCREASED IN THE LAST FEW YEARS

42%39% 39% 40% 40% 41% 41%

46%43%

46%

0%

20%

40%

60%

80%

100%

2007 2008 2009 2010 2011 2012 2013 2014 2015 2016

21% 19% 21% 21% 22% 20% 24% 22%26% 25%

0%

20%

40%

60%

80%

100%

2007 2008 2009 2010 2011 2012 2013 2014 2015 2016

15% 13% 17% 19% 20% 21% 22% 22% 24% 31%

0%

20%

40%

60%

80%

100%

2007 2008 2009 2010 2011 2012 2013 2014 2015 2016

BSA

BARB

CSEW

20

MORE EFFORT BEING PUT IN TO KEEP RESPONSE RATES UP

4.7

3.9

4.1

5.2

4.8

5.2

2.0

2.5

3.0

3.5

4.0

4.5

5.0

5.5

BSA BARB CSEW

2008-9

2015-16

21

MORE EFFORT BEING PUT IN TO KEEP RESPONSE RATES UP

2008/9 2015/16 % increaseExtra calls per

year

BSA (2008-2016) 4.7 5.2 11% c. 4,000

BARB (2009-2015) 3.9 4.8 23% c. 72,000

CSEW (2008-2016) 4.1 5.2 27% c. 55,000

23

INCREASE IN THE PROPORTION OF ADDRESSES WHICH HAVE MORE

VISITS THAN MINIMUM CALL REQUIREMENTS

24%

18%

26%

29% 29%

35%

0%

5%

10%

15%

20%

25%

30%

35%

40%

BSA BARB CSEW

2008-9

2015-16

24

UNPRODUCTIVE ADDRESSES ARE REISSUED TO BOOST RESPONSE

0%

5%

10%

15%

20%

25%

30%

35%

40%

2008 2009 2010 2011 2012 2013 2014 2015 2016

% o

f ad

dre

sses r

eis

su

ed

Survey year

BSA BARB CSEW

25

RESPONSE ON REISSUES IS LOWER ALTHOUGH HAS REMAINED

FAIRLY STEADY

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

2008 2009 2010 2011 2012 2013 2014 2015 2016

Resp

on

se r

ate

Survey year

BSA BARB CSEW

26

CUMULATIVE RESPONSE RATE AFTER X VISITS, 2015-16

28%

42%

46%

57%

67%

67%

46%

63%

72%

0%

10%

20%

30%

40%

50%

60%

70%

80%

0 1 2 3 4 5 6 7 8 9 10 11 12+

Resp

on

se r

ate

After x visits

BSA BARB CSEW

27

RELATIONSHIP BETWEEN RESPONSE RATE AND CALLS

BSA

BARB

CSEW

0%

20%

40%

60%

80%

100%

0%

20%

40%

60%

80%

100%

0 1 2 3 4 5 6 7 8 9

10

11

12

13

14

15

16+

Re

sp

on

se ra

te

% o

f to

tal c

alls

After x visits

Calls Response

0%

20%

40%

60%

80%

100%

0%

20%

40%

60%

80%

100%

0 1 2 3 4 5 6 7 8 9

10

11

12

13

14

15

16

17

18

19

20+

Re

sp

on

se ra

te

% o

f to

tal c

alls

After x visits

Calls Response

0%

20%

40%

60%

80%

100%

0%

20%

40%

60%

80%

100%

0 1 2 3 4 5 6 7 8 9 10

Re

sp

on

se ra

te

% o

f to

tal c

alls

After x visits

Calls Response

30

IMPLICATIONS AND CONCLUSIONS

03

31

CONCLUSIONS

Response rates are falling but the level of effort is increasing. Poses a number

of questions:

What can we realistically control in influencing response?

What is the cost-benefit analysis?

What are the best measures of performance?

32

.aT Y



33

18th May, 2017

AN OPERATIONS’ PERSPECTIVE

F2F INTERVIEWING IN 2017

Sophie Ainsby (NatCen) & Mark McLaughlin (GfK)

With Caroline Baxter (Kantar Public) & Jeremy May (Ipsos MORI)

34

EVIDENCE AND CONTRIBUTIONS SUPPLIED BY…

35

INTERVIEWING DAYS AND INTERVIEWER PANEL YEAR ON YEAR 2010-2016

170,629

128,861

157,522

125,213

133,607 134,085

118,797

154,633

121,121

97,493

80,000

90,000

100,000

110,000

120,000

130,000

140,000

150,000

160,000

170,000

180,000

2010 2011 2012 2013 2014 2015 2016

Inte

rvie

win

g D

ays

Year

RP Days

RL Days

3,967

3,780

3,698

3,367

3,271

3,002

2,797

2,000

2,500

3,000

3,500

4,000

4,500

2010 2011 2012 2013 2014 2015 2016

Inte

rvie

wers

Year

Interviewers

36

What are our operational challenges today?

F2F 2017

RecruitmentResponse to recruitment

advertising fluctuates with few

responses translating to new

panel members. We face

competition for candidates

from a wide range of

alternative employers offering

less challenging work with

guaranteed income. We have

tried alternative payment

methods but this has largely

been unsuccessful. We can still

build panels but it’s

increasingly challenging to do

so quickly.

RetentionNew people need an

introduction to F2F to help

maintain motivation and build

doorstep skills and experience

and this can be challenging

due to work types available.

Payment by results also has an

impact- on first projects

trainees may see their earning

potential limited.

Response Rate TargetsOn paper response rates remain

as they were 10 years ago.

However, saturation of customer

satisfaction and cold calling

makes target achievement more

challenging than it was. Research

frequently makes headlines, but

there isn’t a great deal of public

awareness regarding the vital role

they play.

37

2016 Recruitment: from applicant to interviewer

Operations/Field Challenges Today

c17K

Respond to

adverts

C1.7K

Invited to Train

1180

Join

interviewer

panel

An average of 7% of

applicants actually became

interviewers in 2016

38

JOINERS & LEAVERS 2012-2016

1444

10211117

1322

11801186

1593

1161

13711495

3698

33673271

3002

2797

0

500

1000

1500

2000

2500

3000

3500

4000

2012 2013 2014 2015 2016

Joiners

Leavers

Panel Size

32% 47% 35% 46% 53%

Leavers as % Total Panel

39

INTERVIEWER LENGTH OF SERVICE

2006 vs 2016

0%

5%

10%

15%

20%

25%

30%

Under 1 1 2 3 4 5 to 9 10+

Years

2006

2016

40

Over the last three years

c£3M spent on

mitigating recruitment

& retention challenges.

This cost is fully

absorbed and not

passed on.

MITIGATING THE ISSUES

Increased support for

new people; mentoring

programmes and

engagement measures

Trying to understand

and address our

interviewers’ challenges

via a cross company

interviewer survey

Cost Support Challenges

Operations Actions

41

As an industry we are keen to better understand our interviewers’ challenges

Produced in conjunction with the MRS, the survey began in 2016. 2017

wave has just been completed

93% of interviewers feel pay levels influence why people choose to stop

interviewing

86% feel the challenges of getting participants to agree to be interviewed is

also a significant factor

Experienced interviewers really enjoy random probability work- 81% rate

interest of subject matter highly

INTERVIEWER SURVEY

42

INTERVIEWER COMMENTS ON WHAT THEY FIND CHALLENGING

“It’s now very

difficult to get

people to take part

in a survey, much

more so than in the

past. Consequently if

being paid per

interview, the rate

per hour is less.”

“People are

becoming

untrusting- newer

interviewers get

disheartened.“

“Working unsocial

hours- especially in

winter.”

“Having to

continually work

evenings and

weekends”

43

OVERCOMING OUR CHALLENGES

Response

Rate Targets

Achievable targets impact

on interviewer earnings

and on retention

With the correct resource

levels we can deliver

Raise awareness regarding

the part the public plays in

collection of this valuable

data

Focus on continued efforts,

make more of the value of

research

Encourage commissioners to

be champions of research

Resourcing and maintaining

a field force grows

increasingly expensive

Supporting this infrastructure

requires investment

Awareness &

Engagement

Costs

44

As an industry we face challenges in attracting people to become interviewers

Interviewer retention also requires continued focus, with increasing

polarisation in experience levels making succession planning a challenge

We can still build panels but it has become more difficult and costly to do so

Raising awareness and encouraging promotion of research can help

Continuing to invest in incentive strategies for target samples

Response rate targets need to reflect underlying changes in society

We plan to continue as individual companies and collectively to work to

address this and to engage with interviewers

IN CONCLUSION

45

.aT Y



46

18th May, 2017

AND POPULATION INFERENCE

RESPONSE RATES

Patten Smith (Ipsos MORI) & Joel Williams (Kantar Public)

47

Aims

1. Examine relationship between survey response rates and quality of

population inferences

2. Describe relevant empirical work

3. Draw conclusions for future survey practice

Outline

Conceptual framework

Empirical findings

Practical implications for survey commissioners / practitioners

THIS PRESENTATION

48

CONCEPTUAL

FRAMEWORK1

49

Response rate

AAPOR standard definitions: survey response rate is proportion of eligible

population taking part

Quality of population inference

Use Total Survey Error (TSE) framework - generally accepted framework for

assessment of random probability survey quality

Inventory of survey processes and errors identified with each

Places errors on common metric: mean square error (MSE - error variance

from all sources + square of bias (from all sources))

Hard to measure TSE, but framework clarifies quality issues

DEFINING TERMS

50

SURVEY PROCESSES AND TSE

1) Construct (the

information that

you seek)

2) Measurement

(ways to gather the

information) e.g

questions

3) Response

4) Edited response

Survey data

Validity

Measurement

error

Processing error

Sampling error

Coverage error

a) Defining target

population

b) Finding

Sampling frame

c) Drawing sample

d) Collecting data

from respondents

e) Making post-

survey

adjustments

Non-response error

Adjustment error

51

Non-response bias: difference between estimates for survey respondents and

overall population

Non-response bias (NR bias) for variable Y increases with:

1. correlation between Y and likelihood of responding

2. non-response rate

NR bias:

• is variable-specific; wide range of NR bias values in a single survey

• increases as response rate decreases only if non-zero correlation between

response rates and variables

• but not by much unless correlation is substantial

RESPONSE RATE AND NON-RESPONSE BIAS

52

Expressed algebraically, (P=sample member’s propensity to respond (0-1)):

NR Bias( 𝑌𝑅) ≈𝑐𝑜𝑣(𝑃𝑌)

𝑃=

𝐶𝑜𝑟𝑟(𝑃𝑌)×𝑆𝐷(𝑌)×𝑆𝐷(𝑃)

𝑃


53

EMPIRICAL EXAMINATIONS

OF RESPONSE RATE-TO-2BIAS RELATIONSHIP

54

Relationship between RR and NR bias is open; requires empirical assessment

Two types of empirical study depending on availability of validation data

1. Where good estimate of the “true” value of a variable, compare this with

estimate for survey respondents; difference is estimate of total NR bias

– Aggregate population estimates

– Sample based estimates from “gold-standard” surveys

– Sample frame information about issued cases (direct or linked)

2. Otherwise, assess how survey estimates change with increasing fieldwork

effort (e.g. number of contact attempts, extent of reissuing); assumption - not

a logical necessity! - that this will measure relative NR bias


55

Ideally, measure total NR bias using benchmark validation data

Often measure relative NR bias because:

– Surveys usually commissioned to study the un-benchmarked!

– Benchmarks usually demographic / structural – nothing on opinions &

very little on behaviour (voting excepted)

– Estimating relative NR bias vs that obtained with a ‘gold standard’ RR

(not 100%!) is a more useful way of judging VFM of each component of

data collection effort

TOTAL AND RELATIVE NON-RESPONSE BIAS

56

US: Groves et al (2006/08, US): meta-analysis of studies of total NR bias

UK:

– Census link study: total bias estimates (demographic only) for six surveys

– Hall et al (2013, UK) - relative bias and FW effort in three surveys

– Sturgis, Williams et al (2016, UK) - relative bias and FW effort in 541 non-

demographic variables in six surveys

– Williams (2016) - relative bias and FW effort in CSEW subpopulations

– Unpublished Ipsos analysis of relative bias in 2009-10 Citizenship Survey

EMPIRICAL EVIDENCE

57

Meta-analyses (30 studies / 319

estimates; 59 studies / 959

estimates)

Measured absolute NR bias

Very low correlation between RR

and NR bias

Greater variation within studies

than between them

Preliminary efforts to identify

survey design features that target

NR bias

GROVES (2006 & 2008 W/PEYTCHEVA, US)

58

Three NatCen Surveys: BSA, FRS and HSE (final RRs 51% - 61%)

Compared demographic and survey variables before and after extended FW

efforts (6+ calls to get contact or reissued after initial refusal)

Almost no significant differences between calibration-weighted estimates before

and after these efforts

(As noted earlier, substantial share of total effort allocated to this reissue phase)

HALL, BROWN, NICOLAAS AND LYNN (2013)

59

Six surveys from 2011-14 (all Kantar Public); all (541) non-demographic items

Relative bias (distance from final estimate) measured after each FW effort level:

• 1 call (RRs=7-22%), 2 calls, 3 calls, 5 calls (cf. final RRs = 55-76%)

• Average error after call 1 = 1.6%pts; 1.1%pts after calibration

• Average error after call 2 = 1.0% pts; calibration barely improves on this

Study estimated that a 4-call rule leads to lowest # of total calls (=lowest cost)

On average, questions about beliefs (and attitudes) tended to respond to FW

effort slightly more than did questions about behaviour

Small number of variables with higher relative bias (internet use, freq. of being in

during day, freq. of going to pub) – but convergence after 3 calls

STURGIS, WILLIAMS & BRUNTON-SMITH (2016, UK)

60

IMPACT OF 1ST, 2ND, 3RD, 4TH/5TH CALLS

61

Subpopulation analysis in

CSEW (e.g. age group) – pre-

reissue v final (both calibrated)

Compares error distribution v

null hypothesis error

distribution

Subpopulation error levels

slightly smaller than total

population error levels (greater

within-group homogeneity?)

WILLIAMS (2016, UK)

62

Occasionally extended FW

efforts can reduce NR bias

more substantially

2009-10 Citizenship survey

reissues (50>>>56% RR)

Formal and informal

volunteering prevalence

decreased by 1% point after

reissuing (CI: 0.5-1.5%)

Fits with US experience also

(e.g. see Pew, 2012)

THE POTENTIAL FOR EXCEPTIONS

63

Absolute bias studies: relationship between RR and NR bias generally very weak

Relative bias studies: extended FW efforts increase RRs, but small impact on

survey estimates

Lack of relative bias should not lead to assumption of no bias but should lead to

questions over VFM of additional DC efforts

For some variables extended FW effort makes a difference – but hard to predict

in advance

Most UK work on relative bias based on f2f interview data – unclear whether

findings generalisable to data collection modes with lower ‘maximum’ RRs

BROAD CONCLUSIONS FROM EMPIRICAL EVIDENCE

64

IMPLICATIONS FOR

SURVEY PRACTICE3

65

Question 1: is this a repeat wave of longitudinal survey?

Response rate maximisation often more important because:

–Cumulative attrition produces shrinkage in sample numbers in key

groups

–Can lead to large losses in precision

QUESTIONS WE SHOULD ASK OURSELVES BEFORE SETTING

RESPONSE RATE TARGETS

66

Question 2: how vulnerable to NR bias are your key variables?

Weighted survey estimates for most variables largely unaffected by

response rate variations within observed ranges

Rational starting assumption that response rate levels will make little

difference

For some variables declining RR has greater impact on NR bias: e.g.

volunteering and web usage

Do you have any such variables? Check literature / earlier survey data



67

Question 3: will marginal reductions in NR bias compromise your

conclusions?

Even with highest achievable response rates, levels of absolute NR bias may

still be substantial

Often trends of greater interest than point estimates. NR bias probably

relatively constant over time - little impact on trend lines



68

Question 4: are there better ways of reducing NR bias?

Blanket increase in RR beyond those obtained by reasonable good practice

poor method for reducing NR bias

Address NR bias in other ways? –reduce correlation between key variables

and response propensity?

Increase FW effort for poor responders differing on key variables using

supplementary variables / paradata to identify them



69

Observed impact of response rates on survey estimates usually modest

Pursuit of highest possible response rates may not be best use of limited survey

resources – may be better ways of addressing survey errors

At times making considerable efforts to maximise response rates can be justified:

Waves 2 – n in longitudinal surveys;

where minimising NR bias critical to conclusions

OVERALL CONCLUSION

70

.aT Y



71

18th May, 2017

FOR BETTER DATA

STRATEGIC ISSUES TO CONSIDER

Sharon Witherspoon (Campaign for Social Science)

with Debrah Harding (MRS) and Patten Smith (SRA)

72

IMPLICATIONS AND RECOMMENDATIONS

1

2

3

Issues from the previous sessions

Current trends and observations

Initial thoughts for discussion

73

ISSUES FROM THE

PREVIOUS SESSIONS1

74

ISSUES FROM THE PREVIOUS SESSIONS

Face to face survey responses are declining

– Secular trend arising from social changes e.g. declining privacy, trust,

social capital, participant time

– Strains in the labour market e.g. interviewers harder to recruit and retain

– Importance of understanding methodology -- response rates vs. bias in

affecting descriptive estimates and generalising inference

– Requires understanding and EMPIRICAL data

– Choices in use of resources

– Better informed design

75

CURRENT TRENDS AND

OBSERVATIONS2

76

CURRENT TRENDS AND OBSERVATIONS: I

1. Increasing use of other data sources e.g. admin data

2. Primary data collection still needed for DEPTH of information

3. Different methodologies face different challenges for descriptive

accuracy and generalisability: focus here is on FACE TO FACE

4. Accept competition in the research supplier market

5. Social research projects do not yield large profits

6. SHARED interests in robust quality, design, and maintenance of

capacity

77

CURRENT TRENDS AND OBSERVATIONS: II

7. Current tendering often results in specific research design BEFORE data

collection expertise/ experience brought in

Leads to less than optimal designs, lack of empirical pilots, dilemma of

non-compliant bids, lack of specification of key AIMS of research

8. Importance of linking AIMS with value for money in minimising total

survey error

9. Requires move away from viewing response rates as sole (contractual)

measure of data quality

Alternative approaches in tenders, procurement

Alternative approaches to research design

78

SOME INITIAL THOUGHTS

FOR DISCUSSION3

79

DISCUSSION POINTS: SPECIFIC STUDIES

1. Promote earlier expert engagement with research design

Identify key variables and aims

Commission / tender for work on research design – BEFORE main stage

design fixed

Use existing empirical data to consider total survey error

– What additional data – admin. data, paradata, etc. – available to test

impact of non-response?

Pilot different designs if helpful

Will affect timetable and budget allocation, possibly budget size, but

with better outcome

80

DISCUSSION POINTS: GENERAL & STRATEGIC

2. Strategic engagement with data regulators (ICO) about data linkage, use of

department data sets, consents, opt-ins/outs (stressing data protection but

also consent protections)

3. Strategic engagement of commissioners, data collection experts and

procurement to move away from specification of response rates and focus

on agreed PROCESSES to address non-response bias (and other sources of

error)

Better methodologically

Better allocation of resources

Better ethically / ‘politically’

81

OVER TO WIDER

DISCUSSION4

82

.aT Y

roundtable on research design, data ...as an industry we are keen to better understand our...

Documents