Reasons for Late Response and Nonresponse in Surveys of

Government Agencies Tim Smith (RTI International)

Christian Genesky (RTI International) Danielle Kaeble (Bureau of Justice Statistics) Anthony Whyde (Bureau of Justice Statistics)

www.rti.org RTI International is a registered trademark and a trade name of Research Triangle Institute. 1

Establishment Response Rates

▪ Annual Survey of Public Employment and Payroll (n=50 state governments and about 90,000 local governments)

100 95 90 85 80 75 70 65 60

Res

pons

e R

ate

(%)

2010 2011 2012 2013 2014 2015 2016 Year

▪ Medical Expenditure Panel Survey – Private Sector Establishments

100 95 90 85 80 75 70 65 60

Res

pons

e R

ate

(%)

2010 2011 2012 2013 2014 2015 2016 Year

2

Establishment Response Rates

▪ 2015 Occupational Information Network (O*NET)

-100

80

60

40

20

0 Res

pons

e R

ate

(%)

Number of Employees

▪ Schools and Staffing Survey (SSAS) 2011-12

■ El

Res

pons

e R

ate

(%) 100

80 60 40 20

0 Teacher Listing School Survey Principal Survey Teacher Survey

Form Data Collection

Public Private

3

Risks Due to Late Response/Nonresponse

▪ Higher data collection costs

▪ Longer data collection period

▪ Lower data quality

▪ Staff turn-over

▪ Biased estimates

4

Research Goals

▪ Identify reasons for late response/nonresponse among probation and law enforcement agencies

▪ Assess differences by agency size and other characteristics

▪ Define tailored strategies to encourage survey participation

5

- 95 ~ 0 -~ co 90 0:: Q) 1/) C: 85 0 a. 1/) Q) 80 0::

75 First Second Third Fourth

Agency Size (Probation Population)

---2014 ..... 2015 -+-2016

Methods – Focal Studies

▪ Annual Probation Survey (APS) – Census of 456 agencies nationwide

• 42 provide state-level data • 13 provide data for multiple agencies • 401 provide data for a single agency

– Burden between 30-90 minutes

– Annual confirmation of agency head contact information for survey pre-notification

– Web survey, with telephone follow-up

– 5-month data collection period

– 92% response rate

6

Methods – Focal Studies

▪ Law Enforcement Management and Administrative Statistics (LEMAS) Core Survey – Sample of 3,500 state and local law enforcement agencies

▪ 50 state agencies ▪ 2,640 local police departments ▪ 810 sheriff’s offices

– Burden of 3 hours – Online search for

agency head contact information for survey pre-notification

– Web survey, with telephone follow-up

– 7-month data collection period

– 80% response rate (2016)

--- .....

100

Res

pons

e R

ate

(%) 80

60

40

20

0

Local Police Sheriffs' Office

1

2–4

5–9

10–2

4

25–4

9

50–9

9

100+

1

2–4

5–9

10–2

4

25–4

9

50–9

9

100+

Agency Type and Size (FTE)

2013 2016

7

Methods – Reasons for Late Response/Nonresponse

▪ Codes based on experiential/anecdotal evidence

– Perception that limited resources (e.g., budget and staff), benefits, and role of federal government in state/local issues might impact cooperation

– Tested on 2016 LEMAS Body-worn Camera Supplement Survey

▪ Verbatim reasons

▪ Double-coded into 21 categories, with adjudication

– Refined for APS and LEMAS-Core

8

Methods – Reasons for Late Response/Nonresponse

▪ Final code list – COMPLETE — Claims that survey has been submitted – DUE DATE — Cannot respond by due date – LIMITED TIME/RESOURCES — Not related to due date – APPLICABILITY — Thinks survey does not apply to agency – NO INTEREST – Uninterested in survey topic/goals – NO BENEFIT – No benefit to agency from participation/survey – VOLUNTARY – Participation is not mandated by law – SURVEY FATIGUE — Agency receives too many survey requests – LACK OF DATA (IN PERIOD) — Data are not available during survey period – LACK OF DATA (EVER) — Data do not exist or are not maintained – INACCESSIBLE DATA – Data exist, but are not easily accessible – POOR QUALITY DATA – Data exist, but are of questionable/poor quality – CONFIDENTIALITY – Data cannot be shared outside of agency/authority – FEDERAL ROLE – Federal government should not be involved in local issues/operations – JURISDICTION RULE – Jurisdiction does not participate in research – OTHER – REFUSED TO GIVE REASON FOR DELAY/REFUSAL

9

Findings – Overview

▪ Late respondents – APS: 165 cases; 129 contacts gave reasons – LEMAS: 679 cases; 339 contacts gave reasons (240 “did not receive”)

▪ Nonrespondents – APS: 38 cases; 29 contacts gave reasons – LEMAS: 572 cases; 310 contacts gave reasons (223 “did not receive”)

▪ Five reasons were not cited by late respondent or nonrespondent contacts on either study – APPLICABILITY — Thinks survey does not apply to agency – NO BENEFIT – No benefit to agency from participation/survey – CONFIDENTIALITY – Data cannot be shared outside of agency/authority – FEDERAL ROLE – Federal government should not be involved in local issues/operations – JURISDICTION RULE – Jurisdiction does not participate in research

10

SURVEY FATIGUE

REFUSED

POOR QUALITY DAT A

OTHER

NO INTEREST --1-----------------LIMITED TIME/RESOURCES ------------,

LACK OF DATA (IN PERIOD)

LACK OF DATA (EVER)

INACCESSIBLE DATA --~ ----------------

DUE DATE

COMPLETE I I I I I

0% 25% 50% 75% 100% % Contacts Mentioning Reason

Survey 9 APS

■ LEMAS

Findings – Late Respondents

11

SURVEY FATIGUE

OTHER

NO INTEREST

---.....--------------- -----------------

LIMITED TIME/RESOURCES ------------4

LACK OF DATA (IN PERIOD) - -----------------

LACK OF DAT A (EVER)

INACCESSIBLE DATA ~ - ---------------

DUE DATE

CONFIDENTIALITY

COMPLETE ------------ ■---------

• I I I I

0% 25% 50% 75% 100% % Contacts Mentioning Reason

Survey 9 APS

■ LEMAS

Findings – Nonrespondents

12

POOR QUALITY DATA

OTHER

LACK OF DATA (IN PERIOD)

LACK OF DATA (EVER)

INACCESSIBLE DATA

DUE DATE

COMPLETE I I I I I

0% 25% 50% 75% 100% % Contacts Mentioning Reason

Agency Size - All ■ 01

02 Q3

04

Findings – APS Late Respondents, by Agency Size

13

SURVEY FATIGUE

NO INTEREST

LACK OF DATA (EVER)

INACCESSIBLE DAT A

COMPLETE I I I I I

0% 25% 50% 75% 100% % Contacts Mentioning Reason

Agency Size - All

■ 01 Q2

Q3

Q4

Findings – LEMAS Late Respondents, by Agency Size

14

LUNTARY

OTHER

NO INTEREST

LIMITED TIME/ RESOURCES

LACK OF DAT A (EVER)

INACCESSIBLE DATA

DUE DATE I

0% I I I

25% 50% 75% % Contacts Mentioning Reason

I

100%

Agency Size - All

■ 01 Q2

Q3

Q4

Findings – APS Nonrespondents, by Agency Size

15

SURVEY FA TIGUE

NO INTEREST

LIMITED TIME/RESOURCES ----------

LACK OF DA TA (EVER)

INACCESSIBLE DAT A

CONFIDENTlALITY

COMPLETE I

0% I I I

25% 50% 75% % Contacts Mentioning Reason

I

100%

Agency Size

- All

■ 01 02 03

04

Findings – LEMAS Nonrespondents, by Agency Size

16

.1%

QUARTILE 1

73.5%

QUARTILE 2

8.0% 5.3%

10:69/,

76.1%

QUAR ILE 3

■ Respondent, Did Not Receive ■ Non-Respondent, Did Not Receive ■ Non-Respondent, Received □ Respondent, Received

3.5% 4.6'/o

11.0'k

81.0%

QUARTILE 4

Findings – LEMAS - Did Not Receive Survey Request, by Agency Size

17

Summary

▪ APS and LEMAS demonstrated differential response rates by agency size

▪ Many experiential/anecdotal reasons for nonresponse are not borne out by evidence (e.g., no benefit, confidentiality, federal role)

▪ Reasons for late response and nonresponse vary by agency size

– Large agency late respondents often report data availability

– Small agency late respondents often report staff resource limitations

– Frame address quality did not influence response, even though LEMAS agencies (especially small agencies) often reported not receiving survey materials

18

Lessons

▪ Test data availability when defining survey content

▪ Communication should stress flexibility/support for due dates and mode of data submission

▪ Confirm receipt of survey materials early

19

Limitations

▪ Data are from only two specific kinds of government agencies

▪ Data on reasons restricted to those contacted

▪ Analysis to-date only related to agency size

▪ Solutions are undefined/untested

– Restricted survey content

– Tailored survey correspondence/communication

– Interviewer procedures/materials

20

Next Steps

▪ Identify/analyze additional characteristics

▪ Develop tailored communications strategies

▪ Conduct randomized tests

▪ Investigate with other establishment/agency populations

21

More Information

Tim Smith, Director Security and Resilience Program RTI International, Survey Research Division tksmith@rti.org 919-316-3988

Information on RTI data collection on criminal justice projects: https://www.rti.org/sites/default/files/brochures/rti_data_cj.pdf

22