Do Monetary Incentives IncreaseBusiness Survey Response Rates?
Results from a Large Scale Experiment
Paul Biemer, RTI International and University of North Carolina
Christopher Ellis, Angela Pitts and Kimberly Robbins
RTI International
Outline of the Presentation
Essential information about the establishment survey
Design of the incentive experiment
Results
Conclusions
The O*NET Survey
Occupation Information Network Survey (O*NET)
Provides descriptive ratings on 800+ U.S. occupations
Target population is all U.S. employees in these 800+ occupations
Continuing survey since 1997
Mode of interview
Telephone contact of establishments
Point of contact (POC) is usually Human Resources Manager
PAPI or Internet questionnaires completed by selected employees
The O*NET Survey (cont’d)
Two-stage sample design
Establishments and employees within establishments in selected occupations
~24,000 establishments/year (79% RR)
~36,000 employee respondents/year (69% RR)
O*NET Incentive System
All incentives are prepaid (prior to receipt of questionnaires)
For the establishment: O*NET toolkit (personnel job description aid)
For POC: Desk clock
Framed certificate of appreciation
$20 money order (to be tested)
For employees $10 cash
POC Incentive Experiment: Motivation
Concern that POC incentives not commensurate with POC tasks
Some evidence of diminished POC effort during employee nonresponse follow-up phase
Establishment and employee nonresponse rates were low for some occupations
POC does not receive monetary incentive although employees do ($10)
Overview of Data Collection Protocol
Step 1:
Verification Call to Receptionist
Step 4:
Recruiting Call to POC
Send Information Package
Step 2:
Screening Call to the Point of Contact (POC)
Step 5:
Sampling Call to POC
Step 8:
7-Day Follow-up Call to POC
Step 11:
31-Day Follow-up Callto POC
Step 10:
21-Day Follow-up Callto POC
Step 9:
Send Thank You/Reminder Postcards
Step 12:
Send Replacement Questionnaires
Step 13:
45-Day Follow-up Callto POC
Step 1:
Step 4:
Step 3:
Step 2:
Step 5:
Step 8:
Step 11:
Step 10:
Step 9:
Step 12:
Step 13:
Step 1:
Send Questionnaire Pacakge
Step 6:
Step 7:
Send Toolkit
Step 7:
Overview of Data Collection Protocol
Step 1:
Verification Call to Receptionist
Step 4:
Recruiting Call to POC
Send Information Package
Step 2:
Screening Call to the Point of Contact (POC)
Step 5:
Sampling Call to POC
Step 8:
7-Day Follow-up Call to POC
Step 11:
31-Day Follow-up Callto POC
Step 10:
21-Day Follow-up Callto POC
Step 9:
Send Thank You/Reminder Postcards
Step 12:
Send Replacement Questionnaires
Step 13:
45-Day Follow-up Callto POC
Step 1:
Step 4:
Step 3:
Step 2:
Step 5:
Step 8:
Step 11:
Step 10:
Step 9:
Step 12:
Step 13:
Step 1:
Send Questionnaire Pacakge
Step 6:
Step 7:
Send Toolkit
Step 7:
Incentives sent here
Brochure sent here
Cover of the POC Incentive Brochure
FRONTBACK
Inside of the POC Incentive Brochure
Experimental Design
Split sample design (75/25 split)
Incentive group: POCs offered $20 money order in addition to clock and framed certificate
Control group: POCs offered only clock and framed certificate
Money orders could be made out to POC, the business or a favorite charity
Interviewers (called business liaisons or BLs) handled both incentive and non-incentive cases
Sample Allocations to the Control and Incentive Groups
Control Incentive
Establishments 2,624 7,874
Employees 7,694 22,309
Results
Unweighted Response Rates (%)
Control Trt Diff P-value
Establishment 62.0 61.3 1.7 0.12
Employee 73.6 72.7 0.8 0.45
Other Results
No effect on response rates by
size of establishment
industry grouping
urban or rural regions
occupation of employee
Some evidence of a negative effect for some sectors
Cumulative Employee Response Rates for by Week of Data Collection
Average Employee RR for Waves 1.5, 1.6, 1.7, 1.8, 2.4, 3.3, 3.4, 3.5, 4.3, 4.4, 5.2, 5.3, 6.11, 6.1
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
1 3 5 7 9 11 13 15 17 19
Week of Data Collection
Per
cen
tag
e
Control
Treatment
Summary of Results
No evidence of an effect on establishment cooperation rates
Weak evidence of an effect on employee cooperation rates Both negative and positive effects detected for small
subgroups
Overall, no significant difference
No cost advantage Follow-up costs were the same
Speed of response was the same
Incentive costs were not offset by efficiency gains
Interpreting the Findings
No establishment-level effect
$20 incentive viewed as trivial by POCs Framed certificate and clock could have much larger
perceived value
Decision to participate shared by POC and company
POC cannot accept money in many establishments
No effect at the employee-level
POC has limited ability to affect employee response
$20 ineffective for increasing follow-up intensity