show me the money? effects of preincentives, differential
TRANSCRIPT
University of Wisconsin Survey Center
© 2012. Materials may not be reproduced without permission of the author.
SHOW Me the Money?Effects of Preincentives, Differential Incentives,
and Envelope Messaging in an ABS Mail Survey
Kristen Cyffka1,2, Jennifer Dykema1, John Stevenson1, Kelly Elver1, Karen Jaques1
1University of Wisconsin Survey Center2Department of Statistics, University of Wisconsin-Madison
AAPORMay 18, 2012
University of Wisconsin Survey Center
Background
• RDD telephone surveys are no longer a viable mode for many studies• Decline in last 2 decades in response rates• Inadequate coverage due to cell phones
• Researchers are increasingly looking to alternative designs• Return to mail surveys due to the DSF and ABS
• Effective ways to increase response rates in mail surveys (Edwards et al 2002, Church 1993):• More interesting vs. less interesting questionnaire• Recorded delivery vs. standard• Monetary incentive vs. no incentive
2
University of Wisconsin Survey Center
Research Questions
• How much of an increase will a $5 pre-incentive yield over a $2 pre-incentive?
• Can we attract initial nonresponders with a second incentive?
• What effect will messages on the front of the envelope have on response rates?
• What is the most cost-effective incentive package we can give to increase response rates?
3
University of Wisconsin Survey Center
Study: Survey of the Health of Wisconsin (SHOW)
4
SHOW 1 SHOW 2Year 2011 2012Contacts Initial packet
Reminder postcardSecond packet
Initial packetReminder postcardSecond packet
Sampled cases 2,608 2,616Completed surveys 1,566 1,440Response rate (RR1) 66.9% 66.8%
Experimental conditionsPre-incentiveSecond incentiveEnvelope message
University of Wisconsin Survey Center
Study: Survey of the Health of Wisconsin (SHOW)
5
SHOW 1 SHOW 2Experimental conditions
Pre-incentive
Second incentive
Envelope message
University of Wisconsin Survey Center
Study: Survey of the Health of Wisconsin (SHOW)
6
SHOW 1 SHOW 2Experimental conditions
Pre-incentive - $2- $5
Second incentive
Envelope message
University of Wisconsin Survey Center
Study: Survey of the Health of Wisconsin (SHOW)
7
SHOW 1 SHOW 2Experimental conditions
Pre-incentive - $2- $5
Second incentive - $0- $2
Envelope message
University of Wisconsin Survey Center
Study: Survey of the Health of Wisconsin (SHOW)
8
SHOW 1 SHOW 2Experimental conditions
Pre-incentive - $2- $5
Second incentive - $0- $2
Envelope message - No message- Monetary message
University of Wisconsin Survey Center
Study: Survey of the Health of Wisconsin (SHOW)
9
SHOW 1 SHOW 2Experimental conditions
Pre-incentive - $2- $5
- $2
Second incentive - $0- $2
Envelope message - No message- Monetary message
University of Wisconsin Survey Center
Study: Survey of the Health of Wisconsin (SHOW)
10
SHOW 1 SHOW 2Experimental conditions
Pre-incentive - $2- $5
- $2
Second incentive - $0- $2
- $0- $5
Envelope message - No message- Monetary message
University of Wisconsin Survey Center
Study: Survey of the Health of Wisconsin (SHOW)
11
SHOW 1 SHOW 2Experimental conditions
Pre-incentive - $2- $5
- $2
Second incentive - $0- $2
- $0- $5
Envelope message - No message- Monetary message
- No message- Monetary message- Health message
University of Wisconsin Survey Center
Envelope Message: SHOW 1
12
University of Wisconsin Survey Center
Response Rates: Envelope Messaging
13
57% 57%
11% 9%
0%
10%
20%
30%
40%
50%
60%
70%
No Message Money
Res
pons
e R
ate
SHOW 1
After 2nd SAQAfter 1st SAQ
67% 66%
p=0.600
University of Wisconsin Survey Center
Envelope Message: SHOW 2
14
University of Wisconsin Survey Center
Response Rates: Envelope Messaging
15
54% 52% 51%
14% 14% 15%
0%
10%
20%
30%
40%
50%
60%
70%
No Message Health Money
Res
pons
e R
ate
SHOW 269% 66%
p=0.437
66%
57% 57%
11% 9%
0%
10%
20%
30%
40%
50%
60%
70%
No Message Money
Res
pons
e R
ate
SHOW 1
67% 66%
p=0.60
University of Wisconsin Survey Center
Response Rates: Pre-Incentive
16
53%
61%11%
9%
0%
10%
20%
30%
40%
50%
60%
70%
$2 $5
Res
pons
e R
ate
SHOW 1
64%70%
p=0.005
University of Wisconsin Survey Center
Response Rates: After All Incentives
17
53% 51%
11%18%
0%
10%
20%
30%
40%
50%
60%
70%
Res
pons
e R
ate
SHOW 2
64%69%
p=0.015
$2$0
$2$5
Pre:2nd:
54%
60%
52%
62%11%
9%
10%
9%
0%
10%
20%
30%
40%
50%
60%
70%
Res
pons
e R
ate
SHOW 1
p=0.031
$2$0
$5$0
$5$2
$2$2
Pre:2nd:
70%
63%69%
66%
University of Wisconsin Survey Center
Response Rates: After All Incentives
18
53% 51%
11%18%
0%
10%
20%
30%
40%
50%
60%
70%
Res
pons
e R
ate
SHOW 2
64% 69%
p=0.015
$2$0
$2$5
Pre:2nd:
54% 52%
60% 62%11%
10%
9% 9%
0%
10%
20%
30%
40%
50%
60%
70%
Res
pons
e R
ate
SHOW 1
66%63%
p2nd=0.734
69% 70%
$2$0
$2$2
$5$2
$5$0
Pre:2nd:
pAll =0.030
University of Wisconsin Survey Center
Response Rates: Message and Incentive
19
67% 70%
62%
70%64%
68%
0%
10%
20%
30%
40%
50%
60%
70%
SHOW 2
No MessageHealthMoney
p=0.141
$2$0
$2$5
Pre:2nd:
University of Wisconsin Survey Center
Costs per Complete by Incentive
20
SHOW 1 Pre:2nd:
$2$0
$2$2
$5$0
$5$2
Total Number of Completes 383 371 403 423
Response Rates 65.7% 63.1% 69.0% 70.6%Cost Per Complete $23.31 $26.05 $26.67 $26.28
SHOW 2 Pre:2nd:
$2$0
$2$5
Total Number of Completes 669 741Response Rates 64.4% 69.3%Cost Per Complete $25.49 $28.37
University of Wisconsin Survey Center
Costs per Complete by Incentive
21
SHOW 1 Pre:2nd:
$2$0
$2$2
$5$0
$5$2
Total Number of Completes 383 371 403 423
Response Rates 65.7% 63.1% 69.0% 70.6%Cost Per Complete $23.31 $26.05 $26.67 $26.28
SHOW 2 Pre:2nd:
$2$0
$2$5
Total Number of Completes 669 741Response Rates 64.4% 69.3%Cost Per Complete $25.49 $28.37
University of Wisconsin Survey Center
Costs per Complete by Incentive
22
SHOW 1 Pre:2nd:
$2$0
$2$2
$5$0
$5$2
Total Number of Completes 383 371 403 423
Response Rates 65.7% 63.1% 69.0% 70.6%Cost Per Complete $23.31 $26.05 $26.67 $26.28
SHOW 2 Pre:2nd:
$2$0
$2$5
Total Number of Completes 669 741Response Rates 64.4% 69.3%Cost Per Complete $25.49 $28.37
University of Wisconsin Survey Center
Costs per Complete by Incentive
23
SHOW 1 Pre:2nd:
$2$0
$2$2
$5$0
$5$2
Total Number of Completes 383 371 403 423
Response Rates 65.7% 63.1% 69.0% 70.6%Cost Per Complete $23.31 $26.05 $26.67 $26.28
SHOW 2 Pre:2nd:
$2$0
$2$5
Total Number of Completes 669 741Response Rates 64.4% 69.3%Cost Per Complete $25.49 $28.37
University of Wisconsin Survey Center
Costs per Complete by Incentive
24
SHOW 1 Pre:2nd:
$2$0
$2$2
$5$0
$5$2
Total Number of Completes 383 371 403 423
Response Rates 65.7% 63.1% 69.0% 70.6%Cost Per Complete $23.31 $26.05 $26.67 $26.28
SHOW 2 Pre:2nd:
$2$0
$2$5
Total Number of Completes 669 741Response Rates 64.4% 69.3%Cost Per Complete $25.49 $28.37
University of Wisconsin Survey Center
Costs per Complete by Incentive
25
SHOW 1 Pre:2nd:
$2$0
$2$2
$5$0
$5$2
Total Number of Completes 383 371 403 423
Response Rates 65.7% 63.1% 69.0% 70.6%Cost Per Complete $23.31 $26.05 $26.67 $26.28
SHOW 2 Pre:2nd:
$2$0
$2$5
Total Number of Completes 669 741Response Rates 64.4% 69.3%Cost Per Complete $25.49 $28.37
University of Wisconsin Survey Center
Other Analyses in SHOW 1
Data quality• Item-missing data: no difference between treatments
Differential response• No difference between treatments in:
• Gender• Age• Race• Education• Employment• County
26
University of Wisconsin Survey Center
Discussion
• Envelope messaging• Effects on response rates• Future study
• Incentives• Effects on response rates
• Pre-incentives• Second incentives
• Implications on cost
27
University of Wisconsin Survey Center
Please visit us at:www.uwsc.wisc.edu
Thank You!
For copies of this presentation or more information, contact:
Kristen [email protected]
28
University of Wisconsin Survey Center
Response Rates: After All Treatments
29
53% 56%61% 59%
53% 51%
62% 61%10%13% 6% 11%
10% 10%
9% 8%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Message None Message None Message None Message None
$2 $2 $5 $5 $2 $2 $5 $5
$0 $0 $0 $0 $2 $2 $2 $2
SHOW 1
University of Wisconsin Survey Center
Response Rates: After All Treatments
30
55%50%
54% 53% 54%47%
12%12% 10%
17% 17%20%
0%
10%
20%
30%
40%
50%
60%
70%
80%
None Health Money None Health Money
$2 $2 $2 $2 $2 $2
$0 $0 $0 $5 $5 $5
SHOW 2
University of Wisconsin Survey Center
Response in Initial Nonresponders
31
0%
5%
10%
15%
20%
25%
30%
35%
40%
$0 $5
Res
pons
e R
ate
SHOW 2: Second Incentive
23%
38%
p=0.000
University of Wisconsin Survey Center
Cost analysis – What we included
We included:• Mailings• Postage, printing of all materials• Stuffing• Incentives• Data entry
We did not include:• Survey development and layout of SAQ• Programming of data entry instrument• Project management• Sample management• Data delivery
32
University of Wisconsin Survey Center
Discussion, limitations, and future directions
33
• So ... only $5 vs $2 had an effect on outcomes examined• Insight as to why the
• Additional incentive was not effective• Envelope message was not effective
• Future directions for current data• Mini- meta-analysis of effects of $5 versus $2• Look at responses to key survey questions• Use data from the sample file to try and look at
nonresponse bias• Ideas for future experimentation with ABS mail surveys?
University of Wisconsin Survey Center
Response rate calculation
34
MDE Code Description N rr_v11100 Complete 1580 12111 Other person refusal 1 02112 Known respondent refusal 14 02113 Blank q'naire, implicit refusal 8 02310 Deceased (including USPS: Deceased) 8 02320 Physically or mentally unable 1 03230 Refused by addressee 1 03300 Undeliverable 1 .3311 USPS: Attempted - Addressee not known 74 03313 No such address 5 .3314 Vacant 32 .3315 Not delivered as addressed 44 03320 Moved - left no address 5 04700 Ineligible Age 181 .4701 Non county resident 35 .Blank Noncontact 618 0Total 2608
University of Wisconsin Survey Center
Ineligible and undeliverable rates
35
11.7
4.6
119.8
10.49.2
1110.1
0
2
4
6
8
10
12
14
$5-$2-Env(Y)$5-$2-Env(N)$5-$0-Env(Y)$5-$0-Env(N)$2-$2-Env(Y)$2-$2-Env(N)$2-$0-Env(Y)$2-$0-Env(N)
University of Wisconsin Survey Center
Effects on data quality
• Item missing data• Weak satisficing• Responses to select survey questions• Demographics/nonresponse bias
• There were NO differences due to incentives for ANY of these measures of data quality
36
University of Wisconsin Survey Center
Data Quality: Item-Missing Data
37
Did treatments make a difference in the number of questions completed?
• Used a subset of 102 questions asked of everyone
91.7 92.2 92.5 91.3 91.9 92.0
0
20
40
60
80
100
Message No msg $5 pre $2 pre $2 extra No extra
Num
ber o
f Que
stio
ns C
ompl
eted Pre‐Incentive
p=0.006Second Incentive
p>0.05Messagep>0.05