effects of splitting long surveys into two
TRANSCRIPT
Tweet! #AAPOR @LoveStats
Effects of Splitting
Long Surveys Into Two
Annie Pettit
Chief Research Officer
Tweet! #AAPOR @LoveStats
http://www.people-press.org/2012/05/15/assessing-the-representativeness-of-public-opinion-surveys/
90 77 79 73 72 62
43 40 34 31 21 14
36 28 25 21 15 9
Tweet! #AAPOR @LoveStats
50% of surveys are >=15 minutes
Source: DIY Sampling Panel
5 10 15 20 25 30 35 40 45 50 55 60+
Tweet! #AAPOR @LoveStats
12 Minute Survey
N=300Split
incentive
Method12 Minute
Survey
N=300Split
incentive
20 Minute Survey
N=600Full
incentive
Tweet! #AAPOR @LoveStats
Problem:
Nothing is directly Comparable
• People choose whether to participate
• Each version is different
– Different length
– Different topics
– Different leading/priming questions
Tweet! #AAPOR @LoveStats
Agenda
1. Data quality
2. Data equivalence
3. Responder satisfaction
11/6/2014 6
Tweet! #AAPOR @LoveStats
Data Quality11/6/2014 7
Tweet! #AAPOR @LoveStats
Fewer Red Herrings
2 fake hotels
2 fake airlines
2 fake loyalty programs
Tweet! #AAPOR @LoveStats
0%
10%
20%
30%
40%
50%
FailStraight
1
FailStraight
6
FailStraight
10
FailStraight
64
FailStraight
66
FailStraight
85
FailStraight
91
FailStraight
93
FailStraight
104
Long Short A Short B
Straightlining
P<.05
Tweet! #AAPOR @LoveStats
Follow Instructions Better
Did not follow instruction to “choose only one”
Tweet! #AAPOR @LoveStats
Better Open Ends
Optional
Tweet! #AAPOR @LoveStats
Speeding differences not discernable
Long
Short A
Short B
Standardized Number of Seconds
m=0, sd=1, ceiling outliers removed
Tweet! #AAPOR @LoveStats
Drop the worst 5%
from each dataset
6.1% of Long
6.3% of Short A
5.2% of Short B
Tweet! #AAPOR @LoveStats
Data
Equivalence
Tweet! #AAPOR @LoveStats
Survey Middle: Trips Taken
0%
20%
40%
60%
80%
1 to 3 trips 4 or more trips
Long Short A Short B
Tweet! #AAPOR @LoveStats
Survey Middle: Airline Aware
P<.05P<.05P<.05
P<.05
0%
25%
50%
75%
100%
Long Short A Short B
Tweet! #AAPOR @LoveStats
Survey End: Like Advertising
0%
10%
20%
30%
40%
50%
60%
Youtube Facebook Flyers Billboards
Long Short A Short B
P<.05
Tweet! #AAPOR @LoveStats
Survey End: Privacy Concerns
0%
20%
40%
60%
Top2Q91privacy Top2Q91security Top2Q91family Top2Q91home Top2Q91online
Long ShortB
Tweet! #AAPOR @LoveStats
Survey End: Personality
Tweet! #AAPOR @LoveStats
Respondent
Engagement
Tweet! #AAPOR @LoveStats
Same Completion Trend
25%
50%
75%
100%
Long Short A Short B
Optional
watch a
commercialOptional
watch a
commercial
Grid
Tweet! #AAPOR @LoveStats
More Words in Verbatim
0
20
40
60
Verbatim 1 Verbatim 2 Verbatim 3 Verbatim 4 Verbatim 5 Verbatim 6
Long ShortA ShortB
Optional
P<.05
Tweet! #AAPOR @LoveStats
Possibly Higher Satisfaction
0%
25%
50%
75%
100%
Easy Boring Long RecoLong Short A Short B
P<.05
P<.05
P<.05
Tweet! #AAPOR @LoveStats
More Positive Survey Comments
75%86%
93%
Long Short A Short BPercentage of optional survey comments that were positive
Tweet! #AAPOR @LoveStats
Conclusions
• Not a slam-dunk win
• Data quality is as good or better
• Results are equivalent
• Survey satisfaction is as good or
better
Tweet! #AAPOR @LoveStats
Final Thoughts
• Responders self-select for a 10 or 20 minute
experience
• Differences are in line with sampling error
• Large portions of most surveys do
not need to be answered by everyone.
Plan ahead.
Tweet! #AAPOR @LoveStats
Thank you!
Annie Pettit
Chief Research Officer
Peanut Labs
@LoveStats on Twitter
www.peanutlabs.com
: )