solving the unintentional mobile challenge - c.ymcdn.com · senior vice president, research...
Post on 26-Jul-2018
217 Views
Preview:
TRANSCRIPT
ISO 20252 Certified Gregg Peterson
Senior Vice President, Research Operations
Solving the
Unintentional Mobile Challenge
CASRO Online Research Conference – March 2013
Growth in unintentional mobile survey taking mirrors
growth of internet access via cell phones
0%
5%
10%
15%
20%
25%
30%
Q4 '11 Q1 '12 Q2 '12 Q3 '12 Q4 '12
17%
26%
A current client tracker at Market Strategies:
Mobile survey starts are up 50% in past year
Mobile survey starts
Completes
Page 2
UMR happens...do we care?
> Increased survey length
> Poor usability
> High breakoffs
> Possible mode effects
PROS CONS
> Honor respondent choice
> Improve coverage
> Maximize participation rates
Can a better design minimize the
risks and maximize the benefits?
Page 3
Page 4
HIGH COMPLETION RATES
Fast
Easy
Enjoyable
SAME RESPONSES
COMPARED TO PC
DESIGN GOALS
HIGH ENGAGEMENT
No speeding
Meaningful responses
Mobile
Friendly
Page 6
Test cell 1 – Mobile/legacy (plain html)
Page 7
Test cell 2 – Mobile/new (“mobile friendly”)
Page 8
Test cell 3 – Mobile/numeric (“mobile friendly”)
Page 9
Test cell 4 – Mobile/slider (“mobile friendly”)
Page 10
Test cell 5 – Mobile/drop down (“mobile friendly”)
Page 11
Test cell 6 – PC/legacy (plain html)
Page 12
Test cell 7 – PC/new (“mobile friendly”)
Page 13
Test cell 8 – Mobile/app (Mobile optimized)
Smartphone PC/tablet
Legacy (HTML) 1 – Mobile/legacy 6 – PC/legacy
“Mobile friendly”
2 – Mobile/new
3 – Mobile/numeric
4 – Mobile/slider
5 – Mobile/drop down
7 – PC/new
Mobile App 8 – Mobile/app
Experimental Design
Random assignment at sampling stage
*
*
* Random assignment upon
initial survey entry ~ N = 250 / test cell
Page 14
Hypothesis
> Mobile presentations will have longer survey lengths compared to PC,
regardless of presentation
> All mobile presentations will have higher breakoffs compared to PC
> Mobile friendly versions on smartphones will have shorter survey lengths
and lower break-offs compared to the Mobile/legacy version
> Mobile friendly versions on smartphones will elicit higher user experience
ratings compared to the Mobile/legacy (html) version on a smartphone
> Visual design changes, whether on mobile or PC, have no impact on
measures of engagement
> Response distributions across platforms (PC versus smartphone) will be
more similar when they use a similar design e.g. Cell 2: Mobile/new will
be most similar to cell 7: PC/new
H1
H2
H3
H4
H5
H6
Page 15
Page 16
6- PC/ legacy 7- PC/new
A consistent mobile friendly design – most
comparable across devices?
2 - Mobile/new
H6
best match?
Possible self-selection bias for Smartphone groups Experimental challenges
Page 17
Smartphone PC/tablet
Some self selection for Smartphone groups
Invitations
50% 6% Wrong device
Final participation
rate 6% 12%
0%
60%
0%
60%
Are these groups different?
Age
35-54
55+
18-34
Page 18
weighted
0%
60%
0%
60%
0%
60%
Gender
Education
Income
College
Grad school
100K +
Male Female
50K–100K
Some college/tech
Sig difference A few small differences after weighting
Mobile PC
1 - legacy 2 - new 3 - numeric 4 - slider 5 - drop down 8 - App 6 - legacy 7 - new
PC groups access the internet from phone less often
Page 19
23 24 22 23 22 22 23 22
10
15
20
25
0%
60%
0%
100%
0%
60%
Time spent
on Internet (hours/week)
Cell
Internet
usage (past mo.)
Daily
Weekly
Current
wireless
connection
3G
4G
LTE 4G
WIFI
Mean
40+
20–39
Sig difference
Mobile PC
1 - legacy 2 - new 3 - numeric 4 - slider 5 - drop down 8 - App 6 - legacy 7 - new
Survey length and breakoffs
0 3 6 9 12 15 18 21 24Survey length (minutes)
11.0
8.9
10.2
9.4
10.3
7.1
7.0
> All mobile designs slower than PC
> Legacy mobile more than 50% slower than PC with significantly
higher breakoffs
> Better design of Mobile/new improves on length
13%
5%
4%
7%
5%
4%
7%
No
difference
Survey length Breakoffs
13%
H1 – H3
Page 20
Mobile
1 - legacy
2 - new
3 - numeric
4 - slider
5 - drop down
PC
6 - legacy
7 - new
Traditionally formatted instruments create a poor
user experience for mobile survey takers; R’s liked the
mobile app experience best
Enjoy
completing
the survey
Fast or
slow
Easy or
difficult
to answer
Easy or
difficult
to read Interesting
Top 2-box Bottom 3-box
Mobile
1 - legacy
2 - new
3 - numeric
4 - slider
5 - drop down
Mobile 8 - App
PC
6 - legacy
7 - new
Page 21
H4
Mobile respondents as engaged as PC survey takers;
few significant differences across mobile test cells
Straight-
Lining
1+ grids
Trap
question
Self-
reported
satisficing
Number of
OE char
Provided a
Considered
response
Mobile
1 - legacy
2 - new
3 - numeric
4 - slider
5 - drop down
8 - App
PC
6 - legacy
7 - new
Sig difference
Page 22
H5
Page 23
Strategy for comparing response distributions in this
experiment: Each PC cell versus each Mobile cell
10 0 5
6 - PC/legacy 7 - PC/new 2 - Mobile/new
10 0 5 10 0 5 0%
10%
20%
30%
Difference of top 2-box
%
Very favorable Very unfavorable
Question: “Please indicate your overall feelings toward your electric utility provider.”
Categorical
(15 questions)
0-10 ratings
(12 questions)
CHI-SQUARE ANOVA
Mean absolute
differences
legacy new legacy new Legacy new
7 – PC/new
3 0 4%
Mobile
1 - legacy 1 2 0 0 4% 3%
2 - new 1 3 0 0 3% 4%
3 - numeric 4 5 0 4 5% 6%
4 - slider 10 9 7 9 6% 5%
5 - drop down 6 5 0 2 7% 7%
8 - App 5 4 0 3 6% 6%
Comparing PC and Mobile distributions on
27 key measures
Page 24
versus
H6
6 - PC/
legacy
7 - PC/
new
7 – PC/new
Mobile
1 - legacy
2 - new
3 - numeric X X
4 - slider X X
5 - drop down X X
8 - App X X
Comparing PC and Mobile response distributions on
27 key measures - Summary
Page 25
H6
Novel input designs can produce anomalous
results for selected question/scale types
Page 26
0%
60%
Strongly agree Somewhat agree Neither Somewhat disagree Strongly disagree
3 - Mobile/numeric
4 - Mobile/slider
6 - PC/legacy
7 - PC/new
3 - Mobile/numeric input
4 - Mobile/slider
Vertical presentation of choices via drop
down exaggerates use of top choice
Page 27
0%
30%
10 =Very favorable
9 8 7 6 5 4 3 2 1 0 =Very unfavorable
5 - Mobile/drop down
6 - PC/legacy
7- PC/new
4- Mobile/ drop down
Mobile/legacy and Mobile/new produce the same
results on key measures
Page 28
≈
Non-optimized mobile survey presentations do not
automatically lead to different response distributions
Page 29
0%
30%
10 =Extremelysatisfied
9 8 7 6 5 4 3 2 1 0 =Extremely
dissatisfied
1- Mobile/legacy
2 - Mobile/new
7 - PC/new
Hypothesis
> Mobile presentations will have a longer survey lengths compared to
PC, regardless of presentation
> All mobile presentations have higher breakoffs compared to PC
> “Mobile friendly” versions on smartphones will have shorter survey
lengths and lower break-offs compared to the Mobile/ legacy version
> Mobile friendly versions on smartphones will elicit higher user
experience ratings compared to the HTML version on a smartphone
> Visual design changes, whether on mobile or PC, have no impact on
measures of engagement
> Response distributions across platforms (PC versus smartphone) will
be more similar when they use a similar design e.g. Cell 2: Mobile/new
will be most similar to cell 7: PC/new
H1
H2
H3
H4
H5
H6
X
X Page 30
X
Fast
Easy
Enjoyable
Improve completion rates
Decrease survey length
Improve user experience
Mobile friendly designs…
Have no impact on user engagement
2
The right mobile friendly design…
Does not compromise response quality
Positive user experience better data
Page 31
ISO 20252 Certified
Special thanks to
ResearchNow for their
support on this project
Contact:
Gregg Peterson
Senior Vice President, Research Operations
gregg.peterson@marketstrategies.com
Connect with us:
marketstrategies.com | freshmr.com
Twitter | Facebook | LinkedIn
ISO 20252 Certified
Market Strategies project team:
Joanne Mechling
John LaFrance
Janice Swinehart
Gina Ham
top related