![Page 1: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/1.jpg)
ISO 20252 Certified Gregg Peterson
Senior Vice President, Research Operations
Solving the
Unintentional Mobile Challenge
CASRO Online Research Conference – March 2013
![Page 2: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/2.jpg)
Growth in unintentional mobile survey taking mirrors
growth of internet access via cell phones
0%
5%
10%
15%
20%
25%
30%
Q4 '11 Q1 '12 Q2 '12 Q3 '12 Q4 '12
17%
26%
A current client tracker at Market Strategies:
Mobile survey starts are up 50% in past year
Mobile survey starts
Completes
Page 2
![Page 3: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/3.jpg)
UMR happens...do we care?
> Increased survey length
> Poor usability
> High breakoffs
> Possible mode effects
PROS CONS
> Honor respondent choice
> Improve coverage
> Maximize participation rates
Can a better design minimize the
risks and maximize the benefits?
Page 3
![Page 4: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/4.jpg)
Page 4
HIGH COMPLETION RATES
Fast
Easy
Enjoyable
SAME RESPONSES
COMPARED TO PC
DESIGN GOALS
HIGH ENGAGEMENT
No speeding
Meaningful responses
![Page 5: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/5.jpg)
Mobile
Friendly
![Page 6: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/6.jpg)
Page 6
Test cell 1 – Mobile/legacy (plain html)
![Page 7: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/7.jpg)
Page 7
Test cell 2 – Mobile/new (“mobile friendly”)
![Page 8: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/8.jpg)
Page 8
Test cell 3 – Mobile/numeric (“mobile friendly”)
![Page 9: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/9.jpg)
Page 9
Test cell 4 – Mobile/slider (“mobile friendly”)
![Page 10: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/10.jpg)
Page 10
Test cell 5 – Mobile/drop down (“mobile friendly”)
![Page 11: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/11.jpg)
Page 11
Test cell 6 – PC/legacy (plain html)
![Page 12: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/12.jpg)
Page 12
Test cell 7 – PC/new (“mobile friendly”)
![Page 13: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/13.jpg)
Page 13
Test cell 8 – Mobile/app (Mobile optimized)
![Page 14: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/14.jpg)
Smartphone PC/tablet
Legacy (HTML) 1 – Mobile/legacy 6 – PC/legacy
“Mobile friendly”
2 – Mobile/new
3 – Mobile/numeric
4 – Mobile/slider
5 – Mobile/drop down
7 – PC/new
Mobile App 8 – Mobile/app
Experimental Design
Random assignment at sampling stage
*
*
* Random assignment upon
initial survey entry ~ N = 250 / test cell
Page 14
![Page 15: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/15.jpg)
Hypothesis
> Mobile presentations will have longer survey lengths compared to PC,
regardless of presentation
> All mobile presentations will have higher breakoffs compared to PC
> Mobile friendly versions on smartphones will have shorter survey lengths
and lower break-offs compared to the Mobile/legacy version
> Mobile friendly versions on smartphones will elicit higher user experience
ratings compared to the Mobile/legacy (html) version on a smartphone
> Visual design changes, whether on mobile or PC, have no impact on
measures of engagement
> Response distributions across platforms (PC versus smartphone) will be
more similar when they use a similar design e.g. Cell 2: Mobile/new will
be most similar to cell 7: PC/new
H1
H2
H3
H4
H5
H6
Page 15
![Page 16: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/16.jpg)
Page 16
6- PC/ legacy 7- PC/new
A consistent mobile friendly design – most
comparable across devices?
2 - Mobile/new
H6
best match?
![Page 17: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/17.jpg)
Possible self-selection bias for Smartphone groups Experimental challenges
Page 17
Smartphone PC/tablet
Some self selection for Smartphone groups
Invitations
50% 6% Wrong device
Final participation
rate 6% 12%
![Page 18: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/18.jpg)
0%
60%
0%
60%
Are these groups different?
Age
35-54
55+
18-34
Page 18
weighted
0%
60%
0%
60%
0%
60%
Gender
Education
Income
College
Grad school
100K +
Male Female
50K–100K
Some college/tech
Sig difference A few small differences after weighting
Mobile PC
1 - legacy 2 - new 3 - numeric 4 - slider 5 - drop down 8 - App 6 - legacy 7 - new
![Page 19: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/19.jpg)
PC groups access the internet from phone less often
Page 19
23 24 22 23 22 22 23 22
10
15
20
25
0%
60%
0%
100%
0%
60%
Time spent
on Internet (hours/week)
Cell
Internet
usage (past mo.)
Daily
Weekly
Current
wireless
connection
3G
4G
LTE 4G
WIFI
Mean
40+
20–39
Sig difference
Mobile PC
1 - legacy 2 - new 3 - numeric 4 - slider 5 - drop down 8 - App 6 - legacy 7 - new
![Page 20: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/20.jpg)
Survey length and breakoffs
0 3 6 9 12 15 18 21 24Survey length (minutes)
11.0
8.9
10.2
9.4
10.3
7.1
7.0
> All mobile designs slower than PC
> Legacy mobile more than 50% slower than PC with significantly
higher breakoffs
> Better design of Mobile/new improves on length
13%
5%
4%
7%
5%
4%
7%
No
difference
Survey length Breakoffs
13%
H1 – H3
Page 20
Mobile
1 - legacy
2 - new
3 - numeric
4 - slider
5 - drop down
PC
6 - legacy
7 - new
![Page 21: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/21.jpg)
Traditionally formatted instruments create a poor
user experience for mobile survey takers; R’s liked the
mobile app experience best
Enjoy
completing
the survey
Fast or
slow
Easy or
difficult
to answer
Easy or
difficult
to read Interesting
Top 2-box Bottom 3-box
Mobile
1 - legacy
2 - new
3 - numeric
4 - slider
5 - drop down
Mobile 8 - App
PC
6 - legacy
7 - new
Page 21
H4
![Page 22: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/22.jpg)
Mobile respondents as engaged as PC survey takers;
few significant differences across mobile test cells
Straight-
Lining
1+ grids
Trap
question
Self-
reported
satisficing
Number of
OE char
Provided a
Considered
response
Mobile
1 - legacy
2 - new
3 - numeric
4 - slider
5 - drop down
8 - App
PC
6 - legacy
7 - new
Sig difference
Page 22
H5
![Page 23: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/23.jpg)
Page 23
Strategy for comparing response distributions in this
experiment: Each PC cell versus each Mobile cell
10 0 5
6 - PC/legacy 7 - PC/new 2 - Mobile/new
10 0 5 10 0 5 0%
10%
20%
30%
Difference of top 2-box
%
Very favorable Very unfavorable
Question: “Please indicate your overall feelings toward your electric utility provider.”
![Page 24: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/24.jpg)
Categorical
(15 questions)
0-10 ratings
(12 questions)
CHI-SQUARE ANOVA
Mean absolute
differences
legacy new legacy new Legacy new
7 – PC/new
3 0 4%
Mobile
1 - legacy 1 2 0 0 4% 3%
2 - new 1 3 0 0 3% 4%
3 - numeric 4 5 0 4 5% 6%
4 - slider 10 9 7 9 6% 5%
5 - drop down 6 5 0 2 7% 7%
8 - App 5 4 0 3 6% 6%
Comparing PC and Mobile distributions on
27 key measures
Page 24
versus
H6
![Page 25: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/25.jpg)
6 - PC/
legacy
7 - PC/
new
7 – PC/new
Mobile
1 - legacy
2 - new
3 - numeric X X
4 - slider X X
5 - drop down X X
8 - App X X
Comparing PC and Mobile response distributions on
27 key measures - Summary
Page 25
H6
![Page 26: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/26.jpg)
Novel input designs can produce anomalous
results for selected question/scale types
Page 26
0%
60%
Strongly agree Somewhat agree Neither Somewhat disagree Strongly disagree
3 - Mobile/numeric
4 - Mobile/slider
6 - PC/legacy
7 - PC/new
3 - Mobile/numeric input
4 - Mobile/slider
![Page 27: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/27.jpg)
Vertical presentation of choices via drop
down exaggerates use of top choice
Page 27
0%
30%
10 =Very favorable
9 8 7 6 5 4 3 2 1 0 =Very unfavorable
5 - Mobile/drop down
6 - PC/legacy
7- PC/new
4- Mobile/ drop down
![Page 28: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/28.jpg)
Mobile/legacy and Mobile/new produce the same
results on key measures
Page 28
≈
![Page 29: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/29.jpg)
Non-optimized mobile survey presentations do not
automatically lead to different response distributions
Page 29
0%
30%
10 =Extremelysatisfied
9 8 7 6 5 4 3 2 1 0 =Extremely
dissatisfied
1- Mobile/legacy
2 - Mobile/new
7 - PC/new
![Page 30: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/30.jpg)
Hypothesis
> Mobile presentations will have a longer survey lengths compared to
PC, regardless of presentation
> All mobile presentations have higher breakoffs compared to PC
> “Mobile friendly” versions on smartphones will have shorter survey
lengths and lower break-offs compared to the Mobile/ legacy version
> Mobile friendly versions on smartphones will elicit higher user
experience ratings compared to the HTML version on a smartphone
> Visual design changes, whether on mobile or PC, have no impact on
measures of engagement
> Response distributions across platforms (PC versus smartphone) will
be more similar when they use a similar design e.g. Cell 2: Mobile/new
will be most similar to cell 7: PC/new
H1
H2
H3
H4
H5
H6
X
X Page 30
X
![Page 31: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/31.jpg)
Fast
Easy
Enjoyable
Improve completion rates
Decrease survey length
Improve user experience
Mobile friendly designs…
Have no impact on user engagement
2
The right mobile friendly design…
Does not compromise response quality
Positive user experience better data
Page 31
![Page 32: Solving the Unintentional Mobile Challenge - c.ymcdn.com · Senior Vice President, Research Operations Solving the Unintentional Mobile Challenge ... Age 35-54 55+ 18-34](https://reader031.vdocument.in/reader031/viewer/2022022602/5b5984997f8b9a657c8d533e/html5/thumbnails/32.jpg)
ISO 20252 Certified
Special thanks to
ResearchNow for their
support on this project
Contact:
Gregg Peterson
Senior Vice President, Research Operations
Connect with us:
marketstrategies.com | freshmr.com
Twitter | Facebook | LinkedIn
ISO 20252 Certified
Market Strategies project team:
Joanne Mechling
John LaFrance
Janice Swinehart
Gina Ham