november 11 , 2016 daniele toninelli · 09-13 november 2016, miami, florida, us november 11 th,...
Post on 27-Jun-2020
0 Views
Preview:
TRANSCRIPT
09-13 November 2016,
Miami, Florida, US
November 11th, 2016
Daniele Toninelli University of Bergamo (Italy) - daniele.toninelli@unibg.it
Melanie RevillaRECSM - Universitat Pompeu Fabra (Spain) - melanie.revilla@upf.edu
1. Background
2. Literature
3. Goals & hypotheses
4. Data
5. Methodology
6. Results
7. Conclusions
8. Limits and further research
9. References
“How the Device Screen Size Affects
Data Collected in Web Surveys”
D. Toninelli, M. Revilla
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
Web survey framework
�“unintended mobile respondents” (Peterson, 2012)
Mobile devices: not negligible (Revilla et al., 2015)
�Spain Netquest panel (186 surveys)
� Average mobile participation: 1/3 resp. (Revilla, 2016)
�Different devices characteristics (Sweeney & Crestani,
2006)
� Virtual keyboard
� Speed of Internet connection
� Device & screen sizes → enhanced portability
o Differences within the mobile devices
1. Background: the context
3
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
Mobile devices → affect data collection
�Key factor: screen size
�Higher portability (Brick et al., 2007)
� Higher social desirability bias (Mavletova & Couper, 2013)
� Multitasking (Toninelli & Revilla, 2016)
�Quality and comparability potentially affected
� Response rates reduced (Baker-Prewitt, 2013)
� Increased breakoff rates (Buskirk & Andrus, 2014)
� Longer response times (Mavletova, 2013; Liebe et al., 2015)
� Undesirable differences in responses (Peytchev & Hill,
2008)
2. Literature: previous findings
4
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
Importance of the “screen size”
�Reduced visibility (scrolling) (Peytchev & HIll, 2008)
� Higher effort/burden (de Bruijne & Wijnant, 2013)
�Different completion times (Couper & Peterson, 2015)
� Neg. link screen size/interview length (Liebe et al., 2015)
� Positive correl. screen size/acquiescence tendency (Liebe et al., 2015)
� U-shaped relation error variance/acquiescence
tendency (Liebe et al., 2015)
�Frequent solution: questionnaire optimization(de Bruijne & Wijnant, 2013; Fischer & Bernet, 2014; Mitchel, 2014)
2. Literature: previous findings
5
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
Focus on mobile devices only
�High diversity
Exact screen size
�Measured in inches (diagonal)
More complete view
�Different indicators (4) analyzed
Optimization effect
Updated view
�… quickly growing diversity of devices
3. Goals & hypotheses: contribution
6
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
Effect of the screen size on:
�Completion time (CT)
� Smaller screen size → longer CTs
�Instructional Manipulation Check (IMC)
� Smaller screen size → higher fail rate IMC
�Answer Consistency (AC)
� Smaller screen size → lower AC
�Survey Experience (SE)
� Smaller screen size → more negative SE
Questionnaire optimization effect
3. Goals & hypotheses: hypotheses
7
H1
H2
H3
H4
Hsub
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
Netquest panel (Spain)
�Two-wave survey
� Wave 1 (w1): Feb. 23rd - Mar. 2nd 2015
� Wave 2 (w2): Mar. 9th - Mar. 18th 2015
� Completes: 1,800 (w1; 54.3% of contacted); 1,608 (w2; 89.3%)
�Experimental design
� Survey condition randomly assigned (each wave):
o PC = participation using PC
o MO = participation using mobile devices (quest. optimized)
o MNO = participation using mobile devices (quest. non-optimized)
�Panelists analyzed here: 719 (mobile both waves)
4. Data: the experiment
8
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
Sensitive behaviors (Mavletova & Couper, 2013)
�>100 questions
� Deviant behaviors (justified, done), Immigration
(opinion), Alcohol consumption (frequency, done,
judgement)
� Background variables (e.g. Income, Internet access
Frequency)
� Perceived questions sensitivity
� Survey experience (easy; liked)
�Different layout/scale proposed
� E.g.: yes/no to 11-point scale; grids/separate items
4. Data: the questionnaire
9
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
Overview (screen size)
Step 1: Analysis by group
�Quartiles “screen size” variable
�ANOVA/t test (equality of means)
Step 2: Regression analysis
�Y = indicator, w1 (CT, IMC, AC*, SE)
� Multiple regression (CT, AC*, SE)
� Logistic regression (IMC)
�Robustness: Std. var. + Forward/backward
* For this indicator w1 and w2 data are compared
5. Methodology: analyses
10
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
Screen size stats (w1 / n = 690 / inches)
�Statistics
� Min.: 2.80 / Max.: 10.10
� Avg.: 4.62 (st.dev.: 1.13)
� Median: 4.50 / Mode: 4.00
�By quartile (w1) distrib.
6. Results: overview
11
Freq. %
Q1 (2.8-4.0] 239 34.6
Q2 (4.0-4.5] 137 19.9
Q3 (4.5-5.0] 225 32.6
Q4 (5.0-10.1] 89 12.9
TOTAL 690 100.0
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
Screen size change (w2 vs w1 / n = 686)
�Changes: 48 panelists (7,0%)
6. Results: overview
12
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
ANOVA (quartile classes / w1)
�Significantly
different averages (p = .000)
�The smaller the
screen, the longer
the CTs
�H1 supported
6. Results: completion times
13
H1
Incr
easi
ng s
ize
Average CT = 16.3 min.
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
Regression Y = CT (wave 1)
6. Results: completion times
14
H1
Variables Coeff. p-values(Constant) 1296.20 .000Screen size -37.96 .002Optimization -34.10 .214How Long Acc. Int. 1.81 .475Freq. Acc. Int. -7.72 .004Fare-TimeUse 9.42 .896Fare-Wifi 97.12 .092Conn. speed satisf. 14.79 .376Difficult participation -52.32 .029Dislike survey 24.82 .309Felt easy 37.07 .049Perceived sensit. 9.02 .687Pixel density -.27 .072Age 6.08 .000Educ. Level -21.95 .138
Screen Size:• Significant
• + 1 inch ≈ - 38 seconds
• H1 supported (smaller
screens → longer CTs)
• Previous literature
findings confirmed
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
Regression Y = CT (wave 1)
6. Results: completion times
15
Variables Coeff. p-values(Constant) 1296.20 .000Screen size -37.96 .002Optimization -34.10 .214How Long Acc. Int. 1.81 .475Freq. Acc. Int. -7.72 .004Fare-TimeUse 9.42 .896Fare-Wifi 97.12 .092Conn. speed satisf. 14.79 .376Difficult participation -52.32 .029Dislike survey 24.82 .309Felt easy 37.07 .049Perceived sensitivity 9.02 .687Pixel density -.27 .072Age 6.08 .000Educ. Level -21.95 .138
Optimization:• Not significant
• Vertical scrolling more
relevant
• H1sub (optimization → shortening CTs) not
supported
H1sub
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
ANOVA (quartile classes / w1)
�Non-significantly
different averages (p = .247)
�No direct link
between IMC fail %
and screen size
�H2 not supported
6. Results: instr. manipul. check
16
H2
Incr
easi
ng s
ize
Average IMC = 13.6 %
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
Logistic regression Y = IMC (wave 1)
6. Results: instr. manipul. check
17
H2
Variables Coeff. p-values(Constant) -2.499 .009Screen size -.359 .064Optimization -.546 .029How Long Acc. Int. -.012 .718Freq. Acc. Int. .047 .102Fare-TimeUse .898 .082Fare-Wifi .092 .860Conn. speed satisf. -.110 .426Easy participation -.378 .064Like survey .048 .828Felt easy -.084 .632Perceived sensit. .369 .064Pixel density .002 .152Age .028 .013Educ. Level .172
Screen Size:• Not significant
• H2 (smaller screens → higher IMC fail %) not
supported
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
Variables Coeff. p-values(Constant) -2.499 .009Screen size -.359 .064Optimization -.546 .029How Long Acc. Int. -.012 .718Freq. Acc. Int. .047 .102Fare-TimeUse .898 .082Fare-Wifi .092 .860Conn. speed satisf. -.110 .426Easy participation -.378 .064Like survey .048 .828Felt easy -.084 .632Perceived sensit. .369 .064Pixel density .002 .152Age .028 .013Educ. Level .172
Logistic reg. Y = IMC (wave 1)
6. Results: instr. manipul. check
18
Optimization:• Significant
• Optimized questionnaire → ≈
- 54.6 percentage points in
IMC fails %
• Higher participation quality
• H2sub supported
(optimization → lower IMC
fail %)
H2sub
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
ANOVA (no size change)
(quartile classes/survey
condition)
�Non-significantly
different averages (p = .689/.089)
�No link between AC
and screen size/
survey condition
�H3 & H3sub not
supported
6. Results: answer consistency
19
H3
Incr
easi
ng s
ize
Average AC = 90.1 %
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
Multiple regression Y = AC (wave 2 vs wave 1)Variables Coeff. p-values(Constant) .873 .000Screen size (w1) -.001 .509Screen size change (Δ) -.004 .097Educ. level (w1) .005 .020Easy participat. (w1) .010 .034Easy participat. (Δ) .008 .025Felt easy (Δ) .007 .045Perceived sensit. (w1) -.012 .001Perceived sensit. (Δ) -.010 .005SurveyCond_MO-MO .009 .115SurveyCond_MO-MNO -.002 .727SurveyCond_MNO-MO -.001 .789
6. Results: answer consistency
20
H3
Screen Size (w1 and Δ)
& Survey condition:• Not significant
• Optim. effect for grids
• H3 & H3sub (smaller
screens / optimization → affect AC) not supported
H3sub
Other variables not significant (p-values>.05) are not listed
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
Easy participation
ANOVA (quartile cl./w1)
�Significantly different
averages (p = .005)
�Indirect link between
“easy participation” and
screen size
�H4/a supported
�H4sub/a supported
6. Results: survey experience
21
H4
Incr
easi
ng s
ize
Average = 3.32
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
Multiple regression Y = “easy participation”
6. Results: survey experience
22
H4
Variables Coeff. p-values(Constant) 2.275 .000Screen size .026 .218Optimization .226 .000How Long Acc. Int. .001 .813Freq. Acc. Int. .012 .010Conn. speed satisf. .189 .000Perceived sensit. .174 .000Pixel density .000 .149Age .000 .933Educ. level -.057 .031
Screen Size:• Not significant
• H4 (smaller screens → less easy) not supp.
Optimization:• Significant
• H4sub supported
(optimization → more
easy)
H4sub
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
Like survey
ANOVA (quartile cl./w1)
�Not significantly
different averages (p =
.085)
�No link between “like
survey” and screen size
�H4/b not supported
�H4sub/b supported
6. Results: survey experience
23
H4
Incr
easi
ng s
ize
Average = 3.11
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
Multiple regression Y = “like survey”
6. Results: survey experience
24
H4
Variables Coeff. p-values(Constant) 2.323 .000Screen size .037 .090Optimization .117 .019How Long Acc. Int. .008 .093Freq. Acc. Int. .004 .421Conn. speed satisf. .129 .000Perceived sensit. .220 .000Pixel density .000 .673Age -.001 .754Educ. level -.037 .167
Screen Size:• Not significant
• H4 (smaller screens → lower like) not supp.
Optimization:• Significant
• H4sub supported
(optimization → higher
“like”)
H4sub
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
7. Conclusions: main findings
25
CT IMC AC SE
H(screen size) Affected
Not Affected
Not Affected
Partially Affected
(ANOVA “easy”)
Hsub(optimizat.)
Not Affected
AffectedNot
AffectedAffected
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
… thus?
�Small sized devices do not affect data quality…
� Even if the burden (CTs) and the SE can be affected
�… moreover potential issues (IMC, SE) can be
attenuated using optimized questionnaires
� Positive for the willingness in participating again
� Differently applied by different survey developers
�… current issues are becoming less important
� Bigger devices; higher resolutions; advanced technol.
�Focus on mobile: wider data collection options
7. Conclusions: discussion
26
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
Limits…
�Non-probability
based panel
�Focus on Spain
�Topics not
sufficiently studied
in depth
�Quick evolution of
phenomenon/
technology
8. Limits and further research
27
… & further research
�General population
studies
�Replication studies
�E.g. trends of
experience in using
mobile devices
�Keep on monitoring it (enhanced indicators,
detailed and systematic
paradata collection)
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
• Baker-Prewitt, J. (2013), “Mobile Research Risk: What Happens to Data Quality When
Respondents Use a Mobile Device for a Survey Designed for a PC?”, in CASRO Online
Research Conference 2013 Proceedings, San Francisco, US, 2013, Burke, Cincinnati,
OH, pp. 1-17.
• Brick J.M., Brick P.D., Dipko S., Presser S., Tucker C., Yuan Y. (2007). “Cell Phone
Survey Feasibility in the U.S.: Sampling and Calling Cell Numbers Versus Landline
Numbers”. Public Opinion Quarterly, 71(1), pp. 23-39.
• Buskirk T.D., Andrus C. (2014). “Making mobile browser surveys smarter: results from a
randomized experiment comparing online surveys completed via computer or
smartphone”. Field Methods, published online before print 14 April 2014.
• Couper M.P., & Peterson G.J. (2015). “Exploring Why Mobile Web Surveys Take
Longer”. Presentation at the 2015 GOR conference.
• de Bruijne, M. and Wijnant, A. (2013). “Can Mobile Web Surveys Be Taken on
Computers? A Discussion on a Multi-Device Survey Design”. Survey Practice, Vol. 6,
no. 4.
• de Bruijne, M. and Wijnant, A. (2013). “Comparing Survey Results Obtained via Mobile
Devices and Computers: An Experiment With a Mobile Web Survey on a Heterogeneous
Group of Mobile Devices Versus a Computer-Assisted Web Survey”. Social Science
Computer Review, Vol. 31, no. 4, 482-504.
9. References
28
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
• Fischer, B., & Bernet, F. (2014). “Device effects: How different screen sizes affect answer
quality in online questionnaires”. Paper presented at the 2014 GOR - General Online
Research conference, Cologne, Germany.
• Liebe U., Glenk K., Oehlmann M., Meyerhoff J. (2015). “Does the use of mobile devices
(tablets and smartphones) affect survey quality and choice behaviour in web surveys?”,
Journal of Choice Modelling, Volume 14, March 2015, Pages 17-31.
• Mavletova, A. (2013), “Data quality in PC and mobile web surveys”, Social Science
Computer Review, Vol. 31 No. 4, pp. 725-743.
• Mavletova, A. and Couper, M.P. (2013), “Sensitive Topics in PC Web and Mobile Web
Surveys: Is There a Difference?”, Survey Research Methods, Vol. 7 No. 3, pp. 191-205.
• Mitchel, N. (2014). “When it comes to mobile respondent experience and data quality,
survey design matters”. Quirk’s – Marketing Research Media. Link:
http://www.quirks.com/articles/2014/20140825-3.aspx.
• Peterson, G. (2012), “Unintended mobile respondents”. Paper presented at CASRO
Technology Conference, 31 May, New York, NY.
• Peytchev A., Hill, C.A. (2008). “Experiments in Mobile Web Survey Design”.
Proceedings of the 63rd Annual conference of the American Association for Public
Opinion Research - AAPOR, 2008.
9. References
29
D. Toninelli, M. Revilla
“How the Device Screen Size Affects Data Collected in Web Surveys”
• Revilla M., Toninelli D., Ochoa C., Loewe G. (2015). “Who Has Access to Mobile
Devices in an Opt-in Commercial Panel? An Analysis of Potential Respondents for
Mobile Surveys” in Toninelli, D., Pinter, R., de Pedraza, P. (eds.) “Mobile Research
Methods: Opportunities and Challenges of Mobile Research Methodologies”. Ubiquity
Press, London. Pp. 119–139. DOI: http://dx.doi.org/10.5334/bar.
• Revilla M. (2016). “Analyzing the survey characteristics, participation, and evaluation
across 186 surveys in an online opt-in panel in Spain”. QDET2 conference, 11-14
November 2016, Miami, Florida (US)
• Sweeney, S., & Crestani, F. (2006). Effective search results summary size and device
screen size: Is there a relationship? Information Processing & Management, 42, 1056-
1074.
• Toninelli D., Revilla M. (2016a) “Smartphones vs PCs: Does the Device Affect the Web
Survey Experience and the Measurement Error for Sensitive Topics? - A Replication of
the Mavletova & Couper’s 2013 Experiment”. Survey Research Methods Vol. 10, No. 2,
pp. 153-169. ISSN: 1864-3361. DOI: http://dx.doi.org/10.18148/srm/2016.v10i2.6274.
9. References
30
For further information:
Daniele Toninelli - daniele.toninelli@unibg.it
Melanie Revilla - melanie.revilla@upf.edu
“How the Device Screen Size Affects
Data Collected in Web Surveys”
D. Toninelli, M. Revilla
top related