mobile optimization considerations for survey design a ...kostat.go.kr/iwsm/download/2016/session...
TRANSCRIPT
Mobile Optimization Considerations for
Survey Design –
A Case Study using the U.S. Census
Elizabeth NicholsCenter for Survey Measurement
U.S. Census Bureau
7th International Workshop on Survey MethodologyDaejeon, South KoreaSeptember 28, 2016
This presentation is released to inform interested parties of research and to encourage discussion. The views expressed are those of the authors and not necessarily those of the U.S. Census Bureau.
1
Korea has highest smartphone
ownership at 88%
2
By Lee Hana Korea.net http://www.korea.net/NewsFocus/Sci-Tech/view?articleId=133014
U.S. smartphone ownership at 72%
0
10
20
30
40
50
60
70
80
90
100
2011 2012 2013 2014 2015
Percent
Computer - PC
Smartphone
Tablet
Source: U.S. Census Bureau Internet and Computer Usage Supplement and Pew Research Center
3
Overview of Talk
Four considerations for offering a responsive design – that is, when the design displays optimally on a Personal Computer (PC) and on a mobile device
U.S. Census survey instrumentsMobile-optimized and un-optimized 2011 and 2014 American Community Survey – nationwide -
un-optimized 2014 Census Test – site test in and near Washington D.C. -
un-optimized 2015 Census Test – site test in Savannah, GA area and Maricopa
County, AZ – mobile-optimized 2016 Census Test – site test in Houston, TX and Los Angeles, CA
– mobile-optimized Offered in English, Spanish, Chinese and Korean
4
Consideration 1
Do you need to design your
survey for smaller devices?
5
Consider
Mobile device penetration rates
Whether there will be a bias in your estimates if you do not offer the mobile-optimized version in addition to a PC-optimized version
Respondent expectations
IT resources
6
U.S. Statistics
Device usage when answering the online
census questionnaire
0
10
20
30
40
50
60
70
80
90
100
2011 2014 2015
Percent Mobile device
Smartphone
Tablet
Computer - PC
Source: U.S. Census Bureau 2011 American Community Survey Internet Test, 2014 Census Test, 2015 Census Test
7
Public feedback from the
2016 Census Test
“We would prefer paper-form option but if we get assistance with online option it’s ok.” ~ an older adult, 64 years old or older
“I like the option of being able to get on my phone to fill out the survey.” ~ a college student, between the ages of 18-25
Source: Community Partnership Engagement Program in Harris County, Texas and Los Angeles, California
8
Consideration 2
What should you do before you
field a mobile-optimized design?
9
Consider
Your current un-optimized survey on a mobile phone What features are difficult to see and answer
The conventions of your population In the U.S. we read left to right, up to down
Take a look at popular sites
U.S. mobile standards – draft https://standards.usa.gov/
Draft a design, test with real users, repeat
10
An un-optimized online form
U.S. American Community Survey
11
Optimized for desktop/laptop Not optimized for smartphone
Design considerations
12
U.S. sources of inspiration
13
Usability testing
14
Consideration 3
What are some design features to
evaluate?
15
Preparing for a
mobile-optimized design
Respondents need to:
read question without zooming or horizontal scrolling
record answers without zooming
feel like the entire screen is the survey and not just a portion of the screen
see the question immediately (reduce the size of the banner, navigation, and menu)
16
U.S. convention
Important task needs to go up
Optimized for desktop/laptopOptimized for smartphone
17
No U.S. conventions: hamburger menu,
language icon, menu display
18
U.S. convention:
U.S. names on one line, left to right
19
2015 Census Test 2016 Census Test
U.S. convention:
Respondents are not used to scrolling
horizontally on a smartphone
20
U.S. convention:
Name, telephone number and email
vertically designed for mobile
21
2016 Census Test - English 2016 Census Test - Korean
U.S. Convention: Vertical
scrolling in mobile
22
Smartphone
No U.S. convention for label placement
Label above and near the field
23
Device conventionsExample is a drop down response option
24
iPhone Android
Spacing and symbolism
25
Final design for 2015 Census Test after usability testingInitial design for 2015 Census Test
SpacingNot all vertical design works well
26
Initial design for 2015 Census TestFinal design for 2015 Census Test after usability testing
Consideration 4
How can I evaluate the design
once it is in the field?
27
Consider
Response rates
Paradata analysis after fielding survey
Data associated with the survey completion task
Device used
Time per screen or session
Edit or Alert failures
Changed answers
Break-offs
28
Challenges with paradata results
Mobile takes more time than PC
Mobile increases break-offs compared to PC
Confounds might be self-selection in the U.S.
29
Time-on-task: Not optimized
1.2 tablet - 2014 Census Test
1.4 smartphone - 2014 Census Test
30
From Couper, Antoun, Mavletova Total Survey Error Conference Presentation
Time-on-task: Optimized
1.2 tablet - 2015 Census Test
1.3 smartphone - 2015 Census Test
31
Task completion: Not optimized
32
1.6 smartphone - 2014 Census Test
0.84 tablets - 2014 Census Test
Task completion: Optimized
33
1.14 tablets - 2015 Census Test
1.7 smartphone - 2015 Census Test
Things to consider
We know from usability testing that mobile optimization is more usable than not having it optimized
Usability metrics of time-on-task and task-completion are not necessarily the best measures of usability improvement Why longer?
Scrolling? Connectivity / Load time? Different device displays? Self-selection? Multi-tasking?
Uptick of online forms completed on mobile when optimized Evaluate the demographics of the respondents
34
Summary
Why optimize?
Is there bias and measurement error if you do not
Respondent expectations
How to optimize?
Design, test, redesign
How to measure?
Challenges
35
References-Anderson, M. “Technology Device Ownership: 2015.” Pew Research Center, October, 2015, Available at: http://www.pewinternet.org/2015/10/29/technology-device-ownership-2015
-Baker, R., (2014). Some thoughts on the 2015 Census for mobile. Internal correspondence.
-Couper, M. P., and Peterson, G. (2015). Exploring why mobile Web surveys take longer. Presentation at the GOR 2015 Conference. http://www.gor.de/category/gor-15
-Couper, M. P., Antoun, C. and Mavetova, A (2015). Mobile Web Surveys: A Total Survey Error Perspective. Presentation given at the 2015 International Total Survey Error Conference. Baltimore, Maryland.
-de Bruijne, M. and A. Wijnant. 2013. Comparing survey results obtained via mobile devices and computers: an experiment with a mobile web survey on a heterogeneous group of mobile devices versus a computer-assisted web survey. Social Science Computer Review 31(4): 482–504.
-File, Thom. 2013. “Computer and Internet Use in the United States.” Current Population Survery Reports, P20-568. U.S. Census Bureau,Washington, DC.
-Geisen, E. and Olmsted, M. (2011). 2020 Census Coverage Study: Survey on Mobile Devices Literature Review. RTI International. RTI Project number 0212349.
-Gummer, T., & Rossmann, J. (2015) Explaining Interview Duration in Web Surveys: A Multilevel Approach. Social Science Computer Review 33(2) 217-234
-Horwitz, R., (2015). Usability of the American Community Survey Internet Instrument on Mobile Devices. U.S. Census Bureau. #ACS15-RER-04 Last accessed: https://www.census.gov/content/dam/Census/library/working-papers/2015/acs/2015_Horwitz_01.pdf
-Horwitz, R., (2013). Use of Paradata to Assess the Quality and Functionality of the American Community Survey Internet Instrument U.S. Census Bureau. #ACS12-RER-26-R1 Last accessed: http://www.census.gov/content/dam/Census/library/working-papers/2013/acs/2013_Horwitz_01.pdf
-Jue, A., and Luck, K. (2014). Update: Participation of mobile users in online surveys. Decipher White Paper. Retrieved from https://www.decipherinc.com/n/uploads/images/pages/Decipher_Mobile_Research_White_Paper_Update.pdf.
-Smith, A. (2013). Smartphone Ownership 2013 - Update. Pew Research Center’s Internet & American Life Project. http://pewinternet.org/Reports/2013/Smartphone-Ownership-2013.aspx
-Penzo, M. , 2006 Label Placement in Forms from, http://www.uxmatters.com/mt/archives/2006/07/label-placement-in-forms.php (accessed 8/18/2016)
Pew Internet project (2014). Mobile Technology Fact Sheet as of October 2014. http://www.pewinternet.org/fact-sheets/mobile-technology-fact-sheet/
-Peytchev, A. and Hill, C. A. (2010). Experiments in Mobile Web Survey Design: Similarities to Other Modes and Unique Considerations. Social Science Computer Review. 28(3) 319-335. DOI: 10.1177/0894439309353037.
-Rempel, H. G. and Bridges, L., (2013). That was then, this is now: Replacing the mobile-optimized site with responsive design. Information Technology and Libraries 32(4), 8-24.
-Seckler, M. Heinz, S., Bargas-Avila, J. A., Opwis, K., and Tuch, A. N. (2014). Designing Usable Web Forms – Empirical Evaluation of Web Form Improvement Guidelines DOI: 10.1145/2556288.2557265 Conference: In Proceedings of the 2014 annual conference on Human factors in computing systems (CHI’14). ACM, New York, NY, USA.
36