web survey and forms usability design & testing

Web Survey and Forms Usability

Design & Testing

Jennifer Romano Bergstrom

August 13, 2014

AAPOR webinar

@romanocog @AAPOR

2

@romanocog @AAPOR

What we design for…and reality

Krug, S. (2000) Don’t Make Me Think. Pearson Education *NEW 3rd Edition (2014)*

3

Placement of Instructions

4 http://www.pewinternet.org/three-technology-revolutions/

http://www.pewinternet.org/fact-sheets/mobile-technology-fact-sheet/

@romanocog @AAPOR

Technology Revolutions

% o

f A

me

rica

n a

du

lts

90% of American adults have a cell phone

58% have a smartphone

42% have an tablet

32% have an e-reader

5

@romanocog @AAPOR

The % of cell phone owners who use their cell phone to…

http://www.pewinternet.org/2013/09/19/cell-phone-activities-2013/

6

@romanocog @AAPOR

Nielsen: The Cross-Platform Report, Quarter 2, 2012-US

7

@romanocog @AAPOR

Users read what they need to read

He, Siu, Strohl, & Chaparro (2014). Mobile. In Romano Bergstrom & Schall (Eds). Eye Tracking in User Experience Design. Morgan

Kaufmann.

Usability = “the extent

to which a product can

be used by specified

users to achieve

specified goals with

effectiveness,

efficiency, and

satisfaction in a

specified context of

use.” ISO 9241-11

@romanocog @AAPOR

+ emotions and

perceptions = UX

9 User Experience Design (P. Morville): http://semanticstudios.com/publications/semantics/000029.php

@romanocog @AAPOR

User Experience

10

@romanocog @AAPOR

When to test

11 https://www.youtube.com/watch?v=GrV2SZuRPv0

Low-Fi Paper Prototypes @romanocog @AAPOR

12 https://www.youtube.com/watch?v=GrV2SZuRPv0

Low-Fi Paper Prototypes @romanocog @AAPOR

13 https://uxmag.com/articles/eight-lessons-in-mobile-usability-testing

@romanocog @AAPOR

Mobile Low-Fi Paper Prototypes

14

SUBJECTIVE

+ Satisfaction & difficulty ratings

+ Verbal responses

+ Moderator follow up

+ Real-time +/- dial

@romanocog @AAPOR

UX Data

15

OBSERVATIONAL + Time on page/task

+ Reaction time

+ Selection/click behavior

+ Success/fail rate

+ Conversion rate

SUBJECTIVE

+ Satisfaction & difficulty ratings

+ Verbal responses

+ Moderator follow up

+ Real-time +/- dial

@romanocog @AAPOR

UX Data

16

OBSERVATIONAL + Time on page/task

+ Reaction time

+ Selection/click behavior

+ Success/fail rate

+ Conversion rate

IMPLICIT + Eye tracking

+ Electrodermal activity (EDA)

+ Behavioral analysis

+ Verbalization analysis

+ Implicit associations

+ Pupil dilation

+ Facial expression analysis

SUBJECTIVE

+ Satisfaction & difficulty ratings

+ Verbal responses

+ Moderator follow up

+ Real-time +/- dial

@romanocog @AAPOR

UX Data

@romanocog @AAPOR

Why should we measure implicit?

Traditional UX research is

good at explaining what

people say and do, not

what they think and feel.

@romanocog @AAPOR

Why should we measure implicit?

Explicit data

19

*Satisfaction Questionnaire: Please rate how difficult it was to log in on this device. 1= not difficult at all to 5= extremely difficult.

“Love the picture in the middle of it.”

“It looks very clean and very simple.”

“It looks pretty organized, it's a nice design.”

When asked how they would save information,

four of six participants said they would

bookmark the page or take a screenshot of

the information. Only two mentioned that

they would use the site functionality to save

for later use.

83%

9%

9%

Percentage of Difficulty Ratings*

1 & 2

3

>=4

@romanocog @AAPOR Subjective Data

0

3

6

9

Participant Ratings

Likelihood to Recommend

Not likely at all or

Slightly likely

Moderately Likely

Very Likely

0

3

6

9

Participant Ratings

Likelihood to Use

Not likely at all or

Slightly likely

Moderately Likely

Very Likely

*Satisfaction Questionnaire: How likely would you be to use this site in the future?

How likely would you be to recommend this site to a friend?

Observational data

20

Details page about Military

Physician Assistant career, only

accessible through Military section.

Details page about Civilian

Physician Assistant career.

Six participants clicked on Salary &

Wages when looking for states with the

most jobs before looking under Job

Growth.

• Five participants thought they had

completed the task once they found the

Salary & Wages map on the wrong page.

0

5

10

15

20

Time on Page

Se

co

nd

s

Average time on landing page

@romanocog @AAPOR Observational Data

First click data

21

0

20

40

60

80

100

120

Page 1 Page 2 Page 3

Tim

e (

se

co

nd

s)

Time spent on each page of the instructions before working on form.

Aggregate fixation count heat map across all participants, Page 1.

Participants looked at ‘Purpose of Form’ section the most often.

Implicit Data @romanocog @AAPOR

Romano Bergstrom, J. C. & Strohl, J. (2014). Improving government websites and surveys with usability testing: A comparison of

methodologies. Proceedings from the Federal Committee on Statistical Methodology (FCSM) Conference, Nov 2013,

Washington, DC.

22 22

Trouble with scrolling.

22

@romanocog @AAPOR

Implicit Data

23

Modern eye tracking

Modern Eye Tracking

Co

pyr

igh

t ©

2013

Th

e N

iels

en C

om

pan

y. C

on

fid

enti

al a

nd

pro

pri

etar

y.

24

Example Methodology Participants:

• N = 74 | Average Age = 37

• Mix of gender, ethnicity, income

• Random assignment to diary condition

• New, Old, Prototype, Bilingual

Usability Testing session:

• Participants read a description of the

study.

• The moderator gave instructions and

calibrated the eye tracker.

• Participants completed Steps 1-5 in the

diary at their own pace.

• End-of-session satisfaction questionnaire

• Debriefing interview

Eye Tracker

Moderators worked from another room.

Control Room

Slide from: Walton, L., Romano Bergstrom, J., Hawkins, D. & Pierce, C. (2014). User

Experience and Eye-Tracking Study: Paper Diary Design Decisions. Paper presentation at

the American Association for Public Opinion Research (AAPOR) Conference, Anaheim, CA,

May 2014.

Jarrett, C., & Gaffney, G. (2009) Forms that Work: Designing Web Forms for Usability. Morgan Kaufmann

@romanocog #AAPOR

25

Poor forms lead to: • Increased errors

• Lower response rate

• Higher nonresponse

• Loss of time

• Increased frustrations

• Poor user experiences.

Jarrett, C., & Gaffney, G. (2009) Forms that Work: Designing Web Forms for Usability. Morgan Kaufmann

Good forms = Good UX

@romanocog #AAPOR

26

@romanocog @AAPOR

What can we learn from

eye tracking forms and

surveys?

28

@romanocog @AAPOR

Jarrett & Romano Bergstrom (2014). Forms and Surveys. In Romano Bergstrom & Schall (Eds). Eye Tracking in User Experience

Design. Morgan Kaufmann.

People read pages

with questions on

them differently

than other pages.

The F-shaped eye-tracking pattern of the block of text at the top of the

page is completely different from the eye-tracking pattern on the question

and answer spaces at the bottom of the page.

29

@romanocog @AAPOR

Romano Bergstrom, J. C. & Strohl, J. (2014). Improving government websites and surveys with usability testing: A comparison of

methodologies. Proceedings from the Federal Committee on Statistical Methodology (FCSM) Conference, Nov 2013,

Washington, DC.

Fixation count gaze plot shows the

participant looks back and forth

between the two input options

People

don’t read

instructions.

30

Users focus on the questions.

31

@romanocog @AAPOR

Romano, J. C. & Chen, J. M. (2011). A usability and eye-tracking evaluation of four versions of the online National Survey for

College Graduates (NSCG): Iteration 2. Statistical Research Division (Study Series SSM2011-01). US Census Bureau.

Available online at <http://www.census.gov/srd/papers/pdf/ssm2011-01.pdf>.

Users focus on the

questions.

Participants did not read the text on the log-in

page, but they read the text on the very next page.

32

@romanocog @AAPOR

Error messages must

be useful.

33

@romanocog @AAPOR

Error messages must

be useful.

He, Siu, Strohl, & Chaparro (2014). Mobile. In Romano Bergstrom & Schall (Eds). Eye Tracking in User Experience Design. Morgan

Kaufmann.

34

@romanocog @AAPOR

Error messages must

be useful.

• “How do I advance to the next screen?”

• “It seems like it's stuck on the screen.”

Gaze Plot: After getting an error message, the

participant had to search all over the screen to

find the missing field.

He, Siu, Strohl, & Chaparro (2014). Mobile. In Romano Bergstrom & Schall (Eds). Eye Tracking in User Experience Design. Morgan

Kaufmann.

35

@romanocog @AAPOR

Users do not pay

attention to

multiple error

messages.

Romano, J. C. & Chen, J. M. (2011). A usability and eye-tracking evaluation of four versions of the online National Survey for

College Graduates (NSCG): Iteration 2. Statistical Research Division (Study Series SSM2011-01). US Census Bureau.

Available online at <http://www.census.gov/srd/papers/pdf/ssm2011-01.pdf>.

36

@romanocog @AAPOR

Navigation must be intuitive.

Bristol, K., Romano Bergstrom, J. & Link, M. (2014). Eye Tracking the User Experience of a Smartphone and Web Data Collection

Tool. Paper presentation at the AAPOR Conference, Anaheim, CA, May 2014

Intuitive ‘Next’ button location

• “Where and how you click is a bit counter-intuitive. [It’s] not super

obvious which button to click to get to next sections.”

• “I feel like the ‘Next’ should be at the bottom and not the top.”

Non-intuitive ‘Next’ button location

37

@romanocog @AAPOR

It must be clear

how to get

started.

Users must click “Add a School” to start. The “Add a school”

button does not look like an actionable button. It is not clear that

this is how to start. The form did not look like it was clickable.

Users were unsure how to proceed once they had successfully

brought up two schools side-by-side.

Romano Bergstrom, J. C. & Strohl, J. (2014). Improving government websites and surveys with usability testing: A comparison of

methodologies. Proceedings from the Federal Committee on Statistical Methodology (FCSM) Conference, Nov 2013,

Washington, DC.

@romanocog @AAPOR

Eye tracking is really cool!

Should I always include it

in my research?

@romanocog @AAPOR

Eye tracking is really cool!

Should I always include it

in my research?

@romanocog @AAPOR

Eye tracking is really cool!

Probably not

41

@romanocog @AAPOR

When NOT to track

Jarrett & Romano Bergstrom (2014). Forms and Surveys. In Romano Bergstrom &

Schall (Eds). Eye Tracking in User Experience Design. Morgan Kaufmann.

42

@romanocog @AAPOR

Slot-In answers

Jarrett & Romano Bergstrom (2014). Forms and Surveys. In Romano Bergstrom &

Schall (Eds). Eye Tracking in User Experience Design. Morgan Kaufmann.

43

@romanocog @AAPOR

Gathered answers

Jarrett & Romano Bergstrom (2014). Forms and Surveys. In Romano Bergstrom &

Schall (Eds). Eye Tracking in User Experience Design. Morgan Kaufmann.

44

@romanocog @AAPOR

Created answers

Jarrett & Romano Bergstrom (2014). Forms and Surveys. In Romano Bergstrom &

Schall (Eds). Eye Tracking in User Experience Design. Morgan Kaufmann.

45

@romanocog @AAPOR

Third-party answers

Jarrett & Romano Bergstrom (2014). Forms and Surveys. In Romano Bergstrom &

Schall (Eds). Eye Tracking in User Experience Design. Morgan Kaufmann.

46

Jarrett & Romano Bergstrom (2014). Forms and Surveys. In Romano Bergstrom & Schall (Eds). Eye Tracking in User Experience

Design. Morgan Kaufmann.

@romanocog @AAPOR

See Bristol, K., Romano Bergstrom, J. & Link, M. (2014). Eye Tracking the User Experience of a Smartphone and Web Data

Collection Tool. Paper presentation at the AAPOR Conference, Anaheim, CA, May 2014 for more about the UX across

devices.

• Not necessarily the same UX across all devices

• Different issues occur with different modes

• Match user expectations and survey objectives

• Conduct UX testing with real users

• Use eye tracking to understand the UX…cautiously.

@romanocog @AAPOR

47

Assess the UX across devices Assess the UX across devices.

Thank you!

• Twitter: @forsmarshgroup

• LinkedIn: http://www.linkedin.com/company/fors-marsh-group

• Blog: http://www.forsmarshgroup.com/all/

Jennifer Romano Bergstrom

@romanocog

jbergstrom@forsmarshgroup.com

AAPOR Webinar | August 13, 2014