For a digital product

usability testing

World Usability day colombia 2014Kyle Soucy@kylesoucy

Image source: facit digital

What is Usability Testing?

How do you conduct a usability study?

Lets us begin our

Journey...

Plan StudyDefine goals and objectives

What do you want to learn?

How do teachers shop for books?

Do they use these slider dots?

Do they notice these arrows?

What You Can Learn From A Usability Study

Do users understand how to complete a desired task?

Does the design prevent users from completing tasks more effectively?

Do users notice the buttons or links that you want them to click on?

Are users engaged in the content and understand it?

How long does it take for a user to complete a task?

Does the interface cause users undo frustration or confusion?

What You Can’t Learn From A Usability Study

Opinions or other subjective data(We’re not testing hundreds or thousands of people)

The future(We can only report on what we directly observe)

Design preferences(You won’t get answers to design questions)

Why can’t I just use a focus

group?

Qualitative vs. Quantitative Research• Qualitative Research

- Data gathered directly by observing the user

- Researcher can ask follow-up questions, probe on behavior

- Analysis of data is not mathematical (Why? How? Compare...)

• Quantitative Research

- Data gathered indirectly through surveys, log files, support calls, etc.

- Data can be averaged or statistically testedSource: Christian Rohrer Move, Inc. | Realor.com

Understanding The Domain And The Product

If you don’t know the product, you can’t test it!

Plan StudyDefine what you are testing

Low vs. High-Fidelity Prototypes

Plan StudyTask design

1. Write a list of the most important tasks that users need to accomplish using the product.

2. Turn each task into a scenario.a) Provide some context.b) Supply needed information.c) Don’t give any clues or hints!

3. Decide on the order of tasks.a) Make the first task easier.

Bad Task: Buy a book about genealogy online.

Good Task: You need to get a book about genealogy that covers the reliability of different sources. You want this book in your hands by Thursday. Find and purchase this book.

What makes a good task:

• You’re asking users to do something that they would actually do in real life.

• Good scope. Not too broad, not too specific.

• Has a clear end point.

• Elicits action, not opinion.

Don’t start with “how would you” or “where would you”

Good Task: You need to get a book about genealogy that covers the reliability of different sources. You want this book in your hands by Thursday. Find and purchase this book. Find and purchase this book.

✓ Has a clear end point✓ Elicits action, not opinion✓ Good scope

Your Turn...

1. Take out a sheet of paper (or your computer, phone, or iPad)

2. Write down the most important task your users perform on your product.

3. Turn the task into a scenario.a) Provide some context.b) Supply needed information.c) Don’t give any clues or hints!

Other things to consider...

Restrictions: Consider whether or not you want to place restrictions on how participants try and accomplish the tasks (e.g. “don’t use search” or “stay on the website”).

Printing:If you’re conducting an in-person test, you may want to print out the scenarios so the participant can refer back to it.

Time-Fillers: Create more tasks than you think you have time to complete just incase someone finishes early. Every minute with a participant is an opportunity to learn something.

Visual Aids:If you’re worried that revealing the name of something will bias the results, consider showing the participant a picture instead.

Visual Aid Example:A friend of yours uses on of these to make baby food,

so you’d like to buy one...

Alternating Tasks/Prototypes:Depending on what you’re testing, you may need to alternate tasks. Participants are learning the interface as they go, so the last task may always be the easiest to accomplish unless you alternate them.

Interview-based vs. Predefined Tasks

Interview-based Tasks

1. Interview participant about how they use the product.

2. Use the answers given in the interview to create tasks on the fly.

When in doubt just ask, “what was the last thing you did when you used this product? Can you show me?”

Plan StudyDefine participants

Demographics

Cover all user groups- Prospective as well as

existing users

Be specific and get buy-in from stakeholders!

Recruiting Options

Do It Yourself

Hire a recruiter

Warning: It’s never this easy!

Live-intercept Study

How many to test?

Source: Jakob Nielsen

Budget AllocationWhat NOT to do...

1 Round of Testing

Budget AllocationWhat you SHOULD do...

Round 1 Round 2 Round 3

Design Iteration

Design Iteration

What to pay participants?

Plan Studydefine testing method

Vs.

Remote vs. In-person Testing

Remote Testing Pros & Cons

Participants stay in their native environment.

Accessible to a larger & more diverse pool of participants.

Recruiting is easier. Less no shows & easier last minute replacements.

Opportunity to gather ethnographic data.

Cost & time savings. No travel & lab rental fees.

Easier for observers to “attend” a session.

Risk of compromising data because the participant is using a foreign PC is removed.

Pros:Can’t see the user’s facial expressions.

Can’t see if the user is using peripheral devices.

Dependent on an Internet connection.

Can only recruit participants with a high-speed Internet connection.

Cons:

Remote Testing Tools

• GoToMeeting

• WebEx

• Adobe Connect

• Join.me

• Many more...

Remote Testing Tips• If using a cordless phone, have multiple handsets

charged and ready

• Use a phone recording controller

• Use as few tools as possible

• Back up, back up, back up!

• USB Modem

• Screen sharing backup

• Create a test meeting to make sure the participant can connect beforehand

• Instruct participants to log on early and use a landline (no VOIP)

• Turn off entry/exit chimes for conference call

• Allow only one observer to chat to you

• Use a separate chat/IM client

Unmoderated vs.

Moderated Testing

What is Unmoderated Remote Testing?

Unmoderated Testing Tools

(There are a lot!)

We Won’t Talk About...

We Will Talk About...

Has Anyone Here Used These Tools Before?

What You Can Learn(Depends on the tool and how you use it!)

Web Analytics

Image source: http://www.hiero.com/web-analytics.html

Task Completion Rates

Image source: Loop11

Number of Clicks/Clickstream

Image source: Userlytics

Time on Task/Page

Image source: http://blog.usabilla.com/underdogs-beat-expedia-in-usability-test/

Satisfaction Ratings/Opinion Rankings

Image source: http://wikidoc.org/index.php/Pain

It is simple to use:Strongly Disagree ---1---2---3---4---5---6---7 Strongly Agree

True Intent

Image source: http://ethnio.com/images/live_screener_example_page1.jpg

In-context Feedback

Image source: Loop11

“...I’m only offered a search by model number.

How ridiculous.”

“...I would like to see the products availability and delivery/

shipping information listed.”

How Actionable is the Data?

(Depends on the questions you ask!)

Conducting Unmoderated Remote

Usability Tests

Process

1. Define the study

2. Recruit participants

3. Launch test and send email invites

4. Analyze results

3. Run a Pilot test!>

4 /5 /

Benefits

Test hundreds of people simultaneously while keeping them in their own natural environment.

Test multiple websites simultaneously.

Reduced costs... depending on the tool you use.

Great way to plant the seed of UCD methodologies and introduce usability testing into a company.

UX

Fewer logistics to manage and no need to worry about no-shows and getting last-minute replacements.

Fast Results

Drawbacks

Nothing beats watching participants in real time and being able to ask probing questions about what they are doing as it’s happening.

Some participants may only be interested in earning the honorarium you’ve provided as an incentive.

$$$

You cannot conduct interview-based tasks.

It is simple to use:Strongly Disagree ---1---2---3---4---5---6---7 Strongly Agree

Image source: http://www.hiero.com/web-analytics.html

What participants report on surveys can be very different in comparison to what they actually do.

Image source: http://www.hiero.com/web-analytics.html

It’s possible for participants to think they’ve successfully completed a task when they haven’t.

Image source: http://www.hiero.com/web-analytics.html

Does it matter if it took someone longer to complete a task?

When to Conduct Automated Testing?

Not a replacement for moderated testing or research.

Best used to compliment your qualitative research.

Unmoderated Testing Tools

(A closer look...)

Automated Testing Tools with Recruiting Panels

The danger...

I wonder how a real customer would actually use

this website?

Panelist

UserTesting.com Video

BLOCKED

BLOCKED

The danger...

There are no shortcuts

to conducting good research.

Automated Testing Tools with Other Recruiting Options

Loop11

Treejack

Chalkmark

Chalkmark Results

Automated Tools with Audio and Video

Handout:

Overview of

Unmoderated

Usability

Testing

Tools

/usableinterface

Plan Studyschedule participants

Example Testing Schedule

Plan Studyinvite participants and observers

Key to Success…

Attendance of product decision makers during testing!

Example Remote Participant Invite

Set Up Study

Usability “Labs”

Image source: facit digital

Source: Kent State University

Vs.

Ideal “Lab-Less” Set Up

Should observers be in the room?

Video Recording Options

• Techsmith Morae - $1,995 USD

• Techsmith Camtasia - $99 USD

• Silverback (Mac only) - v2 Free

• Online Meeting Built-in Recorders

Example Video Consent Form

Example Moderator Guide

Moderate Study

Greet Participant and Give Introduction

Preliminary Interview

Evaluation Instruction

Think Aloud Techniques

Think Aloud Techniques

Concurrent (CTA) Retrospective (RTA)

Vs.

Variation of RTA...

Show the participant their video after they’re done testing.

Reminder Prompting

• So...?

• So, what are you thinking?

• What are you seeing here?

• What are you looking at?

• Can you tell me what you think is happening here?

Source: Moderating Usability Testing by Joe Dumas and Beth Loring

Probing Questions

• Is that what you expected?

• What would you do next?

• What did you think about that task?

• Help me to understand more about...

Source: Moderating Usability Testing by Joe Dumas and Beth Loring

Encouraging Statements

• This feedback is really helpful.

• Thank you for thinking out loud.

• That’s good to know.

• You’re doing fine.

Source: Moderating Usability Testing by Joe Dumas and Beth Loring

Remain Neutral

• When in doubt turn the question around:

• “I don’t know what do you think?”

• “What would you do if I wasn’t here?

• Keep acknowledgements limited to: “ah”, “Mm Hmm”, “Ok”, etc.

Source: Moderating Usability Testing by Joe Dumas and Beth Loring and Rocket Surgery Made Easy by Steve Krug

Letting Participants Struggle

Note-taking

Analyze & Report Findings

Formal Usability Testing Report

Findings categorized by

screens or pages

When, What, W

ho, Where, !

and Why Statement!

3-4 !

Positive Finding

s!

3-4 !

Negative Findings!

Major Usability Problem

Video Clip Example

Mobile Usability Testing

Image source: http://img.gawkerassets.com/img/181rzskxifcohjpg/original.jpg

“...the mobile web will be bigger than desktop Internet use by 2015.”

- Morgan Stanley Study

Source: http://mashable.com/2010/04/13/mobile-web-stats/Source: http://mashable.com/2010/04/13/mobile-web-stats/

Image source: http://3.bp.blogspot.com/-Oh6rZE2fG1k/Uct4T4KhE0I/AAAAAAAAARo/3c_HFDkMwAE/s1600/girls-on-their-phone.jpg

Mobile matters!

Mobile In-person Usability Testing

Image source: http://unmatchedstyle.com/news/usability-testing-on-mobile-devices.php

Image source: http://www.90percentofeverything.com/2010/05/07/quick-tip-make-your-own-iphone-usability-testing-sled-for-5/

Testing on paper prototypesImage source: http://www.90percentofeverything.com/2010/05/07/quick-tip-make-your-own-iphone-usability-testing-sled-for-5/

What about remote mobile usability testing?

Current Navigation Tabbed Navigation Hidden Navigation

What about remote mobile usability testing?

unmoderated>

Other Mobile Usability TestingTools

Image source: http://blog.larrybodine.com/uploads/image/a-99-year-old-lady-using-the-ipad.jpg

Fitting Research Into An Agile Process

Waterfall

Analysis Design Coding Testing

Analysis

Design Coding

Testing

Agile

Analysis

Design Coding

Testing

Analysis

Design Coding

Testing

1-4 week sprints

Parallel Sprints

#1

Sprint 1Dev:

UX:

Sprint 2 Sprint 3

Release

Sprint 1 Sprint 2 Sprint 3

Staggered Sprints

#2

Source: http://www.upassoc.org/upa_publications/jus/2007may/agile-ucd.pdf

Tip: Choose a feature to develop first that has a lengthy dev sprint.

Testing On A Small Budget

• Conduct your own recruiting

• Don’t offer large honorariums

• Don’t use a lab

• Don’t video record

• Test less people

• Use surrogate users or just conduct cognitive walkthroughs

• Conduct unmoderated studies

• Don’t write formal reports

Learn more...

• Book: Interviewing Users by Steve Portigal

• Book: Moderating Usability Tests by Joe Dumas & Beth Loring

• Book: Paper Prototyping (chapter on task design)by Caroline Snyder

• Book: Rocket Surgery Made Easy by Steve Krug

• Book: The Moderator’s Survival Guide by Donna Tedesco & Fiona Tranquada

• Article: Seven Common Usability Testing Mistakesby Jared Spoolhttp://www.uie.com/articles/usability_testing_mistakes/

• Article: Interview-Based Tasks: Learning from Leonardo DiCaprioby Jared Spoolhttp://www.uie.com/articles/interview_based_tasks/

• Article: Mobile Testing Toolbox: Part 1&2by Amber Derosahttp://uxmag.com/articles/mobile-testing-toolbox-part-1

• Article: How To Conduct A Usability Test On A Mobile Device by Jeff Saurohttp://www.measuringusability.com/blog/mobile-usability-test.php

• Article: Why And How To Segment Your Customers by Jeff Saurohttp://www.measuringu.com/blog/segment-customers.php

www.usableinterface.com

@kylesoucy

Thank you.

/usableinterface