Week 3

Judy Kay

CHAI: Computer human adapted interaction research group

School of Information Technologies

1. Reflections on work so far

Core + advanced readings

Progress on e-text

• Week 3:– Think-alouds as start of Assignment 1

• Week 2:– Lab consolidation + create affinity diagram

• Week 1: – Studying use of e-textbook, based on auto-

ethnography + observation of other user– System concept statement for e-textbook

2. Reading discussion

Questions to consider

• Form groups – of ~5 all members – within tute class and degree level, – and then split into 2 subgroups

• Compare your concept maps within groups of 2-3– Important common elements– Important differences– Annotate each map * for common !! for diferences

• Then repeat, sharing with other subgroup• Be ready to share this in whole class activity

Think-alouds

The usability tool

"Thinking aloud may be the single most valuable usability engineering method." I wrote this in my 1993 book, Usability Engineering, and I stand by this assessment today. The fact that the same method has remained

#1 for 19 years is a good indication of the longevity of usability methods.

Thinking Aloud: The #1 Usability Toolby Jakob Nielsen on January 16, 2012

When to Use Which User-Experience Research Methodsby Christian Rohrer on October 12, 2014

http://www.nngroup.com/articles/which-ux-research-methods/

Demonstrating think-aloud

Demonstrate Thinking Aloud by Showing Users a Video

by Jakob Nielsen on September 1, 2014 Topics: User Testing

Services

• Userzoom• “can run unmoderated task-based studies with

geographically dispersed participants over any web-based interface (website, prototype, mock-up). Participants take the study simultaneously, in their natural context, using their own PC or device.”

• Users think• User testing

– Let’s watch their video, noting the tasks and critiquing them (just one in class)

Facilitating think aloudmakes you – experimenter -- really valuable

• What are you thinking now?• What do you think that message means?

(only after the user has noticed the message and is clearly spending time on it)

• don't help user except withHow do you think you can do it?• if user appears surprised, Is that what you expected to happen?

So now, onto Assignment 1

First things first

• Do we have the tasks right?• Abstract tasks• Concrete instances of them

– Concrete?– Relevant? – Not lead the user?– Minimalist?– Good coverage?

More preparation

For all user studiesLater in the semester, we will revise these for Assignment 2 and a different interface

Recruiting users

• How representative are they?– similarity to intended user population– Age– Gender– experience in area– interest/motivation– computer literacy

• What effect does user population have for conclusions?

How many users?

• Why You Only Need to Test with 5 Users• by Jakob Nielsen on March 19, 2000 • http://www.nngroup.com/articles/why-you-

only-need-to-test-with-5-users/

Insights: 0 users? 1 user? 2 users?

http://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/

What does this really mean?

• If first test shows catastrophic problems, should you still do 5?

• Never bother with more than 5?• 80% is good enough?

NO! NO! NO! NO!

It is all about budgets, user groups….

• If your interface is to be used by very different groups of people, you need to do the think-aloud with each group eg.– Children– Elderly– Different cultures, languages……

• Iterate!!!!! – Use same budget of effort to test each iteraction

Other views

• Spool, J., & Schroeder, W. (2001, March). Testing web sites: Five users is nowhere near enough. In CHI'01 extended abstracts on Human factors in computing systems (pp. 285-286). ACM.

• Lewis, J. R. (2014). Usability: Lessons Learned… and Yet to Be Learned. International Journal of Human-Computer Interaction, 30(9), 663-684.”

“The consistent finding … has been that observers, individually or in teams, who evaluated the same product discovered very different sets of usability problems.”

• “In the early 1990s … fairly large-sample formative usability studies … determined that the … first five participants observed in a formative usability study should usually reveal about 85% of the problems available for discovery in that iteration, where the properties of the study (type of participants and tasks employed) place limits on what is discoverable. But over time, in the minds of many usability practitioners, the rule became simplified to “All you need to do is watch five people to find 85% of a product’s usability problems.”

Summative evaluation

• As in your Assignment 1• How many participants?• Number of team members * 4

– Why?• User populations?

– Within community interested in UX– Those who are not– Ages, broad range

Stages of running an evaluation

1. Preparation

2. Introduction

3. The test

4. Questionnaire/interview

5. Debriefing

6. Analysis, reflection, summarising, reporting, conclusions for action

Steps 1- 5 done for each user test, as run

Step 6 is applied mainly after several users

Preparation

• Materials for consent• Set up machine, room, environment• Check all of them• Check user instructions• Do a mental run-through

Be sure not to waste user's time because of your lack of preparation!

Introduction

• Welcome user, explain purpose of test– make clear system tested not user– confidentiality– anonymity of reporting– opt out at any time– what is recorded

• Invite any other questions to here– explain procedure– if appropriate, do demo– invite questions

The test

• User works through experiment....– recording– ensure user feels supported– show pleasure at problems identified– critical to help user if stuck

• Questionnaire/interview– open and closed

What data should you collect?

• Observe – direct/indirect– take notes– video/audio/software monitor– software logs for timing

• Questionnaire: – Open

– Best things about <this interface> – If you could change one thing about <this interface>,

what would it be?

– Closed (later in the semester only)

Debriefing

• Thank user• Remind them of usefulness of results• Pause to make sure all data collected• All notes written• May ask user to confirm details

collected

Pitfalls

• Defining the right concrete tasks– Test all key aspects– Multiple tasks for same aspects

• Instructions to the users– Do NOT lead the user– Take particular care not to use words that

are identical to terms on the interface

Benefits of think aloud

• “show what users are doing and why they are doing it while they are doing it in order to avoid later rationalisations”

(Nielsen, Usability Engineering, Academic press 1993, p195)

• Cheap• Slows users down

– studies show users may work faster with fewer errors due to care on critical elements

Problems of think aloud

• Unnatural context and situation (do you talkto yourself?)• People filter, want to please, do not want to look foolish or inept• Hawthorn effect• Experimenter can bias results

• Directly eg via task choice

• Inadvertently eg gasp, brief frown

• Not directly quantitative• Add cognitive load to users• User's “theories” must be interpreted with care• Slows users down• Users are aware they are being observed so behave accordingly

Naturalised think-aloud

• Multi-user interaction– Two (or more) users work on task– Conversation is natural– Observer collects dialogue

Summary

• Top method for formative evaluations• Relatively inexpensive• Can identify major flaws• And may indicate causes of user problems• May give access to user's mental model• Alters activity => meaningfulness• This is a major part of your assignment

Homework

• Get started on Assignment 1• Homework is to work with your group to

get started on the assignment• Define the tasks

• Split the work Think-Alouds on e-textbook evaluation