conducting a usability test - university of michigan · 2005-05-15 · test, then give them tasks...

Jason Withrow ([email protected])

Conducting a Usability Test


Overview• Introducing Usability Testing• The Usability Testing Continuum• Test Designs & Protocols• Recruiting Users • Preparing for the Usability Test• Conducting the Usability Test• Data Analysis & Issues Prioritization


Introducing Usability Testing• First, user testing = usability testing

– User testing is the older term, from the field of HCI (Human-Computer Interaction)

• The best-known user experience method• Brought to the attention of business and the

larger world largely due to Jakob Nielsen• Fundamentally a simple process:

– Have something you can test and someone to test, then give them tasks to complete


‘Discount’ ‘Full Cost’

3-5 users

Done in-person

Notes taken by hand

Focus on qualitative data

Used for the majority of website testing

25+ users

Video, audio, eye-tracking

Can be done remotely

Focus on quantitative data and statistical analysis

Formal usability lab used

The Usability Testing Continuum


Our Focus: Discount Testing• Why only 3-5 people?

– After the first couple of users you will begin to see common patterns in terms of errors, reactions to labeling, etc. You can soon predict what will be problematic for the next user.

– There is a case of diminishing returns for additional users. They identify a new issue here and there and all the same core issues; is the expense worth two new issues?

– Testing fewer users = more rounds of testing; you don’t spend the entire budget at once


Testing a New Website• You’re building a new website. When and

how is testing conducted?– Start by testing competing websites. What did

they do right? What can be improved?– When wireframes and prototypes are built,

those can be tested, revised, then tested again– As the site nears completion, testing is

conducted in a single round.* As time permits fixes are made and testing is done again.

* For most projects this is the only usability testing done.


Testing During a Redesign• Referred to as a pre-test/post-test design.

When and how is testing conducted?– Pre-Test: Start by testing the existing website.

Try to get some hard numbers on user satisfaction, task completion rates, etc.

– When wireframes and prototypes are built, those can be tested, revised, then tested again

– Post-Test: As the site nears completion, testing is conducted using the same tasks from the pre-test. Do the numbers improve?


Test Design: Parallel Comparison• Appropriate for both new websites and

redesigns.• Test multiple site versions and compare

results, then make changes and test again– The ‘parallel’ aspect comes from the various

approaches being developed concurrently• Could be alternate site architectures,

alternate layouts, etc.


Issues with Parallel Comparison• Within-subjects or between-subjects?• Within-subjects means that each user tests

every variation– Issues include fatigue, order effects, and

time/compensation increases– Benefit is that user can compare/contrast

• Between-subjects means that a user only sees one variation– Issues include the number of users required

and the inability of users to compare options


Test Protocols• In usability testing there are two main

protocols (overall approaches to conducting the test) that can be followed:– Think Aloud– Question & Answer

• Choice of protocol can have a significant impact on how a testing session unfolds and the data received


The ‘Think Aloud’ Protocol• The user verbalizes what they are thinking

as they use the website. This is our only way to ‘get inside their head’ during the test.

• Thinking aloud is not a normal activity (for most of us) so the user will often fall silent– It is fine to gently remind them to think aloud,

perhaps asking ‘What are you thinking now?’


The ‘Question & Answer’ Protocol• The facilitator has a list of questions they

ask at specific points during the task, plus they feel free to ask other questions as the situation arises

• Some concerns:– Asking a question about the ‘Site Map’ might

bring that to the user’s attention when otherwise they never would have seen it.

– Questions disrupt the user’s flow/the testing flow and can increase user anxiety


Recommended Protocol• ‘Think Aloud’ during the test with ‘Question

& Answer’ when the task is completed• Don’t assume that you know the problem

already. For example, you could ask:– “I noticed that you stopped for a second before

clicking on the ‘Products’ button. Could you tell me more about that?”rather than:“What was the issue with the ‘Products’ button?”


Recruiting Users• For most websites you can test anyone• For specialized websites targeting a certain

audience (especially if that audience uses specialized language) you would want to be more selective

• In cases where a website has multiple distinct audiences getting users from all audiences would be necessary– Usually 3-5 users from each audience


Recruiting Avenues• Ads in the local newspaper• Flyers• Social network• Recruitment agencies

– Issue of cost– Also concerns about ‘expert users’


Reimbursement• Rates depend on the size of the company

doing the testing as well as the area of the country– Ann Arbor would be $30-$40/hr for the user– California would be $70-$100/hr for the user

• Larger firms (such as IBM) would also pay more, starting at $100 and going up from there


Preparing for the Usability Test• Informed Consent• Demographics Survey• Usability Testing Script• Usability Testing Tasks• Usability Testing Response Sheets• Overall Website Evaluation Form


Informed Consent• Although not a scientific study, we are

observing human behavior so we want their signed consent

• Keep in mind that all data is confidential and should be coded for anonymity

• A sample informed consent document is linked from the workshop materials section


Demographics Survey• Demographics can be included in the

usability testing report, especially if the client requested that specific demographics be met in the user selection/screening

• While not statistically meaningful, you could examine how their self-reported ‘general computer ability’ and ‘web browsing ability’ corresponded to their task performance

• You also want a mix of ability levels


Usability Testing Script• Again, we’re not going for scientific rigor but

we do want all the users to get a standard introduction to the process

• If nothing else, having the script ensures that we don’t miss any important details– For example, you don’t want to be half-way

through the first task and then say “Oh yeah, you should be thinking aloud during this”

• Stress that we’re testing the website, not the person; any difficulties are good to know


Usability Testing Tasks• There are 3 main types of tasks:

– Closed-ended (e.g., locate a known item)– Open-ended (e.g., locate a desired item)– Impressions (e.g., give me your impressions of

the home page)• A usability test can include all 3 types• Ideally you want to select high-priority,

frequent tasks– Consider the 80/20 rule here


For Static Wireframes/Mockups• Focus on ‘impressions’ questions

– What do you see first? Second? – What do each of the links go to?– What would you do if you wanted to…?– What can you do at this website?– How would you rate this website for:

• Ease of Use• Attractiveness• Usefulness• Credibility


Phrasing for Tasks• Brevity is a virtue here

– You read the task aloud to the user and if it is a paragraph long and in multiple parts, they will miss half of it

• Keep these short (1-2 sentences)– The first sentence ‘sets the stage’– The second sentence is the task

• On each response sheet I have the task printed out so I show that to the user too


Number of Tasks & Pilot Testing• The time limit for a usability test is one hour

– Beyond that fatigue starts to impact results• Usually you can fit 10 tasks into an hour;

sometimes you have very short tasks like logging out so you have more than 10

• Pilot test (test a colleague) to:– Determine how many tasks fit into an hour– Catch any wording issues in the tasks

• Don’t make the first user your pilot test


Variability Among Users• Some users will move very quickly, finishing

all the tasks in 30 or 40 minutes– This is fine, just make sure that you are able to

take good notes at the speed they move. It is fine to ask them to retrace their steps as they finished the task.

• Other users will move quite slowly– Generally stop after 75 minutes, thank them for

their participation, and move on to the next user– I once tested an elderly woman who completed

4 tasks in 75 minutes


Handling Transactional Tasks• If you are usability testing a client’s e-

commerce website:– Use a client-provided credit card number so the

user can complete the purchase; coordinate with their billing staff to cancel those orders

– Registration accounts can also be deleted later• When testing competitor sites:

– Don’t complete the purchase or use an approved credit card and charge small amounts

– Registration can be completed with fake info


Usability Testing Response Sheets• Completed after each task• Just like ‘thinking aloud’ you are getting the

thoughts of the user• Since this is after the task the user can also

reflect on the experience and sometimes arrive at new suggestions

• Provides a source of ‘direct quotations’ that can be used in later decision-making


Drawbacks of Response Sheets• The major drawback of the response sheets

are breaks in the testing flow– For some testing I omit these response sheets

because of the disruption. You need to balance this negative against the value of the data gained.

– My example usability testing data is from sessions where I did not use the response sheets


Overall Website Evaluation• Questionnaire completed at the end of the

testing session• The rating scales can be used to arrive at

averages for the various dimensions• Low rankings are certainly areas for

concern and further analysis, but probably are not entirely surprising


Conducting the Usability Test• The Testing Environment• Testing Equipment• What to Look For When Testing• Starting at the Home Page• Recording the Data• Timing the Tasks• Observers


The Testing Environment• The primary consideration is that it is quiet

and free from disturbances– You don’t want to disrupt the flow of the testing

or the user’s process– Noise and interruptions can also increase the

anxiety level of the user– They also disrupt your concentration and you

need to be focused on what the user is saying and doing and be recording that information


Do You Need A Usability Lab?• The University of Michigan Duderstadt Center

(Media Union) has a usability lab with: – PCs and Macs; various browsers– Hardware to record audio/video to a tape– Camtasia (software for capturing A/V for later

burning to disc)– Set up to record user’s expression in a small

picture-in-picture window at lower right of screen• Tec Ed has a usability lab that can be rented• Some companies sell ‘portable labs’


Testing Equipment• All you really need is a computer with an

Internet connection– Preferably a dial-up connection to simulate how

most users experience the website– If the user has dial-up at home and you have a

broadband connection that will positively impacttheir perceptions of the website because they aren’t used to sites loading so fast

• Usually a PC, unless the site is for Mac users (in which case have a Mac)


What to Look For When Testing• User confusion, uncertainty, frustration• The path the user took through the website;

was it roundabout or direct?• How the user interprets labels/links• Don’t focus solely on what doesn’t work,

consider the strengths also– Failing to note the strengths means that there is

a risk of them being thrown out as changes are made to the website


Starting at the Home Page• At the end of each task I ask the user to

return to the home page• This is done to establish a common starting

point for each task• The home page is also a typical user entry

point, so the path from there is important• This also reveals problems getting back to

the Home page from sub-pages


Recording the Data: Audio/Video• Pros:

– “Seeing is believing” (helps to convince decision-makers)

– Reduces the need for detailed note-taking– Perfectly preserves all the details (non-verbals,

exact quotations)


Recording the Data: Audio/Video• Cons:

– Tapes may not be looked at after the session– No one wants to transcribe the data (from

experience I can say with complete conviction that it is a miserable process)

– Technical difficulties; settings may be wrong so audio is quiet, is not recorded, etc.

– Increases user anxiety


Recording the Data: By Hand• Pros:

– No requirement for extra equipment; just a pencil/pen and a pad of paper

– No extra cost incurred

• Cons:– Can be hard to keep up with the user– Even trying your best, some details are missed– Data needs to be typed up later, but that is still

better than transcribing a tape!


Strategies for Hand-Recording Data• Develop your own shorthand for noting user

behaviors (e.g., scrolling, hovering) as well as page names

• For each task you could also have a print-out of the site map (from the website) or the site outline (the deliverable). Put numbers next to page names to note the path taken.– The issue here is inserting quotations and

behaviors in that clicking sequence; where do those go?


Timing the Tasks• Some people prefer to time tasks as a

measurement of site quality/performance• This introduces various issues:

– When does the task start? When you finish reading it aloud? When the user touches the mouse? What if they have a clarifying question at the start of the task?

– When does it end? You can request that they say “I’m done” but often they forget to say that.

– Juggling a stopwatch while taking notes becomes a major physical and mental drain.


Observers• Usability testing offers an excellent

opportunity to introduce people to UCD and to influence their decisions about the website

• An executive may be strongly committed to a design with major usability problems– After observing the first user having problems,

he/she declares that person to be an anomaly– After observing the second and third users

having problems, they start agreeing that problems exist in the design


Accommodating Observers• If in a usability lab:

– They can be in an adjoining room – The room has a one-way mirror or perhaps a

video feed from the computer monitor, so they can watch the session as it happens

• Or they can be in the same room, sitting back a ways from the user– Note that this can cause the user extra anxiety;

try to limit this to one observer


Quantitative Data Analysis• Task Completion Rates / Success Rates• Time on Task• Path Analysis / Error Rates• Clicks to Destination• Individual Task Scores• Overall Site Evaluation


Prioritizing Problems • Problems can be grouped into one of three

severity levels (high, medium, or low)• High Priority: Important user experience

problems that lots of users will encounter (e.g., confusing placement and labeling of the shopping cart)– Need to be resolved as soon as possible– Ideally these are easy to change but

sometimes the changes are expensive (and extensive); the changes still need to be made


Medium & Low Priority Issues• Medium Priority: Not quite as crucial as high

priority issues, but still likely to impact a number of users.– Fixes to these can wait, but they should be

addressed when possible.• Low Priority: Smallest negative impact on

the user experience; may only be seen by a relatively small subset of users.– Fixes are usually easy (e.g., adding a mailto:)

conducting a usability test - university of michigan · 2005-05-15 · test, then give them tasks...

Documents