© simeon keates 2008 usability with project lecture 12 – 20/03/09 susanne frennert

© Simeon Keates 2008

Usability with ProjectLecture 12 – 20/03/09Susanne Frennert


ISO 9241 definition:

“The effectiveness, efficiency and satisfaction with which specified users achieve specified goals in particular environments”

Effectiveness: can people complete their tasks• Task completion: How much got done? Quality of output: Was what got done any good?

Efficiency: how long do people take Deviations from the critical (i.e. shortest) path: How much did the user wander? Error rate: How often did the user make a mistake? How “bad” were the mistakes?

• Time on task: How long did it take?

• Mental workload: How hard did the user have to think?

Satisfaction: the level of comfort that the user feels when using a product and how acceptable the product is to users as a vehicle for achieving their goals

• Qualitative attitude analysis: Did you like it?

• Quantitative attitude analysis: How much did you like it? (e.g. benchmarking)

Feedback from Wednesdays exercise


Feedback from Wednesdays exerciseTypical quantifiable usability measurements:

• The time users take to complete a specific task

• The number of tasks of various kind that can be completed within given time limit

• The number of website features the user can remember during a debriefing after the test

• The proportion of user statement during the test that were positive versus critical towards the website

• The proportion of users who say the would prefer using the improved website over the frozen website

• The number of times the user sidetracked from focusing on the real task



Example of Hypothesis:

The user will find specific products faster at the revised site compared to the “frozen site” (the revised site offer faster task completion times than the “frozen” site”)

or

Task completion time are different between the “frozen” and the revised site

The revised site is more accessible for users who use screen readers

The user satisfaction will differ between the “frozen” and revised site

The revised site is more visually appealing



Effectiveness can be measured by recording whether or not a user could complete a task

•Percentage of tasks completed successfully on first attempt

•Per cent of users able to successfully complete the task

•Number of errors made performing a specific tasks (Errors have to be defined! For example errors is defined as an attempt to click on images, icons and buttons that would not result in completing the task assigned)

•Number of requests for assistance accomplishing tasks



Satisfaction refers to how the user feels about the system. Could be measured by:

•Ratio of positive to negative adjectives used to describe the product

•Per cent of users that the rate the revised site as “more satisfying” than the “frozen” site

•Per cent of customers who feel “in control” of the product



Efficiency can be measured by:

•Time to execute a particular set of instructions

•Time taken on first attempt

•Time to perform a particular task

•Number of key presses taken to achieve task

© Simeon Keates 2008Page 8

User Trials Schedule

28 October 13-15 Pilot study 30 October 13-15 Revision of site and research plan

4 November 13-15 User Trials 6 November 13-15 User Trials

11 November User Trials 13 November User Trials


Conducting sessions with “accessibility” users


Conducting sessions with “accessibility” users

We will consider each of the following principal stages:• Pre-session preparation• Selection and preparation of the trial location• Conduct of the sessions• Follow-up to the sessions


Pre-session preparation

Planning the tasks to be performed• Can the users complete the “usability” tasks?• What is the purpose of the user trial? Identify accessibility issues, usability

issues or usable access issues?• Need to add specific “accessibility” questions/tasks• Example: “Make a peanut butter sandwich” becomes “Can you tell which jar

is the peanut butter?”, etc.• Can the users do them?



Getting ethical approval• Implications of “accessibility” users

• How is approval gained? (Example: user is deaf)

• And recorded? (User cannot sign a consent form)

• How can you be sure that the consent is valid? (Example: user has a cognitive impairment)


Example: Usability test consent form

Please read and sign this form

In this usability test:

You will be asked to perform certain tasks on a website. We will also conduct interview with you. You will be asked to fill in a questionnaire.

Participation in this usability study is voluntary. All information will remain strictly confidential. The descriptions and findings may be used to help improve the web site. However, at no time will your name or any other identification be used. You can withdraw your consent to the experiment and stop participation at any time.

If you have any questions after today, please contact (name) at (phone).

I have read and understood the information on this form and had all of my questions answered

Subject's Signature Date


[Aside] Good consent practice

Make it clear: That the session can be paused at any point

That the participants can withdraw from the session at any point

That the participants can withdraw their data at any point

Outline the data to be collected, how it will be handled and who will be able to access it

Outline the process by which the data will be fully anonymised

Describe the risks to the participants in clear and unequivocal language

Give this to the participant in writing and ask for them to sign


[Aside] Good consent practice

You should also:

Provide examples of consent forms that participants will be expected to sign prior to the session

Ensure that at no point are the participants exposed to potential risks of “harm” (not just physical, but also emotional)



Identifying the needs of the participants:

Navigation • How is the person going to get to the venue?• How is he/she going to navigate around the venue?

Communication• What mode of communication should be used?• Written / spoken / Braille / sign language, etc.?

Assistive technologies• Does the users use AT?• Will they bring their own? Or do you have to supply it?• Will it work with your product?



Conducting pilot studies

As for “usability”

Also test for accessibility of the venue, instructions, etc.

Also do tests to ensure that the AT works



Preparing and distributing pre-session documentation What to include?• All consent forms that the participants will have to sign• Details of any reimbursement offered• Details of the time and location of the trials• Directions to the location where the trials are to be held• The tasks to be performed and details of how the session will be structured

and run (the experimental protocol)

Send these out in good time (allow extra time for reading and replying) Check the correct format• Braille• Audio files/tapes• etc.


Selecting a trial location

Best for you? Best for data collection/analysis? Best for the user?


Challenges of trial locations

Participants who are: Challenges and needs

Blind Navigating to and around trial site (e.g. to toilet)

Getting used to the layout of the lab

Provision for a guide dog

Low vision Large, clear signposting

Deaf Provision for a sign language interpreter

Motor impaired Access to the building (e.g. ramps)


Selecting a trial location – Usability Lab

Advantages: Tightly controlled Already set up (usually) Sufficient space and vantage points

Disadvantages: Unfamiliar for participants Can be intimidating May not be accessible• Getting there• Navigating within


Selecting a trial location – 3rd party locations (e.g. day care centres)

Advantages: Usually known by users Familiar Usually already accessible

Disadvantages: Need to set up (possibly daily) Sufficient space and vantage points? Privacy? Possible interruptions


Selecting a trial location – Remote testing

Advantages: No travel requirement for users Should have necessary AT and optimised computer access Cheap (time, resources and money)

Disadvantages: Reduced feedback from users Difficulty of communication if user is (e.g.) deaf Difficulty observing coping strategies Often difficulties getting software installed, data mailed back, etc.


Conducting the sessions

Need to consider 3 main components

Pre-trial briefing

Performance of the tasks

Interviews and final debrief


Pre-trial briefing

What mode will be used for the briefing?• Spoken / written / sign language, etc.• Find what works best for the user

Ideally, as much material as possible should be sent to the user ahead of the session

Remind users that they are in control

You may need to reassure them


Performance of the tasks

Allow more time for task completion• It may simply take more time

Do not force users into doing tasks that are beyond them• Be prepared for data “incompleteness”

Watch out for tiredness• It is not good for the user or your data collection

Keep reassuring the participants• “No – the computer is not about to blow up!”

Keep obersvers to a miminum• Do not crowd the user

Look out for coping strategies


Interviews and final debrief

As for pre-trial briefing

What mode will be used for the briefing?• Spoken / written / sign language, etc.• Find what works best for the user

Ideally, as much material as possible should be sent to the user ahead of the session


Working with “accessibility” users - Summary

Do not make assumptions about the needs of the participants

If in doubt – ask

Make sure that the most relevant users are selected to participate in the trials

Plan ahead and be prepared for unexpected results


Before the test – Summary(source: Nielsen “Usability Engineering”)

Have everything ready before the user show up Emphasize that it is the system that is being tested, not the user Acknowledge that the website is new and untested, and may have

problems Let the user know that they can stop at any time Explain any recording, keystroke logging, or other monitoring that is

used Tell the user that the test results will be kept completely confidential Make sure you have answered all the user’s questions before

proceeding


During the test – Summary(source: Nielsen “Usability Engineering”)

Try to give the user an early success experience Hand out the test tasks one at a time Keep a relaxed atmosphere in the test room Avoid disruptions: Close the door and post a sign on it. Never indicate in any way that the user is making mistakes or is too

slow Minimize the number of observers at the test Do not allow the users’ management to observe the test If necessary, have the experimenter stop the test if it becomes too

unpleasant


After the test – Summary(source: Nielsen “Usability Engineering”)

End by stating that the user has helped you find areas of improvement

Never report results in such way that individual users can be identified

Only show videotapes outside the usability group with the user’s permission


How many users???


Choosing how many users

For this discussion we will assume:• (a) you have accurately identified the right types of users• (b) there is one major user demographic• (c) the users are fairly homogeneous (i.e. similar) within that demographic


Jakob Nielsen and 5 to 8 users

The most famous answer to this question is from Jakob Nielsen He suggested 5 to 8 users

Logic: Landauer and Nielsen showed that:

Number of usability problems found = N(1-(1-L)n)

where:

N = total number of usability problems in the design

L = proportion of usability problems found by a single user

A typical value for L = 31%



Plotting for L = 31%



Important observations: 0 users find 0 problems! 1 users finds approximately 1/3rd of the problems 15 users find approximately all of the problems

Notes: These numbers are based on L=31%, that may not always be the true

value• Depends on the size and complexity of the product

Nielsen recommends 5 users (not 10 or 15) to save costs• They’ll find approximately 85% of the problem


Nielsen and card sorting

Nielsen does not recommend 5 users for everything He recommends 15 to 20 users for card sorting (higher is better since card

sorting uses the statistical method of cluster analysis)

He reasons that there’s only ever a 75% correlation between the results from 5 users and the ultimate results

15 users give 90% correlation 20 users give 93% correlation 30 users give 95% correlation• i.e. an additional 5% for twice the number of people!

60 people reach 98% correlation


Nielsen and card sorting

Why the difference from 5-8 to 15-20? User trials are for evaluations• i.e. trying to find out how good something that exists is

Card sorting is a generative method• i.e. trying to establish the parameters for a new product

Greater variability in the data to be collected and analysed• e.g. different perspectives of the fundamental problem• different mental models to be captured and synthesised • etc.

Can (and should) combine card sorting and user trials


Other perspectives

Nielsen and Landauer recommend 5 to 8 users

Laurie Faulkner recommends 15 users• “Beyond the 5 user assumption: Benefits of increased sample sizes in

usability testing”

Still more people recommend other numbers So what should it be?

Let’s look at what we are trying to achieve…


Why don’t we do large numbers in usability? (source: Frank Spillers “Demystifying usability”

We are looking for behavioural insight (what people do) Statistics tell you how many people failed, but not necessarily why they

did so The objective is usually to apply findings to fix design problems in a

corporate context (not academic analysis) Research shows that even with low numbers, you can gain valid data• Note: this is especially true for “accessibility” testing

Usability testing has been used for 25 years• Experts, authors and academics put their reputations behind the

methodology


Behaviour vs. opinion

Market research is largely opinion-driven You need to ask lots of people what they think Needs very large samples – often 00s or 000s

Usability research is behaviour-driven You observe what people do in a restricted context, not what they say If 10/15 users are confused, you can assume many more will be Behaviour is governed by cognitive factors (biological and otherwise)

not opinion which is more emotional (and thus variable)


Faulkner’s approach

60 users were involved in usability trials Assumed that 100% of problems were found

Taking random collections of 5 people out of the 60 Some sets found 99% of the problems Some found only 55%

Taking random collections of 10 people out of the 60 Lowest success was 80%

Taking 20 people out of 60 Lowest success was 95%

Note – these results depend on the quality of the 60 users selected


Suggested sample sizes

Spillers suggests the following for corporate user research Surveys (phone and web) = ~240 to ~1000+ Focus groups = 15 to 20 (depending on goals of study) Usability trials = 10 to 15 participants Field studies = 15 to 40 participants Card sorting = 15 to 30 (higher is better because of the analyses

methods needed)

Accessibility trials – typically as many as you can get hold of


Exercise


Exercise – part 1

Prepare the testing protocol for evaluating the accessibility and usability of your web-site

Also, address any additional research aims identified in your research plan from Wednesday


Exercise – part 2

You need to consider the following: Pre-session briefing• Prepare your welcome statement• What you are doing and why• Privacy issues and right to withdraw• Any initial questions you wish to ask• Prepare a consent form

Tasks• Identify at least 5 tasks for each user on each site• Ensure you do not introduce systematic errors• Prepare any likely questions you may wish to ask• Remember to add/amend tasks for the “blind” test


Exercise – part 3

You need to consider (continued) Post-session de-brief• Ask any remaining questions needed to address your research issues• Thank the user for their time

E-mail your protocol to Simeon, Stina and me

Remember – you will be putting this to the test next week!


Exercise – suggestions for tasks

Exploring the site / describe each page• Great for getting users used to what is where

Completing a guided product selection task• Find “this” product

Completing an unguided product selection task• Find “any” product of your choice

Changing your mind• You decide you do not want this

How many types of [x] (example: tea)?


Exercise – additional points

Decide whether all users do the same tasks in the same order or not• Be on the lookout for “order” effects

You should randomise the presentation of the sites• ½ do site 1 first• ½ do site 2 first

© simeon keates 2008 usability with project lecture 12 – 20/03/09 susanne frennert

Documents