Pragmatic Usability Evaluation

Overview

Heuristic EvaluationEvaluatorsUsability Metrics

Heuristic Evaluation

‘heuristic’ - ‘used of problem solving techniques that proceed by trial and error’ (related to Greek ‘eureka’)

Longman Concise English Dictionary, 1985

a method of usability evaluation where an analyst finds usability problems by checking the user interface against a set of supplied heuristics or principles

Lavery, Cockton and Atkinson, 1996

Who should do heuristic evaluation?

Use more than one evaluatorIdeally should not be the designerIdeally should be a usability specialist

technical authors also usefulEach carries out independent inspection, then

aggregate findingsEvaluators may need help

unless ‘walk-up-and-use’ application could provide typical usage scenario

Effectiveness of increasing number of evaluators

0

25

50

75

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

number of evaluators

per

cen

tag

e o

f u

sab

ilit

y p

rob

lem

s fo

un

d

(taken from Nielsen, 1993)

Evaluator Performance

Nielsen 1992: same interface

evaluated by 3 groups novice (knowledge

about computers only)

usability experts usability and

specialist domain experts

22%

41%

60%

0%

25%

50%

75%

100%

pro

ble

ms

no

vic

es

sin

gle

ex

pe

rts

do

ub

lee

xp

ert

s

Performing Heuristic Evaluation

Go through interface, compare against recognised usability principles first pass flow of interaction and general

scope subsequently focus on specific elements

Typically 1 - 2 hours in totalOutput: list of usability problems cross-

referenced to usability guidelines

Usability Metrics

learnability

memorability

errors

subjective satisfaction

efficiency

Setting Usability Metrics

skill and intuition...better than last versionbetter than the competitionclient targetsset a range of levels

unacceptable minimum target ideal

Usability Profile

unacceptable minimum target ideal

learnability

efficiency

memorability

errors

satisfaction

Usability Testing - Planning

draw up a test plan see separate handout ‘Checklist for

usability test plans’ informal use by the test team formal use for QA procedures

consider whether to use video/audio recording

Usability Testing - Users

test users should represent target users remember sales staff as a special user group may need to give basic training

getting hold of users internal users should be easy customers from user groups may help paid volunteers : students, classified ads..

take account of older users if relevant

Pay-off ratio for user testing(after Nielsen, 1993)

0

20

40

60

80

1 2 3 4 5 6 7 8 9 10 11 12

number of test users

rati

o o

f b

en

efi

t to

co

st

Usability Testing - designing test tasks

representative & provide reasonable coverage

do-able but not trivialconsider relating to a larger scenarioprovide a written task descriptionpresent in increasing level of difficultydecide whether to use verbal protocols

Relative effectiveness

Karat et al (1992)expert individual & group walkthroughs

used guidelines and tasksusability testing

users identified and described problemstesting identified most problems

including some severe ones missed by experts

Effectiveness continued

walkthroughs useful when resources linited, or for early design

team walkthroughs better than individuals

techniques are complementarycost effectiveness similaralso formal experimental trials

Usability testing - procedure

preparation remember to switch off screen-savers, email, etc.

introductiontestingdebriefing

questionnaires if used also ask about the testing process

write up quickly

usability metrics

learnability time to reach specified

level of proficiency e.g. complete a specified,

representative task

note that learning is a continuum

memorability test users on commands

after trial session

errors number of errors in

completing specified task

subjective satisfaction rating scales physiological measures

efficiency times for experts to

complete specified task(s)

frequency of ‘non-productive’ actions

ratio of used to unused commands