1/43 practical results from large-scale web usability testing rolf molich dialogdesign
TRANSCRIPT
2/43
http://www.dialogdesign.dk/cue2.htm
Slides in Microsoft PowerPoint 97 format
Download Test Reports and Slides
3/43
A recent survey shows that
80% of all Danish drivers think that their driving skills are above average.
How about usability testers?
How It All Started...
4/43
Too much emphasis on one-way mirrors and scan converters
Little knowledge of REAL usability testing procedures - mainly beautified descriptions
Too little emphasis on usability test procedures and quality control (”Who checks the checker?”)
How It All Started...
5/43
Who Checks the Checker?
When did YOU last have an objective check of your usability testing skills?
Who would you trust as an evaluator of your skills?
6/43
Test End Test object Student Professional
teams teams
1 Oct 97 9 Danish web-sites 50 0
2 Dec 97 CUE-1: Win Calendar Progr. 0 4
3 Oct 98 9 Danish web-sites 50 0
4 Dec 98 CUE-2: www.hotmail.com 2+3 7
5 Mar 99 Web Text - Encyclopedia 0 4
Comparative Evaluations
7/43
Introductory couse in Human-Computer Interaction at the Technical University of Copenhagen
Two courses in the fall of 1997 and 1998
120 students per course
Fifty teams of one to three students
2 x 9 Danish web-sites tested by four to nine teams with at least four test participants
Three weeks to complete test and write report
Student Tests
8/43
Quality of Usability tests and reports is acceptable considering that most teams used 20-50 hours
Some teams wrote quite professional reports after just one month of the course (Surprise?)
Few false problems and opinions
Limited overlap between findings
Can Students do Usability Testing?
10/43
Buttons in lower right corner:
Empty shopping basket
Change order
Continue shopping
Go on with your purchase
Would a human bookseller act like this?
www.bokus.com - Bookstore
11/43
Inhuman treatment of users on many e-commerce web-sites
On-site searching seldom works. Users are better off without on-site searching
Many web-sites focus on the company, not the user
Conclusions
-
13/43
User task:
You want to take your business to BG Bank. Make an appointment with the bank
Hard to find in menu structure
Users entered ”appointment” as keyword for Search
Problem Example
15/43
Tolerate user input errors
Provide human error messages (constructive)
Recommend index, site-map
Special handling of frequent keywords
Show user search keywords in context
How to Improve Search
16/43
CUE-1Comparative Usability Evaluation 1
Four professional teams usability tested the same Windows calendar program
Two US teams (Sun, Rockwell), one English (NPL) and one Irish (HFRG, Univ. Cork)
Results published in a panel and a paper at UPA98
Main conclusions similar to CUE-2
17/43
CUE-2Comparative Usability Evaluation 2
Nine teams have usability tested the same web-site– Five professional teams
– Two semi-professional teams
– Two student teams
– (plus three student teams from TUD)
Test web-site: www.hotmail.com
18/43
Purposes of CUE-2
Provide a survey of the state-of-the art within professional usability testing of web-sites.
Set a benchmark against which other usability labs can measure their usability testing skills.
Investigate the reproducibility of usability test results
Give participating teams an idea of strengths and weaknesses in their approach to usability testing
20/43
Basis for Usability Test
Web-site address: www.hotmail.com
Client scenario (written by Erika Kindlund and Meeta Arcuri)
Access to client through intermediary (Erika Kindlund)
One month to carry out test -Web-site adress not disclosed until start of test period
21/43
What Each Team Did
Familiarize with Hotmail
Define test scenarios
Define user profile; recruit test participants
Run a suitable number of tests, determined by the team
Write usability test report in standard company format and anonymize it
22/43
Problems Found
CUE-1 CUE-2 Total number of problems 141 300
Found by seven teams - 1 Found by six teams - 1 Found by five teams - 4 Found by four teams 1 4 Found by three teams 1 15 Found by two teams 11 49 Found only by one team 128 (91%) 226 (75%)
23/43
Comparison of Tests
Based mainly on test reports
Focus on significant differences
Selection of parameters for comparison based on two generally recognized textbooks:
– Dumas and Redish, ”A Practical Guide to Usability Testing”
– Jeff Rubin, ”Handbook of Usability Testing”
24/43
Resources
Team A B C D E F G H J
Person hours used for test 136 123 84 (16) 130 50 107 45 218
# Usability professionals 2 1 1 1 3 1 1 3 6
Number of tests 7 6 6 50 9 5 11 4 6
25/43
Usability Test Reports
Team A B CD E F G H J
# Pages 16 36 105 36 19 18 11 22
Exec summary Y Y NN N Y N Y Y
# Screen shots 10 0 80 1 2 1 2 0
Severity scale 2 3 21 2 1 1 3 4
26/43
Usability Results
Team A B C D E F G H J
# Positive findings 0 8 4 7 24 25 14 4 6
# Problems 26 150 17 10 58 75 30 18 20
% Exclusive 42 71 24 10 57 51 33 56 60
% Core problems (100%=26) 38 73 35 8 58 54 50 27 31
Person hours used for test 136 123 84 NA 130 50 107 45 218
27/43
Results
There are overwhelmingly many usability problems.
There are many ”serious” usability problems.
Limited overlap between team findings.
28/43
Conclusions
In most cases, no form of cost-effective testing will find all or most of the problems - or even most of the serious ones
Claims like ”Method x finds at least 80% of all serious usability problems” are not in accordance with the results of this study
29/43
Problems Found in CUE-2
Total number of different usability problems found 300
Found by seven teams 1 Found by six teams 1 Found by five teams 4 Found by four teams 4 Found by three teams 15 Found by two teams 49 Found only by one team 226
30/43
Problem Found by Seven Teams
During the registration process Hotmail users are asked to provide a password hint question. The corresponding text box must be filled.
Most users did not understand the meaning of the password hint question. Some entered their Hotmail password in the text box.
Clever but unusual mechanisms like the password hint question must be explained carefully to users.
32/43
Problem Example
Users consistently glanced briefly at this screen and then without hesitation clicked the button ”I Accept”
The button ”I Accept” is very conveniently placed (”usable”), but the text is quite difficult to read. The text is written in legalese, not in webbish.
Users want text that they can ”Skim, skim, and read”.
Do unusable ”Terms of Service” have any legal value?
34/43
Difficult to read - legalese, not English Does not answer important user questions
about privacy, cost Not in native language
Signals ”Don’t waste your time on this”: Button ”I agree” is too usable No information on how to return to
Terms of Service
Problems with Terms of Service
37/43
Examples of language related problems that were detected by European teams
Send Mail: Term "Compose" difficult to understand. Use "Create new message" or "Write Mail” (5/9)
Create new account: "State/Province" textbox is required but does not make sense in many countries (2/9)
Language Related Problems
38/43
Some language related problems suggested by US teams were not confirmed by European test teams
Change "last name" to "family name" Meaning of "U.S. Residents only" and
"Non-U.S. Residents Only" is unclear
Language Related Problems
40/43
Problems listed with severity, #users Distinguish clearly between
– Personal opinions, – Expert opinions, – User opinions, – User findings
Advice for a Usable Usability Report
41/43
No power user test, although four teams also recruited power users
Few tests that require complicated setup. Examples: Attachments; boundary testing, e.g. large number of e-mails in in-box
Teams completed their usability tests within schedule, but they never compared their results to those from the other teams
Some State-of-the-Art Boundaries
42/43
The total number of usability problems for each tested web-site is huge,much larger than you can hope to find in one series of usability tests
Usability testing techniques can be improved considerably
We need more awareness of the usability of usability work
Conclusions
43/43
http://www.dialogdesign.dk/cue2.htm
Slides in Microsoft PowerPoint 97 format
CUE-2 Panel:Tuesday at 4.30 p.m.
Download Test Reports and Slides