11/2/2010

1

Diseño y Evaluación de Sistemas InteractivosCOM-14112-001

Usability Tests28 de Octubre de 2010

Dr. Víctor M. González y Gonzálezvictor.gonzalez@itam.mx

Agenda

1. Comments on Partial Exam / next week.2. Usability Tests

11/2/2010

2

Usability TestsChapter 10 – Observing the User Experience by Mike Kuniavsky

Usability Tests / What to Test?

• Usability testing should be used to examine the functionality of individual features.

• It is better to highlight potential misunderstandings or errors inherent in the way features are implemented rather than to evaluate the entire user experience.

• Usability testing is never one-time event: every round of testing can focus on a small subset of features

11/2/2010

3

Usability Tests / When to Test?

• A good time to start usability testing is when the development cycle is somewhat underway, but not so late that testing prevents the implementation of extensive changes.

• Ocassionally, usability testing reveals problems that require a lot of work to correct.

• Tests scheduled throughout the development process.

Usability Test ScheduleTiming Activity

t – 2 weeks Determine the test audience; start recruiting immediately

t – 2 weeks Determine feature set to be tested

t – 1 week Write first version of the script; construct test tasks; discuss with the development team; check on recruiting

t – 3 days Write second version of script; review tasks; recruiting should be completed

t – 2 days Complete script; schedule practice test, setup and check all equipment

t – 1 day Do practice test in the morning; adjust script and tasks as appropriate

t Test (usually 1-2 days, depending on schedule)

t + 1 day Discuss with observers; collect copies of all notes

t + 2 days Relax; take a day off and do something else

t + 3 days Watch all tapes; take notes

t + 1 week Combine notes; write analysis

t + 1 week Present to development team; discuss and note directions for further research

11/2/2010

4

Recruiting

• Time consuming, Time constrained: You need to find the right people and to match their schedules to yours

• The best people to invite are those who are going to need the service your are providing in the near future or who have used a competing service in the recent past. High level of interest and knowledge.

Example of Recruiting Criteria

• Web site for Harley-Davison USA

– Men or women, preferably men– 25 years old or older, preferably 35-50– Have internet access at home or work– Use the Web five or more hours a week– Have bought something online in the last three months– Are interested in Harley Davison Products

11/2/2010

5

Recruiting

• For a focused task-based usability testing you should have a least five participants for each round of testing. – Jakob Nielsen has shown that the cost-benefit cutoff for

usability testing is about five users per target audience.

• Time slots management: Double book; leave at least half an hour between participants.

• For 60-min test you can do four or five in a single day.

Choosing Features

• Features -> Tasks -> Order of Test• Five features can be tested in a given 60-90

minute test.• A test duration in average is one to two

hours.• Together with the development team: Making

a list of the top five features to test.

11/2/2010

6

Choosing Features

• Look for features that are:– Used Often – New– Highly publicized– Considered Troublesome, based on feedback from

earlier versions– Potentially dangerous or have bad side effects if

used incorrectly– Considered important by users

Choosing Features

• Competitive Testing– Although comparing two interfaces is more time

consuming that testing a single-interface, it can reveal strengths and weakness between products.

– Performing the same tasks with an existing interface and a new prototype can reveal whether the new design is more functional.

11/2/2010

7

Creating Tasks

• Tasks need to be representative of typical user activities and sufficiently isolated to focus attention on a single feature

• Good task should be – Reasonable: typical of the kinds of things that people will

do.– Described in terms of end-goals: phrase your tasks as

something related to evaluator’s life.– Specific: explicitly articulated end goal.– Doable: avoid frustration, ask what it’s possible– Domain neutral: everybody is able to do it, but not

necessarily know how to do it.– A reasonable length: 90 min test – 50-70 for tasks, 12

min average.

Metrics

• Primarily a qualitative exercise • Performance metrics: investigate the relative

efficiency of different designs– The speed with which someone completes a task– How many errors they make– How often they recover from their errors– How many people complete the task succesfully

11/2/2010

8

Scripts / Protocol / Guide

• A script for the moderator to follow so that the interviews are consistent and everything gets done.

• The script is divided into three parts:– The introduction and preliminary interview– The tasks– The wrap-up

Tecnología Anoto TM

El papel

– Facilidad de uso.– Transporte.– Bajo costo en su elaboración.– Flexibilidad de navegación a

través de documentos.

Las tecnologías digitales

• El manipular, almacenar y recuperar grandes cantidades información de manera sencilla.

• Replicar información.• Compartir información.

11/2/2010

9

Área de aplicación: Ambiente medico

• Se maneja gran cantidad de información

• Uso intensivo del papel

18

Hoja de Enfermería

11/2/2010

10

19

Prototipo de diseño

20

Sección de Signos Vitales

11/2/2010

11

21

Sección de Medicamentos

22

Sección de Envió de mensajes

11/2/2010

12

23

Evaluación

22 estudiantes de enfermería – 5to. Semestre de licenciatura– Sin experiencia en el uso de plumas

digitales

5 computadoras 2 plumas digitales (Logitech io1)3 plumas tradicionales (verde, rojo y azul)1 audífonos1 impresora (HP Láser 600)4 tabla con clip tamaño carta

Satisfaction - SUS

• The System Usability Scale (SUS) was originally developed by John Brooke while working at Digital Equipment Corporation (1986)

• Ten statements to which participants rate their level of agreement. A 5-point scale of agreement.

• No attempt is made to assess different attributes of the system (e.g., usability, effectiveness, etc.). The intent is that one look at the combined rating.

11/2/2010

13

Design the Test• Where: Determine the Location of the Tests

Design the Test

• Where: Typical equipment and facilities

11/2/2010

14

Costs• Where: Costs of facilities:

From www.webcredible.co.uk

• Hire of our usability lab, observation room and sofa-filled break out area costs a flat fee of $1400 + TAX for a full day (9am-5.30pm). Alternatively, the rooms can be hired out for $200 + TAX per hour. The labs can also be hired outside of working hours.

• For these prices, you'll receive:– Full use of the usability lab and observation room – Exclusive access to our break out area, complete with comfy sofas – Technical support staff on hand in case of any problems – Light refreshments – Full session recordings on DVD

CostsMexico City

• Usability Lab: $ 510 daily• Expert: $ 650 daily• Assistant: $ 300 daily• User recruitment: $ 100 each (high - mid)

$ 60 each (low end)• Report: $ 1000• Expert Analysis: $ 650 + 300 daily• Video equipment: $ 300 - 400 daily• Video Analysis: $ 400 daily• Translator: $ 600 daily