Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh

Mikael B. SkovDepartment of Computer Science

Aalborg University, Denmark

Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-

Aware Mobile Systems in the Field

2Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh

Motivation

We have to investigate into the criteria, methods, and data collection techniques for usability evaluation of mobile systems (Johnson 1998)

Often it is assumed that usability evaluations of mobile devices should be done in the field “… the scaling dimensions that characterize context-aware

systems makes it impossible to use traditional, contained usability laboratories …” Abowd and Mynatt (2000)

Kjeldskov and Graham (2003) found that 71% of mobile device evaluations were done in laboratory experiments

3Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh

Motivation

4Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh

Aim

to compare the outcome of evaluating the usability of a mobile system in a laboratory respectively in the field

to describe techniques used for improving the realism of laboratory settings by including mobility and context and support high-quality video data collection in the field

5Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh

System: MobileWARD

Wireless access to EPR on handheld computer

Information and functionality adapted to location, time and the nurse’s assignments

WARD 273

WARD 274

WARD 271 WARD 275

WARD 276WARD 272WARD 270OFFICE

RECEPTION

RINSEROOM

UNCLEAN

RINSEROOMCLEAN

NURSINGROOM

LIVINGAREA

A

B

C

D

E

F

6Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh

System: MobileWARD

In the corridor Overview of all patients, assigned

patients and pending tasks Direct access to reading details about

each individual patient’s history

Entering a ward Overview of the patients in the ward

Scanning patient’s wrist band Access to entering new measures

General: Button size to allow interaction w. finger or pen

7Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh

Method

Laboratory evaluation Lab at Aalborg University, Denmark 6 test subjects (trained nurses) Tasks derived from user study Laboratory furnished as hospital,

divided into two wards + corridor

Field evaluation Frederikshavn Hospital, Denmark 6 test subjects (trained nurses) No specified tasks Involving real work activities

8Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh

Method Mobile usability equipment

enabling the capturing of video and audio

The usability problems were classified as cosmetic, serious or critical (Molich, 2000)

All sessions were analyzed in random order by two teams of trained usability evaluators

The two teams produced two lists of usability problems and these were merged into one complete list.

9Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh

Findings (1)

37 different usability problems Lab evaluation resulted in 36 problems

• 8 critical, 18 serious, and 10 cosmetic Field evaluation resulted in 23 problems

• 7 critical, 10 serious, and 6 cosmetic Primarily more serious and cosmetic problems

Critical Serious Cosmetic

Lab

Fie

ld

10Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh

Findings (2)

More problems per session 18.8 (2.0) problems versus 11.8 (3.3) problems (U=2.651,

p<0.01)• Critical: 5.3 (1.2) and 4.5 (2.2) problems• Serious: 7.5 (1.0) and 4.5 (0.8) problems• Cosmetic: 6.0 (0.9) and 2.8 (1.0) problems

Identified significantly more serious (U=2.79, p<0.01) and cosmetic problems (U=2.84, p<0.01)

11Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh

Field Evaluations Revisited (1)

Little added value of taking the evaluation into the field

Same problems in the laboratory Field contribution: Validity of data

entered into the system Lack of control undermined the

extendibility of the field None of the field subjects used the

note taking facility The higher number of identified

problems in the lab condition could be a result of irrelevant usability problems

12Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh

Field Evaluations Revisited (2)

Both the lab and field revealed context-aware problems

All seven context-aware related problems in both conditions

All field subjects got confused when the system automatically updated information or functionality according to the physical location

The clip-on camera facilitated data collection of mobile use

The configuration allowed subjects to move freely in the environment while at the same time still providing a close-up view of the interaction

However, problems of placing the devices between use

WARD 273

WARD 274

WARD 271 WARD 275

WARD 276WARD 272WARD 270OFFICE

RECEPTION

RINSEROOM

UNCLEAN

RINSEROOMCLEAN

NURSINGROOM

LIVINGAREA

A

B

C

D

E

F

13Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh

Conclusions

Was it worth the hassle? Not really, at least not for usability problem

identification However, the real use situation provided additional

information on use Replicating the context – always possible?

Lab evaluation without context replication Field evaluation with task assignments

14Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh

Questions…