i. scott mackenzie york university · 2009-11-18 · 1 1 york university – department of computer...

1

1

York University – Department of Computer Science and Engineering

ISO9241 – Part 9

I. Scott MacKenzieYork University

2


What is ISO 9241-9

• ISO 9241• “Ergonomic design of visual display terminals

(VDTs) used for office work”

• Seventeen parts• Part 9: “Requirements for non-keyboard input

devices”• Part 9 - Annex B – “Performance testing”

2

3


Annex B – Performance Tests

• Task #1 - One-directional tapping test• Task #2 - Multi-directional tapping test• Task #3 - Dragging test• Task #4 - Path following test• Task #5 - Tracing test• Task #6 - Free-hand test• Task #7 - Grasp and park (homing) test

4


Task #1One-directional Tapping Test

3

5


Task #2Multi-Directional Tapping Test

6


Task #3Dragging Test

• Variation on• One-directional tapping test• Multi-directional tapping test

• An object is “dragged” and dropped in the destination target

• See MacKenzie, Sellen, & Buxton paper from CHI 1991

4

7


Task #4Path Following Test

8


Task #5Tracing Test

5

9


Task #6Free-hand Input Test

etc.

10


Task #7Grasp and Park Test

• “The subject performs a simple pointing task and operates a key on the keyboard between each pointing task with the same hand.”

6

11


Device Assessment Questionnaire

• ISO 9241-9 also includes a comprehensive qualitative evaluation

• Uses 13 questions and a 5-point Likert response scale

• Assesses aspects of operation, fatigue, comfort, and overall usability

Next slide

12


7

13


14


8

15


ISO Metric for Performance Testing

Throughput

16


Throughput (1)

• Combines speed and accuracy in a single measure

• Based on Fitts’ law (Fitts, 1954)• Endorsed by ISO as the metric for evaluating

pointing devices• Calculations and procedures are tricky

9

17


Throughput (2)

• Benefits are substantial• Emerging use in industry• Aka “index of performance” or “bandwidth”• We’ll explain shortly, but first…

18


Fitts’ Index of Difficulty (1)

10

19


Fitts’ Index of Difficulty (2)

20


ID Examples

11

21


ID vs. Movement Time

22


Throughput

12

23


Some Data

Each score is the mean over repeated trials and multiple participants.

24


Something is Missing

• The measure (2.66 bits/s) does not reflect the accuracy of participants’ responses

• This seriously weakens within-study and between-study comparisons

• A solution was proposed by Crossman (1956) and endorsed by Fitts (1964), Welford (1968), and others

13

25


The Solution

• Transform W into We

• We is the “effective target width”• We = width of the distributions of “hits”• W reflects what participants were asked to do• We reflects what participants actually did

26


Effective Target Width

• SDx – standard deviation in selection coordinates along axis of approach

• We is the central 96% of the spatial distribution• Assumption: distribution is normal (research shows that

the assumption holds reasonably well)• Nominal error rate is 4%• H = = 4.133 σ (Shannon, 1949)• See MacKenzie (1992a, 1992b)

We = 4.133 SDx

σπ e2

14

27


4% Errors

28


Less Than 4% Errors

15

29


Greater Than 4% Errors

30


Frequency Distribution Example1

1 From…

MacKenzie, I. S., Sellen, A., & Buxton, W. (1991). A comparison of input devices in elemental pointing and dragging tasks. Proceedings of the CHI `91 Conference on Human Factors in Computing Systems, pp. 161-166. New York: ACM. (Click here to view) (Google Scholar citations)

16

31


“Effective” Index of Difficulty

• ID is the difficulty of the task presented to the participants

• IDe is the difficulty of the task achieved by the participants

• IDe reflects what participants actually did• May use De instead of D, if participants did not “go the

distance” (check data to determine this)

IDe = log2(D / We + 1)

32


Some Data (Revisited)

17

33


Compare

• Speed only:• Throughput = 2.66 bits/s

• Speed and accuracy:• Throughput = 1.81 bits

• Difference:• +47%

34


So…

• We know how to calculate Throughput• Where do we begin?

18

35


What is the Throughput for the Mouse?

• The question above is meaningless unless the test conditions are specified

• Conditions• Generic mouse• Standard mouse driver• Nominal gain setting• Serial task• Expert participants• Etc.

• Answer: TPMOUSE ≈ 4.5 bits/s• The above is a “baseline condition”

36


Baseline Conditions

• Serve as a check on experimental methods; i.e., • Apparatus• Procedures• Data collection• Data analysis

• Obtaining a known and established result for a baseline condition means the methodology is sound

• Strengthens within-study comparisons• Strengthens between-study comparisons

19

37


Clean up the Mess

Baseline condition (A) is the same in each study

38


Clean up the Mess

Comparisons are valid, since baseline condition (A) is consistent between studies

20

39


Example Study

• Four device conditions• Remote pointing

• Gyration GyroPoint (air)• Interlink RemotePoint

• Desktop pointing• Gyration GryoPoint (desk)• Microsoft Mouse 2.0

Reference:

MacKenzie, I. S., & Jusoh, S. (2001). An evaluation of two input devices for remote pointing. Proceedings of the Eighth IFIP Working Conference on Engineering for Human-Computer Interaction - EHCI 2001. pp. 235-249. Heidelberg, Germany: Springer-Verlag. (Click here to view) (Google Scholar citations)

40


Gyration GyroPoint

21

41


Interlink RemotePoint

42


Results

22

43


Questionnaire – “Ease of Use” (Q13)

5.0 5.0 3.1 1.9

44


ISO 9241-9 Task #1 Demo

23

45


ISO 9241-9 Task #2 Demo

46


Thank you

Questions?

i. scott mackenzie york university · 2009-11-18 · 1 1 york university – department of computer...

Documents