Lecture #3TESTING WITHOUT USERS II.

*39TUR Winter 2012/2013

(2)

Human Computer Interaction

Three main objectives– Improve the access to the computers

– Reduce the complexity of this access

– Reduce the probability of errors while using computers

Can be done by:– Direct observation of using computers

– Use of mental models

– Mathematical models of using computers

(3)

Cognitive Science

Part of psychology investigating mental processes such as:– Perception, memory, thinking, learning, problem solving,

language, …

Extensive applications in other fields– Engineering disciplines

• Purpose is to improve the design

– Major interest by• Human Computer Interaction• Human Factors (Cognitive Ergonomics)

(4)

Mental Models

How do we understand the way how the world works around us?– Memory (past experience)– Expectation of the behavior

We have certain understanding of the world’s mechanics– How the objects react? – Which is the relationship among them?

When throwing a ball at a friend– How fast should I do that?– How do they know where to stand?

(5)

Mental Models

A cognitive structure– “It’s in one’s head.”

Describes how certain aspects of the world work– How objects of certain class mutually interact with

objects of different class– How objects change their properties during an

interaction

Each person has their own unique mental model of the world

Reality …

FARE INSPECTORS!!

THIS STATIONIS SOMEWHEREAROUND HERE

Y39TUR

ELECTRICITYCOURSES

A FRIEND OFMINE USED TO

LIVE AROUND HERE

TOURISTS

… and a mental model

(8)

Mental Model Example: The Subway

Having a mental model, we can ask:– “How far are we from station X?”– “Where do we need to change when going to Y?”– “What is the next station?”– “Which way to go to my buddy’s?”

Mental model contains information AND knowledge of uncertainty– “I don’t know what’s on the B-line.”

(9)

Mental Model

How the things are understood– Not how they actually work!

(10)

How do we use mental models?

Prediction of the world’s behavior– “Two more stops. We still have time to read three pages

of the book”

Mental models are fuzzy– Can lead to incorrect predictions

We change the mental models during their use– We learn and adapt

(11)

Mental Models in HCI

Understanding mental models:– Important for understanding the user’s interaction

with the system– Understanding how the computers are understood– Understanding how the data are understood

Usability tests are used to uncover users’ mental models

Mismatch between mental models and reality leads to usability issues– User’s mental models can explain their behavior

(12)

Mental Models in HCI (contd.)

Users make use of the mental model of the user interface, e.g.:– What to expect when clicking a button

• “We should be one click away from that big dialog window that tells me the CD is burning.”

– To tell we did the right/wrong thing• “Oops, we should not get to this page. I should have clicked the link

below.”

• “Oh no, I got on the wrong train.”

(13)

Example: Mismatch of Mental Models

Telephone banking in an imaginary bank– Task: Make a payment using your credit card– Device: Telephone DTMF menu– A menu would say:

• “For account balance, press 1”

• “For payments, press 2”

• “For credit card operations, press 3”

Send money (from my account, using CC…)

Receive moneySee how much I have

Checking AccountsDepartment

Credit CardsDepartment

DESIGNER’S MENTAL MODELUSER’S MENTAL MODELBANK SERVICES

(14)

Mental Models in Usability Testing

When preparing a test, we must not make the user accept our mental model– Instructions for user is not a user guide

• Suggestive instructions

– A real user has no existing mental model of the tested application

• Making use of a previous experience

• That’s why the recruitment from the target group is so important

(15)

Mental Models in Usability Testing

A use case– “Making a payment using a credit card.”

Incorrect wording of the task in a usability test– “Choose ‘credit card payments’ from the menu of the

telephone service.”

Correct wording of the task in a usability test– “Try using the credit card to make a payment in the

menu of the telephone service.”

Cognitive Science in UI Testing

(17)

It is easy to describe a machine …

More-or-less, all diagrams of a computer look like this:

Prediction of function is easy Can we find such a simple description for a human

being?

Input

Memory

Proc. Output

(18)

Human Information Processor Model

An approach to model how information is handled by the user.– A technicist approach

First formulations 1980s– Card, S.K., Moran T.P., & Newell, A. (1983).– Tiffany Jastrembski and Neil Charness (2007)

Human Information Processor Model

“input”

“output”

“processing”

Human Information Processor

Eye movement: 230 ms Visual capacity: 17 letters Auditory capacity: 5 letters Effective working memory capacity: 7 chunks

…

(20)

(21)

Human Information Processor Model for Testing

Basic idea:– Similar to cognitive walkthrough– When stimuli are known, what will be the corresponding

human behavior?

No need for implementation or even prototypes No need for real users Gives a scientific foundations for a design

– Like for other engineering disciplines

(22)

Cognitive Theories in HCI

Models based on Human Information Processor Model:– Fitt’s Law

• How long it takes to select a target

• Evaluation of input devices

– Hick’s Law• Time to choose, depending on the number of choices

– KLM (Keystroke-level Model)• Efficiency of the user interfaces assessed through low-level actions

– GOMS (Goals, Operators, Methods, Selectors)• Higher level than KLM

• Structure and hierarchy of tasks

FITT’S LAW

(24)

Fitt’s Law

Paul Fitt (1954) Based on ergonomics

– How fast would a person reach a target with their hand?

Prediction of time needed to acquire a target, based on:– Distance to the target (D)– Dimension of the target (W)

Something Other thing

Large

Small

(25)

Fitt’s Law

D, W … distance, width (amplitude) Device-dependent constants

– a … operation cost (e.g. time needed to press a button)– b … inherent speed of the device (how fast can we

move around?)

Originally for 1-D problems

(26)

Fitt’s Law

Target 1 Target 2

Index of Difficulty

ID equalFrom: Marti Hearst, User Interface Design & Development

(27)

Fitt’s Law

Target 2Target 1

Index of Difficulty

ID smaller

(28)

Fitt’s Law

Target 2Target 1

Index of Difficulty

ID bigger

(29)

Fitt’s Law for Testing

Additional design heuristic for Heuristic Analysis – Used as a qualitative suggestion.

Fitts’ law often used for determining best case for new kinds of input methods– Used as a theoretical framework for conducting

experiments where two approaches are compared.

(30)

Design Heuristics based on Fitt’s Law

Things done more often should be assigned a larger button.– Size of the Enter key– Sides of the screen and corners have “infinite size”– Possible problems of consistency

Things done more often should be “closer to each other”– Context menu– Frequency-based order vs. logic-based order

(31)

Infinitely Large Objects

Hick’s Law

(33)

Hick’s Law

t ~ b * log(N+1)– N … number of choices– +1 … binary decision whether or not to proceed– b … a context-dependent value

The more options, the longer time

Hick’s Law

Interpretations:– Reduce the number of things the user can do– Reduce the number of items in the menu

Limit the decisions the user needs to make– “The more choices we eliminate, the more enjoyable the experience

will be.” (http://uxdesign.smashingmagazine.com/2012/02/23/redefining-hicks-law/)

“Minimalist Design” rule

(34)

Keystroke-level Model (KLM)

(36)

KLM != Koninklijke Luchtvaart Maatschappij

(37)

Keystroke-level Model

Cognitive Walkthrough, Heuristic Evaluation– Good source of qualitative findings of:

• Usability issues• No ability to tell the time taken by …

– No data on actual performance

To measure time:– Large number of users (statistically valid data)– Expensive (time and people)

Is there a cheap alternative?

(38)

Keystroke-level Model

Purpose of the KLM– A discount usability method– A method of evaluation of the UI– KLM defines a metric of performance of a UI– Provides an estimate of minimal duration of a UI walkthrough

• Will be worse in reality

Based on a model of “virtual user”– On Human Information Processor Model– Formalism of behavior

Focuses on performance only

(39)

Keystroke-level Model

Experimental basis of the KLM– Low-level model of a “typical user”– A large number of people observed while using generic GUI– Measurement of:

• Reaction times

• Duration of elementary actions

(40)

Keystroke-level Model

Input:– Similarity to Cognitive Walkthrough:

• Detailed description of sequence of actions is needed

– Main difference:• Description is at much lower level

Output:– Estimate of the lowest time possible

(41)

Keystroke-level Model

Set of operators– Further indivisible actions

• Based on the current application domain

– Physical actions• Reach for mouse, drive the cursor somewhere, etc.

– Mental actions• Make a decision, select one item of many, etc.

(42)

Keystroke-level Model

Uses of KLM– Determine what is the minimum time of a UI walkthrough– Compare two different walkthroughs of UI leading to the

same result– Compare performance of users of different profiles

• Beginner user (no shortcuts, all commands from the menu)

• Intermediate user (minimum amount of shortcuts)

• Advanced user (ample use of shortcuts + command line)

– Calculate the potential volume of savings• Is it useful to invest into any UI optimizations?

• Is it useful to train the users?

(43)

Keystroke-level Model

Theoretical support of a particular design change suggestion– Comparison of the current state and the design– It is possible to formally support a claim on correctness

of the designed solution

(More on this in the NUR course)

(44)

Keystroke-level Model

An ideal walkthrough is considered– We test the correct and minimal solution

• No confused users

• No mistakes

• No roll-backs

– Time that we calculate is minimal possible• Reality will probably be worse

(45)

KLM Operators

K Keystroke– Key hit, pressed, or released.– Also for the mouse button– 0.08 s – 1.20 s, based on the skill level

P Point on a target– 1.10 s, average performance

H Home the input device– 0.40 s

M Mental preparation– 1.35 s

R System reaction time– Whatever time the system takes

Card et al. (1983)

(46)

KLM Simple Walkthrough Example Example: Make “The cat” in sentence

“The cat sat on the mat” bold.– Note: K = .60 (average typist)

Steps– Select “The cat”

• Reach for the mouse (H = .40)• Point to “The” (P = 1.10)• Double-click and hold down the button (K

= .60)• Move to “cat” (P = 1.10)• Release the mouse button (K = .60)

– Set to bold• Press Ctrl (K = .60)• Press “B” (K = .60)• Release Ctrl (K = .60)

Total = 5.6 seconds

(47)

KLM Example: Alternative – Using a Menu

Total = 7.2 seconds

Example taken from Newman & Lamming (1995) Interactive System Design

Select “The cat” (see previous slide)

Set to boldface• Point to “Format” menu (P = 1.10)• Press mouse button (K = 0.60)• Move to bold (P = 1.10)• Release mouse button (K = 0.60)

(48)

More on KLM OperatorsMore on KLM Operators

K – Keystroke– Determined by typing speed

• T = 0.08 s … excellent typist (155 WPM = 775 CPM)

• T = 0.28 s … average typist (40 WPM = 200 CPM)

• T = 1.20 s … worst typist; unfamiliar keyboard

P – Pointing– Average value T = 1.10 s given– For any action of pointing– More precise value must be determined by an

experiment (e.g. using Fitt’s Law)

(49)

More on KLM OperatorsMore on KLM Operators

H – Homing– Switch between the keyboard and the mouse– Not applicable if each hand operates one device– T = 0.40 s

M – Mental preparation– T = 1.35 s– How determined:

• Numerous tasks observed and analyzed

• Total time minus time spent on physical operations (K, P, H)

• Divide by number of “mental activities”

(50)

When to use the “M” operatorWhen to use the “M” operator

“When do people think?”

Place M before each K and P– K MK– P MP

• “People need to think where to move next, what to type next, etc.”

Remove M between the letters of a typed word– MKMKMK MKKK

• “People don’t think before each letter”

Remove M between compound actions (“point-and-click”)– MPMK MPK

• “People take point and click as a single operation”

(51)

KLM Example #2 Delete words using selection

M P [start of selection] K [click] M P [end of selection] K [shift] K [click] H [rt hand on Delete] M P [Delete]

7.37 seconds

Delete words, letter-by-letter

M P [first letter] K [click] H [rt hand on Delete] M K × 14

8.36 seconds

(52)

KLM VerificationKLM Verification

(52)

Card, Moran and Newell, “The Keystroke Level Modelfor User Performance Time with Interactive Systems”

(53)

Limitations of KLM

The task is analyzed as if performed by an expert user– That is, without an error

Only predicts the effectiveness– Not the learnability– Not the memorability

What is ignored– Parallel execution– Task interleaving– Mental (cognitive) load– Planning and problem-solving (“How do I choose what to do?”)

GOMS

(55)

GOMS

GOMS = Goals, Operators, Methods, and Selection rules

Useful for design as well as for testing– Provides a model of users’ behavior for detailed analysis

of tasks and UI modeling– Here we focus only on testing

(56)

GOMS as an extension of KLM

More complex model of the interaction– Hierarchy of goals

• “To set a text in bold, we first need to type the text and then to specify the formatting.”

– Alternatives how sub-goals can be handled• “We can switch to bold using menu or using a keyboard shortcut”

– Can describe more complex tasks

(57)

GOMS: ComponentsGOMS: Components

An interaction with a UI modeled using these four concepts:An interaction with a UI modeled using these four concepts:– GoalsGoals

• The state to be reachedThe state to be reached

– OperatorsOperators• Elementary perception, cognitive, or motoric actionsElementary perception, cognitive, or motoric actions

• E.g.: Moving the mouse cursorE.g.: Moving the mouse cursor

– MethodsMethods• Procedures to reach the subgoalsProcedures to reach the subgoals

• Sequences of operators and subgoalsSequences of operators and subgoals

– Selection rulesSelection rules• if-then rules determining which method is to be usedif-then rules determining which method is to be used

More details in the More details in the course course User Interface Design (User Interface Design (Návrh uživatelského Návrh uživatelského rozhraní)rozhraní)

(58)

GOMS Example: Making a slide

(59)

GOMS Example: Selecting a menu itemGOMS Example: Selecting a menu item

Structure of menu (width vs. depth)Structure of menu (width vs. depth) Condition 1: Width 16, Depth 1.Condition 1: Width 16, Depth 1.

GoalGoal: Carry out sequence of commands: Carry out sequence of commandsGoalGoal: Carry out a command: Carry out a command

GoalGoal: Determine which command to carry out: Determine which command to carry outOperator: See the screen, determine the command

GoalGoal: Execute the command: Execute the commandSelect: How do we make our selection

IF item # between 1 & 9 THEN use 1-KEY METHOD

Operator: Use 1-Key MethodOperator: Press Enter

Result: Average count of steps = 33Result: Average count of steps = 33

Loops

(60)

CPM–GOMS as an extension of KLM

CPM = Critical Path Method Parallel processes taken into consideration

– Certain activities run in parallel– Separate processes (channels)

• Motoric processHand with the mouse

Keyboard

Eye motion

• Perception processPerceiving stimuli from the environment

• Cognitive processProcessing information

Decisions

(61)

CPM–GOMS

Purpose– Provide an estimate of the UI performance– Similar to KLM– Different underlying model

Requirements for testing– Similar to KLM– Describe the sequence of actions

(62)

CPM–GOMS

Critical Path Method– Economic theory of optimal resource allocation– Find the fastest sequence of actions, leading to the desired result

Elementary operators are very simple actions– Perception (PP)– Cognitive (CP)– Motor (MP)

Explicit modeling of parallelism, considering PP, CP, MP and Memory system

The time to carry out a task is predicted, based on CPM– The longest path through the task, based on cognitive limitations

and information availability

(63)

CPM–GOMS: Critical Path Example: Critical Path Example

Example:– Point + Shift-click– “Move the mouse pointer to desired location, then click,

while holding shift.”

Note:

– The same operation in KLM:• M P K

Source:Daniel Gutierrez, MIT, 2008

(64)

CPM–GOMS: Critical Path Example: Critical Path Example

PP

CP

MPright

MPleft

MPeye

0

start eye move

move eye to target

start mouse move

50 50

30

perceivetarget

move mouse

100

480

perceivecursor

start Shift press

verifytarget

pressShift

100

50 50

100

Adapted from Rob Miller

(65)

CPM–GOMS: Critical Path Example: Critical Path Example

PP

CP

MPright

MPleft

MPeye

0

start eye move

move eye to target

start mouse move

50 50

30

perceivetarget

move mouse

100

480

perceivecursor

start Shift press

verifytarget

pressShift

100

50 50

100

Adapted from Rob Miller

(66)

CPM–GOMS Example: ParallelismCPM–GOMS Example: Parallelism

A telephone operator introduced new workstations for the A telephone operator introduced new workstations for the operatorsoperators– Amount of keystrokes reducedAmount of keystrokes reduced– The most frequent keys moved closer togetherThe most frequent keys moved closer together– New design by 4% slowerNew design by 4% slower– 1 sec/call = $3 million/year1 sec/call = $3 million/year

KLM: No explanationKLM: No explanation CPM-GOMS:CPM-GOMS:

– Removed keystrokes were not on the critical pathRemoved keystrokes were not on the critical path– Keys used during opening of the callKeys used during opening of the call– Then moved from the initial phase of the task to later phaseThen moved from the initial phase of the task to later phase

• To the critical phase

Adapted from Rob Miller

(67)

Limitations of CPM–GOMSLimitations of CPM–GOMS

Perfect performance expectedPerfect performance expected– Expert user expectedExpert user expected

Tasks must have precisely defined goalsTasks must have precisely defined goals Does not account for:Does not account for:

– Problem solvingProblem solving– User’s attempts, atd. User’s attempts, atd.

• Novice usersNovice users

• Trial and errorTrial and error

Complex tasks are difficult to modelComplex tasks are difficult to model

(68)

Ilinkin, Kim (2010)

“Evaluation of Text Entry Methods for Korean Mobile Phones, a User Study”

Chon-ji-inA

EZ-HangulB

SKYC

(71)

Ilinkin, Kim (2010)

Analytical comparison– KLM-GOMS

• A (1111.39 ms/char)

• B (1003.92 ms/char)

• C (995.82 ms/char)

User study