Anna Kolesnichenko

Songmei Han

The Growth of Cognitive The Growth of Cognitive Modeling in HCI Since GOMSModeling in HCI Since GOMS

OverviewOverview

GOMS as cognitive modelingAdvances in modeling specific serial

componentsExtensions of the basic frameworkWhat Cognitive Modeling in HCI can

and cannot do

Cognitive modelingCognitive modelingthe progress in modeling the kind of

cognition involved in HCI– basic and advanced sets of parameters

account for the time of given activities

– formal modeling in grammars and production systems

Error production Time to learn Savings from previous learning

– critical path analysis Specification of interacting processes and their

durations

Cognitive ModelsCognitive ModelsPredict how users will interact with

proposed designsConstrain the design spaceAnswer specific design decisionsEstimate total time for task performanceProvide base for calculating training time

and designing training documentationDetermine stages of activity that take the

longest time or produce the most errors

Cognitive modeling (cont.)Cognitive modeling (cont.)Method used for design, evaluation and

trainingGaps in understanding the process of

interacting with computers– Human learning– Design of consistent user interfaces– Error production and management– Interpretation of visual displays for meaning– Concurrent vs. sequential processes

Gaps in cognitive theoryGaps in cognitive theory

Fails to capture– User’s fatigue– Individual differences– Mental workload– Change expected in work life– User’s judgment of the acceptability of the

software

Analytic models of human Analytic models of human performance with computersperformance with computers

1980-1983 – Card, Moran and Newell - significant advance from modeling in cognitive psychology– modeled together many of the processes

contributing to the full cycle of perception to action

– described in enough detail the knowledge necessary to perform a task

– Enabled to generate predictions about human behavior in real, naturalistic tasks

FrameworkFramework

Two key components– Model Human Processor (MHP)

General characterization of the human information-processing system

– System architecture

– Quantitative parameters of component performance

– GOMS A way of describing what the user needs to know to

perform computer-based tasks

GOMSGOMS

A family of modelsDescribes

– The knowledge necessary– Four cognitive components of skilled

performance in tasks Goals Operators Methods Selection rules

Original GOMS FrameworkOriginal GOMS Framework

Focus: Selection of

methods from memory Time to specify

and execute an action

Assumptions Skilled user Serial sequence of independent cognitive operations and

motor activities

The method Predict a time it takes a user to execute a task A task is based on retrieving plans from long-term memory A method is chosen from available methods depending on

the features of the task Execute motor movements necessary

Time parameters for external actions were estimated from empirical data

The GOMS MethodThe GOMS Method

ExampleExample

Parameters: k – keystroke: 280 msec M - mental operator: 1.35 sec P – pointing: 1.1 sec H – moving hands: 400 msec

Example:

sum ( a , b )

Mkkk MkMkMkMkMk

Total = 6M’s + 8 k’s = 6(1.35) + 8(.280) = 10.34 sec

Limitations of GOMSLimitations of GOMS Limited range of domains Applied to skilled users only Accounts for performance but neither learning nor

recall Focused on errorless performance Gives little account of cognitive processes Focused of sequential tasks while many processes

occur in parallel Does not address mental workload Disregards fatigue that users experience Does not account of individual differences among

users

Advances in modeling specific Advances in modeling specific serial componentsserial components

1983 – further research based on GOMS methodology– Serial processing– Time parameters are constant across tasks

Incorporated relevant cognitive psychology factors

Empirical work based on studies of entering editor commands, formulas in spreadsheets, etc.

Classes of parametersClasses of parameters

motor movementperceptionmemorycognition

Motor MovementsMotor Movements

Keying

Moving a mouse

Hand movement

KeyingKeyingTime to enter a keystroke in a normal

typing taskValue depends on

– The skill level of the typist– Frequency with which a key is used– Predictability and continuity of the text

Example– Skilled typist – 80msec/keystroke– User unfamiliar with the keyboard – 1200 msec

/keystroke

Moving a MouseMoving a MousePointing with a mouse at objects at various

distances and of various target sizes.Derived from empirical experiments.Fitts’s law:

– T = 1.03+.096 log2(D/S+.5)

– applied to nested menus:

T = .81+.21 log2(D/S+.5)

Hand MovementsHand Movements

Time needed to move from the space bar of the keyboard until the pointing control begins to move the cursor.

Large-muscle movementCharacterized by Fitts’s lawEmpirically, T = 360 msec

PerceptionPerception

Recognition of features of the current task and assessment of some parameters necessary to do a task

Examples:– Time to respond to a brief light = 100 msec

(50-200 msec depending on intensity)

– Time to recognize a 6-letter word = 340 msec– Time for the eye to jump to next location

= 320 msec

Memory and Cognitive Memory and Cognitive ProcessesProcesses

Memory retrieval

Executing steps in a mental procedure

Choosing among methods

Memory RetrievalMemory Retrieval

Time to retrieve the next unit of information– well-known units– from long-term memory to working memory

A repeated act speeds up memory access

Memory Retrieval ExampleMemory Retrieval Example

Retrieve a command name or delimiter 1350msec

Retrieve a random command abbreviation

1200msec

Retrieve the next part of a formula 1100msec

Repeated retrieval of same command 660msec

Executing Steps in a TaskExecuting Steps in a Task

GOMS catalogues:the retrieval of goal and its subgoalsthe decision to select a methodthe retrieval of the motor movementsthe execution of each command

componentProduction system formalism - explicit

representation

Choosing Among MethodsChoosing Among Methods

The more choices – the longer the expected response time

Empirical estimations of time vary

(1.3 – 4.6 sec)

Composite PerformanceComposite Performance

A task:

enter a block of values(2 digits)Mouse method- enter each value, point to

the next cell with a mouseMenu method - <ret> key advances cursor

automatically to the next cell. Use mouse only to go to the next line

Empirical solutionEmpirical solution

Empirical results:Mouse method

4.19 sec per cellMenu method

2.46 sec per cell

2.81 sec to start each line

GOMS solutionGOMS solutionMouse method

moving the hand to the mouse 360 msec

clicking the mouse 230 msec

moving the hand to the keyboard 360 msec

retrieving digits 1200 msec

typing digits 460 msec

retrieving the end action 1200 msec

typing the <ret> key 230 msec

Total 4040 msec

3% error of 4.19 sec empirical result

Menu method – starting a new line:

moving hand to mouse 360 msec

pointing to a new line 1500 msec

clicking the mouse 230 msec

moving hand to keyboard 360 msec

Total 2450 msec

13% error of 2.81sec empirical result

GOMS solution (cont.)GOMS solution (cont.)

GOMS solution (cont.)GOMS solution (cont.)

Menu method – typing a number into a cell:

retrieving two digits 1200 msec

typing two digits 460 msec

retrieving the end action 1200 msec

typing the <ret> 230 msec

Total 3090 msec

26% error of 2.46 sec empirical result

Pros and cons of the methodPros and cons of the method

Challenged based on inclusion or exclusion or an operation (esp. mental)

Achievements– within an average of 14% error of the observed

values– accurate enough to be useful

SummarySummary

Problems with GOMS:– Serial process assumption– Independent task assumption

Served well in a variety of basic computer-based tasks

Extension of the Basic Extension of the Basic FrameworkFramework

Learning and Transfer– Time to learn– Transfer from one system to the other

Analysis of errors: Workload in Working Memory

Parallel ProcessesModeling parallel processes with critical

path analysis

Modeling in extended workModeling in extended work

Modeling of grammatical rules– What knowledge a user must have before

translating from goals to actions in a system?– Similar to goal decomposition and methods in

GOMS.– Provide a countable entity:

The number of rules

Task-Action Grammar (TAG)Task-Action Grammar (TAG)

Feature Possible values

Direction Forward, backward

Unit Character, word

Task[direction, unit] Symbol[Direction]+ Letter [Unit]

Symbol[forward] “cntl”

Symbol[backward] “meta”

Letter[word] “W”

Letter[character] “C”

Task-Action Grammar (TAG)Task-Action Grammar (TAG)

Goals Action

Move cursor one character forward

Cntl-C

Move cursor one character backward

Meta-C

Move cursor one word forward

Cntl-W

Move cursor one word backward

Meta-W

Production system Production system

Make underlying knowledge explicitOnce written, the accuracy and

completeness can be checked by running the program

Program can be used to predict both errors and learning time behavior

Rules for writing a SQL join queryRules for writing a SQL join query

Rule 1: StartUp. SeeIfJoinNeededIF

GOAL SeeIfJoinNeeded

Not (NOTE SeeingIfJoinNeeded TRUE

THEN

Add NOTE SeeingIfJoinNeeded TRUE

Add STEP CountTables

Rules for writing a SQL join queryRules for writing a SQL join query

Rule 2: CountTablesIF

GOAL SeeIfJoinNeeded

STEP CountTables

THENDo Task Count NumberOfTables * NumberOfTables

Add NOTE NumberOfTables * NumberOfTablesDelete STEP CountTablesAdd STEP AddJoinNote

Rules for writing a SQL join queryRules for writing a SQL join query

Rule 4: IfNumberOfTablesNot =2, ThenCleanUp

IF GOAL SeeIfJoinNeededSTEP AddJoinNoteNOTE NumberOfTables 1

THENDelete SETP AddJoinNoteDelete NOTE NumberOfTables ?NumberOfTablesAdd STEP Cleanup

Use of Production RulesUse of Production Rules

IF part:Check for a match between the rule’s condition and the current goal and the current notes in working memory (WM).

THEN part:If there is a match, execute THEN part action to add and delete NOTES and STEPS to or from WM.

Time to LearnTime to Learn

Kieras & Polson– Study learning under highly restrictive and

controlled condition– Determine the number of steps in a procedure – The time to learn each step is 30 second– The start-up time is 30 to 60 minutes.

Time to Learn (cont)Time to Learn (cont)

Results from from widely different situations and labs are at the same order of magnitude.

The number of rules is less critical than whether the features of those rules follow real-world features encoded in user’s memory.

Problem: how to quantify the learning time in more naturalistic situations.

Transfer of TrainingTransfer of Training

Production is the unit of learning. The number of productions shared by the two

systems can predict the amount of transfer. Time to master a new procedure is a function of

the number of new production to be learned. Specify the exact effects of consistent design

across system and assess the relative costs of different degrees of consistency among procedures.

Transfer Predicted by the number Transfer Predicted by the number of new rules to be learnedof new rules to be learned

Analysis of ErrorsAnalysis of Errors

Cause of errors:– Working Memory (WM) overload– Production systems are used to estimate the

contents of WM and the resident duration of each piece of information in WM.

– The more items in WM, the greater the likelihood of errors.

Systems with different WM Systems with different WM workloadworkload

Lotus 1-2-3– User has to find and remember the coordinates

of cells in the formula. (e.g. D23)

Interactive Financial Planning System (IFPS)– User can refer to a cells by name with adjective

(e.g. previous) to indicate relative location. No need to remember the coordinates.

WM Load for Different WM Load for Different SystemsSystems

Col Name

Row No.

Operator Var Name

Var Name

Var Name

Pointer Pointer Pointer Pointer Pointer Row No.

WM

Time

Position Naming (e.g. B22)Formula with cells in the same column

Operator Var Name

Var Name

Var Name

Pointer Pointer Pointer Pointer Pointer Row No.

WM

Time

Keyword Naming (e.g. Previous Sales)Formula with cells in the same column

WM Load and Error WM Load and Error ProbabilityProbability

The higher the WM load, the more errors– Lotus: 19 items in WM, 14% errors– IFPS: 14 items in WM, 6% errors.

When there are few than 8 items in WM, errors would be eliminated.– Magic number: 7

WM Load and Error WM Load and Error Probability (cont)Probability (cont)

SQL join query:

the more the intervening restriction statements, the higher the probability of forgetting the last crucial join statement.

No intervening restriction, 1.7% errors

2 intervening statements, 4.2% errors.

Problems with Error AnalysisProblems with Error Analysis

Do not distinguish between the peak load in WM and the duration of each item in WM.

People might use strategy to reduce WM load. (Notes, clues from the environment)

WM overload is not the only cause of errors.

Parallel Processes Parallel Processes

Reasons a serial model– Easy to quantify:

task time = sum of subcomponent time– Tradition in Cognitive modeling

Reasons for a parallel model– External signals appear in parallel. – Mental events can occur in parallel.– External actions can be done in parallel.

Examples of Parallel Examples of Parallel ProcessesProcesses

Rapid typing.Bank clerk enter the handwritten amount on

the check into the computer.Menu search in expert users.

The amount of parallel processes in a task depends on the skill level of the user.

Questions in Modeling Parallel Questions in Modeling Parallel ProcessesProcesses

What processes are occurring in parallel?Which processes depend on each other so

that they must occur in serial?The speed of which process is the limiting

factor for the task?

Critical Path AnalysisCritical Path Analysis

Analyze non-serial processes in timed cognitive tasks.

Model cascading mental and external process.

Analyze how three parallel processor resources (perceptual, cognitive, and motor) work with sequential dependencies.

Critical Path Analysis (cont)Critical Path Analysis (cont)

Parameters needed in critical path analysis – Component processes– Duration of each component process– Dependencies among the component processes

Total time taken by the task can be determined once the above parameters are all known.

Critical Path Analysis for 2 Critical Path Analysis for 2 TypistsTypists

Perceptual processor

Cognitive

Motor processor

Critical Path Analysis for Different UsersCritical Path Analysis for Different Users

What cognitive modeling in What cognitive modeling in HCI can and can not doHCI can and can not do

What cognitive model in HCI can do but more research is needed– Non skilled or casual users– Learning– Errors and mental workload– Cognitive processes– Parallel processes– Individual difference

What cognitive modeling in HCI What cognitive modeling in HCI can and can not do (cont)can and can not do (cont)

What cognitive modeling in HCI can not do– Estimate the impact of fatigue on performance– Assess people’s acceptance to the interface– Divide functionality between computer and

human: what human do best, what computer do best.

– How computer changes work and organizational life.

topic studied Extension possible

insufficient information

Different model

Non skilled user X

Learning X X

Errors X X

Cognitive Process X X X

Parallel Process X X X

Mental workload X

Functionality X

Fatigue X

Individual difference X

Acceptance X

Organizational life X