models of human performance - revision
DESCRIPTION
Models of Human Performance - revision. CSCI 4800/6800 Spring 2006 Kraemer. Objectives. Introduce theory-based models for predicting human performance Introduce competence-based models for assessing cognitive activity Relate modelling to interactive systems design and evaluation. - PowerPoint PPT PresentationTRANSCRIPT
Models of Human Models of Human Performance - revisionPerformance - revision
CSCI 4800/6800CSCI 4800/6800Spring 2006Spring 2006
KraemerKraemer
ObjectivesObjectives Introduce theory-based models for Introduce theory-based models for
predicting human performancepredicting human performance Introduce competence-based models Introduce competence-based models
for assessing cognitive activityfor assessing cognitive activity Relate modelling to interactive Relate modelling to interactive
systems design and evaluationsystems design and evaluation
Some Background ReadingSome Background ReadingDix, A et al., 1998, Human-Computer Interaction Dix, A et al., 1998, Human-Computer Interaction
(chapters 6 and 7) London: Prentice Hall(chapters 6 and 7) London: Prentice HallAnderson, J.R., 1983, The Architecture of Cognition, Anderson, J.R., 1983, The Architecture of Cognition,
Harvard, MA: Harvard University PressHarvard, MA: Harvard University PressCard, S.K. et al., 1983, The Psychology of Human-Card, S.K. et al., 1983, The Psychology of Human-
Computer Interaction, Hillsdale, NJ: LEAComputer Interaction, Hillsdale, NJ: LEACarroll, J., 2003, HCI Models, Theories and Carroll, J., 2003, HCI Models, Theories and
Frameworks: towards a multidisciplinary science, Frameworks: towards a multidisciplinary science, (chapters 1, 3, 4, 5) San Francisco, CA: Morgan (chapters 1, 3, 4, 5) San Francisco, CA: Morgan KaufmanKaufman
Why Model Performance? Why Model Performance? Building models can help develop theoryBuilding models can help develop theory
– Models make assumptions explicitModels make assumptions explicit– Models force explanationModels force explanation
Surrogate user:Surrogate user:– Define ‘benchmarks’Define ‘benchmarks’– Evaluate conceptual designsEvaluate conceptual designs– Make design assumptions explicitMake design assumptions explicit
Rationale for design decisionsRationale for design decisions
Why Model Performance?Why Model Performance? Human-computer interaction as Applied Human-computer interaction as Applied
ScienceScience– Theory from cognitive sciences used as basis Theory from cognitive sciences used as basis
for designfor design
– General principles of perceptual, motor and General principles of perceptual, motor and cognitive activitycognitive activity
– Development and testing of theory through Development and testing of theory through modelsmodels
Pros and Cons of ModellingPros and Cons of Modelling PROSPROS
– Consistent description through (semi) formal Consistent description through (semi) formal representationsrepresentations
– Set of ‘typical’ examplesSet of ‘typical’ examples– Allows prediction / description of performanceAllows prediction / description of performance
CONSCONS– Selective (some things don’t fit into models)Selective (some things don’t fit into models)– Assumption of invariabilityAssumption of invariability– Misses creative, flexible, non-standard activityMisses creative, flexible, non-standard activity
Generic Model Process?Generic Model Process?
Define system: {goals, activity, tasks, Define system: {goals, activity, tasks, entities, parameters}entities, parameters}
Abstract to semantic level Abstract to semantic level Define syntax / representationDefine syntax / representation Define interactionDefine interaction Check for consistency and completenessCheck for consistency and completeness Predict / describe performancePredict / describe performance Evaluate resultsEvaluate results Modify modelModify model
User Models in DesignUser Models in Design BenchmarkingBenchmarking
Human Virtual MachinesHuman Virtual Machines
Evaluation of conceptsEvaluation of concepts
Comparison of conceptsComparison of concepts
Analytical prototypingAnalytical prototyping
BenchmarkingBenchmarking What times can users expect to take What times can users expect to take
to perform taskto perform task
– Training criteriaTraining criteria– Evaluation criteria (under ISO9241)Evaluation criteria (under ISO9241)– Product comparisonProduct comparison
Human Virtual MachineHuman Virtual Machine How might the user perform?How might the user perform?
– Make assumptions explicitMake assumptions explicit– Contrast viewsContrast views
Evaluation of ConceptsEvaluation of Concepts Which design could lead to better Which design could lead to better
performance?performance?
– Compare concepts using models prior to Compare concepts using models prior to building prototypebuilding prototype
– Use performance of existing product as Use performance of existing product as benchmarkbenchmark
Performance vs. CompetencePerformance vs. Competence Performance ModelsPerformance Models
– Make statements and predictions about Make statements and predictions about the time, effort or likelihood of error the time, effort or likelihood of error when performing specific tasks;when performing specific tasks;
Competence ModelsCompetence Models– Make statements about what a given Make statements about what a given
user knows and how this knowledge user knows and how this knowledge might be organised.might be organised.
Notions of MemoryNotions of Memory Procedural Procedural
– Knowing howKnowing how– Described in ACT by Production RulesDescribed in ACT by Production Rules
DeclarativeDeclarative– Knowing thatKnowing that– Described in ACT by ‘chunks’Described in ACT by ‘chunks’
Goal StackGoal Stack– A sort of ‘working memory’A sort of ‘working memory’– Holds chunks (goals)Holds chunks (goals)– Top goal pushed (like GOMS)Top goal pushed (like GOMS)– Writeable Writeable
Production SystemsProduction Systems
Rules = (Procedural) KnowledgeRules = (Procedural) Knowledge
Working memory = state of the worldWorking memory = state of the world
Control strategies = way of applying Control strategies = way of applying knowledgeknowledge
Production SystemsProduction Systems
Architecture of a production system:Architecture of a production system:
Rule base Working Memory
Interpreter
The Problem of ControlThe Problem of Control Rules are useless without a useful way to Rules are useless without a useful way to
apply themapply them
Need a consistent, reliable, useful way to Need a consistent, reliable, useful way to control the way rules are appliedcontrol the way rules are applied
Different architectures / systems use Different architectures / systems use different control strategies to produce different control strategies to produce different resultsdifferent results
Production RulesProduction Rules
IF conditionIF conditionTHEN actionTHEN action
e.g., e.g., IF ship is dockedIF ship is docked
And free-floating shipsAnd free-floating shipsTHEN launch shipTHEN launch shipIF dock is freeIF dock is free
And Ship matchesAnd Ship matchesTHEN dock shipTHEN dock ship
The Parsimonious Production The Parsimonious Production Systems Rule NotationSystems Rule Notation
On any cycle, any rule whose conditions are On any cycle, any rule whose conditions are currently satisfied will firecurrently satisfied will fire
Rules must be written so that a single rule will Rules must be written so that a single rule will not fire repeatedlynot fire repeatedly
Only one rule will fire on a cycleOnly one rule will fire on a cycle
All procedural knowledge is explicit in these All procedural knowledge is explicit in these rules rather than being explicit in the interpreterrules rather than being explicit in the interpreter
Why Cognitive Architecture?Why Cognitive Architecture? Computers architectures:Computers architectures:
– Specify components and their Specify components and their connectionsconnections
– Define functions and processesDefine functions and processes
Cognitive Architectures could be seen Cognitive Architectures could be seen as the logical conclusion of the as the logical conclusion of the ‘human-brain-as-computer’ ‘human-brain-as-computer’ hypothesishypothesis
General RequirementsGeneral Requirements Integration of cognition, perception, and actionIntegration of cognition, perception, and action
Robust behavior in the face of error, the Robust behavior in the face of error, the unexpected, and the unknown unexpected, and the unknown
Ability to run in real timeAbility to run in real time
Ability to LearnAbility to Learn
Prediction of human behavior and performancePrediction of human behavior and performance
Adaptive Control of Thought, Adaptive Control of Thought, Rational (ACT-R)Rational (ACT-R)
http://act.psy.cmu.eduhttp://act.psy.cmu.edu
Adaptive Control of Thought, Rational Adaptive Control of Thought, Rational (ACT-R)(ACT-R)
ACT-R symbolic aspect realised over ACT-R symbolic aspect realised over subsymbolic mechanismsubsymbolic mechanism
Symbolic aspect in two parts:Symbolic aspect in two parts:– Production memoryProduction memory– Symbolic memory (declarative memory)Symbolic memory (declarative memory)
Theory of rational analysisTheory of rational analysis
Theory of Rational AnalysisTheory of Rational Analysis Evidence-based assumptions about Evidence-based assumptions about
environment (probabilities)environment (probabilities)
Deriving optimal strategies (Bayesian)Deriving optimal strategies (Bayesian)
Assuming that optimal strategies reflect Assuming that optimal strategies reflect human cognition (either what it actually human cognition (either what it actually does or what it probably ought to do)does or what it probably ought to do)
(Very simple) ACT(Very simple) ACT Network of propositionsNetwork of propositions
Production rules selected via pattern Production rules selected via pattern matching. Production rules coordinate matching. Production rules coordinate retrieval of chunks from symbolic memory retrieval of chunks from symbolic memory and link to environment.and link to environment.
If information in working memory matches If information in working memory matches production rule condition, then fire production rule condition, then fire production ruleproduction rule
ACT*ACT*
Declarativememory
Proceduralmemory
Working memory
Retrieval Storage Match Execution
OUTSIDE WORLD
Encoding Performance
Addition-Factsix U (4); T (1); H (0)
eight
addend1 sum
addend2
Knowledge Representation
1618 +_____34_____1
Goal buffer: add numbers in right-most columnVisual buffer: 6, 8Retrieval buffer: 14
Symbolic / Subsymbolic levelsSymbolic / Subsymbolic levels Symbolic levelSymbolic level
– Information as chunks in declarative memory, Information as chunks in declarative memory, and represented as propositionsand represented as propositions
– Rules as productions in procedural memoryRules as productions in procedural memory Subsymbolic levelSubsymbolic level
– Chunks given parameters which are used to Chunks given parameters which are used to determine the probability that the chunk is determine the probability that the chunk is neededneeded
– Base-level activation (relevance)Base-level activation (relevance)– Context activation (association strengths)Context activation (association strengths)
Conflict resolutionConflict resolution Order production rules by preferenceOrder production rules by preference Select top rule in listSelect top rule in list Preference defined by:Preference defined by:
– Probability that rule will lead to goalProbability that rule will lead to goal– Time associated with ruleTime associated with rule– Likely cost of reaching goal when using Likely cost of reaching goal when using
sequence involving this rulesequence involving this rule
ConclusionsConclusions ACT use simple production systemACT use simple production system
ACT provides some quantitative ACT provides some quantitative prediction of performanceprediction of performance
Rationality = optimal adaptation to Rationality = optimal adaptation to environmentenvironment
States, Operators, And States, Operators, And Reasoning (SOAR)Reasoning (SOAR)
http://www.isi.edu/soar/soar.htmlhttp://www.isi.edu/soar/soar.html
States, Operators, And Reasoning States, Operators, And Reasoning (SOAR)(SOAR)
Sequel of General Problem Solver (Newell Sequel of General Problem Solver (Newell and Simon, 1960)and Simon, 1960)
SOAR seeks to apply SOAR seeks to apply operatorsoperators to to statesstates within a within a problem spaceproblem space to achieve a to achieve a goalgoal. .
SOAR assumes that actor uses all SOAR assumes that actor uses all available knowledge in problem-solvingavailable knowledge in problem-solving
Soar as a Unified Theory of Soar as a Unified Theory of CognitionCognition
Intelligence = problem solving + Intelligence = problem solving + learninglearning
Cognition seen as search in problem Cognition seen as search in problem spacesspaces
All knowledge is encoded as productions All knowledge is encoded as productions a single type of knowledgea single type of knowledge
All learning is done by chunking All learning is done by chunking a single type of learninga single type of learning
Young, R.M., Ritter, F., Jones, G. 1998 Young, R.M., Ritter, F., Jones, G. 1998
"Online Psychological Soar Tutorial" "Online Psychological Soar Tutorial"
available at: available at: http://www.psychology.nottingham.ac.uk/staff/http://www.psychology.nottingham.ac.uk/staff/Frank.Ritter/pst/pst-tutorial.htmlFrank.Ritter/pst/pst-tutorial.html
SOAR ActivitySOAR Activity
Operators:Operators: Transform a state via some action Transform a state via some action
State:State: A representation of possible stages of A representation of possible stages of progress in the problemprogress in the problem
Problem space:Problem space: States and operators that can States and operators that can be used to achieve a goal.be used to achieve a goal.
Goal:Goal: Some desired situation. Some desired situation.
SOAR ActivitySOAR Activity Problem solving = applying an Operator to a Problem solving = applying an Operator to a
State in order to move through a Problem Space State in order to move through a Problem Space to reach a Goal. to reach a Goal.
Impasse = Where an Operator cannot be applied Impasse = Where an Operator cannot be applied to a State, and so it is not possible to move to a State, and so it is not possible to move forward in the Problem Space. This becomes a forward in the Problem Space. This becomes a new problem to be solved.new problem to be solved.
Soar can Soar can learnlearn by storing solutions to past by storing solutions to past problems as problems as chunkschunks and applying them when it and applying them when it encounters the same problem againencounters the same problem again
SOAR ArchitectureSOAR ArchitectureChunkingmechanism
Production memory
Pattern ActionPattern ActionPattern Action
Decision procedure
Working memoryManager
Preferences Objects
Conflict stackWorking memory
ExplanationExplanation Working MemoryWorking Memory
– Data for current activity, organized into Data for current activity, organized into objectsobjects
Production MemoryProduction Memory– Contains production rulesContains production rules
Chunking mechanismChunking mechanism– Collapses successful sequences of Collapses successful sequences of
operators into chunks for re-useoperators into chunks for re-use
3 levels in soar3 levels in soar Symbolic – the programming levelSymbolic – the programming level
– Rules programmed into Soar that match Rules programmed into Soar that match circumstances and perform specific actionscircumstances and perform specific actions
Problem space – states & goalsProblem space – states & goals– The set of goals, states, operators, and The set of goals, states, operators, and
context.context. Knowledge – embodied in the rulesKnowledge – embodied in the rules
– The knowledge of how to act on the The knowledge of how to act on the problem/world, how to choose between problem/world, how to choose between different operators, and any learned chunks different operators, and any learned chunks from previous problem solvingfrom previous problem solving
How does it work?How does it work? A problem is encoded as a current state A problem is encoded as a current state
and a desired state (goal)and a desired state (goal) Operators are applied to move from one Operators are applied to move from one
state to anotherstate to another There is success if the desired state There is success if the desired state
matches the current statematches the current state Operators are proposed by productions, Operators are proposed by productions,
with preferences biasing choices in with preferences biasing choices in specific circumstancesspecific circumstances
Productions fire in parallelProductions fire in parallel
ImpassesImpasses If no operator is proposed, or if there is a If no operator is proposed, or if there is a
tie between operators, or if Soar does not tie between operators, or if Soar does not know what to do with an operator, there is know what to do with an operator, there is an impassean impasse
When there are impasses, Soar sets a new When there are impasses, Soar sets a new goal (resolve the impasse) and creates a goal (resolve the impasse) and creates a new statenew state
Impasses may be stackedImpasses may be stacked When one impasse is solved, Soar pops up When one impasse is solved, Soar pops up
to the previous goalto the previous goal
ConclusionsConclusions It may be too "unified"It may be too "unified"
– Single learning mechanismSingle learning mechanism– Single knowledge representationSingle knowledge representation– Uniform problem stateUniform problem state
It does not take neuropsychological evidence It does not take neuropsychological evidence into account (cf. ACT-R)into account (cf. ACT-R)
There may be non-symbolic intelligence, e.g. There may be non-symbolic intelligence, e.g. neural nets etc not abstractable to the neural nets etc not abstractable to the symbolic levelsymbolic level
Comparison of ArchitecturesComparison of Architectures
ACT-RACT-R SOARSOARTypeType HybridHybrid SymbolicSymbolic
TheoryTheory Rational Rational analysisanalysis
Problem Problem solvingsolving
BasisBasis Cog. Psy.Cog. Psy. AIAI
LTMLTM Productions; Productions; factsfacts
ProductionsProductions
WMWM Goal stackGoal stack Working Working memorymemory
LearningLearning YesYes YesYes