cogtool a tool for interface design and act-r research
DESCRIPTION
CogTool A tool for interface design and ACT-R research. Bonnie E. John HCI Institute Carnegie Mellon University. CogTool is an open source tool where you can describe an environment in a storyboard…. CogTool is an open source tool where you can describe an environment in a storyboard…. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/1.jpg)
CogToolA tool for interface design and ACT-R research
Bonnie E. JohnHCI Institute
Carnegie Mellon University
![Page 2: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/2.jpg)
CogTool is an open source tool where you can describe an environment in a storyboard…
![Page 3: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/3.jpg)
CogTool is an open source tool where you can describe an environment in a storyboard…
![Page 4: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/4.jpg)
…and demonstrate a task
![Page 5: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/5.jpg)
CogTool automatically creates an ACT-R model of a skilled person doing this task and produces predictions of task execution time.*
* Based on Card, Moran and Newell’s Keystroke-Level Model (1980)
![Page 6: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/6.jpg)
And you can look under thehood to see what ACT-R is doing.
![Page 7: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/7.jpg)
What is prediction of skilled task execution time good for? Some examples of similar analyses
• PERCs “time travel” evaluation at IBM– DARPA set requirement to show 10x productivity improvement over
2002, which we can credibly demonstrate with CogTool• Saved NYNEX from making a $160 million workstation purchase that
would have COST them $2 million/year in operating costs• IRS procurement of new IT system turned on “value” calculated, in part,
using this type of analysis (IBM lost $700 million contract to AT&T’s $1.4 billion)
• NextGen airspace will have economic consequences of time to execute cockpit tasks
• Carlsbad Police reduced injuries and loss of life in their in-vehicle information systems.
– SAE Recommended Practice J2365, Calculation of the Time to Complete In-Vehicle Navigation and Route Guidance Task.
![Page 8: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/8.jpg)
Why should researchers at the ACT-R workshop care?
• If you are not in academia– Your organization may builds systems and this could be directly
useful for evaluating them
• If you are in academia– Consulting
• this may be a quicker way to evaluate new systems than directly coding ACT-R
– Teaching• Human Factors or User Interface Design classes and this is one
technique you could teach your students• Psychology class, part of which may be applied• Want a gentle introduction to cognitive modeling to get students
excited– Research
• Rapid environment construction• “Rapid Theory Prototyping”
![Page 9: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/9.jpg)
• The variability between novice modelers has been reduced by 70%• This makes modeling the least variable of all usability
techniques!!!
First a word about teaching with CogToolKLMs “By-Hand” v. KLMs with CogTool (BRIMS 2010)
By-Hand KLM:Average CV=22%
CogTool KLM:Average CV=7%
![Page 10: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/10.jpg)
Rapid Environment Construction
• Use CogTool’s storyboarding to construct an environment for your ACT-R model• Export the ACT-R code CogTool
creates• Put in your own ACT-R model
![Page 11: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/11.jpg)
Rapid Environment Construction
• Use CogTool’s storyboarding to construct an environment for your ACT-R model• Export the ACT-R code CogTool
creates• Put in your own ACT-R model
• The interface for doing this isn’tas easy as I would like
• The environment isn’t yet a trueACT-R device model
• Anybody who would like to helpand contribute to our opensource code, please contact me
![Page 12: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/12.jpg)
Rapid Theory Prototyping
• Or how I explored 7 theories before breakfast
![Page 13: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/13.jpg)
Rapid UI Prototyping
UI Prototype
Prototypers = UI designers
Far easier, quicker to build than fully-functional UI
Limited to a few specific tasks
Need only be “good enough” to testuser behavior with a proposed UI
“cheating”(e.g., wizard-of-oz)
Sufficient to suggest what’s important enough to begin implementation and what should be given more thought
Tool for thought and communication
![Page 14: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/14.jpg)
Rapid UI Prototyping Rapid Theory Prototyping
UI Prototype
Prototypers = UI designers
Far easier, quicker to build than fully-functional UI
Limited to a few specific tasks
Need only be “good enough” to testuser behavior with a proposed UI
“cheating”(e.g., wizard-of-oz)
Sufficient to suggest what’s important enough to begin implementation and what should be given more thought
Tool for thought and communication
Theory Prototype
Prototypers = Theory developers
Far easier, quicker to build than fully-functional theory
Limited to a few specific tasks
Need only be “good enough” to testtheory’s behavior against human data
“cheating”
Sufficient to suggest what’s important enough to begin implementation andwhat should be given more thought
Tool for thought and communication
![Page 15: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/15.jpg)
General approach to rapid theory prototyping
1. Start with the simplest, most generic model of a theory and ask it to do a task
2. When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledge
3. Repair the the model through rapid theory prototyping
Repeat steps 2-3 until the model is failing only by chance 4. Analyze all the failures and decide:
• Whether prototyped theory or mechanism matches human performance• If so, prioritize effort to improve the original simple theory to validate and
then move into the tool for design
![Page 16: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/16.jpg)
Example of an aviation task
1. Start with the simplest, most generic model of a theory and ask it to do a task
Simplest, most generic model of a theory for exploring an aviation device:Information Foraging Theory (Pirolli & Card, 1999)
Augmented with the Minimal Model of Visual Search (Halverson & Hornof, 2007)
Using a general knowledge corpusEmbodied in CogTool-Explorer (Teo & John, 2008)
A task:A three step procedure to set the approach reference speed and flap angle, using the CDU in a 777.
![Page 17: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/17.jpg)
17
Example Run of CogTool-ExplorerExample Run of CogTool-Explorer
![Page 18: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/18.jpg)
7 theory prototypes in 4 hrsThe final one approaches human behavior (HFES09)
0
20
40
60
80
100
0 50 100 150 200 250
Minutes to prototype the theory
% Success
Total task
P’t
ype-1
P’t
ype-2
P’t
ype-3
P’t
ype-4
P’t
ype-5
P’t
ype-6
P’t
ype-7
Change knowledge (aviation vocabulary)
Elaboration + How it works knowledge
Baseline(simplest,most generic)
![Page 19: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/19.jpg)
Different prototypes target improvement in different steps
0
20
40
60
80
100
0 50 100 150 200 250
Minutes to prototype the theory
% Success
Total task
1st step
2nd step
P’t
ype-1
P’t
ype-2
P’t
ype-3
P’t
ype-4
P’t
ype-5
P’t
ype-6
P’t
ype-7
![Page 20: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/20.jpg)
Example: Step 1
0
20
40
60
80
100
0 50 100 150 200 250
Minutes to prototype the theory
% Success
1st step
P’t
ype-1
P’t
ype-2
P’t
ype-3
P’t
ype-4
P’t
ype-5
P’t
ype-6
P’t
ype-7
![Page 21: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/21.jpg)
Step 1
Given the goal:“select landing flap and reference air speed for a approach”
Hit the INIT REFbutton
![Page 22: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/22.jpg)
Example: Step 1
0
20
40
60
80
100
0 50 100 150 200 250
Minutes to prototype the theory
% Success
1st step
P’t
ype-1
P’t
ype-2
P’t
ype-3
P’t
ype-4
P’t
ype-5
P’t
ype-6
P’t
ype-7
Change knowledge (aviation vocabulary)
Elaboration + How it works knowledge
![Page 23: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/23.jpg)
Example: Step 1, Theory Prototype 4
0
20
40
60
80
100
0 50 100 150 200 250
Minutes to prototype the theory
% Success
1st step
P’t
ype-1
P’t
ype-2
P’t
ype-3
P’t
ype-4
P’t
ype-5
P’t
ype-6
P’t
ype-7
Hierarchical visual regions & recovery
![Page 24: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/24.jpg)
Theory Prototype 4:Hierarchical visual regions + recovery
• We “cheated” by prototyping this theory in the CogTool storyboard, not the ACT-R code
– CogTool runs ACT-R models on a UI description called a storyboard.
• States, which contain widgets (e.g., buttons), and transitions between states represent human actions on widgets (e.g., pressing a button)
• Flat representation of widgets, so visual search and information foraging considers them equally
– Cheat: Prototype visual regions = large “buttons” on the regions• When the correct region is chosen, it transitions to a state with
only the buttons in that region • When an incorrect region is chosen, it transitions to a state
with only the regions that have not been chosen– 1 hour change
![Page 25: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/25.jpg)
Original storyboard for this task:Each state has the CDU’s 69 buttons
• How we “cheated” by prototyping this theory in the CogTool storyboard:
– Prototyping visual regions = large “buttons” on the regions
• When the correct region is chosen, it transitions to a frame with only the buttons in that region
• When an incorrect region is chosen, it transitions to a frame with only the regions that have not been chosen
– 1 hour change
![Page 26: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/26.jpg)
Theory Prototype 4:Hierarchical visual regions
![Page 27: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/27.jpg)
Theory Prototype 4:Prototyped as buttons in storyboard
![Page 28: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/28.jpg)
Theory Prototype 4:Correct region buttons in region
INITREF
RTEDEPARR
ALTN VNAV
FIX LEGS HOLDFMC
COMMPROG
MENUNAVRAD
![Page 29: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/29.jpg)
Theory Prototype 4:Incorrect region other regions
INITREF
RTEDEPARR
ALTN VNAV
FIX LEGS HOLDFMC
COMMPROG
MENUNAVRAD
No green = no cycling
![Page 30: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/30.jpg)
Theory Prototype 4:Tremendous improvement on 1st step
0
20
40
60
80
100
0 50 100 150 200 250
Minutes to prototype the theory
% Success
1st step
P’t
ype-1
P’t
ype-2
P’t
ype-3
P’t
ype-4
P’t
ype-5
P’t
ype-6
P’t
ype-7
Tremendous
improvement
![Page 31: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/31.jpg)
Theory Prototype 4:But still insufficient to complete task
0
20
40
60
80
100
0 50 100 150 200 250
Minutes to prototype the theory
% Success
Total task
1st step
P’t
ype-1
P’t
ype-2
P’t
ype-3
P’t
ype-4
P’t
ype-5
P’t
ype-6
P’t
ype-7
Tremendous
improvement
Not sufficient to do a
multi-step task
![Page 32: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/32.jpg)
General approach to rapid theory prototyping
1. Start with the simplest, most generic model of a theory and ask it to do a taskFlat visual search
2. When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledgeHierarchical visual regions + recovery
3. Give the model this mechanism or knowledge through “rapid theory prototyping”Quick change to CogTool’s storyboard, not ACT-R code
Repeat steps 2-3 until the model is failing only by chance7 iterations, described in the paper
4. Analyze all the failures and decide:
1. Whether prototyped theory matches human performance2. If so, prioritize effort to improve the underlying theory
High priority: Hierarchical visual regions + recovery
![Page 33: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/33.jpg)
General approach to rapid theory prototyping
1. Start with the simplest, most generic model of a theory and ask it to do a taskFlat visual search
2. When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledgeHierarchical visual regions + recovery
3. Give the model this mechanism or knowledge through “rapid theory prototyping”Quick change to CogTool’s storyboard, not ACT-R code
Repeat steps 2-3 until the model is failing only by chance7 iterations, described in the paper
4. Analyze all the failures and decide:
• Whether prototyped theory matches human performance• If so, prioritize effort to improve the underlying theory
![Page 34: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/34.jpg)
General approach to rapid theory prototyping
1. Start with the simplest, most generic model of a theory and ask it to do a taskFlat visual search
2. When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledgeHierarchical visual regions + recovery
3. Give the model this mechanism or knowledge through “rapid theory prototyping”Quick change to CogTool’s storyboard, not ACT-R code
Repeat steps 2-3 until the model is failing only by chance7 iterations, described in the paper
4. Analyze all the failures and decide:
• Whether prototyped theory matches human performance• If so, prioritize effort to improve the underlying theory
![Page 35: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/35.jpg)
General approach to rapid theory prototyping
1. Start with the simplest, most generic model of a theory and ask it to do a taskFlat visual search
2. When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledgeHierarchical visual regions + recovery
3. Give the model this mechanism or knowledge through “rapid theory prototyping”Quick change to CogTool’s storyboard, not ACT-R code
Repeat steps 2-3 until the model is failing only by chance7 iterations, described in the paper
4. Analyze all the failures and decide:
• Whether prototyped theory matches human performance• If so, prioritize effort to improve the underlying theory
![Page 36: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/36.jpg)
General approach to rapid theory prototyping
1. Start with the simplest, most generic model of a theory and ask it to do a taskFlat visual search
2. When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledgeHierarchical visual regions + recovery
3. Give the model this mechanism or knowledge through “rapid theory prototyping”Quick change to CogTool’s storyboard, not ACT-R code
Repeat steps 2-3 until the model is failing only by chance7 iterations, described in the HFES 2009 paper
4. Analyze all the failures and decide:
• Whether prototyped theory matches human performance• If so, prioritize effort to improve the underlying theory
![Page 37: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/37.jpg)
General approach to rapid theory prototyping
1. Start with the simplest, most generic model of a theory and ask it to do a taskFlat visual search
2. When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledgeHierarchical visual regions + recovery
3. Give the model this mechanism or knowledge through “rapid theory prototyping”Quick change to CogTool’s storyboard, not ACT-R code
Repeat steps 2-3 until the model is failing only by chance7 iterations, described in the paper
4. Analyze all the failures and decide:
• Whether prototyped theory matches human performance• If so, prioritize effort to improve the underlying theory
High priority: Hierarchical visual regions + recovery
See Leonghwee Teo’s talk on Sunday
![Page 38: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/38.jpg)
General approach to rapid theory prototyping
1. Start with the simplest, most generic model of a theory and ask it to do a taskFlat visual search
2. When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledgeHierarchical visual regions + recovery
3. Give the model this mechanism or knowledge through “rapid theory prototyping”Quick change to CogTool’s storyboard, not ACT-R code
Repeat steps 2-3 until the model is failing only by chance7 iterations, described in the paper
4. Analyze all the failures and decide:
• Whether prototyped theory matches human performance• If so, prioritize effort to improve the underlying theory
High priority: Hierarchical visual regions + recovery
See Leonghwee Teo’s talk on Sunday
![Page 39: CogTool A tool for interface design and ACT-R research](https://reader035.vdocument.in/reader035/viewer/2022070410/568145db550346895db2dd5d/html5/thumbnails/39.jpg)
General approach to rapid theory prototyping
1. Start with the simplest, most generic model of a theory and ask it to do a taskFlat visual search
2. When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledgeHierarchical visual regions + recovery
3. Give the model this mechanism or knowledge through “rapid theory prototyping”Quick change to CogTool’s storyboard, not ACT-R code
Repeat steps 2-3 until the model is failing only by chance7 iterations, described in the paper
4. Analyze all the failures and decide:
• Whether prototyped theory matches human performance• If so, prioritize effort to improve the underlying theory
High priority: Hierarchical visual regions + recovery
See Leonghwee Teo’s talk on Sunday
Other examples of Rapid Theory Prototyping1. My poster with Tiffany Jastrzembsk for modeling aging aduts (sort of)2. Paper at the ASSETS conference modeling blind users of screen readers