lecture 6: multitasking › ~salvucci › courses › cs630... · the multitasking continuum...

1 CS 630: Cognitive Systems. Dario Salvucci, Drexel University.

Lecture 6: Multitasking

CS 630: Cognitive Systems. Dario Salvucci, Drexel University. 2

Our Multitasking World

talking & driving

cooking & reading a book

writing paper & reading email

watching game & talking to friends

listening & note-taking


The Challenges of Multitasking

■  Technological challenges –  user interfaces, hardware, networking...

■  Scientific challenges –  how do we multitask? –  when is multitasking easy or difficult? –  how does it affect task performance?

■  Societal challenges –  when is multitasking useful? –  when is multitasking inappropriate? –  when is multitasking dangerous?


The Multitasking Continuum

talking & driving






The Multitasking Continuum

Concurrent Multitasking (e.g., PRP, driver distraction)

Sequential Multitasking (e.g., task interruptions)

seconds hours minutes

Time between Task Switches

talking & driving






Threaded Cognition

■  Goal: Unifying theory of multitasking –  ... across the multitasking continuum –  ... across laboratory and real-world domains –  ... across different levels of abstraction

■  Approach: Computational cognitive modeling (obviously J) –  Threaded cognition –  in the ACT-R cognitive architecture


Threaded Cognition

■  Your brain is a “Thought Kitchen” –  with resources and processes

•  central resource: the cook •  other resources: oven, stove, mixer, etc.


Threaded Cognition

■  Concurrent multitasking is a basic skill—best represented by a simple general mechanism

■  Threaded cognition... –  allows concurrent execution of multiple “streams of thought” = threads

–  takes models A, B... predicts behavior of A+B

■  Theoretical components –  (1) Resources that perform relevant processing

•  derived from the ACT-R architecture

–  (2) Processing principles that define task allocation


Processing Principles

(1) Threaded Processing –  Cognition maintains a set of active goals that

produce “threads” of processing •  many domains are nicely represented as threads

–  some are more obvious – e.g., driving + dialing –  some are less obvious – e.g., list-memory tasks

–  In ACT-R terms, this means maintaining multiple goals at a time

•  In the past, ACT-R had only one goal at a time •  then it had a “goal stack” (inspired by tasks with a

robust subgoal structure, like Tower of Hanoi) •  now, several active goals



(2) Resource Exclusivity –  Resources execute processing requests serially,

exclusively for one request/task at a time •  resources can be massively parallel themselves,

within the resource –  e.g., visual processing

•  but resources can serve only one goal at a time

–  (caveat: what about resources like motor? — are the hands independent? fingers? hands from feet? etc.)



(3) Resource Usage –  Threads acquire and release resources in a

greedy, polite manner. •  greedy: used as soon as available •  polite: threads free resources ASAP

(4) Conflict Resolution –  When threads contend for the procedural

resource, the thread with the highest urgency proceeds.

•  highest urgency = least recently used •  simple mechanism for balancing thread processing


Threaded Models

■  Now let’s look at some models that use threaded cognition…


Threads-1

(set-‐task "cs630.ThreadsTask” (add-‐dm (type-‐number isa type-‐number) (answer-‐phone isa answer-‐phone) (goal-‐focus type-‐number) (goal-‐focus answer-‐phone)

(p type-‐number*Bind-‐Birst =goal> isa type-‐number current-‐x nil ?visual-‐location> state free buffer empty ?visual> state free buffer empty ==> +visual-‐location> isa visual-‐location screen-‐x lowest )

(p type-‐number*Bind-‐next =goal> isa type-‐number current-‐x =x ?visual-‐location> state free buffer empty ?visual> state free buffer empty ==> +visual-‐location> isa visual-‐location screen-‐x lowest > screen-‐x =x )


Threads-1

(p type-‐number*encode =goal> isa type-‐number =visual-‐location> isa visual-‐location screen-‐x =x ?visual> state free buffer empty ==> +visual> isa move-‐attention screen-‐pos =visual-‐location =goal> current-‐x =x )

(p type-‐number*type =goal> isa type-‐number =visual> isa text value =digit ?manual> state free ==> +manual> isa press-‐key key =digit )

(p type-‐number*done =goal> isa type-‐number current-‐x =x ?visual-‐location> state error ==> -‐goal> )


Threads-1

(p answer-‐phone*encode =goal> isa answer-‐phone =aural-‐location> isa audio-‐event ?aural> buffer empty state free ==> +aural> isa ring event =aural-‐location )

(p answer-‐phone*done =goal> isa answer-‐phone =aural> isa ring ?vocal> state free ==> +vocal> isa speak string "Hello!" -‐goal> )


Threads-1 0.000 vision unrequested [vision~62]! 0.000 procedural start! 0.050 procedural ** TYPE-NUMBER*ENCODE ** [type-number]! 0.050 vision move-attention! 0.135 vision encoding-complete [text~66]! 0.185 procedural ** TYPE-NUMBER*TYPE ** [type-number]! 0.185 motor press-key "1"! 0.235 procedural ** TYPE-NUMBER*FIND-NEXT ** [type-number]! 0.235 vision find-location [vision~72]! 0.285 procedural ** TYPE-NUMBER*ENCODE ** [type-number]! 0.285 vision move-attention! 0.300 audio audio-event [audio-event~63]! 0.300 audio unrequested [audio-event~63]! 0.350 procedural ** ANSWER-PHONE*ENCODE-SOUND ** [answer-phone]! 0.350 audio attend-sound! 0.370 vision encoding-complete [text~75]! 0.435 motor preparation-complete! 0.485 motor initiation-complete! 0.585 motor output key 1! 0.735 motor finish-movement! 0.785 procedural ** TYPE-NUMBER*TYPE ** [type-number]! 0.785 motor press-key "2"! 0.835 procedural ** TYPE-NUMBER*FIND-NEXT ** [type-number]! 0.835 vision find-location [vision~84]! 0.850 audio audio-encoding-complete [ring~78]! 0.885 procedural ** TYPE-NUMBER*ENCODE ** [type-number]! 0.885 vision move-attention! 0.935 motor preparation-complete!


Threads-1 0.935 procedural ** ANSWER-PHONE*DONE ** [answer-phone]! 0.935 declarative store chunk [answer-phone]! 0.935 speech speak "Hello!"! 0.970 vision encoding-complete [text~87]! 0.985 motor initiation-complete! 1.085 motor output key 2! 1.085 speech preparation-complete! 1.135 speech initiation-complete! 1.135 speech output-speech "Hello!"! 1.235 motor finish-movement! 1.285 procedural ** TYPE-NUMBER*TYPE **! 1.285 motor press-key "3"! 1.335 procedural ** TYPE-NUMBER*FIND-NEXT **! 1.335 vision find-location [vision~95]! 1.385 procedural ** TYPE-NUMBER*ENCODE **! 1.385 vision move-attention! 1.435 speech finish-movement! 1.435 motor preparation-complete! 1.470 vision encoding-complete [text~98]! 1.485 motor initiation-complete! 1.585 motor output key 3! 1.735 motor finish-movement! 1.785 procedural ** TYPE-NUMBER*TYPE **!...!


Threads-2

(set-‐task "cs630.ThreadsTask") (add-‐dm (start isa goal) ) (goal-‐focus start)

(p start-‐multitasking =goal> isa goal ==> +goal> isa type-‐number +goal> isa answer-‐phone )

... < same as before>


Threads-2 0.000 vision unrequested [vision~62]! 0.000 procedural start! 0.050 procedural ** START-MULTITASKING **! 0.050 declarative store chunk [start] (start isa goal)! 0.050 declarative store chunk [type-number~65]! 0.100 procedural ** TYPE-NUMBER*ENCODE ** [type-number~65]! 0.100 vision move-attention! 0.185 vision encoding-complete [text~70]! 0.235 procedural ** TYPE-NUMBER*TYPE ** [type-number~65]! 0.235 motor press-key "1"! 0.285 procedural ** TYPE-NUMBER*FIND-NEXT ** [type-number~65]! 0.285 vision find-location [vision~76]! 0.300 audio audio-event [audio-event~63]! 0.300 audio unrequested [audio-event~63]! 0.335 procedural ** TYPE-NUMBER*ENCODE ** [type-number~65]! 0.335 vision move-attention! 0.385 procedural ** ANSWER-PHONE*ENCODE-SOUND ** [answer-phone~67]! 0.385 audio attend-sound!...! 0.835 procedural ** TYPE-NUMBER*TYPE ** [type-number~65]! 0.835 motor press-key "2"! 0.885 audio audio-encoding-complete [ring~82]! 0.885 procedural ** TYPE-NUMBER*FIND-NEXT ** [type-number~65]! 0.885 vision find-location [vision~88]! 0.935 procedural ** ANSWER-PHONE*DONE ** [answer-phone~67]! 0.935 declarative store chunk [answer-phone~67]! 0.935 speech speak "Hello!"!...!


Threads-3

(set-‐task "cs630.ThreadsTask") (add-‐dm (type-‐number isa type-‐number) ) (goal-‐focus type-‐number) ...

(p handle-‐sound*encode =goal> =aural-‐location> isa audio-‐event ?aural> buffer empty state free ==> +aural> isa ring event =aural-‐location +goal> isa handle-‐sound +goal> =goal )

(p handle-‐sound*phone =goal> isa handle-‐sound =aural> isa ring ?vocal> state free ==> +vocal> isa speak string "Hello!" +goal> isa converse )


Threads-3 0.000 vision unrequested [vision~62]! 0.000 procedural start! 0.050 procedural ** TYPE-NUMBER*ENCODE **! 0.050 vision move-attention! 0.135 vision encoding-complete [text~66]! 0.185 procedural ** TYPE-NUMBER*TYPE **! 0.185 motor press-key "1"! 0.235 procedural ** TYPE-NUMBER*FIND-NEXT **! 0.235 vision find-location [vision~72]! 0.285 procedural ** TYPE-NUMBER*ENCODE **! 0.285 vision move-attention! 0.300 audio audio-event [audio-event~63]! 0.300 audio unrequested [audio-event~63]! 0.350 procedural ** HANDLE-SOUND*ENCODE **! 0.350 declarative store chunk [type-number]! 0.350 audio attend-sound! 0.370 vision encoding-complete [text~75]! 0.435 motor preparation-complete! 0.485 motor initiation-complete! 0.585 motor output key 1! 0.735 motor finish-movement! 0.785 procedural ** TYPE-NUMBER*TYPE ** [type-number~80]! 0.785 motor press-key "2"! 0.835 procedural ** TYPE-NUMBER*FIND-NEXT ** [type-number~80]! 0.835 vision find-location [vision~87]! 0.850 audio audio-encoding-complete [ring~81]! 0.885 procedural ** TYPE-NUMBER*ENCODE ** [type-number~80]! 0.885 vision move-attention!


Threads-3 0.935 motor preparation-complete! 0.935 procedural ** HANDLE-SOUND*PHONE ** [handle-sound~79]! 0.935 declarative store chunk [handle-sound~79]! 0.935 speech speak "Hello!"! 0.970 vision encoding-complete [text~90]! 0.985 motor initiation-complete! 1.085 motor output key 2! 1.085 speech preparation-complete! 1.135 speech initiation-complete! 1.135 speech output-speech "Hello!"! 1.235 motor finish-movement! 1.285 procedural ** TYPE-NUMBER*TYPE ** [type-number~80]! 1.285 motor press-key "3"! 1.335 procedural ** TYPE-NUMBER*FIND-NEXT ** [type-number~80]! 1.335 vision find-location [vision~100]! 1.385 procedural ** TYPE-NUMBER*ENCODE ** [type-number~80]! 1.385 vision move-attention! 1.435 speech finish-movement! 1.435 motor preparation-complete! 1.470 vision encoding-complete [text~103]! 1.485 motor initiation-complete! 1.585 motor output key 3! 1.735 motor finish-movement! 1.785 procedural ** TYPE-NUMBER*TYPE ** [type-number~80]! 1.785 motor press-key "4"!!< “converse” goal is still active and proceeds here... >!


To the Laboratory...

■  Let’s look at threaded cognition in two laboratory tasks: –  tracking & choice –  dual-choice tasks


Tracking & Choice

■  Manual tracking appears in many forms


Tracking

■  In many experiments, it’s much more controlled –  either: try to keep a pointer on a target

–  or: try to keep a cursor within a target range

–  while the movement is generated using a pseudo-random forcing function


Tracking & Choice

■  Experiment: Martin-Emerson & Wickens (1992) –  tracking: keep the cursor in

the target area •  hard vs. easy tracking,

depending on forcing function

–  choice: see an arrow pointing {left, right}, press key to respond

•  arrow separated from target area by offset that varies between 0° and 35° of visual angle

target area

cursor

choice stimulus

offset


Tracking & Choice

■  Experiment: Martin-Emerson & Wickens (1992)


Track.java

public void start (){ processDisplay(); Utilities.shuffle (offsetIndices); offsetIndex = 0; offsetCount = 0; lastArrowTime = 5.0;

addPeriodicUpdate (.020);}

void updateTarget (double time){ double pi2 = 2 * Math.PI; if (easy) { tx = (2.86 * Math.sin (0.00 + (pi2 * (time / 16.670)))) + (1.15 * Math.sin (1.57 + (pi2 * (time / 6.250)))) + (0.57 * Math.sin (3.93 + (pi2 * (time / 9.091))));

ty = (2.29 * Math.sin (0.79 + (pi2 * (time / 8.000)))) + (1.72 * Math.sin (4.72 + (pi2 * (time / 11.110)))) + (1.72 * Math.sin (2.36 + (pi2 * (time / 50.000)))); } else ...

<< move target to (tx,ty) >>}


Track (set-‐task "cs630.Tracking") (sgp :v nil :emma t ) (start-‐hand-‐at-‐mouse) (add-‐dm (track-‐goal isa track) (choice-‐goal isa choice) ) (goal-‐focus track-‐goal) (goal-‐focus choice-‐goal)


Track ;; Tracking Task (p Bind-‐target =goal> isa track ?visual-‐location> state free -‐ buffer requested ?visual> state free buffer empty ==> +visual-‐location> isa visual-‐location kind cross )

(p move-‐to-‐target =goal> isa track =visual-‐location> kind cross ?visual> state free buffer empty ?manual> state free ==> +visual> isa move-‐attention screen-‐pos =visual-‐location +manual> isa move-‐cursor loc =visual-‐location )

(p repeat-‐track =goal> isa track =visual> isa cross ?manual> state free ==> )


Track ;; Choice Task (p Bind-‐arrow =goal> isa choice ?visual-‐location> state free -‐ buffer requested ?visual> state free buffer empty ==> +visual-‐location> isa visual-‐location kind text :attended nil )

(p arrow-‐not-‐found =goal> isa choice ?visual-‐location> state error ==> -‐visual-‐location> ) (p encode-‐arrow =goal> =visual-‐location> kind text ?visual> state free buffer empty ==> +visual> isa move-‐attention screen-‐pos =visual-‐location )

(p respond-‐left =goal> isa choice =visual> isa text value "<" ?manual> state free ==> +manual> isa punch hand left Binger pinkie ) (p respond-‐right ... )


Track 1849.957 procedural ** FIND-ARROW ** [choice-goal]! 1849.957 vision error! 1850.007 procedural ** ARROW-NOT-FOUND ** [choice-goal]! 1850.020 vision unrequested [vision~27824]! 1850.057 procedural ** FIND-TARGET ** [track-goal]! 1850.057 vision find-location [vision~27827]! 1850.107 procedural ** MOVE-TO-TARGET ** [track-goal]! 1850.107 vision move-attention! 1850.107 motor move-cursor vision~27827! 1850.107 motor preparation-complete! 1850.148 vision encoding-complete [cross~27832]! 1850.157 motor initiation-complete! 1850.242 eye preparation-complete [cross~27832]! 1850.315 eye execution-complete [cross~27832]! 1850.529 motor move cursor (255 10)! 1850.579 motor finish-movement! 1850.629 procedural ** REPEAT-TRACK ** [track-goal]! 1850.679 procedural ** FIND-ARROW ** [choice-goal]! 1850.679 vision find-location [vision~27836]! 1850.729 procedural ** ENCODE-ARROW ** [track-goal]! 1850.729 vision move-attention! 1850.864 eye preparation-complete [text~27839]! 1850.995 eye execution-complete [text~27839]! 1851.023 vision encoding-complete [text~27839]! 1851.073 procedural ** RESPOND-LEFT ** [choice-goal]! 1851.073 motor punch left pinkie! 1851.123 procedural ** FIND-TARGET ** [track-goal]! ...!


Track

■  Process timeline


Tracking & Choice

■  Results: Choice response time


Tracking & Choice

■  Results: Tracking error


Dual-Choice Tasks

■  A choice task means –  get a (simple) stimulus –  produce a (simple) response

■  Dual-choice tasks ask a person to do 2 choice tasks at almost the same time

■  Several factors are often varied in experiments using this paradigm –  perceptual modality: visual / aural –  motor modality: manual / vocal –  cognitive difficulty of stimulus à response

mapping


Dual-Choice Tasks

■  Example (with consistent SàR mappings) –  visual-manual task

–  aural-vocal task

O – – – – O – – – – O –

low tone

one

mid tone

two

high tone

three


Dual-Choice Tasks

■  In dual-choice tasks, there’s usually a delay between one stimulus and the other –  called the “SOA” = “stimulus onset asynchrony” –  e.g., if we start the aural-vocal task at time = 0,

we might present the visual-manual task at time = {.000 .050 .150 .250 .500 1.000}

■  Compared to the single-task case, how long will the visual-manual task take when... –  SOA is big? (stimuli far apart)

–  SOA = 0? (concurrent stimuli)

–  somewhere in between?


The PRP Effect

■  Psychological Refractory Period (PRP) effect –  “refractory” from analogy with cells returning

to normal after excitation (not a great analogy, but it stuck)


The PRP Effect

■  What causes the PRP effect? ■  For a long time, it was assumed to be a solid

indicator of a “cognitive bottleneck” ■  A box-diagram depiction:

–  this is the “response-selection” bottleneck –  it’s also a bit misleading...

Is there really an inherent cognitive bottleneck?


Experiments

■  Schumacher et al. (2001), Experiment 1 –  people can achieve perfect time-sharing!


Experiments

■  Schumacher et al. (2001), Experiment 1 –  people can achieve perfect time-sharing! –  table with final results after learning only...

Single-Task Dual-Task

Aural-Vocal 446 ms 456 ms

Visual-Manual 281 ms 283 ms


Experiments

■  Schumacher et al. (2001), Experiment 2 –  PRP effect comes from instructions /

constraints: do Task 1, then do Task 2


Perfect Time-Sharing

■  Perfect time-sharing follows readily from threaded cognition –  in this case, everything works out perfectly; no

interference, no dual-task/PRP effect!


Dual-Choice Tasks

■  An inconsistent SàR mapping requires an extra step to retrieve the mapping

O – – – – – – O


Dual-Choice Tasks

■  An inconsistent SàR mapping requires an extra step to retrieve the mapping –  with possible interference, as here (A)


Dual-Choice Tasks

■  Increased perceptual difficulty makes perception for the tasks run into each other

O O O O O O O O


Dual-Choice Tasks

■  Increased perceptual difficulty makes perception for the tasks run into each other –  with possible interference, as here (B)


Problem State

■  Problem state = temporary information required during task execution –  roughly speaking, a task’s “mental context” –  in ACT-R: stored in the imaginal buffer

■  Example: Solving 3+4 –  encode “3”, then “+”, then “4” –  all this is now held in the problem state /

imaginal buffer –  in this case, used to pass along information for

retrieval –  can also be used to remember new information

•  i.e., associate 3, +, 4, and then 7 with one another


Problem State

■  Example: Writing a paper –  what do you need to keep in mind as you’re

writing a... •  sentence? •  paragraph? •  section?

–  where do you maintain the least amount of information?

■  Example: Tracking, or Driving –  no problem state needed!


Memory & Interruptions

■  Memory clearly plays an important role in interruptions

■  What are the (at least two) important features of human memory? –  information strengthens with use –  information decays over time


Memory for Goals

■  Memory for goals theory (Altmann & Trafton, 2002) + ACT-R memory theory (Anderson et al., 2004) –  to suspend a task, people encode (rehearse) the

current goal until it’s readily available in memory •  in ACT-R, each retrieval boosts a chunk’s activation,

making it easier to recall •  e.g., rehearse “I’m ordering a platypus” a few times

–  to resume the task, people simply recall the goal •  in ACT-R, associated cues can facilitate recall •  e.g., seeing computer, or browser on platypus web page


Memory for Goals

■  Memory for goals as threads...

■  Encoding

■  Retrieval

Primary task Secondary task Primary task


Memory for Goals

■  Memory for goals as threads...

■  Encoding

■  Retrieval

Rehearsal Retrieval Primary task

Secondary task

Primary task


Memory for Goals

■  How many retrievals/rehearsals is a “good” number for a typical interruption?


Interruption Study

■  Monk, Trafton, & Boehm-Davis (2008) –  explored effects of interruption duration &

demand on primary-task resumption

–  primary task: programming a VCR –  interruption duration: 3, 8, or 13 seconds –  interrupting task

•  no-task: just wait •  track: manual tracking task •  n-back: compare current and previous letters (<,>)


Interruption Study

■  Model –  model for primary task

•  full model not needed… •  specified declarative chunk to represent the goal •  estimated time parameter for performing first action

–  models for interrupting tasks •  no-task: trivially waits •  tracking: does the tracking

•  n-back: simplified from previous work (Juvina & Taatgen, 2007)

–  uses declarative resource to retrieve last item!

–  model for interruption process described earlier –  but when exactly should encoding occur?


Interruption Study

■  Encoding strategies –  S1: Encode during the entire interruption –  S2: Encode for n seconds, concurrently with

secondary task –  S3: Encode until retrieval takes no more than n

seconds, concurrently with the secondary task –  S4: Encode for n seconds prior to the

interruption, ending at the onset of the interruption

–  S5: Encode for a few (3) retrievals prior to the interruption, ending at the onset of the interruption


Interruption Study

Monk et al. (2008)


Interruption Study

Monk et al. (2008) Model – S1

(rehearse entire interruption)

no effect of duration


Interruption Study


(rehearse few times before interrupt)

duration effect too large; no n-back interaction


Interruption Study


(rehearse for n sec before interrupt)

no n-back interaction


Interruption Study


(rehearse until retrieval < n sec)

strange n-back interaction (interference forces��too much encoding!)


Interruption Study


(rehearse for n sec after interrupt)

Yes! n-back interaction due to��declarative interference

R2=.94


Interruption Study

■  ACT-R model for S2 –  interleaved with tracking


Interruption Study

■  ACT-R model for S2 –  interleaved with the N-back task


Interruption Study

■  Tracking error –  data: slight effect for 3-sec interruption across

three experiments (albeit not conclusive) –  model: S2 & S3 show this effect due to encoding

■  Bottom line: Memory-intensive interruptions (like N-back) are especially disruptive because they interfere with rehearsal

lecture 6: multitasking › ~salvucci › courses › cs630... · the multitasking continuum...

Documents