Dynamic attention and predictive tracking

Todd S. HorowitzVisual Attention Laboratory

Brigham & Women’s Hospital

Harvard Medical School

Lomonosov Moscow State University Cognitive Seminar, 6/10/2004

lab photo

Jeremy Wolfe

David FencsikGeorge Alvarez

Sarah Klieger

Randy Birnkrant

Jennifer DiMase

Helga Arsenio Linda Tran (not pictured)

Multi-element visual tracking task (MVT)

• Devised by Pylyshyn & Storm (1988)

• Method for studying attention to dynamic objects

Multi-element visual tracking task (MVT)

• Present several (8-10) identical objects

• Cue a subset (4-5) as targets

• All objects move independently for several seconds

• Observers asked to indicate which objects were cued

Demo

mvt4

demo

Interesting facts about MVT

• Can track 4-5 objects (Pylyshyn & Storm, 1988)

• Tracking survives occlusion (Scholl & Pylyshyn, 1999)

• Involves parietal cortex (Culham, et al, 1998)

• “Clues to objecthood” - Scholl

Accounts of MVT performance

• FINSTs (Pylyshyn, 1989)

• Virtual polygons (Yantis, 1992)

• Object files (Kahneman & Treisman, 1984)

• “Object-based attention”

These are all (partially) wrong

• FINSTs (Pylyshyn, 1989)

• Virtual polygons (Yantis, 1992)

• Object files (Kahneman & Treisman, 1984)

• “Object-based attention”

Common assumptions

• Low level (1st order) motion system updates higher-level representation– FINST– Object file– Virtual polygon

• Continuous computation in the present

Overview

• MVT and attention

• Tracking across the gap

• Tracking trajectories

MVT and attention

• Clearly a limited-capacity resource

• Attentional priority to tracked items (Sears & Pylyshyn)

• Hypothesis: MVT is mutually exclusive with other attentional tasks

George Alvarez, Helga Arsenio, Jennifer DiMase, Jeremy Wolfe

MVT and attention

• Clearly a limited-capacity resource

• Attentional priority to tracked items (Sears & Pylyshyn)

• Hypothesis: MVT is mutually exclusive with visual search

MVT and attention

• Clearly a limited-capacity resource

• Attentional priority to tracked items (Sears & Pylyshyn)

• Hypothesis: MVT is mutually exclusive with visual search

• Method: Attentional Operating Characteristic (AOC)

AOC Theory.

% Correct (Search)

% C o r r e c t ( T r a c k i n g )

chanceperformancefor each task

tracking aloneperformance

s e a r c h a l o n ep e r f o r m a n c e

completeindependence

mutuallyexclusive data

fall on this line

General methods - normalization

• Single task = 100

• Chance = 0

• Dual task performance scaled to distance between single task performance and chance

General methods - staircases

• Up step (following error) = 2 x down step

• Asymptote = 66.7% accuracy

• Staircase runs until 20 reversals

• Asymptote computed on last 10 reversals

General methods - tracking

• 10 disks

• 5 disks cued

• Speed = 9°/s

AOC Theory.

% Correct (Search)

% C o r r e c t ( T r a c k i n g )

chanceperformancefor each task

tracking aloneperformance

s e a r c h a l o n ep e r f o r m a n c e

completeindependence

mutuallyexclusive data

fall on this line

AOC reality

• Tasks can interfere at multiple levels

• Interference can occur even when resource of interest (here visual attention) is not shared

• How “independent” are two attention-demanding tasks which do not share visual attention resources?

Gold standard: tracking vs. tone detection

Gold standard method

• Tracking– Duration = 6 s

• Tone duration– 10 600 Hz tones– Onset t = 1 s– ITI = 400 ms– Distractor duration = 200 ms– Task: target tone longer or shorter?– Target duration staircased (31 ms)– Dual task priority varied

N = 10

0 25 50 75 100 1250

25

50

75

100

125

tone accuracy

Gold standard AOC

Tracking + search method

• Tracking– Duration = 5 s

• Search– 2AFC “E” vs. “N”– Distractors = rest of alphabet– Set size = 5– Duration staircased (mean = 156 ms)– Onset = 2 s

N = 9

Tracking + search method

.

E

T

R

B

H

Tracking + search AOC

0 25 50 75 100 1250

25

50

75

100

125

search accuracy

Tracking + search AOC

0 25 50 75 100 1250

25

50

75

100

125visual search tone

tone|search accuracy

Does tracked status matter?

T

L

L

L

T

L

method

• Tracking– Duration = 3 s

• Search– 2AFC left- or right-pointing T– Distractors = rotated Ls– Set size = 5– Duration staircased (mean = 218 ms)– Onset = 1 s

N = 9

search inside tracked set

T

LT

LL

L

L

search outside tracked set

T

LTL

L

L

L

mixed blocked

search inside tracked set

search outside tracked set

inside vs. outside AOC

0 25 50 75 100 1250

25

50

75

100

125 mixed - out

blocked - out

mixed - in

search accuracy

Does spatial separation matter?

E

F

V

H

P

method

• Tracking– Duration = 5 s

• Search– 2AFC “E” vs. “N”– Distractors = rest of alphabet– Set size = 5– Duration = 200 ms– Onset = 2 s

N = 9

spatial separation AOC

0 25 50 75 100 1250

25

50

75

100

125

search accuracy

search v track summary

0 25 50 75 100 1250

25

50

75

100

125

tone|search accuracy

MVT and search

• Clearly not mutually exclusive

• Not pure independence

• Close to gold standard

• MVT and search use independent resources?

Two explanations

• Separate attention mechanisms

• Time sharing

Predictions of time sharing hypothesis

• Should be able to leave tracking task for significant periods with no loss of performance

• Should be able to do something in that interval

Track across the gap method

Track across the gap method

• Track 4 of 8 disks

• Speed = 6°/s

• Blank interval onset = 1, 2, or 3 s

• Trajectory variability: 0°, 15°, 30°, or 45° every 20 ms

• Blank interval duration staircased (dv)

• N = 11

0 10 20 30 40 50

350

400

450

500

550

variability (°)

track across the gap asymptotes

Predictions of time sharing hypothesis

• Should be able to leave tracking task for significant periods with no loss of performance (see also Yin & Thornton, 1999) - confirmed

• Should be able to do something (e.g. search) in that interval

search during gap method

• AOC method• Tracking task same as before• Search task in blank interval

– Target = rotated T– Distractors = rotated Ls– Set size = 8– 4AFC: Report orientation of T

• Duration of search task staircased (326 ms)

0 25 50 75 100 1250

25

50

75

100

125

search accuracy

search during gap AOC

0 25 50 75 100 1250

25

50

75

100

125

tone|search accuracy

Predictions of time sharing hypothesis

• Should be able to leave tracking task for significant periods of time with no loss of performance (see also Yin & Thornton, 1999) - confirmed

• Should be able to do something (e.g. search) in that interval - confirmed

Summary

• MVT and visual search can be performed independently in the same trial

• May support independent “visual attention” mechanisms

• May support time-sharing

Summary

• Tracking across the gap data support time sharing

• Tracking across the gap data raise new questions

What is the mechanism?

• Not a continuous computation in the present

• Not first order motion mechanisms

• Not apparent motion

Randall Birnkrant, Jennifer DiMase, Sarah Klieger, Linda Tran, Jeremy Wolfe

None of these theories fit

• FINSTs (Pylyshyn, 1989)

• Virtual polygons (Yantis, 1992)

• Object files (Kahneman & Treisman, 1984)

What is the mechanism?

• Some sort of amodal perception? (e.g. tracking behind occluders, Scholl & Pylyshyn, 1999)

• … but there are no occlusion cues!

Scholl & Pylyshyn, 1999

Maybe the gap is just an impoverished occlusion stimulus

• No occlusion/disocclusion cues

• Synchronous disappearance

Predictions of impoverished occlusion hypothesis

• Occlusion cues will improve performance

• Asynchronous disappearance will improve performance

Method

• Track for 5 s• Speed = 12°/s• Track 4 of 10 disks• Independent variables (blocked)

– Gap duration:107 ms, 307 ms, 507 ms– Occlusion cues absent, present– Disappearances synchronous, asynchronous

• N = 15

synchronous disappearance

all items reappear simultaneously

items invisible but continue to move

synchronous disappearance + occlusion

occlusion begins

disocclusion begins

Occlusion/Disocclusion

asynchronous disappearance

item reappears

one item at a time disappears but continues to move

asynchronous disappearance + occlusion

... moves while invisible...

... then disoccludes

one item at a time begins to be occluded...

synchronous asynchronous0.75

0.80

0.85

0.90

0.95

1.00 disappearocclude

comparing cue types

Occlusion hypothesis fails

• Occlusion cues don’t help

• Asynchronous disappearance doesn’t help

Method

• Track for 5 s• Speed = 12°/s• Synchronous condition only• Independent variables (blocked)

– Gap duration:107 ms, 307 ms, 507 ms– Occlusion cues absent, present– Track 4, 5, or 6 of 10 disks

• N = 11

comparing cue types

4 5 60.75

0.80

0.85

0.90

0.95

1.00disappearocclude

number of targets

Occlusion hypothesis fails

• Occlusion cues don’t help

• Occlusion cues can actually harm performance

• Asynchronous disappearance doesn’t help

What is the mechanism?

• Not a continuous computation in the present

• Not first order motion mechanisms

• Not apparent motion

• Not amodal perception (occlusion)

How do we reacquire targets?

• remember last location (backward)

• store trajectory (forward)

David Fencsik, Sarah Klieger, Jeremy Wolfe

location-matching account

Memorizedpre-gap targetlocation.

Nearest tomemorizedlocation:identified as target.

First Post-Gap Frame

trajectory-matching account

Memorizedpre-gap targettrajectory.

On target trajectory: identified as target.

First Post-Gap Frame

Shifting post-gap location

d

d

0

Last visible pre-gap location

opposite of expected location

-1

Expected post-gap location

+1

= Stimulus trajectory

shifting post-gap location predictions

-1 0 10.60

0.65

0.70

0.75

0.80locationtrajectory

shift

Shifting post-gap location methods

• track for 5 s

• speed = 8°/s

• track 5 of 10 disks

• gap duration = 300 ms

• post-gap location condition blocked

• stimuli continue to move after gap

shifting post-gap location

-1 0 10.60

0.65

0.70

0.75

0.80

shift

Location vs. trajectory-matching

• support for location-matching– see also Keane & Pylyshyn 2003; 2004

• but advantage for -1 is suspicious

Location vs. trajectory-matching

time +1.0

+time +2.0

++

time +1.5

shift & stop methods

• track for 4-6 s

• speed = 9°/s

• track 2 or 5 of 10 disks

• gap duration = 300 ms

• post-gap location condition blocked

• stimuli stop after gap

moving vs. static after gap

-1 0 10.60

0.65

0.70

0.75

0.80

motion after gap

shift

moving vs. static after gap

-1 0 10.60

0.65

0.70

0.75

0.80

motion after gap

static after gap

shift

2 vs. 5 targets

-1 0 10.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00 5 targets2 targets

shift

Location vs. trajectory-matching

• support for location-matching

• However...– conditions are blocked– observers might see their task not as tracking

across the gap, but learning which condition they’re in

– might not tell us about normal target recovery

Location vs. trajectory-matching

• can subjects use trajectory information?

• always have items move during gap

• vary whether trajectory information is available or not

moving condition

invisible motion

static condition

invisible motion

manipulate pre-gap information methods

• track for 4 s

• speed = 9°/s

• track 1 to 4 of 10 disks

• gap duration = 300 ms

manipulate pre-gap information

0 1 2 3 4 50.75

0.80

0.85

0.90

0.95

1.00

movingstatic

targets

manipulate pre-gap information

0 1 2 3 4 50.75

0.80

0.85

0.90

0.95

1.00

movingstatic

targets

Location vs. trajectory-matching

• observers can use trajectory information

• unlimited (or at least > 4) capacity for locations

• smaller (1 or 2) capacity for trajectories

Conclusions

• Flexible attention system allows rapid switching between MVT and other attention-demanding tasks

• Some representation allows recovery of tracked targets after 300-400 ms gaps

• This representation includes location and trajectory information

Speculation

• MVT reveals two mechanisms, rather than just one

• Frequently (but perhaps not continuously) updated location store

• Attention to trajectories