a computational model of the reviewing of object-files
DESCRIPTION
A computational model of the reviewing of object-files. Michael Liddle Alistair Knott Anthony Robins. Introduction. Selective visual attention. Object-files and the object-specific advantage (OSA). Computational modeling of cortical vision. - PowerPoint PPT PresentationTRANSCRIPT
1
A computational model of the reviewing of object-files
Michael Liddle
Alistair Knott
Anthony Robins
2
Introduction
• Selective visual attention.
• Object-files and the object-specific advantage (OSA).
• Computational modeling of cortical vision.
• A neural network model of object-file reviewing (first of its kind?)
3
Selective visual attention
4
Managing limited resources
• Retinal image contains an enormous amount of information.
• Processing complexity subject to combinatorial explosion.
• Solution: only bother processing information about one object at a time.
5
Explaining the solution
• What actually happens in the brain when we “attend” to an object?
• Experiments indicate that attention is the means by which feature “conjunction” and “binding” occurs (Treisman & Gelade, 1980).
• What is the medium of this binding? Object-files!
6
Object-files and the object-specific advantage
7
Object-files
• Kahneman & Treisman (1984)
• Provide stable repositories for visual information about four or five objects.
• Maintain identity and continuity of objects during a perceptual episode.
• Analogy: police files for investigations.
8
Object-files
• When attending to an object for the first time: “open” an object-file.
• When reattending to an object: “review” the information in its object-file.
• Reviewing involves reconciling old information with new.
9
The object-specific advantage
• Evidence for a object-specific type of priming (Kahneman, Treisman, & Gibbs, 1992), linked to object-file reviewing.
• Facilitation for perceptually coherent objects, greater than general priming.
• Suggestion is that previous perception of an object allows stored information to speed recognition.
10
Example: Preview
V
Q
11
Example: Linking
12
Example: SO condition
V
13
Example: DO condition
Q
14
Example: NM condition
S
15
Recognition times
450
460
470
480
490
500
510
520
RT (ms)
SO
DONM
16
Computational models of cortical vision
17
Providing a foundation
• Object-files must exist at a relatively high level of visual perception.
• Important to consider both current thought about neurology of visual attention, as well as existing computational models.
18
Models of object detection and recognition
• Models of detection:– Retinotopic maps of salient regions
(saliency maps).– Guide attentional processes.
• Models of recognition:– Hierarchical structures (increasing
selectivity/receptive field size)– Output encoding of feature conjunctions.
19
A neural network model of object-file reviewing
20
Neural network modeling
• Connect collection of simple “neuron-like” components via weighted “synapse-like” components.
• Basic neuron sums its inputs, and applies an “activation-function” to determine output.
• Output is interpreted as a firing rate.
21
Modeling the object-specific advantage
• Need a recognition procedure that can be subject to facilitation (i.e. involves a time course).
• Need to store “bottom-up stimulus” information in an object-specific way.
• Need to provide “top-down expectation” based on stored information for currently attended object.
22
Modeling the OSA
• Correct expectation should lead to facilitation.
• Incorrect expectation should not destroy general priming.
23
Modeling facilitation
• Use type based classification: when type is known, recognition is complete.
• Enforce single winning type by lateral competition.
• Winner is called the “stimulus type”.
• Enhance time factor by using “cascaded activation” neurons.
24
Modeling facilitation
V
Q
S
J
Hierarchical feature encoderType layer
V
25
Modeling facilitation
V
Q
S
J
Hierarchical feature encoderType layer
V
26
Modeling facilitation
V
Q
S
J
Hierarchical feature encoderType layer
V
27
Modeling facilitation
V
Q
S
J
Hierarchical feature encoderType layer
V
Recognised
28
Storing stimulus: object specificity
• FINSTs (Fingers of INSTantiation) identify “proto-objects” in the scene (Pylyshyn, 1989)
• Track their proto-objects as they move and change size/shape.
• Set of four or five FINSTs constantly assigned/reassigned from saliency map
• Provide candidates for attention.
29
Storing stimulus: object specificity
• Associate a neuron with each FINST.• Selecting a FINST for attention
activates its neuron.• Associate stimulus type with current
FINST.• Thus a level of indirection is introduced
between retinal location and mental representation.
30
Storing stimulus: object specificity
V
Q
S
J V
Q
Association“stuff”
31
Storing stimulus: object specificity
V
Q
S
J V
Q
Association“stuff”
32
Association stuff?
V
Q
S
J
33
Association stuff
FINST
Feedforward
Feedback
*Excitatory connections shown only
Type
V
Q
S
J
“Object-file”
34
Storing stimulus: feedback “stuff”
FINST
Feedforward
Feedback
*Excitatory connections shown only
Type
V
Q
S
J
“Object-file”
V
Q
35
Storing stimulus: feedback “stuff”
FINST
Feedforward
Feedback
*Excitatory connections shown only
Type
V
Q
S
J
“Object-file”
V
Q
36
Providing expectations: feedforward “stuff”
FINST
Feedforward
Feedback
*Excitatory connections shown only
Type
V
Q
S
J
“Object-file”
V
Q
37
FINST
Feedforward
Feedback
*Excitatory connections shown only
Type
V
Q
S
J
“Object-file”
V
Q
Providing expectations: feedforward “stuff”
38
Storing stimulus:opening an object-file
39
Storing stimulus
FINST
Feedforward
Feedback
*Excitatory connections shown only
Type
V
Q
S
J
“Object-file”
V
Q
40
Storing stimulus
FINST
Feedforward
Feedback
*Excitatory connections shown only
“Object-file”
V
Q
Type
V
Q
S
J
41
Storing stimulus
FINST
Feedforward
Feedback
*Excitatory connections shown only
“Object-file”
V
Q
Type
V
Q
S
J
42
Storing stimulus
FINST
Feedforward
Feedback
*Excitatory connections shown only
“Object-file”
V
Q
Type
V
Q
S
J
43
Storing stimulus
FINST
Feedforward
Feedback
*Excitatory connections shown only
“Object-file”
V
Q
Type
V
Q
S
JRecognised
44
Storing stimulus
FINST
Feedforward
Feedback
*Excitatory connections shown only
“Object-file”
V
Q
Type
V
Q
S
JRecognised
45
Storing stimulus
FINST
Feedforward
Feedback
*Excitatory connections shown only
“Object-file”
V
Q
Type
V
Q
S
JRecognised
Stored
46
Providing correct expectation:
the SO condition
47
Providing correct expectation (SO)
FINST
Feedforward
Feedback
*Excitatory connections shown only
Type
V
Q
S
J
“Object-file”
V
48
Providing correct expectation (SO)
FINST
Feedforward
Feedback
*Excitatory connections shown only
“Object-file”
V
Type
V
Q
S
J
49
Providing correct expectation (SO)
FINST
Feedforward
Feedback
*Excitatory connections shown only
“Object-file”
V
Type
V
Q
S
J
50
Providing correct expectation (SO)
FINST
Feedforward
Feedback
*Excitatory connections shown only
“Object-file”
V
Type
V
Q
S
JRecognised
51
Providing incorrect expectation:
the DO and NM conditions
52
Providing incorrect expectation (DO/NM)
FINST
Feedforward
Feedback
*Excitatory connections shown only
Type
V
Q
S
J
“Object-file”
Q
53
Providing incorrect expectation (DO/NM)
FINST
Feedforward
Feedback
*Excitatory connections shown only
“Object-file”
Q
Type
V
Q
S
J
54
Providing incorrect expectation (DO/NM)
FINST
Feedforward
Feedback
*Excitatory connections shown only
“Object-file”
Q
Type
V
Q
S
J
55
Providing incorrect expectation (DO/NM)
FINST
Feedforward
Feedback
*Excitatory connections shown only
“Object-file”
Q
Type
V
Q
S
J
56
Correcting incorrect expectation (DO/NM)
FINST
Feedforward
Feedback
*Excitatory connections shown only
“Object-file”
Q
Type
V
Q
S
J
57
Correcting incorrect expectation (DO/NM)
FINST
Feedforward
Feedback
*Excitatory connections shown only
“Object-file”
Q
Type
V
Q
S
J
58
Correcting incorrect expectation (DO/NM)
FINST
Feedforward
Feedback
*Excitatory connections shown only
“Object-file”
Q
Type
V
Q
S
J
59
Correcting incorrect expectation (DO/NM)
FINST
Feedforward
Feedback
*Excitatory connections shown only
“Object-file”
Q
Type
V
Q
S
JRecognised
60
Correcting incorrect expectation (DO/NM)
FINST
Feedforward
Feedback
*Excitatory connections shown only
“Object-file”
Q
Type
V
Q
S
JRecognised
Corrected
61
Preserving general priming:the DO condition
62
Preserving general priming (DO)
V
Q
S
J
63
Recognition times: model
RT
SO
DO
NM
64
Recognition times: empirical
RT
SO
DONM
65
Results
• Got the right order of time courses:– SO < DO < NM
• Got close to the right proportions for time courses: – SO << DO < NM
• Thus:– OSA implemented.– Standard priming retained.
66
Side effects (predictions?)
• Recognition is slowed in DO and NM conditions (interference).
• Corrections are persistent (persistence).• Corrections lost if not given enough time
(masking).• Unassigned FINSTs take on attended
type when attention to one object is prolonged.
67
More to be done…
• Move away from types towards feature-encoding (requiring recognition to be done at a higher level).
• Implement a biologically plausible FINST module on top of a saliency map.
• Combine with hierarchical feature encoding models.
68
The end
Got any questions?
69
References
• Itti, L., & Koch, C. (2001). Computational modeling of visual attention. Nature Reviews Neuroscience, 2(3), 94-203.
• Pylyshyn, Z. (1989). The role of location indexes in spatial attention: a sketch of the FINST spatial-index model. Cognition, 32, 65-97.
70
References
• Kahneman, D., & Treisman, A. (1984). Changing view of attention and automaticity. In R. Parasuraman, & D. Davies (Eds.), Varieties of attention (pp. 29-61). New York: Academic Press.
• Kahneman, D., Treisman, A., & Gibbs. B. (1992). The reviewing of object files: object-specific integration of information. Cognitive Psychology, 24, 174-219.
71
References
• Riesenhuber, M., & Poggio, T. (1999). Hierarchical models of object recognition in cortex. Nature Neuroscience, 2(11), 1019-1025.
• Treisman, A., & Gelade, G. (1980). A feature-integration theory of attention. Cognitive Psychology, 12, 97-136.
72
The REAL end!
73
This slide intentionally left blank
74
Variations
• Experiments involving more preview objects:– Object-specific advantage decreased for more than four
objects.– Corroborates limited number claim.
• Experiments involving moving frames in linking display:– Results consistent with “static” conditions.– Suggests Object-files associated with objects not locations.
75
Storing stimulus: association “stuff”
• For each FINST neuron:– Feedback layer: raised threshold neurons forcefully transmit
stimulus type to object-file layer. – Object-file layer: self-stabilising (competitive) neurons
“remember” stimulus type for the currently attended FINST.
• Winner in object-file layer called “stored type”.• Type only stored in attended FINST’s object-file.
76
Providing expectations: more association “stuff”
• For each FINST neuron:– Feedforward layer: raised threshold
neurons transmit stored type back to type layer.
• Expectation only provided from attended FINST’s object-file.
77
Side effects (predictions?)
• What: recognition is slowed in DO and NM conditions.
• Why: interference in the type layer due to incorrect expectation.
• Test: compare recognition times for non-previewed objects.
• Conclusion: seems a plausible “surprise” effect.
78
Side effects (predictions?)
• What: corrections are persistent.• Why: the expectation provided for an
object is the currently stored type.• Test: an OSA should exist for the most
recently observed type only.• Conclusion: seems a plausible
prediction if object-file theory is accurate.
79
Side effects (predictions?)
• What: correction is not made if attention to a changed object is too short.
• Why: correction is not instantaneous, corrected feedback is required.
• Test: OSA should only be “updated” is change was seen for long enough.
• Conclusion: seems a plausible object level analogue to masking effects in iconic memory.
80
Side effects (predictions?)
• What: prolonged attention to an object causes unassigned FINSTs to store current type.
• Why: unassigned FINSTs are very plastic because they have no stored type.
• Test: not really testable.• Conclusion: probably a problem with the
current implementation.