reading assignmentilab.usc.edu/classes/2002cs564/lecture_notes/24-reaching-grasping … · •sma...
TRANSCRIPT
1Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
CS564 - Brain Theory and Artificial IntelligenceUniversity of Southern California
Lecture 24.The FARS model of Control of Reaching and Grasping
Reading Assignment:Fagg, A. H., and Arbib, M. A., 1998, Modeling Parietal-Premotor Interactions in Primate Control of Grasping, Neural Networks, 11:1277-1303.
TMB 2.2, 5.3
2Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
Final Exam
Tuesday, December 17, 11:00am – 1:00pm, VKC-100
No books, no questions, work alone, everything seen in class.
3Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
"What" versus "Where →→→→ How"
VisualCortex
ParietalCortex
InferotemporalCortex
How (dorsal)
What (ventral)
reach programming
grasp programminggraspexecution
reachexecution
“What” versus “Where”: Mishkin and Ungerleider“What” versus “How”: Goodale and Milner
4Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
Introducing AIP and F5 (Grasping) in Monkey
F5 - grasp commands inpremotor cortexGiacomo Rizzolatti
AIP - grasp affordancesin parietal cortexHideo Sakata
5Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
The Sakata Protocol
6Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
Grip Selectivity in a Single AIP Cell
A cell that is selective for side opposition (Sakata)
7Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
Differential Timing of Activity Peaks in Different AIP Neurons
Note the need for a broad database of many cells within each region to see that cells are not just “pattern recognizers” but also have a relationship to the time course of the ongoing behavior.
8Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
Size Specificity in a Single AIP Cell
This cell is selective toward small objects, somewhat independent of object type ( Hideo Sakata)
Note: Some cells show size specificity; others do not.
9Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
Grasp Specificity in an F5 Neuron
Precision pinch (top)
Power grasp (bottom)
(Data from Rizzolatti et al.)
10Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
FARS (Fagg-Arbib-Rizzolatti-Sakata) Model Overview
AIP
F5
dorsal/ventral streams
Task Constraints (F6)
Working Memory (46)
Instruction Stimuli (F2)
Task Constraints (F6)Working Memory (46?)Instruction Stimuli (F2)
AIPDorsalStream:Affordances
IT
VentralStream:Recognition
Ways to grab this “thing”
“It’s a mug”PFC
AIP extracts the set of affordances for an attended object.These affordances highlight the features of the object relevant to physical interaction with it.
11Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
Secondary Somatosensory Cortex (SII)
In the grasp versus point comparisonin a PET study of humans, we found a marked increase of activity in the secondary somatosensory cortex (SII).
Ablation of SII in non-human primates results in decrements in tactile discrimination and impaired tactile learning.
Focal lesions of the parietal operculum in humans characteristically produce tactile agnosia without loss of simple tactile sensation, or motor control. This deficit can include the inability to sort objects based on size or shape, although sorting on texture is preserved.
The model relates the augmented response to higher order tactile feedback or tactile expectation.
12Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
Motor Commands, Expectations, and Feeedback
F5(grasp type)
MI
hand
(muscle assemblies)
(elementary sensory features)
SI
SII
expectation
motor commands
sensory info
(sensory hyperfeatures)
A7(internal model)
13Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
Interaction of AIP and F5 During the Sakata Task
Activation Connection
Inhibitory Connection
Priming Connection
AIP precision-related cell
AIP power-related cell
set extension flexion hold release
Precision Pinch
F5
AIP
Go Signal 2nd Go Signalcontact with object
maximum aperture reached
visual
motor
Visual Inputs
14Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
The Problem of Serial Order in Behavior (Karl Lashley)
If we tried to learn a sequence likeA → B → A → Cby reflex chaining, what is to stop A triggering B every time, to yield the performance A → B → A → B → A → ….. (or we might get A → B+C → A → B+C → A → …..)A solution:Store the “action codes” (motor schemas) A, B, C, … in one part of the brain (F5 in FARS) and have another area (pre-SMA in FARS) hold “abstract sequences” and learn to pair the right action with each element:
(pre-SMA): x1 → x2 → x3 → x4 abstract sequence
(F5): A B C action codes/motor schemas
Hypothesis: The “Sakata-Protocol Sequencing” is not mediated within F5 --Sequences are stored in pre-SMA and administered by the Basal Ganglia (BG)
15Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
The “Visual Front End” of the FARS Model
Visual Cortex
Parietal Cortex
VIP
PIPAIP
F4
How (dorsal)
IT
What (ventral)
(position)
(shape, size, orientation)
(object/grasp transform)
F5
(grasp type)
(arm goal position)
16Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
PIP →→→→ IT connections are hard-wired for a simple set of objects
Cylinder Box Sphere
PIP
general
diameter
length
general
width
length
general
diameter
height
IT
Cylinder Box Sphere
PIP
general
diameter
length
general
width
length
general
diameter
height
IT
Cylinder Box Sphere
PIP
general
diameter
length
general
width
length
general
diameter
height
IT
Note the use of coarse coding
17Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
Coarse Coding/Population Coding
To code some variable x lying in an interval [a,b) we could take n cells, with cell i (i = 0, …, n-1) firing if and only if the current value of x lies in the ith subinterval
In coarse coding, we achieve much greater discrimination by taking into account the continuously varying firing level fi of each cell, and then we can decode values of x actually varying across each interval, using some such formula as
��
���
� +−+−+n
iaban
iaba )1)((,)(
�
�−
=
−
=��
���
�
−−+
1
0i
1
0i
f
1f
n
i
n
ii
naba
18Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
PIP →→→→ AIP connections are hard-wired for a simple set of affordances
Cylinder
general
diameter
length
narrow wide
short long
precisionprecision aperture = 20mm
precision precision aperture = 60mm
PIP
AIP
A B C D
19Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
IT →→→→ AIP
The mapping from object identity in IT to maps directly to both the grasp type and the aperture of grasp in AIP when the nature of the object implies such data:
E.g., in the case of AT, the projection from IT can provide the necessary grasp type and parameters for a lipstick but not for a cylinder.
precisionprecision aperture = 20mm
precision precision aperture = 60mm
IT
AIP
A B C D
narrow cylinder (bottle cap) wide cylinder (jar top)
A bottle cap activates a precision grasp with a narrow aperture.A jar top maps to a precision grasp with a wide aperture.
20Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
PIP →→→→ IT connections are hard-wired for a simple set of objects
Cylinder Box Sphere
PIP
general
diameter
length
general
width
length
general
diameter
height
IT
Cylinder Box Sphere
PIP
general
diameter
length
general
width
length
general
diameter
height
IT
Cylinder Box Sphere
PIP
general
diameter
length
general
width
length
general
diameter
height
IT
Note the use of coarse coding
21Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
F5 activity during execution of a precision graspThe top two traces show the position of the thumb and index finger.Left: The next five traces represent the average firing rate of five F5 neurons (set-, extension-, flexion-, hold-, and release-related). The remaining five traces represent the various external (Ready, Go, Go2) and internal (SII) triggering signals.
Right: Illustrating the temporally distributed coding of F5 cells.
22Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
Positioning F2, F6 and Areas 46 and SII in Monkey
23Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
Prefrontal Influences on F5
F4
Inferior Premotor
Cortex
F5(grasp type)
F6
46
(arm goal position)
F2 (abstract stimuli)
pre-SMA
Dorsal premotor cortex
Frontal Cortex
24Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
Grasp Selection in F5
Within F5, the active grasps compete through a winner-take-all
Area 46, working in conjunction with F6, supplies task-dependent biases for grasp selection in F5, based upon
• task requirements (such as what is going to be done after the grasp), or
• a working memory of a recently executed grasp.
The biasing can • be on the class of grasp (e.g. power versus precision), or • include the parameters of the grasp (e.g. width of aperture) mechanism which incorporates any biases that might be received from area 46.
25Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
Supplementary Motor Area (SMA) in Monkey
• a unilateral lesion of the SMA disrupts the monkey's ability to allocate his hands to different subtasks of a bimanual task (Brinkman, 1984)
• SMA is involved in the temporal organization of complex movements (Tanji, 1994).
Luppino, Matelli, Camarda, & Rizzolatti subdivide SMA:
SMA-proper (F3; the caudal region) has heavy projections to the limb regions of F1 and related portions of the spinal cord.
F6 (pre-SMA) does not project to the spinal cord, and has only moderate projections to areas F3 and F2 (the dorsal premotor cortex), but has a very heavy projection to area F5.
26Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
Pre-SMA
F6 (pre-SMA) has a very heavy projection to area F5. Inputs into area F6 include VIP, and area 46.F6 contains neurons that become active when an object that the monkey is about to grasp moves from being out of reach into the peripersonal space of the monkey
Interpretation: this class of pre-SMA (F6) neuron is responsible for generating a go signal when it is appropriate for the monkey to begin a reaching movement.
F4
Inferior Premotor Cortex
F5(grasp type)
F6
46
(arm goal position)
F2 (abstract stimuli)
FrontalCortex
pre-SMA
Dorsal premotor cortex
27Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
Area 46
Area 46 has been implicated as a working memoryin tasks requiring information to be held during a delay period (Quintana & Fuster, 1993).
This memory has been posited to participate in learning tasks involving complex sequences of movements (Dominey, 1995).
Area 46 projects to F6, and also exchanges connections with area F5 (Luppino, et al., 1990; Matelli, 1994).
When a human is asked to imagine herself grasping an object, activated areas (PET or fMRI) involved include:
• Area 46 (Decety, Perani, Jeannerod, Bettinardi, Tadary, Woods, 1994) • Area 44 (a possible F5 homologue)• A site along the intra-parietal sulcus (Grafton, et al., 1996).
28Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
Role of Area 46 for Grasping in the Dark
During the performance of the task in the light, area 46 maintains a memory of those F5 cells that participate in the grasp.
To simulate performance in the dark, PIP and IT are then cleared.If a new trial is initiated soon enough, area 46 provides positive support to those F5 cells that were active during the first trial.
The area 46 working memory provides a static description of the grasp that was recently executed, in the sense that the temporal aspects of the grasp are not stored - only a memory of those units that were active at some time during the execution.
[Area 46 is involved in human imagination of grasp execution.]
F4
Inferior Premotor Cortex
F5(grasp type)
F6
46
(arm goal position)
F2 (abstract stimuli)
FrontalCortex
pre-SMA
Dorsal premotor cortex
29Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
Conditional Tasks and Area F2
Dorsal premotor cortex (F2) is thought to be responsible for the association of arbitrary stimuli (an IS) with the preparation of motor programs (Evarts, et al., 1984; Kurata & Wise, 1988; Mitz, Godshalk, & Wise, 1991; Wise & Mauritz, 1985).
In a task in which a monkey must respond to the display of a pattern with a particular movement of a joystick:
• some F2 neurons respond to the sensory-specific qualities of the input. However, • many F2 units respond in a way that is more related to the motor set that must be
prepared in response to the stimulus. When a muscimol lesion in this region is induced, the monkey loses the ability to correctly make the arbitrary association.
F4
F5(grasp type)
F6
46
(arm goal position)
F2 (abstract stimuli)
pre-SMA
Dorsal premotor cortex
In addition to simulating the Sakata task, we simulate conditional tasks.
30Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
The Complete FARS Model
Visual Cortex
Parietal Cortex
VIP
PIPAIP
F4
How (dorsal)
ITWhat (ventral)
Inferior Premotor
Cortex(position)
(shape, size, orientation)
(object/grasp transform)
F5(grasp type)
MI
hand
(muscle assemblies)
(elementary sensory features)
F6
46
SI
SII
expectation
motor commands
sensory info
(arm goal position)
(sensory hyperfeatures)
F2 (abstract stimuli)
A7(internal model)
31Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
To complete your study of this material
Study the paper:
Fagg, A. H., and Arbib, M. A., 1998, Modeling Parietal-Premotor Interactions in Primate Control of Grasping, Neural Networks, 11:1277-1303.
to ensure that you understand the supplementary slide-set:
18+. FARS Model 1 [Spares] (2000)
32Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
F5 activity during execution of a precision graspThe top two traces show the position of the thumb and index finger.Left: The next five traces represent the average firing rate of five F5 neurons (set-, extension-, flexion-, hold-, and release-related). The remaining five traces represent the various external (Ready, Go, Go2) and internal (SII) triggering signals.
Right: Illustrating the temporally distributed coding of F5 cells.
33Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
The Complete FARS Model
Visual Cortex
Parietal Cortex
VIP
PIPAIP
F4
How (dorsal)
ITWhat (ventral)
Inferior Premotor
Cortex(position)
(shape, size, orientation)
(object/grasp transform)
F5(grasp type)
MI
hand
(muscle assemblies)
(elementary sensory features)
F6
46
SI
SII
expectation
motor commands
sensory info
(arm goal position)
(sensory hyperfeatures)
F2 (abstract stimuli)
A7(internal model)
34Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
Thumb and index finger temporal behavior as a function of cylinder size
35Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
F5 cell responses during precision grasps of seven different apertures
This particular cell is active only for narrow precision pinches
36Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
F5 movement-related cell (A) and a hold-related (B) cell during the perturbation experiment
20mm/30mm traces correspond to presentation and grasping of a 20mm and a 30mm cylinder, respectively; traces labeled 20→30 and 30 → 20 indicate perturbation trials, in which a 20mm cylinder is switched for a 30mm cylinder, and a 30mm cylinder for a 20mm one, respectively.
37Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
Two objects that map to the identical grasp
38Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
Comparison of population responses towards two different objects (but identical grasps).
Lighted movement task, AIP (A) and F5 (B) cells; and
AIP populations during fixation (C) and dark movement (D) tasks.
39Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
F5 Feedback to AIP
Visual-related AIP receive object-specific inputs; motor-related cells receive recurrent inputs from F5, which do not demonstrate object-specific activity.
40Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
A single object mapping to two possible grasps
Before execution, one grasp must be selected based upon the current context (e.g., based upon an Instruction Stimulus).
41Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
Two F5 units (A, B) in response to the four conditions: (c,pr), (c,pw), (nc,pr), and (nc,pw). c = conditional; nc = non-conditional; pr = precision pinch; pw = power grasp.
42Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
Four boxes of different dimensions
Grasping is performed along the horizontal axis. The two blocks in the left column are grasped using a precision pinch of a 10mm aperture; the blocks on the right require a 20mm aperture.
43Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
Comparison of AIP visual responses for objects of the same (A) and different (B) widths; and AIP motor-related responses (dark movement condition) for objects of same (C) and different (D) widths.