![Page 1: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/1.jpg)
Premotor cortex, action control, and language
Arthur M. Glenberg
University of Wisconsin-Madison
Ago, ergo Cogito
![Page 2: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/2.jpg)
Action control is amazinghitting a tennis ball playing piano walking down steps
Ago, ergo Cogito
hitting a tennis ball, playing piano, walking down steps
coordination among muscles - hierarchical controlover serial order
continuous recalibration
exquisitely precise timingexquisitely precise timing
slow sensory feedback and irrelevant feedback
Evolution and the brain have done a wonderful job solving a set ofvery tricky problems.
Might those solutions be used to solve other tricky problems?
![Page 3: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/3.jpg)
Today’s talk
Ago, ergo CogitoA solution to action control problems: controller/predictor models of action control, Wolpert’s HMOSAIC t ti i tHMOSAIC as a starting point.
Relating HMOSAIC to the brain: Canonical and Mirror Neurons
A marvelous confluence: The HMOSAIC model of action control,with minimal changes, can be a model of language learning,
d ti d h iproduction, and comprehension.
Data from several research projects testing the claim that languagp j g g gcomprehension is intimately related to action control.
![Page 4: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/4.jpg)
Wolpert’s MOSAIC
feedbackAgo, ergo Cogito
lift 150 g
socialenvironmentpredictorlift 500 g
lift 350 g
GainControl
Predictions &probabilities
environment
body/perceptual
predictorsensory/envpredictor
perceptualstates
h i l
controllercontroller +
physicalenvironment
efference copypypredictor
faster than feedbackselects relevant feedback (attention)
feedback & errorlearning predictor selects relevant feedback (attention)
helps learning controllerenhances perception
glearning controllerlearning responsibilitie
![Page 5: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/5.jpg)
drink
feedback: move eyesfeedback: extendHMOSAIC
Ago, ergo Cogitosocialenvironment
sensory/envpredictor
drink
Gain<post<post.
body/perceptualcontroller
predictor
Predictions &probabilities
GainControleyes,…>
i
extend,…>
perceptualstates
controller <prioreyes,…><priorextend,…>
physicalenvironment
+efference copyeyesgraspextend
efference copy
t d grasp lifteyes extendbasic motor repertoire; motor cortex:
![Page 6: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/6.jpg)
feedback
HMOSAIC and language
Ago, ergo Cogito
i ldi t
drink
feedback
socialenvironment
predictor
Predictions &
GainControl
body/perceptual
states
controller
Predictions &probabilities
physicalenvironment
+grasp
efference copy
canonical neuronsmirror neurons
![Page 7: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/7.jpg)
canonical neurons
Ago, ergo Cogitomirror neurons
purposeaction recognitionintent/theory of mindenhance perceptionenhance perceptioncommunication?
Human datafMRI, TMS,
behaviorBroca’s & handBroca s & handBuccino, Tettamanti,Aziz-Zadeh, more
Fadiga et al., speech MNFrom, Rizzolatti & Arbib, 1998
![Page 8: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/8.jpg)
feedback
HMOASIC and languageBroca’s area: speech and hand; Fadiga et al. speech MN
Ago, ergo Cogito
i ldi t
drink
feedback
/ “drink”
socialenvironment
predictor
Predictions &
GainControl
speech
body/perceptual
states
controller
Predictions &probabilities
hBroca’s area
physical
speech
bi directional assoc
control over serialorder in speech
environment
+grasp /dr/bi-directional assocof speech and actio
gesture
efference copy
canonical neuronsmirror neurons
grounding of meaning
![Page 9: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/9.jpg)
feedbackLearning nouns
socialenvironment
predictor
bottle / “bottle”Ago, ergo Cogito
environment
body/
speechPredictions &probabilities
GainControl
body/perceptual
states
controller
speech
p
Assume child knows how to interact with a
physicalenvironment
how to interact with a bottle: move eyes, grasp
Joint attention: eyes+ Speech mirror neuron
graspeyes
Canonical neurons
efference copyphonology & meaning ->
a construction
![Page 10: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/10.jpg)
feedback
Learning verbs
Ago, ergo Cogito
i ldi t
drink / “drink”
socialenvironment
predictor
speechPredictions &
GainControl
body/perceptual
states
controller
h
Predictions &probabilities
physical
speech
Assume childknows how to drinkenvironment
+Auditory mirror&
grasp /dr/
knows how to drink
efference copy
&Action mirror
![Page 11: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/11.jpg)
Dad says, “Give Dada the cup.”
feedback
Ago, ergo Cogito
di t
give cup / “give”social
predictor
speechPredictions &
GainControl
environment
controller
h
probabilities body/perceptual
statesAssume child knows
“cup” ->
speech
physicalenvironment
how to give a cup
peyesgrasp
Action
releaseextendgrasp +Action
efference copy
![Page 12: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/12.jpg)
give5/“give”i 4/“ i ” 3 “Give Lavinnia the candy ”
5 “Give Dada the spoon”4 “Give Dada the candy”
In other words: Behavior consistent with having learned an abstract, double-object construction emerges from integrating (by responsibilities)give4/“give”
give3/“give”give2/“give”
Ago, ergo Cogito2 “Give Momma the bottle.”3 Give Lavinnia the candy.
1 “Give Dada the cup.”feedback
emerges from integrating (by responsibilities) actions learned for specific, concrete situations. Syntax (and compositionality) emerges from
ti t lgive2/ give
give1/“give”Learning speech/action paramepredictor
feedback
socialenvironment
action control.
Attend with eyes to speaker
Attend with eyes to object,controller
speech Predictions &probabilities
GainControl body/
perceptualstates
Attend with eyes to speaker
Attend with eyes to object, set kinematics, graspspeech
states
physicalenvironment
+
Now “Give Dada the bottle”
y pspeaker
object
graspExtend arm to speaker
+
controller
predictorspeech
bottle / “bottle”
Now, Give Dada the bottlespeaker
act
controllerspeech
graspeyes
![Page 13: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/13.jpg)
The Data!
Ago, ergo CogitoLanguage acquisitionAngrave & Glenberg (2007)
Action systems affect language comprehensionGlenberg, Sato, Cattaneo, Riggio, Palumbo, & Buccino (200Glenberg, Sato, Cattaneo, Riggio, Palumbo, & Buccino (200Glenberg & Cattaneo (in preparation)
Language comprehension affects action systemsBorghi, Glenberg, & Scorolli (in preparation)
![Page 14: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/14.jpg)
A & Gl b (2007)
Does the child know about bottles before learning “bottle?”
Ago, ergo Cogito
Angrave & Glenberg (2007)Analysis of MacArthur-Bates CDI for gesture & speech
• A01 armshow Look • C14 pushtoycarPush• A01. armshow Look • A02. reachout Look • A03. points See
A09 requestarm Get
• C14. pushtoycarPush • C15. throwball Throw • E01. sweep Sweep
E02 keyindoor Put• A09. requestarm Get • A12. Shrugsallgone Finish • B02. pattycake Clap
B05 sing Sing
• E02. keyindoor Put • E03. Poundhammer Hit • E06. pretendread Read
E09 pla m sinstr Pla• B05. sing Sing • B06. dance Dance • C01. Eatspoon Eat
C02 d i k D i k
• E09. playmusinstr Play • E10. pretenddrive Drive • E13. writepen Write
E15 t l P t• C02. drinkcup Drink • C05. wipeownface Clean
• E15. putonglasses Put
![Page 15: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/15.jpg)
28
A12Verb onset month = .61 x Action onset month + 17 monthsp < .001
26E01 E02
E13E15
pwrite
mon
th
24
tesnhtnom
A01A02
A09B02
B05
B06C05
C14C15 E03
E10clap throw
sweep
rb o
nset
m
22
breVO
A03
E06E09
drinkread
Ve
20
C01
C02 read
4 6 8 10 12 14 16Action Onset monthAction onset month
![Page 16: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/16.jpg)
feedback
Why the lag between comprehension and production?
Ago, ergo Cogito
i ldi t
drink / “drink”
socialenvironment
predictor
speechPredictions &
GainControl
body/perceptual
states
controller
h
Predictions &probabilities
physical
speech
environment
+grasp /dr/
efference copy
![Page 17: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/17.jpg)
Controller and predictor for Art’s knees
Ago, ergo CogitoFatigue muscles systemsfeedback
Bean task!
socialenvironmentpredictor
step
G i
/“step”
environment
body/perceptual
statescontroller
Predictions &probabilities
GainControl
speech
states
physicalenvironment
+step /st/
speech
efference copy
+step /st/
efference copy
![Page 18: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/18.jpg)
2500
Con Sent PNCon Sent Tow (Art gave you the pen)You see the coins in the fountain
John see the coins in his hand.
2250
mse
c)
Con Sent Far
Abs Sent Away (You told Liz the story)
Abs Sent Tow (Liz told you the story)
Con Sent Away (You gave Art the pen)You see the coins in the fountain.
2000
men
t Tim
e (m
y ( y)
1750
Judg
m
Con Sent YouCon Sent Near
You see the coins in your hand.You see the coins in your hand.
1500
12501250 Tow Away Tow Away Tow Away Tow Away Tow Away
beans lexical decision beans beans beansExp 1 Exp 2 Exp 3 Exp 3 Exp 3
![Page 19: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/19.jpg)
Conclusions
In designing primate action systems, evolution figured Ago, ergo Cogito
g g p y , gout solutions to difficult problems of timing, learning, serial order, and hierarchical control.
Many of those same problems occur in language, and it appearsthat evolution adapted mechanisms of action control, in Broca’s at lto language.
Data supporting these claims come frompp glanguage acquisitionaffects of action systems on language comprehensionaffects of language comprehension on use of action systemaffects of language comprehension on use of action system
Language can be used as a tool because linguistic meaning i d t i d b d t d t b dil ti i th ldis determined by and connected to bodily action in the world.
![Page 20: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/20.jpg)
Bologna collaborators: Anna Borghi, Claudia Scorolli, Annalisa SettiLeipzig collaborators: Shirley-Ann Rüschemeyer, Wolfgang Prinz, Simone Bosbach
Many Thanks to:
The Wisconsin Laboratory for Embodied Cognition:
Ago, ergo Cogito
Parma collaborators: Mark Sato, Luigi Cattaneo, Daniele Palumbo, Lucia Riggio, Givoanni Buccino, Vittorio Gallese, Giacomo Rizzolatti
The Wisconsin Laboratory for Embodied Cognition:Doctoral Students: Post-doc:
Megan Brown Lawrence AngraveDavid HavasDavid Havas
Erica Kesin RichmondLisa Lindeman
Undergraduate Students:gJenna Dittmar Laura KleinTiana Gutierrez Stephen MlinazLauren Hazen Josh MoonMaia Jacobs Emily MouilsoBeth Jaworski Michal Rischall Scott Johnson
Funding: National Science Foundation, University of Wisconsin, Max Plank Institute for Human Cognitive and Brain Sciences, & University of Parma, Department of Neuroscience
![Page 21: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/21.jpg)
Iacoboni (2005)
Ago, ergo Cogito
= predictor= controller controller
![Page 22: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/22.jpg)
lenberg, Sato, Cattaneo, Riggio, Palumbo, & Buccino (2007)
Ago, ergo Cogito
Trans-cranial magnetic stimulation (TMS)
TMS DesignTransfer vs No-Transfer vs NonsenseConcrete vs AbstractPulse at Verb or End
record MEP from OPrecord MEP from OPNB: NO ARM MOVEMENT FOR RESPONSE
![Page 23: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/23.jpg)
Ago, ergo Cogito
0.4 TransferNo Trans
0.2
0
-0.2
Concrete Abstract
-0.4Verb End Verb End
Pulse LocationPulse timePulse time
![Page 24: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/24.jpg)
Language comprehension affects motor systems
Hamilton, Wolpert, & Frith (2004) “Your own action
Ago, ergo Cogito
influences how you perceive another person’s action”lift 15 kg
lift 150 gfeedback from vision
lift 150 g
Predictions &
socialenvironmentpredictor
/lift 500 g
lift 350 g
GainPredictions &probabilities
body/perceptual
statescontroller
sensory/envpredictor
+
GainControl
states
physicalenvironment
controller +
efference copy
Borghi, Glenberg, & Scorolli (in preparation)
![Page 25: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/25.jpg)
Videos from Bosbach, Cole, Prinz, & Knoblich
Ago, ergo CogitoLarge box weights:3 6 12 18 kg
, ,(2005)
3, 6, 12, 18 kgSmall box weights.05, .3, .6, .9 kg
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.Heavy lift: Move the bowling ball from the cart to the tablefrom the cart to the table.Light lift : Move the wine glass from the cart to the table.Descriptive: The wine glass is on thetable.Is the object on the table breakable?
![Page 26: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/26.jpg)
Ago, ergo Cogito
1on
0.75
ion
Hea Video
Cor
rela
tio 0.5
Cor
rela
ti
Light Video
Heavy Video
C
0.25
lift lift
0Descriptive Heavy Light
lift liftSentence Type
![Page 27: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/27.jpg)
Rüschemeyer, Glenberg, Kaschak, & Friederici
Ago, ergo CogitoTable 1Examples of the sentences with English translations
Towards Away StationaryDas Auto fährt auf Dich zu Das Auto Fährt von Dir weg Das Auto sieht großDas Auto fährt auf Dich zu.Das Auto Fährt von Dir weg. Das Auto sieht großThe car drives towards you. The car drives away from you. The car looks bigDer Pfeil fliegt auf Dich zu. Der Pfeil fliegt von Dir weg. Der Pfeil sieht klien aus.Th fli t d Th fli f Th l kThe arrow flies towards you. The arrow flies away from you. The arrow looks Der Ball rollt auf Dich zu. Der Ball rollt von Dir weg. Der Ball sieht groß aus.The ball rolls towards you. The ball rolls away from you. The ball looks big.
![Page 28: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/28.jpg)
Ago, ergo Cogito
![Page 29: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/29.jpg)
“Give momma the bottle.”“Give momma the bottle.”Dad says,
Ago, ergo Cogito
di t
give / “give”feedback
socialpredictor
speechPredictions &
GainControl
environment
“momma” ->eyes
controller
h
probabilities body/perceptual
states
“bottle” ->
reachspeech
physicalenvironment
eyesgraspreleaseextendgrasp
eyes to mommaeyes to bottlegain control gain control+
gain controlact!
efference copyreleaseextendgrasp
![Page 30: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/30.jpg)
Language comprehension & simulation“The girl gives the horse an apple.”
give3/“give”give2/“give”
Ago, ergo Cogito
speech activates motor systemgive1 /“give”
feedback
p y
gain control set
“give”-> speech MN & controller
predictor
speech
give1 / give
P di ti &Gain
Control
socialenvironment
give -> speech MN & controllerpredictor generates
expectations of a recipiencontroller
h
speech Predictions &probabilities
Controlbody/
perceptualstates
speech
physicalenvironment+
lh
“the horse”-> speech MN & conpredictor generates
sensory feedback of horse
Efference copy -> A mental modelwith perspective
applehorse sensory feedback of horse
with perspectiveEmergence of working memory
eyes grasp
![Page 31: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/31.jpg)
“Transfer X the Y.”
Ago, ergo Cogito
Learning kinematicspredictorhand/“hand”
feedback
socialenvironment g
Hand X the Y. (hand priors)Th X th Y (th i )controller
predictor
speech Predictions &probabilities
GainControl
environment
body/perceptual
Throw X the Y. (throw priors)controller
speech
perceptualstates
physicalenvironment
Result: Easily learned action schemas for verbs of concrete
gainlocate X
locate Y
environment+
schemas for verbs of concrete transfer.
locate Y
kinematics
gain Syntactic/semantic bootstrappingusing the motor system!
![Page 32: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/32.jpg)
“Transfer X the Y.”
Ago, ergo Cogito
Learning kinematicspredictortell/“tell”
feedback
socialenvironment g
Tell X the Y. (tell priors)Radio X the Y (radio priors)controller
predictor
speech Predictions &probabilities
GainControl
environment
body/perceptual Radio X the Y.(radio priors)
Delegate X the Y. (delegate prioetc.
controller
speechstates
physicalenvironment
Results: Easily learned action schema for
gainlocate X
locate Y
+
Easily learned action schema for verbs of abstract transfer.
locate Y
kinematics
gain
![Page 33: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/33.jpg)
1400Ago, ergo Cogito
1300
1400No Transfer
Away Toward
ACE interaction
1200
1300 ACE interaction& main effect of C t1200 Concreteness
1100
Concrete Abstract
Motor system forunderstanding
or1000
Response Location
Yes-near Yes-farYes-near Yes-farMotor imagery &responding
Response Location
![Page 34: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/34.jpg)
Ago, ergo Cogito
The Plan:
Replicate Glenberg & Kaschak using Italian
Use TMS to determine if action effect is during understanding or responding.
Collaborators: Marc Sato Luigi Cattaneo LuciaCollaborators: Marc Sato, Luigi Cattaneo, Lucia Riggio, Daniele Palumbo, and Giovanni Buccino
Dipartimento di Neuroscienze, Università di Parma
![Page 35: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/35.jpg)
Triad of Concrete sentences
Toward: Marco/ ti/ da' /le carte. (Marco gives you the papers.)
Ago, ergo CogitoAway: Tu/ dai /le carte/ a Marco. (You give the papers to Marco.)
No Transfer: Tu/ leggi/ le carte/ con Marco. (You read the papers with Marco.)
Triad of Abstract sentences
Toward: Anna/ ti/ delega/ le responsabilità. (Anna delegates the responsibilities to
you )you.)
Away: Tu/ deleghi/ le responsabilità/ ad Anna. (You delegate the responsibilities
to Anna.)
No Transfer: Tu/ discuti/ delle responsabilità/ con Anna. (You discuss the
responsibilities with Anna.)
Nonsense Sentences
Giovanni/ canta/ le carte/ con te. (Giovanni sings the papers with you.)
Daniele/ ti/ pulisce/ la responsabilità. (Daniele cleans the responsibilities to you.)Daniele/ ti/ pulisce/ la responsabilità. (Daniele cleans the responsibilities to you.)
Note: The slashes indicate the segments used in Experiment 2.
![Page 36: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/36.jpg)
Simulate temporal course of giving (away) vs receiving
t fi t l tAgo, ergo Cogito
at first laterAway: grasp open
Toward: open grasp
Pulse at EndPulse at Verb
AbstractConcreteAbstractConcrete0 5
0.75
p g p
AbstractConcreteAbstractConcrete
0.25
0.5
0
0 5
-0.25
-0.5Toward Away No Trans Toward Away No Trans Toward AwayNo Trans Toward AwayNo Trans
Sentence Direction
![Page 37: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/37.jpg)
Action and the embodiment of concrete language
Ago, ergo CogitoHauk, Johnsrude, & Pulvermüller (2004)
Tettamanti et al (2005)Tettamanti et al. (2005)
Aziz-Zadeh et al. (2006)
What about the embodiment of abstract language?Space:Space:Richardson, Spivey, Barsalou, & McRae (2003) Boroditsky and Ramscar (2002)
Action: Gl b & K h k (2002)Glenberg & Kaschak (2002)
![Page 38: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/38.jpg)
Other Predictions
Ago, ergo Cogito
eye movements and language
development of gesture & languagep g & g g
signed languages
![Page 39: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/39.jpg)
eye movements and languageTanenhaus visual world paradigm
Ago, ergo Cogito
Tanenhaus visual world paradigmAltmann & Kamide
Richardson & colleagues
![Page 40: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/40.jpg)
P dd t l (i )Ago, ergo Cogito
Padden et al. (in press)
In Al-Sayyid Bedouin Sign Language (ABSL), signs for both yy g g g ( ) gconcrete and abstract verbs of transfer involve arm movemeof away from the body.
( )
Other signed languages?
Karen Emmorey (personal communication) “it is true for ASL and probably for many signed languages”
![Page 41: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/41.jpg)
Conclusions
Ago, ergo CogitoThe data strongly imply that concrete language (about
transfer) is grounded in the motor system
The data suggest that abstract language (about transfer)is also grounded in the motor systemis also grounded in the motor system
Models of language built on models of motor controlModels of language built on models of motor controlprovide a unique understanding of languagephonology, syntax, and semantics as arising f hi hi l h i f t t lfrom hierarchical mechanisms of motor control
![Page 42: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/42.jpg)
Why might abstract language understanding involvemotor system? A developmental hypothesis:
Ago, ergo Cogitolearning nounsattention (move eyes); affordances
i l ?
learning verbs of transfer: parameters of actionsource object kinematics (force direction mode)
canonical neurons?
source, object, kinematics (force, direction, mode)mirror neurons?
generalization of parametersgeneralization of parameterssource, objects, kinematics --> transfer schema
syntactic/sematic boot-strapping using the motor system!y pp g g yYou tell Liz the story.
![Page 43: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/43.jpg)
“Give me the cup.”“Give momma the bottle.”“Give Lavinnia the candy ”
Ago, ergo Cogito
Give Lavinnia the candy.
Learning action parameters:predictor
give/“give”
feedback
socialenvironment g p
Gain control (move only eyes)controller
predictor
speech Predictions &probabilities
GainControl
environment
body/perceptual
Attend with eyes to destination{momma, me, Lavinnia}
controller
speech
perceptualstates
physicalenvironment
gainlocate X
locate Y
Attend with eyes, set kinematic{bottle, cup, candy}
environment+
locate Y
kinematics
gainGain control (ready to move)
Result: A double object action schema (verb argument construction) for giveAct Now!
![Page 44: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/44.jpg)
Linda Smith (2005) “Action alters shape categories”
Ago, ergo Cogito
Action
0.9
1
0 5
0.6
0.7
0.8
Sy
0.2
0.3
0.4
0.5As
0
0.1
S1 S2 S3 A1 A2 A3
Similarity of Test Object to Exemplar
Thus, perceived similarity, categorization, and namingare influenced by action.
![Page 45: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/45.jpg)
Glenberg & Kaschak 2002
Action systems affect language comprehension
Ago, ergo Cogito
Glenberg & Kaschak, 2002
Task: Judge if a sentence is sensibleIV 1: Implied Direction: Toward vs. AwayIV 2: Sentence type
ImperativeOpen the drawer. / Close the drawer.
Concrete TransferConcrete TransferCourtney handed you the notebook. / You handed Courtney the
notebook.Andy delivered the pizza to you. / You delivered the Pizza to Andy.
Ab t t T f tAbstract Transfer sentences Liz told you the story. / You told Liz the story.The policeman radioed the message to you. / You radioed the
message to the policeman.g pNonsense
Open the plate./You drank the house to Joe.
IV 3: Actual Response Direction: Yes-is-near vs Yes-is-IV 3: Actual Response Direction: Yes is near vs. Yes isfar
![Page 46: Premotor cortex, action control, and languagease.tufts.edu/psychology/documents/conf2007Glenberg.pdf · 2014-04-16 · 2 “Give Momma the bottle.” Ago, ergo Cogito Give Lavinnia](https://reader033.vdocument.in/reader033/viewer/2022042812/5faa6b220b614e64787ce0ae/html5/thumbnails/46.jpg)
e
1400Imperative
1850
Abstract Transfer
Concrete Transfer
1850A
n R
eadi
ng T
ime
13501800
1750
1800
AwayToward
AwayToward
AwayToward
Tri
mm
ed M
ean
13001700
17501750
1700
Toward
You told Liz the stor
T
1250Yes-is-near Yes-is-far
1650Yes-is-near Yes-is-far
1650Yes-is-near Yes-is-far
Response Direction
Is the effect caused by comprehension processes or response pro