language embedded in systems of perception and action rolf a. zwaan florida state university
TRANSCRIPT
![Page 1: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/1.jpg)
Language Embedded Language Embedded
in Systems in Systems of Perception and of Perception and
ActionAction
Rolf A. ZwaanRolf A. Zwaan
Florida State UniversityFlorida State University
![Page 2: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/2.jpg)
Experiential TracesExperiential Traces
![Page 3: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/3.jpg)
Experiential TracesExperiential Traces
“referential” linguistic
![Page 4: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/4.jpg)
Experiential AssociationsExperiential Associations
“intra-referential” intra-linguistic
![Page 5: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/5.jpg)
Experiential AssociationsExperiential Associations
cross-domain
![Page 6: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/6.jpg)
Hebbian LearningHebbian Learning
Co-occurrence primary factor in the Co-occurrence primary factor in the forming of associationsforming of associations Intra-linguistic;Intra-linguistic; Intra-referential;Intra-referential; Cross-domain (grounding).Cross-domain (grounding).
![Page 7: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/7.jpg)
Experiential TracesExperiential Traces
Linguistic TracesLinguistic Traces AuditoryAuditory MotorMotor VisualVisual TactileTactile
Referential TracesReferential Traces VisualVisual MotorMotor AuditoryAuditory OlfactoryOlfactory TactileTactile SomatosensorySomatosensory
![Page 8: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/8.jpg)
LSA/Encyclopedia_all
00.050.1
0.150.2
0.250.3
0.350.4
0.450.5
0 1000 2000 3000 4000distance from Miami (miles)
cosine
Capturing Capturing Second-order Second-order CorrelationsCorrelations
r2 = .30
![Page 9: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/9.jpg)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0 1 2 3
distance from the Netherlands (# of intervening countries)
LSA cosine
r2 =.65
Capturing Capturing Second-order Second-order CorrelationsCorrelations
![Page 10: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/10.jpg)
Utility of Linguistic Utility of Linguistic Traces/AssociationsTraces/Associations
Additional layer of associations.Additional layer of associations. May enhance “fluency.”May enhance “fluency.”
““Referential bootstrapping.”Referential bootstrapping.”
![Page 11: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/11.jpg)
Referential Referential BootstrappingBootstrapping
Name this animal:Name this animal: What has horns like a giraffe,What has horns like a giraffe, a deer head,a deer head, a a
horse neck,horse neck, and legs like a zebra? and legs like a zebra? http://www.westerville.k12.oh.us/ZOO/Orlich/pages/Okafrtpg.htmhttp://www.westerville.k12.oh.us/ZOO/Orlich/pages/Okafrtpg.htm
![Page 12: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/12.jpg)
Referential Referential BootstrappingBootstrapping
![Page 13: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/13.jpg)
ResonanceResonance
Experiential traces “resonate” with Experiential traces “resonate” with linguistic input.linguistic input. Visual resonance.Visual resonance. Motor resonance.Motor resonance.
The role of resonance in The role of resonance in comprehension.comprehension.
![Page 14: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/14.jpg)
PredictionsPredictions
Prediction 1Prediction 1 Language comprehension affects Language comprehension affects
subsequent perceptual performance.subsequent perceptual performance. Prediction 2Prediction 2
Incidentally acquired visual Incidentally acquired visual representations affect subsequent representations affect subsequent language processing.language processing.
Prediction 3Prediction 3 Language comprehension affects Language comprehension affects
concurrent motor performance.concurrent motor performance.
![Page 15: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/15.jpg)
Comprehension Affects Comprehension Affects Perceptual Performance Perceptual Performance
(Zwaan, Stanfield, & Yaxley, 2002)(Zwaan, Stanfield, & Yaxley, 2002)
The ranger saw the eagle in the nest.[SAW, RANGER, EAGLE][IN, EAGLE, NEST]
The ranger saw the eagle in the sky. [SAW, RANGER, EAGLE][IN, EAGLE,
SKY]
![Page 16: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/16.jpg)
The ranger saw the eagle in its nest. The ranger saw the eagle in the sky.
![Page 17: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/17.jpg)
Picture Recognition Picture Recognition and Naming Latenciesand Naming Latencies
(Zwaan, Stanfield & Yaxley, 2002)(Zwaan, Stanfield & Yaxley, 2002)
600
650
700
750
800
Recognition Naming
MatchMismatch
RT (ms)
![Page 18: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/18.jpg)
Language may Involve Language may Involve Dynamic Dynamic
RepresentationsRepresentations(Zwaan. Madden, Yaxley, & Aveyard, 2004)(Zwaan. Madden, Yaxley, & Aveyard, 2004)
The midfielder passed the ball to you.
![Page 19: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/19.jpg)
![Page 20: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/20.jpg)
![Page 21: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/21.jpg)
![Page 22: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/22.jpg)
![Page 23: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/23.jpg)
Zwaan, Madden, Yaxley, & Aveyard, 2004
![Page 24: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/24.jpg)
Prediction 2Prediction 2
Incidentally acquired visual Incidentally acquired visual representations affect subsequent representations affect subsequent language processing.language processing.
![Page 25: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/25.jpg)
The Visual-Memory The Visual-Memory Paradigm Paradigm (Aveyard, Zwaan, Radach, & (Aveyard, Zwaan, Radach, &
Vorstius, in preparation)Vorstius, in preparation) Memory PhaseMemory Phase
Word-picture verification (4 times)Word-picture verification (4 times) Reading Phase Reading Phase (ostensibly unrelated)(ostensibly unrelated)
Eye movements trackedEye movements tracked
![Page 26: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/26.jpg)
Memory PhaseMemory Phase
eagle
![Page 27: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/27.jpg)
Memory PhaseMemory Phase
![Page 28: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/28.jpg)
Reading PhaseReading Phase
The ranger was in the park on his The ranger was in the park on his morning round. It was a bright day. morning round. It was a bright day. In the sky there was an eagle, In the sky there was an eagle, soaring above the prairie.soaring above the prairie.
![Page 29: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/29.jpg)
First FixationsFirst Fixations
300
350
400
450
500
550
600
650
In the sky an eagle wassoaring
above the…
MatchMismatch
![Page 30: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/30.jpg)
Gaze DurationsGaze Durations
500
550
600
650
700
750
800
850
In the sky an eagle wassoaring
above the…
MatchMismatch
![Page 31: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/31.jpg)
Visual World Visual Visual World Visual MemoryMemory
SituatedSituated Visual SearchVisual Search Auditory Auditory
PresentationPresentation DV = eye fixations DV = eye fixations
during display during display viewingviewing
Re-situatedRe-situated Memory SearchMemory Search Visual Visual
PresentationPresentation DV = eye fixations DV = eye fixations
during readingduring reading
![Page 32: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/32.jpg)
Prediction 3Prediction 3
Motor traces are activated by Motor traces are activated by linguistic input.linguistic input.
![Page 33: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/33.jpg)
Action Compatibility Action Compatibility EffectEffect
(Glenberg & Kaschak, 2002)(Glenberg & Kaschak, 2002) Sensibility judgmentsSensibility judgments
Close the drawer.Close the drawer. Andy delivered the pizza to you.Andy delivered the pizza to you. Liz told you the story.Liz told you the story. Break the cloud.Break the cloud.
![Page 34: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/34.jpg)
Action Parameters Action Parameters and Motor Resonanceand Motor Resonance
Actions have various parameters.Actions have various parameters. Direction.Direction. Force.Force. Manner (e.g., open hand vs. closed hand).Manner (e.g., open hand vs. closed hand). Speed.Speed. Duration.Duration.
Which action parameters exhibit Which action parameters exhibit resonance in language comprehension?resonance in language comprehension?
![Page 35: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/35.jpg)
Measuring Response Measuring Response ForceForce
![Page 36: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/36.jpg)
Measuring Response Measuring Response ForceForce
05000
100001500020000
2500030000350004000045000
Time
Force (kPa)
10%
T
90%A
Atmospheric pressure = 100 kPa
![Page 37: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/37.jpg)
Response ForceResponse Force(Zwaan & Taylor, in prep.)(Zwaan & Taylor, in prep.)
Sensibility judgments.Sensibility judgments. Verbs.Verbs.
He pushed the car.He pushed the car. (high) (high) He started the car.He started the car. (low) (low) He admired the car.He admired the car. (no) (no)
Nouns.Nouns. He broke the baseball bat. He broke the baseball bat. (high)(high) He broke the bread. He broke the bread. (low)(low) He broke the promise. He broke the promise. (no)(no)
![Page 38: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/38.jpg)
Response Force: VerbsResponse Force: Verbs
15000
16000
17000
18000
19000
20000
21000
Force
HighLowNo
ResponseAmplitude(kPa)
![Page 39: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/39.jpg)
Response Force: NounsResponse Force: Nouns
15000
15500
16000
16500
17000
17500
18000
18500
19000
Force
HighLowNo
ResponseAmplitude(kPa)
![Page 40: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/40.jpg)
Manner of ActionManner of Action
Sustain vs. ReleaseSustain vs. Release He gripped the hammer.He gripped the hammer. He dropped the hammerHe dropped the hammer..
PredictionsPredictions Sustain faster than Release (ACE).Sustain faster than Release (ACE). Sustain longer than Release (ACE).Sustain longer than Release (ACE).
![Page 41: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/41.jpg)
Sustain vs. ReleaseSustain vs. Release
Response SpeedResponse Speed
1000
1100
1200
1300
1400
1500
1600
Sustain Release
SustainRelease
![Page 42: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/42.jpg)
05000
100001500020000
2500030000350004000045000
Time
Force (kPa)
10%
Sustain vs. ReleaseSustain vs. Release
Response DurationResponse Duration
![Page 43: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/43.jpg)
Sustain vs. ReleaseSustain vs. Release
Response Duration >10%Response Duration >10%
200
220
240
260
280
300
320
Sustain Release
SustainRelease
![Page 44: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/44.jpg)
Motor ResonanceMotor Resonance
What is the role of motor What is the role of motor representations in language representations in language comprehension?comprehension? Instrumental Instrumental (Glenberg)(Glenberg) Ornamental Ornamental (many skeptics)(many skeptics) Useful/necessary in specific cases Useful/necessary in specific cases
(Jackendoff)(Jackendoff)
![Page 45: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/45.jpg)
Motor ResonanceMotor Resonance
What is the role of motor What is the role of motor representations in language representations in language comprehension?comprehension? Useful/necessary when actions areUseful/necessary when actions are
ConcreteConcrete PunctatePunctate SimpleSimple
Let’s first ask: How is motor resonance Let’s first ask: How is motor resonance modulated by linguistic input?modulated by linguistic input?
![Page 46: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/46.jpg)
How is motor resonance How is motor resonance modulated by linguistic modulated by linguistic
input?input? ((Zwaan & Taylor, 2006, JEP General)Zwaan & Taylor, 2006, JEP General)
Reading by rotating.Reading by rotating. Every 4 deg. new frame.Every 4 deg. new frame.
He /realized /that /the music /was He /realized /that /the music /was /too loud/so he //too loud/so he /turned downturned down/the /the /volume./volume.
60 psychology undergraduates.60 psychology undergraduates.
![Page 47: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/47.jpg)
Reading-by-RotationReading-by-Rotation
300
310
320
330
340
350
360
370
380
He...he turned down the volumeSentence Region
Reading Time
MatchMismatch
![Page 48: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/48.jpg)
Is Motor Resonance Is Motor Resonance
Short-lived or Focus-bound?Short-lived or Focus-bound? (Taylor & Zwaan, in prep.)(Taylor & Zwaan, in prep.)
Perspective changePerspective change.. After/lighting/the candles/for
the/romantic/evening,/he/dimmed/the/lights. Maintaining focus on the action.Maintaining focus on the action.
He lit/the candles/for the/romantic/evening./He noticed/the bright/lights,/which he/dimmed/slowly.
Reading-by-rotating.Reading-by-rotating.
![Page 49: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/49.jpg)
Motor Resonance is Focus-Motor Resonance is Focus-boundbound
![Page 50: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/50.jpg)
Strong Test Strong Test of Linguistic Focus of Linguistic Focus
HypothesisHypothesis Action-modifying vs. subject-Action-modifying vs. subject-
modifying adverbsmodifying adverbs … … which he turned down slowly.which he turned down slowly. ……which he turned down obediently.which he turned down obediently.
Stay tuned.Stay tuned.
![Page 51: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/51.jpg)
Conclusions & OutlookConclusions & Outlook
Language is embedded in systems of Language is embedded in systems of perception and action.perception and action. Rapid exchange of information.Rapid exchange of information.
But it needs to be determined to But it needs to be determined to what extent it is in bed with these what extent it is in bed with these systems.systems. Perceptual & motor resonance Perceptual & motor resonance
ornamental or instrumental?ornamental or instrumental?
![Page 52: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/52.jpg)
AcknowledgementsAcknowledgements Grants from the National Science Foundation Grants from the National Science Foundation
(NSF) and the National Institute of Mental (NSF) and the National Institute of Mental Health (NIMH)Health (NIMH)
Graduate students (past & present)Graduate students (past & present) Rob Stanfield, Carol Madden, Rich Yaxley, Mark Rob Stanfield, Carol Madden, Rich Yaxley, Mark
Aveyard, Larry TaylorAveyard, Larry Taylor Postdoctoral Fellows (past & present)Postdoctoral Fellows (past & present)
Barbara Kaup, Dave Therriault, Johanna Kaakinen, Barbara Kaup, Dave Therriault, Johanna Kaakinen, Tobias RichterTobias Richter
Faculty CollaboratorsFaculty Collaborators Mike Kaschak, Ralph Radach, Katinka Dijkstra Mike Kaschak, Ralph Radach, Katinka Dijkstra
![Page 53: Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University](https://reader036.vdocument.in/reader036/viewer/2022062423/56649c6d5503460f9491f7ca/html5/thumbnails/53.jpg)