How do we process text with spatial information?

Marijn E. Struiksma*, Matthijs L. Noordzij**,

Bas F.W. Neggers*** & Albert Postma*

*Universiteit Utrecht**Radboud Universiteit Nijmegen

*** Universitair Medisch Centrum Utrecht

•Spatial language: spatial configuration of the world• Simple sentences

• Search directions, identifying

people

• Complex route finding instructions

•Generate mental map• Also blind people

Spatial Language

•Aim: gain insight in underlying process and nature of mental representations

• Research with sighted and blind

• Different input modalities

Spatial Language

•Propositional model• Verbal strategy

•Strategic model• Verbal strategy

or

• Visuo-spatial strategy

Spatial Language Processing

•Noordzij et al. (2005)•Sentence-sentence•Sentence-picture•80% expectancy

Different Strategies

Different Strategies

+

triangle left of circle

+

triangle left of circle

80%

20%

Spatial condition

Different Strategies

+

triangle and circle

+

triangle and circle

20%

80%Non-spatial condition

Different Strategies

• Spatial: slower on unexpected pictures

• Propositional model: unexpected stimuli – general switch cost Identical for spatial and non-spatial

• Strategic model: unexpected stimuli – general switch cost and incompatibility for both sentence and picture

Fig. 1 from Noordzij et al. (2005)

•Automatic propositional representation

•Additional visual-spatial representation• Strategically dependent on context

•Neuroimaging parietal areas:• Understanding spatial terms

• Visual-spatial representation

Dual-representational model

•Noordzij et al. 2006•Similar sentence-sentence and sentence-picture paradigm

•Similar behavioral pattern•Event-related potentials (ERPs)

EEG: processing spatial sentences

•ERPs

EEG: processing spatial sentences

Fig. 4 from Noordzij et al. (2006)

Parieto-occipital activation for spatial sentences, expecting a picture.

visual-spatial strategy

•Sentence-sentence and sentence-picture paradigm

•100% expectancy for S2•RT faster for picture than sentence, especially for spatial

fMRI: processing spatial sentences

•fMRI

fMRI: processing spatial sentences

Fig. 2 from Noordzij et al. (2008)

Spatial > Non-spatial for both stimulus modalities: activity in left Supramarginal gyrus (SMG)

•How is spatial language processed in the absence of vision?

•Is language processing different for different input modalities?

•Is the nature of mental representations modality specific?

Nature of mental representations

•Sentence-sentence paradigm• Comparing spatial and non-spatial

•Auditory version•Scanner-details:

• 3T Philips Achieva scanner

• PRESTO-SENSE sequence

• TR = 500ms

• Voxel-size: 4*4*4 mm

fMRI blind & sighted

•13 congenitally blind• Age 36.5 ± 9.8

• 7 male, 6 female

• 5 right-handed, 5 left-handed, 3

ambidexter

•13 sighted controls• Age 37.2 ± 11.2

• 8 male, 5 female

• 6 right-handed, 4 left-handed, 3

ambidexter

Subjects

•Compare two sentences• Do they describe the same

situation?

Spatial Sentence Comprehension

•Block-design: 4 sessions, 18 blocks per session, 2 trials of 7.5s per block

•4 conditions

Spatial Nonspatial

Compound preposition

left of right of (LR)

together with (TW)

Compound adverb

smaller than taller than (ST)

older than younger than (YO)

Spatial Sentence Comprehension

Position = Left of/Right of

Size = Taller than/Smaller than

Conjunction = Together with

Age = Older than/Younger thanPosition

(15s)

Interval (6-9s)

Size(15s)

Age(15s)

Conjunction(15s)

Interval (6-9s) Interval

(6-9s) Interval (6-9s)

Presented pseudo-randomly

Spatial Sentence Comprehension

Performance for all conditions for blind and sighted

0,880

0,900

0,920

0,940

0,960

0,980

1,000

1,020

Position Size Combination Age

Dimension

Pe

rfo

rma

nc

e (

%*1

00

)

CB

SC

Spatial Non-spatial

Performance

• Performance is good• No difference CB or SC

Response times for four conditions

500

550

600

650

700

750

Position Size Combination Age

Dimension

Res

pons

e tim

e (m

s)

Spatial Non-spatial

Behavioral Results: RT

• Main effects of Space and Category• No group difference

•Contrast: Position > Combination

•ROI around SMG from Noordzij et al. (2008)

•Conjunction CB and SC

Results: Supramarginal Gyrus

ROI, T=2.5, p = .045 (corrected)

•Contrast: Relational > Combination

•ROI around SMG from Noordzij et al. (2008)

•Conjunction CB and SC

Results: Supramarginal Gyrus

ROI, T=2.5, p = .045 (corrected)

Contrast Estimates SMG

Average parameter estimate in SMGfor blind and sighted

0

2

4

6

8

10

Position Size Combination Age

Dimension

Average ß (SE)

CB

SC

fMRI results

• Evidence for reorganization• Difference CB and

SC for task vs.

rest

• Occipital areas• No dimension modulation

Whole brain, T=4.0,k=10 voxels, p < 0.03

•Behavioral results: sighted can generate propositional and visual-spatial representations

•Target area: left SMG•Left SMG not influenced by

• context (picture or sentence)

• input modality (visual or auditory)

Conclusions 1

•Blind also activate left SMG and visual areas

•Visual areas are not modulated by Space or Category

•Left SMG important for processing spatial prepositions• Other function: ordering on 1 dimension

•Activation in left SMG is modality-independent + hardwired

Conclusions 2

Questions ?

•Block-design•3 types of blocks

• Rest

• Non-words

• Words

Verb Generation

Results• Conjunction CB+SC

• Language areas• bilateral Broca, Wernicke

• Covert verb generation:• precentral gyrus• supplementary motor area

• CerebellumWord - Nonword

Whole brain, T=4.0,k=20 voxels, p < 0.007

Results: Reorganization•Difference

CB-SC•CB significantly activate occipital cortex • BA 19, 31, 37CB vs. SC

Whole brain, T=4.5, k=10 voxels, p < 0.019