some articulatory and cerebral correlates of · pdf filesome articulatory and cerebral...

SOME ARTICULATORY AND CEREBRAL CORRELATES

OF PROSODIC FOCUS IN FRENCH

Hélène Lœvenbruck

Speech and Cognition Department, GIPSA-lab (ex ICP)

Grenoble, France

GrenobleImagesParoleSignal

Automatique

� Prosodic focus

- draws the listener/viewer’s attention to relevant elements in the discourse.

- a means to designate, identify or select an element

(e.g. Halliday, 1967; Gussenhoven, 1983; Selkirk 1984; Nølke, 1994; Birch & Clifton, 1995; Ladd, 1996)

=> Related to pointing, or deixis

DiscussionMultimodal pointing

-acoustic-articulatory -cerebralcorrelates

Vocal pointing:

Manual and ocularpointing

Introduction

What’s pointing?

- a communicative device which orients the attention of another person so that an object / person / direction / event becomes the shared focus of attention



Vocal pointing:


Introduction

- performed with the index finger and arm extended in the direction of the interesting object and withthe other fingers curledinside the hand

(Butterworth, 2003)

Pointing is ‘special’ because it is…

…ubiquitous

- observed in everyday interactions (work, vacation), in most cultures of the world

- observed in oral as well as signed communication- used linguistically for referential indexing (pronouns) in some sign languages (Klima & Bellugi, 1988)

…probably universal

- index finger pointing claimed to be a universal ability (e.g. Povinelli & Davis, 1994)-- in some cultures, index finger pointing can be replaced with lip pointing (Barai in Papua New Guinea, see Wilkins, 2003)



Vocal pointing:


Introduction

Pointing is ‘special’ because it…

…has a long historical past

- European art:

Leonardo da Vinci (late XVth century) Jean Fouquet (early XVth century)Bayeux Tapestry (end of the XIth century)

- Asian art:

Sculptures of pointing buddhas ( XVIIIth century Burmese art)



Vocal pointing:


Introduction

Pointing is ‘special’ because it…

…is performed by evolved animals, capable of some degree of imitation and inter-individual communication

- Chimpanzees in captivity can point at unreachable food.

- They sometimes use ‘canonical’ index finger pointing, but most often, they point with all fingers extended.

- They do so without explicit training.

(Leavens, Hopkins & Bard, 2005)



Vocal pointing:


Introduction



Vocal pointing:


Introduction

Pointing is ‘special’ because it is… multimodal

� digital

� ocular

� vocal

� labial (Kuna indians in Panama, Arrernte in Australia, Awtuwand Barai in Papua New Guinea, Ewe in Ghana, Navajo in North America; cf. Wilkins, 2003)

� with other body parts(head, lip, chin, elbow, foot, arm, hand: Kendon, 1996)

ThatThat’’ss Andy!Andy!

Pointing is ‘special’ because it is…

…involved in stages of language development



Vocal pointing:


Introduction

- Deictic gaze (8-9 months)

mutual attention between child and adult(e.g. Tomasello, 1999)

- Finger pointing (9-11 months)

lexicon construction: « what’s that? »(e.g. Butterworth & Morissette , 1996)

- Word + pointing gesture (16-20 months)

morphosyntax emergence

(Volterra & Caselli, 1986; Goldin-Meadow, 1999; Bates & Dick, 2002; Butterworth, 1997)

Pointing is involved in stages of language development



Vocal pointing:


Introduction

Demonstratives in acquisition:

Syntactic precursors

- « That! » presentational

- « That’s Mum! » presentational construction

- « That’s the sugar that goes in there! » relative construction

- « That’s Mum that said that the sugar goes in there! »

(Jisa & Kern, 1998; Diessel & Tomasello, 2000; Tomasello, 2003)

� Pointing (ocular, manual) is involved at many stages

of human language development, from mutual attention

emergence (gaze) to syntactic acquisition through

lexicon construction.

� What do we know about the somatosensory and

cerebral correlates of vocal pointing?

� The role of (ocular and manual) pointing in language

development suggests that vocal pointing may share

features in common with ocular and manual pointing,

and typically may be grounded in close cerebral tissues.



Vocal pointing:


Introduction

Could vocal pointing share cerebral features in common

with ocular and manual pointing?

What are the cerebral correlates

of manual and ocular pointing?



Vocal pointing:


Introduction

Manual pointing:

-Preparation phase: prepare to point with right index finger to an cued location (left or right)

- Execution phase: when target flashes, point towards target

Ocular pointing (saccades):

- Preparation phase: prepare to look at cued location (left or right)

- Execution phase: when target flashes, look at target location



Vocal pointing:


Introduction

- Bilateral occipital cortex: transient sensory response to cue

- Bilateral frontal cortex: sustained responseat the junction of precentral and superior frontal sulci, i.e. human frontal eye field (FEF, involved in oculomotorpreparation)

- Bilateral posterior parietal cortex: sustained response along horiz. segment of intraparietal sulcus (IPS)



Vocal pointing:


Introduction

- Bilateral FEF: same as in ocular pointing

- Bilateral IPS: same as in ocular pointing

- Additional left H. activation for manual pointing:angular gyrus, supramarginal g., superior parietal lobule (PrCu), dorsal precentral g., superior temporal sulcus

Results hold, even when left hand is used



Vocal pointing:


Introduction


of vocal pointing?

In French, vocal pointing can be conveyed by:

- syntactic extraction:

C’est Madeleine qui m’amena

(It’s/that’s Madeleine who/that brought me along)



Vocal pointing:


Introduction

- Prosodic focus:

MADELEINEFm’amena

(MADELEINEF brought me along)

MADELEINE m’amena



Vocal pointing:


Introduction

� Acoustically sophisticated:

Prosodic focus = F0 raising + lengthening on focal constituent

F0 lowering on pre-focal constituent

Deaccentuation of post-focal constituent

Rhythmical effects, e.g. increased duration of

focal constituent

See e.g. Astésano (2001), Dahan & Bernard (1996), Delais-Roussarie et al. (2002),

Di Cristo (1998), Dohen & Lœvenbruck (2004), Jun & Fougeron (2000), Rossi

(1999), Touati (1987)

MADELEINE m’amena

RA NI MA

� Rythmically:

- Increase in duration of focal constituent

- Increase in duration of focused syllable (onset)

- Sometimes anticipatory increase in duration of pre-focal

syllable

- No decrease in duration of post-focal constituent (no

dephrasing)

� Prosodic structure:

- Sometimes restructuring of focused AP into one IP or into

several APs.



Vocal pointing:


Introduction

� Are there articulatory correlates of prosodic pointing?


-acoustic-articulatory-cerebralcorrelates

Vocal pointing:


Introduction

(old) EMA acquisition protocole

5 coils in sagittal plane:

– 1 reference (sup. incisor)

– 3 on the tongue

– 1 for the jaw (lower incisor)

Fe = 500 Hz



Vocal pointing:


Introduction

Position of Ymid(mm)

Velocity(mm/s)

Acceleration

(mm/s2)

Beg. /la/

Peak velocity

Max displact

end /la/



Vocal pointing:


Introduction

Displacement in [la]

0

2

4

6

8

10

12

utt. initial utt. central utt. final

mm unfocused

focused

Duration of [la]

0

50

100

150

200

250

utt. initial utt. central utt. final

ms unfocused

focused

Lœvenbruck (1999)

Another articulatory monitoring tool: Liptrack

Lipopening(/a/)

Lip closure(/m/)

LipProtrusion

Inter-lip areaDiscussionMultimodal

pointing-acoustic-articulatory-cerebralcorrelates

Vocal pointing:


Introduction

� Articulatorily, prosodic pointing requires accurate motor control:

-increased articulation and duration of the focal constituent

-with optional reduced articulation of post-focal constituent

Dohen et al. (2004); Dohen & Lœvenbruck (2006)

unfocused /ma/

focused /ma/

post-focal /ma/



Vocal pointing:


Introduction


of vocal pointing (deixis)?



Vocal pointing:


Introduction

16 right-handed male French speakers

(1) Baseline condition

Madeleine m’amena /ma.d�.l�n.ma.m�.na/

(Madeleine brought me around)

(2) Prosodic pointing condition

MADELEINEF m’amena

(3) Syntactic pointing condition

C’est Mad’leine qui m’am’na /s�.ma.dl�n.ki.ma.mna/

(It’s Mad’leine who brought me ’round)



Vocal pointing:


Introduction

Madeleine m’amena

3s +

3s x

3s

MADELEINE m’amena

3s

…14 silent productions

of the sentence = 42s

� Audio validation

Duration and F0 analyses using Praat software.

Checked that pre- and post-scan audio recordings were similar.

� Paradigm and task:

-extensive training (inner speech) a day before the experiment

-pre- and post-scan audio recordings

to assess subject’s performance inside scanner

-3 functional scans using a block paradigm

-each scan = 8 epochs (each condition twice) of 42 s

-order of presentation alternated between scans and subjects



Vocal pointing:


Introduction

�MRI acquisition

-1.5 MR imager (philips NT) with EPI

-25 adjacent, axial, slices (5mm) were imaged 10 times during each

epoch

-imaging volume parallel to the bi-commissural plane

-major MR parameters:

TR = 3700 ms

TE = 45 ms

pulse angle = 90°

acquisition matrix = 64x64

reconstruction matrix = 128x128

field-of-view = 256x256 mm2

-high-resolution 3D anatomical MR scan between first and second

functional scans.



Vocal pointing:


Introduction

� MRI data processing

SPM-99 software

Pre-processing steps:

- Motion correction

- Spatial normalization with MNI template

- Spatial smoothing

� Statistics

Contrasts between conditions using the GL Model.

Statistical significance threshold p = 0.001.

Clusters of activated pixels were identified, based on the intensity of

the individual responses and the spatial extent of the clusters.

Finally, a significance threshold of p = 0.05 (corrected) was applied

for identification of the activated clusters.

Fixed effect group analysis.



Vocal pointing:


Introduction



Vocal pointing:


Introduction

LIFG + insula

Premotorcortex

Wernicke

SMG

sensory

expectation

auditory

descriptionaction analysis

RR LL

Prosodic pointing recruits a left temporo-parieto-frontal network

Cf. TPF in action imitation (Iacoboni et al., 2003)



Vocal pointing:


Introduction

� The description of the actions to be imitated is handled by the

superior temporal sulcus (STS) and sent to the posterior parietal

cortex, where it is combined with additional somatosensory

information.� This completed description is then sent to the frontal cortex where

the goal of the actions to be imitated is coded.

� Reafferent copies of the imitated actions are sent back to the STS

for action monitoring.

Temporo-Parieto-Frontal network

in vocal pointing



Vocal pointing:


Introduction

� A left temporo-parieto-frontal network might be recruited in the

organization of vocal pointing actions from auditory representations.

� The inferior parietal region seems to function as a sensory

integrator forming representations necessary in the organization of

linguistic pointing at targets.

� In analogy with visually-guided manual pointing, prosodic pointing

may need integrated representations (auditory and articulatory) to

be formed via the superior temporal and parietal regions in order to

organize articulation and phonation in an adequate prosodic pattern.

� Prosodic pointing, i.e. expressive orofacial pointing may be

considered in continuity with ocular and manual pointing.

…To

From…



Vocal pointing:


Introduction

Pointing with syntax

Premotorcortex

LIFG + insula

� Syntactic (with or without prosody) pointing is grammaticalised,

automatized.

It does not recruit the entire TPF network and is handled solely by

the left IFG.

� No activation of the LSMG or Wernicke’s area (no sensory or

auditory expectations)

X

LLRR



Vocal pointing:


Introduction


of multimodal pointing?

Multimodal Pointing

The aim of this fMRI study of prosodic, syntactic,

manual and ocular pointing

is to test the hypothesis that these different pointing

modalities could activate neighbouring cerebral areas.

Lœvenbruck et al. (2009)



Vocal pointing:


Introduction

Material

� 3 Tesla MR imager (Bruker)

� Overt speech: Oral productions recorded using an fMRI compatible microphone



Vocal pointing:


Introduction

Noise reduction: - Microphone outside scanner, at one end of a wave guide

- Other end of guide connected to a mask placed over subject’s mouth

Subject training

Inside the tunnel: the subject wears a mask and can see the stimuli on the rear-view mirror

Cardboard tunnel and paper video-projection screen



Vocal pointing:


Introduction

3T fMRI session

The subject wears the mask and can see the stimuli on the rear-view mirror.

The elbow lies on an arm rest during manual pointing, finger movements are monitored using a digital camera (outside the room, behind window).

Eye-movement are monitored using an ASL 504 eye-tracker during ocular pointing.

� 15 right-handed speakers of French

(9 women and 6 men), aged 18-55 yrs

� 4 sessions: - prosodic focus (vocal pointing)

- syntactic extraction (vocal pointing)

- index finger pointing (digital pointing)

- eye pointing (ocular pointing)

� event-related fMRI paradigm

� Each session : 5 conditions (pointing vs. control, preparation vs. execution, null event) lasting 4.5s each



Vocal pointing:


Introduction



Vocal pointing:


Introduction

Underlying question: - Does Lise hold the book?

- No, it’s JULES who holds the book.

= pointing with the eye, the index or the voice

-Yes.

= no pointing

Prosodic modality Syntactic modality

Control “Lise tient le livre.” “Lise tient le livre.”condition “Lise holds the book.” “Lise holds the book.”

Pointing “JULES tient le livre.” “C’est Jules qui tient le livre.”Condition “JULES holds the book.” “It’s Jules who holds the book.”



Vocal pointing:


Introduction

Digital modalitymodality Ocular modality

Control

downward finger movement downward look

Pointing finger pointing eye pointingtowards Jules towards Jules



Vocal pointing:


Introduction

Control condition utterance

Thanks to the mask, the sentence pronounced by the subject (Thanks to the mask, the sentence pronounced by the subject (““LiseLise tienttient le le livrelivre””) is clearly audible, despite the ambient noise (115dB). ) is clearly audible, despite the ambient noise (115dB). F0 tracking is even possible.F0 tracking is even possible.



Vocal pointing:


Introduction

Prosodic pointing utterance

An F0 rise on “JULES” and a post-focal deaccentuation are observed

here, typical of a deictic intonation.



Vocal pointing:


Introduction



Vocal pointing:


Introduction

Syntactic pointing utterance

Digital pointing

- Superior parietal lobule (BA 7)bilaterally, with left predominance

- Inferior parietal lobule (BA 40) bilaterally, with left predominance

- Left frontal cortex (BA 4, 6)

LLRR

Ocular pointing

- Left postcentral gyrus (BA 3)

- Bilateral frontal lobe (BA 4, 6)

- Bilateral occipital areas (BA 17, 18)

LLRR

Prosodic pointing

- Superior parietal lobule (BA 7) bilaterally, with left dominance

- Left postcentral gyrus (BA 2, 3)

- Bilateral perisylvian areas (BA 47, 13, 42)

LLRR

Yellow : digital

Cyan : ocular

Red : prosodic

In the superior parietal lobule:

Green: digital + ocular

Purple: ocular + prosodic

Syntactic pointing

- No left parietal activation (right BA 40 only slightly activated)

- Right perisylvian areas (BA 44, 47, 41)

- Bilateral premotor cortex (BA 6)

x



Vocal pointing:


Introduction

Discussion

- Activation of left superior parietal lobule (BA7 and neighbouring BA

3) in manual, ocular and prosodic pointing.

- Ocular pointing more anterior than prosodic pointing, itself more

anterior than digital pointing.

- Speakers may use multisensory (auditory, somatosensory)

representations to produce prosodic pointing, just as they do in

manual and ocular pointing. These representations may be formed

via the activation of associative parietal areas.

- Non-significant parietal activation during syntactic pointing

(grammaticalisation = frozen construction = no multisensory

representation necessary = deparietalisation)



Vocal pointing:


Introduction

Conclusion

These results suggest that the left superior parietal lobule could be a

potential substrate for pointing with the finger, the eye, and the

larynx.

They are in line with our conjecture that linguistic on-line pointing

(prosodic focus) is grounded in the same cerebral network as

gestural (manual and ocular) pointings.

They shed light on the complementary relationship between syntax

and prosody, with syntactic focus conceived as more ‘frozen’ or

‘stored’ and prosodic focus as more ‘built on-line’.

Many thanks to all our participants and to:

MonicaBaciu

Christian Abry

Marion Dohen

LaurentLamalle

ChristophSegebarth

Jean-Luc Schwartz

Coriandre Vilain

CédricPichat

Marie-AgnèsCathiard

FrancescaCarota

Pascal Perrier

Christophe Savariaux

A handful of thanks to Jean-François Dars, a point photographer

some articulatory and cerebral correlates of · pdf filesome articulatory and cerebral...

Documents