Gesture and speech are controlled by the same systemGesture and speech are controlled by the same system

Maurizio GentilucciMaurizio Gentilucci

Dipartimento di Neuroscienze, Università di Parma, Italy

RTM (Rete Multidisciplinare Tecnologica), IIT (Istituto Italiano di Tecnologia) and Università di Parma

Gesture and speech belong to two distinct communication systems (Krauss e Hadar 1999). Gesture functions as an auxiliary support when verbal expression is temporally disrupted or word retrieval is difficult. Gesture and speech belong to the same communication system (Goldin-Meadow 1999, McNeil 1992). They are linked to the same thought processes even if the expression modality differs.

GESTURE: GESTICULATION (DEICTIC, MOTORIC, REPRESENTATIONAL), ICONIC, SYMBOLIC, SIGN. (Kendon 2004, McNeill 2000)

Monkey grasps with the mouth

Monkey grasps with the right handMonkey grasps with the left hand

The link between gesture and speech (and in general language) supporting the view that gesture and speech are controlled by a same system can be the result of the activity of a system evolved from a class of neurons recorded in monkey premotor area F5 (Rizzolatti et al. 1988).

This system (the dual hand-mouth command system) can be involved in transferring aspects of the control (and meaning) of manual actions to the control of mouth postures and vice versa aspects of the control (and meaning) of mouth actions to the control of hand postures.

The role of understanding the meaning of hand and mouth gestures can be carried out by a system evolved from the class of the mirror neurons (Gallese et al., 1996; Ferrari et al., 2003)

Grasp of a small and large object and simultaneous opening of the mouth.

Grasp of the small and large object with the mouth and simultaneous opening of the fingers

From an evolutionary point of view, this circuit of commands to both hand and mouth might have been

used by humans to transfer an initial arm gesture communication system from hand to mouth

(Armstrong, Stokoe and Wilkox 1995, Gentilucci and Corballis 2006). It can also be used to couple gestures

to speech (Gentilucci et al. 2008).

A system transferring aspects of the grasp control to speech seems to be active in modern human behavior

Grasp of a small and large object and simultaneous pronunciation of syllables.

THE CONTROL OF THE GRASP WITH THE HAND AND THE MOUTH AFFECTS THE CONTROL OF POSTURES OF THE MOUTH AND THE HAND.

THE CONTROL OF THE GRASP WITH THE HAND AFFECTS THE PRONUNCIATION OF PHONEMIC UNITS

Effects of hand postures, mouth postures, foot postures, and vocalizations on the kinematics of reach to grasp of differently sized

objects executed with hand and mouth

Can assumed postures of one effector (mouth or hand) and specifically

configurations mimicking the interaction with objects of different size affect the

control of the simultaneous grasp with another distal effector (hand or mouth)? An

affirmative response to the question can support the idea that representations of

the interactions of a distal effector with objects are sufficient to affect the control

of the grasp of another distal effector. In addition, it excludes the possibility about

a simple temporal coupling between the movements of the two effectors. Second,

is the reciprocal interaction between distal effectors restricted to hand and mouth

or can be it extended to foot postures?

Experiment 1: reaching with the mouth and grasping with the mouth after assuming the posture of power grasp, precision grasp and relaxed hand.Experiment 2: reaching with the mouth and grasping with the mouth after assuming the posture of extended, flexed and relaxed fingers

Experiment 3: reaching-grasping with the hand after opening. closing and relaxing the mouth

Experiment 4: reaching-grasping with the hand after extending, flexing, relaxing the toes.

We examined the possibility that speech can affect grasp by considering that vocalizations require particular postures of the internal mouth (Leoni and Maturi 2002). The open vowels, such as /a/, are related to large internal mouth apertures, whereas closed vowels, such as /i/, are related to small internal mouth apertures. If external mouth postures affect the control of grasp (experiment 3), it is possible that even specific internal mouth postures assumed for pronunciation of vowels affect the control of grasp.

Experiment 5: reaching to grasp the objects while producing different vocalizations, i.e. /a/, /ɔ/ and /i/.

THE CONTROL OF POSTURES OF HAND AND MOUTH RATHER THAN TOES, AFFECTS THE CONTROL OF GRASP WITH MOUTH AND HAND.

VOCALIZATIONS (OPEN AND CLOSED VOWELS) AFFECT THE CONTROL OF HAND GRASP

We found that deictic gestures (HERE, THERE, Chieffi et al. 2009) as well as communicative gestures (CIAO,NO, STOP,Barbieri et al. 2009; Bernardis and Gentilucci, 2006; Gentilucci et al. 2006) reciprocally interact with the words of the same meaning.

We tested the hypothesis whether a system relating hand gestures to internal mouth postures used to vocalize exists and it can be considered precursor of a system relating gestures to words.

Effects of the production of representational gestures LARGE and SMALL (unimanual and bimanual gestures) on

1) Production of vowels (“A”, “I”; experiment 1)

2) Pronunciation of words (“GRÀNDE”, large, “PÌCCOLO”, small) (experiment 2)

3)Pronunciation of pseudo-words (“SCRÀNTA”, “SBÌCCARA”, experiment 3)

Production of the vowels “A” and “I”

Pronunciation of the words “GRÀNDE” (large) and “PÌCCOLO” (small)

Pronunciation of the pseudo-words “SCRÀNTA” and “SBÌCCARA”

A system coupling unimanual gestures to vocalization exists and may be considered precursor of a more distributed system relating gestures to words.

This system, on the basis of word meaning categorized the gestures in two classes (LARGE and SMALL) creating a size representation independent of effector. This influenced the pronunciation of the vowels included in the words. The gesture LARGE (bimanual condition) induced an increase in formant 1 (F1). The increase in F1 corresponds to an increase in internal mouth aperture. Conversely, the words influenced the hand aperture.

Pronouncing pseudo-words was responsible for a size representation not yet independent of the gesturing effectors.

fMRI (Willems et al. 2007, Xu et al. 2009) and rTMS (Gentilucci et al. 2006) experiments suggest that il circuit formed by Pars opercularis, Pars triangularis, and Pars orbicolaris of Inferior Frontal Gyrus is involved in the integration between gesture and speech.

Thank you! and…