Temporal Processing for Syntax Acquisition: A simulation study

Jean-Marc Blanc (blanc@isc.cnrs.fr)Christelle Dodane (dodane@isc.cnrs.fr)

Peter Ford Dominey (dominey@isc.cnrs.fr)Institut des Sciences Cognitives

UMR 5015 CNRS-Université Claude Bernard Lyon 167, boulevard Pinel 69675 BRON cedex

Abstract

Early perceptual processing capabilities are likely tocontribute to the categorization of lexical vs.grammatical words by newborns. This lexicalcategorization could be performed by detectingdifferences in the prosodic structure of these wordcategories. Here we demonstrate that a TemporalRecurrent Network (TRN) that allows realistic treatmentof the dynamic temporal aspect of prosody performs thislexical categorization task on French and English. Wethen examine the functional relation between thiscapability, and non-linguistic temporal discrimination.We reduce sensitivity to temporal structure in the TRNby increasing the network time constants. This yields (1)a reduction in performance in the lexical categorizationtask, and (2) a deficit in the processing of brief auditoryevents similar to that observed for children with SLI.While our principle result is that the TRN can performlexical categorization based on prosodic structure, it is ofinterest that the impaired TRN suggests a functional linkbetween impaired temporal processing, and impairedlexical categorization in SLI.

IntroductionOne of the most challenging questions in child languageacquisition is how children learn syntax. Acquiringlanguage involves classifying lexical items intosyntactic categories. Four main sources of informationin linguistic input have been proposed as potentiallyuseful in this categorization: Distributional Information(Redington, Chater & Finch 1998), SemanticBootstrapping, Phonological Constraints (Kelly 1995)and prosodic information (Shi, Werker & Morgan1998). With respect to prosodic information coded inthe fundamental frequency (F0), and lexicalcategorization, we note that in French, content wordswithin a prosodic group (i.e. not at a group boundary)are marked with F0 peaks, and the F0 of content wordsin English are marked with a principal accent (Hirst &Di Cristo 1998).

The current research tests the hypothesis that theseaspects of the temporal structure of prosody can beprocessed by a temporal recurrent network (TRN), thusproviding insight into lexical categorization andlanguage acquisition.

Prosodic Foundations for SyntaxJusczyk et al. (1996) proposed that infants are able tomake immediate use of their sensitivity to prosodicmarkers as a means for organizing the input. We willnow examine several investigations that have beenrealized to confirm this hypothesis.

Sensitivy to prosodic structureAdults: Bagou et al. (2002) evaluated the relative

contribution of two prosodic cues, lengthening and F0contour, in the processes of speech segmentation andstorage of new words. Their results showed thatprosodic information facilitates the acquisition of a newmini-language. Kelly (1995) demonstrated thatAmerican subjects were able to exploit phonologicalcues to identify unknown words as verbs or nouns.

Children: Though children frequently fail toproduce function morphemes in their earliest utterances,Gerken and McIntosh (1993) have suggested thatchildren by the age of 2 years have a representation ofsome specific function morphemes, and the context inwhich they appear. We will now consider the ability ofnewborns to exploit prosodic cues for the early basis ofsyntax.

Newborns: French newborns can discriminatebetween English and Japanese sentences (differentrhythm classes), but not between Dutch and Englishsentences (same rhythm class). This discrimination waspossible despite the speech being low-pass filtered (at400 Hz), highlighting the role of prosody (Nazzi,Bertoncini & Mehler, 1998). Moreover, newborninfants are able to use probabilistic combinations ofacoustic and phonological information to perceptuallyseparate English word tokens into function/contentcategories (Shi et al. 1999). At a later stage, such anability could potentially help bootstrap infants intoacquisition of grammar by allowing them to detect andrepresent classes of words on the basis of perceptiblesurface cues. However, the specific cues used bynewborns to make this distinction have not beenprecisely determined.

Simulation Shi et al. (1998) investigated if various“presyntactic cues” (such as number of syllables,

145

presence of a complex syllable nucleus, presence ofsyllable coda, and syllable duration, to name only a fewphonologically relevant cues) are sufficient to guide theassignment of words to rudimentary grammaticalcategories. Their investigation of English, MandarinChinese and Turkish shows that “sets of distributional,phonological, and acoustic cues distinguishing lexicaland functional items are available in infant-directedspeech across such typologically distinct languages asMandarin and Turkish” (Shi et al. 1998). Thusgrammatical words tended to be acoustically and/orphonologically minimized in comparison to lexicalwords.

Durieux and Gillis (2000) proposed an artificiallearning system for lexical categorisation with English,based on phonological and prosodic information.However in these two simulation studies, a number ofspecific cues were extracted from the speech by ahuman expert. Here, we investigate whether a neuro-realistic system can automatically extract cues from thecontour of the fundamental frequency itself.

Material & Methods

CorporaLSCP: This corpus contains 54 French sentences readby a single native speaker. The segmentation providedgroups of adjacent content and function words (~ 200for each category; Ramus et al. 1999). The corpusserved in part for Experiments 1 and 3.

MULTEXT Experiment 2 used French and Englishspeech from the MULTEXT multilingual corpusdeveloped for the study of prosody (Campione &Veronis, 1998). Stories were read by 20 differentspeakers (5 males and 5 females per language) whichlead to a total of 8236 words for English, and 6945words for French.

Fundamental Frequency (F0)Extraction Fundamental frequency (Raw data) wasobtained from the speech signal autocorrelation(PRAAT software; Boersma 1993). This description ofF0 was provided as input to the TRN.Processing To obtain an acceptable perceptualrepresentation of intonation based on raw values of F0,we apply the MOMEL quadratic spline smoothingalgorithm (Hirst & Espesser (1993), yielding a smoothcontinuous curve reflecting the intonation contour withwhich the utterance is spoken.

Temporal Recurrent Network (TRN)Architecture: The TRN is based on the dynamiccoding properties of recurrently connected leakyintegrator neurons with sigmoid output functions that

are organized in three layers: the Input, consisting ofsixty units, that transmits prosodic information to theState layer, which projects to the StateD layer (SeeFigure 1). Recurrently connected State and StateDlayers are each made up of 25 units, and StateD neuronsinclude several different time constants to allowsensitivity to different time scales.Principles: The connections between the differentlayers are randomly initialized (-0.5; +0.5) at the outsetand remain fixed. Weights are then selected from apopulation of networks based on performance. Learningconsists of evaluating the representation of inputsequences in the activities of State and StateD after asequence has been presented.Treatment of prosody: The dynamic system thatresults from this recurrent network has beendemonstrated to be sensitive to the temporal and serialorder of input sequences, and was shown todiscriminate between languages of different rhythmclasses based on their prosodic structure (Dominey &Ramus 2000).

Figure 1: Temporal Recurrent Network Architecture

Representation of inputs F0 was coded in Hz at 5mssampling intervals as a population of activity with aGaussian distribution, where the unit with the highestactivation encodes the mean frequency.

Exp. 1: Development of LexicalIdentification

The purpose of Experiment 1 was the identification,based on peaks in F0, of two lexical categories:Content words that have a meaning-related componentsuch as nouns, verbs, adjectives, and adverbs, andFunction words that are primarily structural, such asarticles, prepositions, and auxiliaries.

Exp 1a: Linguistics InvestigationsThe location of F0 peaks is based on the detection of achange in the sign of the difference between twoadjacent values of F0. Thus local minima and maxima(F0 peaks) are detected. To eliminate irrelevant peaksdue to deviations in the raw F0 data, the MOMEL

F0 StateD

StateInput

146

smoothing algorithm was applied. We observed that92% of F0 peaks and 61% of F0 valleys occurred incontent words, with the opposite distribution forfunction words. Applying the rule that the presence of aF0 peak indicates that a word belongs to Contentcategory, whereas the absence of this cue revealed aFunction word yields the results in Table 1, wherechance is 50,6%.

Table 1: French lexical categorization with LSCP

Corpus Method F0 % correctRandom 50,6F0 peaks Raw data 62,1F0 peaks MOMEL 82,8TRN Raw data 83,6

LSCPFrench

TRN+SOM Raw data 84,1

Exp 1b: Temporal Neuro-inspired SimulationsThe TRN was trained on half of the function (F) andcontent (C) words at group boundaries. Afterpresentation of a F or C word, the vector of activity inState/StateD was sampled, and averaged into aPrototype vector, yielding prototype vectors for F and Crespectively. Validation consisted of presenting thesecond half of the corpus, and identifying words basedon their distance from the prototypes. We alsoconducted experiment with a Self-Organized Map toclassify these vectors in a unsupervised way. Table 1presents performance for the best of a population of 50networks on the validation, for supervised (TRN) andunsupervised learning (TRN+SOM).

Exp 2: Application to larger CorporaExperiment 2 applies the Peak and TRN methods ofidentification to a larger corpus (MULTEXT) with tendifferent speakers. The speech signal is divided intowords rather than groups of words belonging to thesame lexical category and the speech is closer tounconstrained speech than that in the LSCP corpus.

FrenchTable 2 demonstrates that the TRN performs theContent/Function distinction within the same range asthe explicit detection of F0 peaks, superior to chance.The LSCP and MULTEXT French corpora yieldeddifferent performance, and their syntactic contentdiffers notably. The LSCP corpus contains onlysentences between 15 and 21 syllables, whereas theMULTEXT corpus is based on short passages includingsyntactic structure specific to spontaneous speech.Nonetheless, despite these differences, lexicalcategorization remains possible.

EnglishWe subsequently tested whether F0 peaks were alsocharacteristic of content words in English. Table 2indicates that this intonation contour could be used toobtain the same lexical identification in English. Weconclude that, the TRN can perform lexicalcategorization at a level comparable to human infants(Shi et al. 1999). However, as pointed out by Shi et al.1998, the contribution of a given cue to lexicalcategorization may vary across languages. Thus thepresence of an F0 peak may have less impact inEnglish than in French.

Table 2: Lexical categorization with MULTEXT

Language Method F0 % correctRandom 52Peaks MOMEL 73,1FrenchTRN Raw data 70,3Random 54Peaks MOMEL 64,5EnglishTRN Raw data 62,8

Impaired Prosodic Foundations for SyntaxIn contrast to normally developing children, there existsa subgroup of children characterized by a significantlimitation in reading and/or language development andability without the presence of an overt underlyingcondition such as low overall IQ or impaired hearing.This condition is often referred to as Specific LanguageImpairment (SLI). Several investigators have proposedthat individuals with SLI have special problems withthe acquisition of functional categories (Eyer &Leonard, 1995; Leonard 1995; Bishop 1979). Van derLely (1996) suggested that children with SLI hadgreater difficulty assigning roles such as agent andtheme or theme and goal on the basis of syntacticstructure alone.

The origin of the processing deficits in languageimpaired populations is still a question of debate. Tosummarize, it could be due to basic sensory processingdeficits (Tallal et al. 1993) or to higher-level cognitiveprocesses. The first hypothesis postulated thatabnormalities in the neurophysiological encoding ofacoustic differences in speech are responsible for thisdeficit. This impairment of auditory temporalprocessing was tested in several auditory experimentsemploying non-verbal stimuli. When stimuli are eitherbrief or rapid, children with SLI have difficulty indiscriminating them, although they have no difficulty indifferentiating the same stimuli when they arelengthened or presented at slower rate. At least asubgroup of children with SLI exhibits a deficit forprocessing rapid auditory events at the same time that

147

they have trouble with the manipulation of functionwords and phonemes (Tallal 1998).

These observations lead us to three importantexperimental considerations: Exp 1a: Prosodic cues canallow the distinction of function versus content words;Exp 1b and Exp 2: This prosodic structure can beprocessed by a temporal neural network model toperform this identification; Exp 3: A deficit in temporalprosodic processing could produce impairments both inlexical categorization and in auditory temporaldiscrimination tasks based on pure tones.

Exp. 3: Simulation of a Temporal DeficitIn Exps 1 and 2, the processing of temporal contextlikely plays an important role, as F0 peaks are definedwithin a progressive rising and falling context, implyinga required sensitivity to peaks and the surroundingtemporal context. In this context, the TRN is sensitiveto serial and temporal structure. Thus, the objective ofthis experiment is to determine if perturbation of thetemporal sensitivity of the network will yield the sameimpairments as observed in children with SLI for themanipulation of syntax (i.e. lexical categorization), andalso for auditory processing of rapid events.

Impaired Lexical CategorizationA population of 50 control networks was able toperform lexical categorization with a mean of 75%correct on the LSCP corpus. To reduce sensitivity totemporal structure in the TRN we increased the networktime constants to produce 50 distorted networks. Noneof the distorted networks were able to perform thecategorization task (mean of 50%, and maximum for53%), whereas these models could perform a sequenceidentification task based on 3-element sequences withelements of long duration (800 ms).

Auditory Repetition TaskThe Rapid Perception Test was elaborated to test theability of children with SLI to handle rapid auditoryevents. Tallal and Piercy (1973) demonstrated thatnormal children were able to identify the order of two75ms tones separated by an inter-stimulus interval (ISI)as short as 8ms, while individuals with SLI required anISI exceeding 300ms (Fig 2).

These experiments were replicated on the twopopulations of TRN (control and distorted), based oncomparing the State/StateD vectors in a discriminationmeasure. Figure 2 reveals that the same disorder wasobserved for the distorted network which failed in thecase of short ISIs. As observed in children with SLI,performance improves rapidly for longer ISI durationsfor the distorted networks.

Backward MaskingWright et al. (1997) tested the hypothesis that theauditory temporal processing deficits observed in SLIchildren could be related to impairments in thedetection of temporally interacting stimuli as in tasks offorward and backward masking. They reported thatcontrol children showed a significantly lower tonethresholds than SLI children for detecting a tone in thebackward masked condition, in which the target toneimmediately follows a white noise mask.

The model performs this backward masking task inwhich the presence or absence of a 20 ms tone (codedby activation of specific F0 units) should be detectedafter a 300 ms noise mask (coded by uniform activationof F0 units) (Fig 3). Wright et al. (1997) demonstratedthat children with SLI required significantly higherintensity levels for the tone than normal children toperform this task. This experiment reveals the samedistinction between the normal and distorted networks,as between normal and SLI children, as illustrated inFigure 3.

Discussion

Implication for a Simulated Acquisition ofLanguageThe first two experiments demonstrated that theintonation contour could serve as basis for anidentification of Function vs. Content words, that couldbootstrap the acquisition of syntax. Furthermore wedemonstrated that a biologically plausible recurrentnetwork system could extract reliable information fromthe fundamental frequency, without human labeling ofspeech samples (as was required in Durieux & Gillis2000; Shi et al. 1998). This observation has beendemonstrated on two languages, English and French.

In this context it would of interest to study ChildDirected Speech (CDS), that is known to includingmore salient information. Indeed CDS displays anexaggeration of prosody, in particular in the contour ofF0. In addition to previous work, supplementaryinformation (like a representation of spectrum) could beprovided to the network to increase the performance ofthe lexical categorization.

Neurophysiological ImplicationThe TRN is a an extension of a model originallydeveloped to simulate primate behavioralelectrophysiology experiments (Dominey et al. 1995).In the current context it demonstrated how temporalprocessing can contribute a rudimentary basis forsyntax in the form of lexical categorization. Thisconfirmed the TRN’s utility in processing prosodic anddistributional regularities (Dominey & Ramus 2000).

148

Figure 2: Comparison of children performance versus TRN on the Auditory Repetition Task (Tallal & Piercy 1973)

Figure 3: Tone threshold for TRN and children for the first two conditions of the task in Wright et al. 1997.

All these experiments lead to the conclusion that afine-grained treatment of temporal structure in speech isof primary importance for initiating syntax acquisition.

Deficits in Prosodic Processing, and SLISeveral studies have reported the difficulties faced bychildren with SLI with manipulation of function words,together with a deficit in treatment of rapid auditoryevents. The time window implied in this deficientprocessing has been proposed to be roughly theduration of a syllable (Tallal & Piercy, 1973). In thiscase it will be difficult to localize F0 peaks within theword structures of sentences. In this context the TRNfailed to identify Content and Function words, when itssensibility to fine temporal structure was altered. Tallaland al. (1998) indicated that this auditory deficit wasreversible, indeed after training on the ART task,

thresholds approached the normal tens-of-ms range.These results pose interesting simulation goals, and weare currently investigating the effects of training anddiscrimination threshold modifications for improvingtask performance in our SLI simulations.

ConclusionThree objectives were achieved with these experiments.First, two basic categories (Function and ContentWords) could be identified from the prosodic structureof speech. Second, this distinction could be performedby a Temporal Recurrent Network, that was developedin a functional neurophysiology framework (Domineyet al. 1995). Third, the TRN could be perturbed tosimulate the behavior of children with SLI for detectionof rapid auditory events, while at the same time itslexical categorization based on prosody is severelyattenuated. This provides the basis for future studies of

T R N S i m u l a t i o n

50

60

70

80

90

100

110

8 30 150 428 1466 3023

IS I (ms)

Pef

orm

ance

(%

)

Cont ro l 0 ,01 Distor ted 0 .5

C h i l d r e n

5 0

6 0

7 0

8 0

9 0

1 0 0

1 1 0

8 3 0 1 5 0 4 2 8 1 4 6 6 3 0 2 3

I S I ( m s )

Per

form

ance

(%

)

C o n t r o l C h i l d r e n w i t h S L I

T R N S im u la t ion

0

2 0

4 0

6 0

8 0

1 0 0

l o n g - t o n e b a c k w a r d

C o n tro l 0 .01 D is t o r t e d 0 . 5

Children

0

20

40

60

80

100

long-tone backward

Th

resh

old

(d

B)

Control Children withSLI

149

the contribution of temporal structure of speech tonormal and impaired language acquisition.

AcknowledgmentsJMB is supported by a Doctoral fellowship from theRégion Rhone-Alpes, CD by a Post-Doctoral fellowshipfrom the HFSP Organization, and PFD by the ACIIntegrative and Computational Neuroscience, theOHLL and the Eurocores OMLL Projects. We thank F.Ramus, A. Christophe and E. Dupoux for insightfuldiscussion and access to the LSCP corpus.

ReferencesBagou, O., Fougeron, C., & Frauenfelder, U.H. (2002).

Contribution of Prosody to the Segmentation andStorage of "Words" in the Acquisition of a NewMini-Language, Prosody 2002, 11-13 April 2002,159-162.

Bishop, D.V.M. (1979). Comprehension indevelopmental language disorder. DevelopmentalMedicine and Child neurology, 21, (pp. 225-238).

Boesrma, P., Accurate short-term analysis of thefundamental frequency and the harmonics-to-noiseratio of a sampled sound. Proceedings of the Instituteof Phonetic Sciences 17, pp. 97-110.

Campione, E., & Véronis, J., (1998). A multilingualprosodic database, Proc. of ICSLP'98, Sidney.

Dominey, P.F., Arbib M.A., & Joseph J.P. (1995). AModel of Cortico-Striatal Plasticity for LearningOculomotor Associations and Sequences, Journal ofCognitive Neuroscience, 7:3, 311-33.

Dominey P.F., & Ramus F. (2000). Neural networkprocessing of natural language: I. Sensitivity to serial,temporal and abstract structure of language in theinfant. Language and Cognitive Processes, 15(1)87-127.

Durieux, G., & Gillis, S. (2000). Predictinggrammatical classes from phonological cues: Anempirical test. In Approaches to bootstrapping:phonological, syntactic and neurophysiologicalaspects of early language acquisition, ed. B. Höhle,J. Weissenborn, pp. 189-232. Amsterdam: Benjamins

Eyer J., Leonard L., (1995). Functional categories andspecific language impairment: A case study.Language Acquisition, 4, 177-203.

Gerken, L. A., & McIntosh, B. J. (1993). The interplayof function morphemes and prosody in earlylanguage. Developmental Psychology, 29, 448-457

Hirst D., & Di Cristo A., (1998). A survey of intonationsystems. in Hirst & Di Cristo (eds). IntonationSystems : A Survey of Twenty Languages.,1-44.

Hirst, D. & Espesser, R.(1993). Automatic modelling offundamental frequency using a quadratic splinefunction. Travaux de l'Institut de Phonétique d'Aix15, 71-85.

Jusczyk, P.W., Kelmer Nelson, D.G. (1996). Syntacticunits, prosody, and psychological reality during

infancy, In JL Morgan, K Demuth (Eds) Signal toSyntax: Bootstrapping from speech to grammar inearly acquisition, Lawrence Erlbaum, Mahwah NJ.

Kelly, M.H. (1995). The role of phonology ingrammatical category assignments. In J.L. Morganand K. Demuth (Eds.), Signal to syntax:Bootstrapping from speech to grammar in earlyacquisition (pp. 249-262). Mahwah, NJ: LawrenceErlbaum Associates.

Van der Lely, H. K. J. (1996). Specifically languageimpaired and normally developing children: Verbalpassive vs. adjectival passive sentence interpretation,Lingua 98-4, 243-272.

Leonard, L. (1995). Functional categories in thegrammars of children with specific languageimpairment. Journal of Speech and Hearing Research,38, 1270-1283. Leonard L. (1998). Children withspecific language impairment. Cambridge, MA: MITPress.

Nazzi, T., Bertoncini, J., & Mehler, J. (1998). Languagediscrimination by newborns: towards anunderstanding of the role of rhythm. Journal ofExperimental Psychology: Human Perception andPerformance.

Ramus, F., Nespor, M., & Mehler, J. (1999). Correlatesof linguistic rhythm in the speech signal. Cognition,73(3), 265-292.

Redington, M., Chater, N. & Finch, S. (1998).Distributional information: A powerful cue foracquiring syntactic categories. Cognitive Science, 22,425-469.

Shi R., Morgan J.L., & Allopenna P. (1998).Phonological and acoustic bases for earliestgrammatical category asssignment : a cross linguisticperspective. Journal of child language 25 (1998),169-201.

Shi R., Werker J.F., & Morgan J.L. (1999). Newborninfants' sensitivity to perceptual cues to lexical andgrammatical words, Cognition, Volume 72, Issue 2,B11-B21.

Tallal P., & Piercy, M. (1973). Defects of non-verbalauditory perception in children with developmentalaphasia. Nature. Feb 16;241(5390):468-9

Tallal, P. (1998). Children with language impairmentcan be accurately identified using temporalprocessing measures: A response to Zhang andTomblin, Brain and Language, 65, 395-403 BrainLang. 1999 Sep;69(2):222-9.

Tallal, P., Merzenich, M., Miller, S., & Jenkins, W.(1998). Language Learning Impairments: IntegratingBasic Science, Technology and Remediation,Experimental Brain Research, v. 123, p 210-219.

Wright, B.A., Lombardino, L.J., King, W..M, Puranik,C.S., Leonard, C.M., & Merzenich, M.M. (1997).Deficits in auditory temporal and spectral resolutionin language-impaired children. Nature. May 8;387(6629):176-8.

150