Simulation of Language Acquisition

Walter Daelemans

(CNTS, University of Antwerp)walter.daelemans@ua.ac.behttp://www.cnts.ua.ac.be/~walter

EMLAR 2005 Utrecht

Overview• Theories, computational models and

simulations• Machine Learning

– Generalization versus abstraction– Eager versus lazy learning

• Memory-based models of language acquisition and processing

• Case Study 1: Stress acquisition• TiMBL crash course and demonstration• Case Study 2: German plural

Simulation (1)

• Theory– Explains and predicts empirical data

(observations, experimental results)– Cogsci: in terms of knowledge representation,

acquisition, and processing framework– Problems

• Verbal• Sometimes vague, underspecified• Every theoretical description, however exact, turns

out to contain errors when you try to implement it (~ Hugo Brandt Corstius, second law of Computational Linguistics)

Simulation (2)• Computational Model

– Translation of a theory into specific symbol representation and processing framework (algorithms and data structures)

– Advantages• Precise formulation• Explicit in all details• Consistence and completeness can sometimes be

proven• Falsifiable through simulations

• Simulations– A computational model with specific

parameter settings used to mimic specific empirical data

Machine Learningas a model for acquisition

• Cognitive architecture– Competence (knowledge representation)– Performance (search)– Acquisition (search)

• Bias – Restrictions on input and output

representations– Restrictions on learning algorithm– Restrictions on knowledge representation

formalism

Output Input

Performance Component

Ri RkRl

Rj

Learning Component

Search

Experience

BIAS

Generalisation Abstraction

+ abstraction

- abstraction

+ generalisation - generalisation

Rule InductionConnectionism

StatisticsHandcrafting

Table LookupMemory-Based Learning

…(Fill in your most hated

linguist here)

Nativism Rule-Based

nativist

empiricist

+ rule-based - rule-based

Innate mental rules

Rule Induction ConnectionismStatistics

Memory-Based Learning

Hard-wired neural networksInnate probabilities?

Innate exemplars?

Machine Learning crash course

The field of machine learning is concerned with the question of how to construct computer programs that automatically learn with experience. (Mitchell, 1997)

• Dynamic process: learner L shows improvement on task T after learning.

• Getting rid of programming.• Handcrafting versus learning.• Machine Learning is task-independent.

Machine Learning: Roots

• Information theory• Artificial intelligence • Pattern recognition • Took off during 70s • Major algorithmic improvements during

80s • Forking: neural networks, data mining

Machine Learning: 2 types

• Theoretical ML (what can be proven to be learnable by what?) – Gold, identification in the limit – Valiant, probably approximately correct learning

• Empirical ML (on real or artificial data) – Evaluation Criteria:

• Accuracy• Quality of solutions • Time complexity• Space complexity• Noise resistance

Empirical ML: Key Terms 1• Instances: individual examples of input-output

mappings of a particular type• Input consists of features• Features have values• Values can be

– Symbolic (e.g. letters, words, …)– Binary (e.g. indicators)– Numeric (e.g. counts, signal

measurements)

• Output can be– Symbolic (classification: linguistic symbols, …)– Binary (discrimination, detection, …)– Numeric (regression)

Empirical ML: Key Terms 2

• A set of instances is an instance base• Instance bases come as labeled training sets or unlabeled

test sets (you know the labeling, the learner does not)

• A ML experiment consists of training on the training set, followed by testing on the disjoint test set

• Generalization performance (accuracy, precision, recall, F-score) is measured on the output predicted on the test set

• Splits in train and test sets should be systematic: n-fold cross-validation– 10-fold CV– Leave-one-out testing

• Significance tests on pairs or sets of (average) CV outcomes

Empirical ML: 2 Flavors

• Eager– Learning

• abstract model from data

– Classification• apply abstracted model to new data

• Lazy– Learning

• store data in memory

– Classification• compare new data to data in memory

Eager vs Lazy Learning

Eager:– Decision tree induction

• CART, C4.5

– Rule induction• CN2, Ripper

– Hyperplane discriminators

• Winnow, perceptron, backprop, SVM

– Probabilistic• Naïve Bayes,

maximum entropy, HMM

– (Hand-made rulesets)

Lazy:– k-Nearest

Neighbour• MBL, AM• Local regression

-etje

-kje

Coda last syl

Nucleus last syl

Rule Induction

?

-etje

-kje

Coda last syl

Nucleus last syl

MBL

Eager vs Lazy Learning

• Decision trees keep the smallest amount of informative decision boundaries (in the spirit of MDL, Rissanen, 1983)

• Rule induction keeps smallest number of rules with highest coverage and accuracy (MDL)

• Hyperplane discriminators keep just one hyperplane (or vectors that support it)

• Probabilistic classifiers convert data to probability matrices

• k-NN retains every piece of information available at training time

Eager vs Lazy Learning

• Minimal Description Length principle:– Ockham’s razor– Length of abstracted model (covering core)– Length of productive exceptions not covered by core

(periphery)– Sum of sizes of both should be minimal– More minimal models are better

• “Learning = compression” dogma• In ML, length of abstracted model has been

focus; not storing periphery

Eager vs Lazy: So?• Highly relevant to language modeling• In language data, what is core? What is periphery?• Often little or no noise; productive exceptions• (Sub-)subregularities, pockets of exceptions• “disjunctiveness” and “polymorphism”• Some important elements of language have different

distributions than the “normal” one• E.g. word forms have a Zipfian distribution• Hard to distinguish noise from exceptions on the

basis of– Frequency– Typicality

ML and Natural Language

• Apparent conclusion: ML could be an interesting tool to do psycholinguistic modeling– Next to probability theory, information theory,

statistical analysis (natural allies)

• More and more annotated data available• Skyrocketing computing power and

memory

Case Study

Exemplar-based acquisition of Dutch Stress

(Durieux / Gillis / Daelemans)

This “rule of nearest neighbor” has considerable elementary intuitive appeal and probably corresponds to practice in many situations. For example, it is possible that much medical diagnosis is influenced by the doctor's recollection of the subsequent history of an earlier patient whose symptoms resemble in some way those of the current patient. (Fix and Hodges, 1952, p.43)

MBL: Use memory traces of experiences as a basis for analogical reasoning, rather than using rules or other abstractions extracted from experience and replacing the experiences.

MBL Acquisition

• Language process is represented by a set of exemplars in memory– Exemplars act as models– Learning is incremental storage of exemplars– Compression and Metrics

• Exemplar consists of set of (mostly symbolic) features

MBL Processing

• New instances of a performance process are solved through – Memory retrieval– Analogical (Similarity-Based) Reasoning

• Similarity metric– Language (faculty) - independent– Adaptive (feature and exemplar weighting)

Operationalization• Basis: k nearest neighbor algorithm:

– store all examples in memory– to classify a new instance X, look up the k

examples in memory with the smallest distance D(X,Y) to X

– let each nearest neighbor vote with its class– classify instance X with the class that has the most

votes in the nearest neighbor set• Choices:

– similarity metric– number of nearest neighbors (k)– voting weights

The Overlap distance function

• “Count the number of mismatching features”

The MVDM distance function

• Estimate a numeric “distance” between pairs of values– “e” is more like “i” than like “p” in a phonetic

task– “book” is more like “document” than like “the”

in a parsing task

Feature weighting in the distance function

• Mismatching on a more important feature gives a larger distance

• Factor in the distance function:

Entropy & IG: Formulas

Exemplar weighting

• Scale the distance of a memory instance by some externally computed factor

• Smaller distance for “good” instances• Bigger distance for “bad” instances

Distance weighting

• Relation between larger k and smoothing

• Make more distant neighbors contribute less in the class vote– Linear inverse of distance (w.r.t. max)– Inverse of distance– Exponential decay

Learning word stress:A case study

• Learn primary stress • Compare MBL with P&P/UG• Match acquisition and processing data

• Durieux, G. (2003) “Computermodellen en klemtoon.” Fonologische Kruispunten, BICN.

• Daelemans, W., Gillis, S., and Durieux, G. (1994). The acquisition of stress: A data-oriented approach." Computational Linguistics 20: 421-451.

• Daelemans, W., Gillis, S., Durieux, G., and Van den Bosch, A. (1993). Learnability and markedness: Dutch stress assignment. In T.M. Ellison and J.M. Scobbie (Eds.), Computational Phonology . Edinburgh Working Papers in Cognitive Science, 8, pp. 157-178.

MBL for psychology

• Similarity metric– Analogy engine

• Feature weighting– Relevance assignment– Information fusion

• Value weighting– Implicit concept formation

• Exemplar weighting– Recency, priming

• Distance-weighted extrapolation– Distributions, probabilities

• Local modeling– Heterogeneity and density

Dominant Linguistic Approach

• Principles and Parameters, UG– Typology– Acquisition

• Formalism: Metrical trees, metrical grids• Stress = prominence relations between

constituents in a hierarchical structure

YOUPIE (Dresher & Kaye, 1990)

• Assumptions– 11 parameters (216 “languages”)– Task-specific system for learning stress (domain

knowledge)– Core grammar only

• Learning– Cue-based parameter setting results in a grammar of

stress

• Performance– Generate tree with grammar and algorithmically

determine stress location

1 0 1 0 0 0 0 1 1 0 1UG-stressGrammar and Assignment rules

word

PLD

Cue-based Learning

Parameters (with setting for Dutch)

Parameter Value

P1 Word tree right/left dominant

P2 Binary/unbound feet

P3 Feet assigned from the left/right edge

P4 Feet right/left-dominant

P5 Feet are / are not quantity-sensitive

P6 Feet are quantity-sensitive w.r.t. rime / nucleus

P7 Strong node in foot must / mustn’t branch

P8A There isn’t / is an extra-metrical syllable

P8 Left / Right-most syllable is extra-metrical

P9 Weak foot looses / doesn’t loose foot status in a clash

P10 Feet are / aren’t assigned iteratively

MBL

• Assumptions– Lexical storage and generalization – Generic learning method, no task-specific linguistic

knowledge– Core and periphery

• Learning– Based on storage of exemplars

• Performance– Similarity-based reasoning with feature weighting on

stored exemplars

Syllable-structure representationsRetrieval orSimilarity-based reasoning on exemplars

word

PLD

Storage

Stress pattern

YOUPIE tested• Experimental design

– 216 languages– 117 items per language generated by YOUPIE performance

component (no exceptions, core only)– For each language, grammar learned with YOUPIE cue-

based learning component

• Results– For 60% of the languages, YOUPIE reconstructs the original

parameter setting with which the words were generated– For 21% convergence is to a compatible setting– For 19% of the languages errors in one or more stress

patterns

• Upper Boundary!– Perfect input, no exceptions to be learned

MBLP vs.

Youpie

System and level

Score

Sd Accuracy

MBLP-words 104 15.01

89%

YOUPIE-words

105 28.24

90%

MBLP-syllables

3.7 97%

YOUPIE-syllables

11.88

95%

MBLP-languages

89 41%

YOUPIE-languages

176 81%

Discussion

• No significant quantitative difference in performance

• Clear qualitative difference– YOUPIE: more languages perfectly learned– MBLP: fewer errors per language

• Issues:– Real language data– Core and periphery– Acquisition– Processing

Dutch stress

• Stress on one of the last three syllables • Predictable, but not completely

– E.g., py-a-ma ca-na-da pa-ra-plu

• Words not covered by the parameter-configuration for Dutch need lexical marking with exception features (one, two or completely idiosyncratic)

MBLP on Dutch data

• CELEX, 4868 monomorphemes• Exemplar encoding schemes For each of the three final syllables:

– S1: syllable weight (SL, L, H, SH)– S2: nucleus and coda (complete rhymes, VC)– S3: nucleus and coda (separate features, phonemes)– S4: onset, nucleus, and coda (phonemes)

• Class: final, penultimate, ante-penultimate

Results

Language Acquisition• Learning rules or learning lexical items?• Rules (Hochberg ‘88 Spanish, Nouveau ‘93 Dutch)

– Lexical learning lacks generalization capacity– Lexical learning incompatible with acquisition data

• Imitation task– Errors increase with irregularity– Tendency to regularization (but irregularization

occurs)• By stress shift• By changing structure of repeated word

Error Percentages

Discussion• MBLP error correlates with markedness like

children’s errors• MBLP has a tendency for regularization like children

– Direction of stress shifts – Structural changes from inspection of nearest neighbors

• Irregularization and differences 3 and 4 year-olds on marked patterns hard to explain in rule-based context

Rule learning is not the only possible explanation for the language acquisition data

Adult processing

• Rule-based: stress grammar and set of irregular words, marked in the lexicon– Known words: rule application except when

blocked by lexicon– Unknown words: rule application

• MBLP: lexical storage and analogy– Known words: look-up– Unknown words: analogy

Experimental set-up

• Stimuli– Create pseudo-words and transcribe them

(encoding 4)– Have a machine learner assign stress

(regular or irregular)

Bisyllabic Trisyllabic

Regulars 60 60

Irregulars 60 60

Experimental set-up

• Method– 18 adult participants– Reading task– 3 independent judges, consensus

• Result– Main effect for regularity-variable (ANOVA p

< .001); regular stress only in regular conditions– In all conditions, participants do the same as

model prediction (ANOVA p < .001)

Results

Results

Discussion

• Adult speakers sometimes prefer marked stress patterns for non-words

• These cases are partially predictable with an MBLP model and are problematic in a rule-based model (regularization only)

• BUT:– MBLP has a significantly better match with participant

behavior in the regular conditions– Hypothesis: differences between mental lexicon and

celex• Using a set-up with a population of machine ‘learners’

using different samples from celex explains the variability

Summary• Goal: put MBLP to the test on a concrete linguistic

problem of sufficient complexity by comparing it to– Linguistic theory– Child language acquisition data– Adult processing data

• Results:– MBLP and YOUPIE (P&P/UG) comparable– MBLP can learn core as well as periphery using superficial

representations– MBLP shows same errors and tendencies as children

learning stress placement– MBLP better predictor of human adult behaviour with non-

words

Overall Conclusion

• Exemplar-based models should be taken as a serious alternative for rule-based/P&P/UG/dual route type theories– Workable operationalisation of analogy– Adequacy

• Similar results in morphology and syntax (grammatical relations, chunking, pp-attachment)

• We’ll see …

Simulation with TiMBL

Demonstration: German plural

TiMBLhttp://ilk.uvt.nl/timbl

• Tilburg Memory-Based Learner• Available for research and education• Lazy learning, extending k-NN and IB1• Optimized search for NN

– Internal structure: tree, not flat instance base– Tree ordered by chosen feature weight– Many built-in optional metrics: feature weights,

distance function, distance weights, exemplar weights, …

Current practice

• Default TiMBL settings: – k=1, Overlap, GR, no distance weighting– Work well for some morpho-phonological tasks

• Rules of thumb:– Combine MVDM with bigger k– Combine distance weighting with bigger k– Very good bet: higher k, MVDM, GR, distance

weighting– Especially for sentence and text level tasks

usage: Timbl -f data-file {-t test-file} [options]

Algorithm and Metric options:

-a n : algorithm.

0 or IB1 : IB1 (default)

1 or IG : IGTree

2 or TRIBL : TRIBL

3 or IB2 : IB2

4 or TRIBL2 : TRIBL2

-m s : use feature metrics as specified in string s:

format: GlobalMetric:MetricRange:MetricRange

e.g.: -mO:N3:I2,5-7

D: Dot product. (Global only. numeric features implied)

O: weighted Overlap. (default)

M: Modified value difference.

N: numeric values.

I: Ignore named values.

-w 0 : No Weighting.

1 : Weight using GainRatio. (default)

2 : Weight using InfoGain

3 : Weight using Chi-square

4 : Weight using Shared Variance

f : use Weights from file 'f'.

-b n : number of lines used for bootstrapping (IB2 only).

-d val : weight neighbors as function of their distance:

Z : all the same weight. (default)

ID : Inverse Distance.

IL : Inverse Linear.

ED:a : Exponential Decay with factor a. (no whitespace!)

ED:a:b : Exponential Decay with factor a and b. (no whitespace!)

-k n : k nearest neighbors (default n = 1).

-q n : TRIBL treshold at level n.-L n : MVDM treshold at level n.-R n : solve ties at random with seed n.-t f : test using file 'f'.-t leave_one_out: test with Leave One Out,using IB1.-t cross_validate: Cross Validate Test,using IB1. @f : test using files and options described in file 'f'. Supported options: d e F k m o p q R t u v w x % - -t <file> is mandatory

Input options:-f f : read from Datafile 'f'.-f f : OR: use filenames from 'f' for CV test-F format: Assume the specified inputformat. (Compact, C4.5, ARFF, Columns, Binary, Sparse

)-l n : length of Features (Compact format only).-i f : read the InstanceBase from file 'f'. (skips phase 1

& 2 )-u f : read value_class probabilities from file 'f'.-P d : read data using path 'd'.-s : use exemplar weights from the input file-s0 : silently ignore the exemplar weights from the

input file

Output options:-e n : estimate time until n patterns tested.-I f : dump the InstanceBase in file 'f'.-n f : create names file 'f'.-p n : show progress every n lines. (default p = 100,000)-U f : save value_class probabilities in file 'f'.-V : Show VERSION.+v or -v level : set or unset verbosity level, where level is s: work silently. o: show all options set. f: show Calculated Feature Weights. (default) p: show MVD matrices. e: show exact matches. as: show advanced statistics. (memory consuming) cm: show Confusion Matrix. cs: show per Class Statistics. (implies +vas) di: add distance to output file. db: add distribution of best matched to output file k: add a summary for all k neigbors to output file (sets -x) n: add nearest neigbors to output file (sets -x and --) You may combine levels using '+' e.g. +v p+db or -v o+di

-W f : save current Weights in file 'f'.+% or -% : do or don't save test result (%) to file.-o s : use s as output filename.-O d : save output using path 'd'.

Internal representation options:-B n : number of bins used for discretization of numeric feature values-c n : clipping frequency for prestoring MVDM matrices-D : Don't store distributions. (saves memory, but disables +vDB option)+H or -H: write hashed trees (default +H)-M n: size of MaxBests Array-N n: Number of features (default 2500)-T n : ordering of the Tree : DO: none. GRO: using GainRatio IGO: using InformationGain (… and many others)+x or -x : Do or don't use the exact match shortcut. (IB only, default is -x)

Data & Representation• Symbolic features

– segmental information (syllable structure)– stress– gender

• German Plural (~ 25,000 from CELEX)Vorlesung (lecture) l e - z U N F en

Classes: e (e)n s er - U- Uer Ue

Cognitive Architectures of Inflectional Morphology

• Dual Route (Pinker, Clahsen, Marcus …)

– Rules for regular cases• (over)generalization• default behaviour

– Associative memory for exceptions• irregularization / family effects

• Single Route (R&M, MacWhinney, Plunkett, Elman, …)

– Frequency-based regularity

Dual Route

PatternAssociator Rule

Input Features

Suffix-class

MemoryFailure

German Plural

• Notoriously complex but routinely acquired (at age 5)

• Evidence for Dual Route ? -s suffix is default/regular (novel words,

surnames, acronyms, …)

-s suffix is infrequent (least frequent of the five most important suffixes)

Class Frequency Umlaut Frequency Example(e)n 11920 Abarte 6656 no 4646 Abbau

yes 2010 Abdampf - 4651 no 4402 Aasgeier

yes 249 Abwasserer 974 no 287 Abbild

yes 687 Abgangs 967 Abonnement

The default status of -s

• Similar item missing Fnöhk-s• Surname, product name Mann-s• Borrowings Kiosk-s• Acronyms BMW-s• Lexicalized phrases Vergissmeinnicht-s• Onomatopoeia, truncated roots, derived nouns, ...

Discussion• Three “classes” of plurals: ((-en -)(-e -er))(s)

the former 4 suffixes seem “regular”, can be accurately learned using information from phonology and gender

-s is learned reasonably well but information is lacking• Hypothesis: more “features” are needed (syntactic, semantic,

meta-linguistic, …) to enrich the “lexical similarity space”

• No difference in accuracy and speed of learning with and without Umlaut

• Overall generalization accuracy very high: 95% (90%)

• Schema-based learning (Köpcke).

*,*,*,*,i,r,M e

Acquisition Data:Summary of previous studies

• Existing nouns: (Park 78; Veit 86; Mills 86; Schamer-Wolles 88; Clahsen et al. 93; Sedlak et al. 98)

– Children mainly overapply -e or -(e)n– -s plurals are learned late

• Novel words: (Mugdan 77; MacWhinney 78; Phillis & Bouma 80; Schöler & Kany 89)

– Children inflect novel words with -e or -(e)n – More “irregular” plural forms produced than

“defaults”

MBLP simulation

• model overapplies mainly -en and -e

• -s is learned late and imperfectly

• Mainly but not completely parallel to input frequency (more -s overgeneralization than -er generalization)

Bartke, Marcus, Clahsen (1995)

• 37 children age 3.6 to 6.6• pictures of imaginary things,

presented as neologisms– names or roots

– rhymes of existing words or not

– choice -en or -s

• results:– children are aware that unusual

sounding words require the default

– children are aware that names require the default

MBLP simulation

• sort CELEX data according to rhyme

• compare overgeneralization– to -en versus to -s

– percentage of total number of errors

• results:– when new words don’t rhyme

more errors are made

– overgeneralization to -en drops below the level of overgeneralization to -s

Conclusions

• Computational models in language acquisition shouldn’t necessarily be connectionist– From rule induction to exemplar-based

models

• TiMBL may be useful as software for computational psycholinguistics