The dynamics of incremental sentence comprehensionA situation-space model

Stefan FrankDepartment of Cognitive, Perceptual and Brain SciencesUniversity College Londons.frank@ucl.ac.uk

sentence comprehension

cognitive modelling

information theory

Sentence comprehension as mental simulation

• The mental representation of a sentence’s meaning is not some symbolic structure

• But an analogical and modal simulation of the described state of affairs (e.g., Barsalou, 1999; Zwaan, 2004)

• Comparable to the result of directly experiencing the described situation

• Central property of analogical representations: direct inference

Sentence comprehension as mental simulationStanfield & Zwaan (2001)

John put the pen in the cup

John put the pen in the drawer

Was this object mentioned in the sentence?

fast RT

fast

RT

Direct inference results from the analogical nature of mental representation

A model of sentence comprehensionFrank, Haselager & Van Rooij (2009)

• Formalization of analogical representations and direct inference

• Any state of the world corresponds to a vector in situation space

• These representations are analogical: Relations between the vectors mirror probabilistic relations between the represented situations

• In practice, restricted to a microworld

The microworldConcepts and atomic situations

• 22 Concepts, e.g.,- people: charlie, heidi, sophia- games: chess, hide&seek, soccer- toys: puzzle, doll, ball- places: bathroom, bedroom, street, playground- predicates: play, place, win, lose

• 44 atomic situations, e.g.,– play(charlie, chess)– win(sophia)– place(heidi, bedroom)

The microworldStates of the world

• Atomic situations and boolean combinations thereof refer to states of the world:

– play(sophia, hide&seek) place(sophia, playground)∧“sophia plays hide-and-seek in the

playground”– lose(charlie) lose(heidi) lose(sophia)∨ ∨

“someone loses”• Interdependencies among states of the world affect

probabilities of microworld states:– sophia and heidi are usually at the same place– the person who wins must play a game

Representing microworld situations

• Automatic generation of 25,000 observations of microworld states.

• Unsupervised Competitive Layer yields a situation vector μ(p) [0,1]150 for each atomic situation p

• Any state of the world can be represented by Boolean operations on vectors: μ(p), μ(pq), μ(pq)

• Probability of a situation can be estimated from its representation: P(z) ≈ ∑iμi(z)/150

Representing microworld situationsDirect inference

• The conditional probability of one situation given another, can be estimated from the two vectors:

P(p|z) = P(pz)/P(z)• From the representations μ(play(sophia, soccer)),

μ(play(sophia, ball)), μ(play(sophia, puzzle)) it follows that• P(play(sophia, ball)|play(sophia, soccer)) ≈ .99 • P(play(sophia, puzzle)|play(sophia, soccer)) ≈ 0

• Representing sophia playing soccer is also representing her playing with ball, not puzzle

The microlanguage

• 40 words• 13,556 possible sentences, e.g.,

– girl plays chess– ball is played with by charlie– heidi loses to sophia at hide-and-seek– someone wins

• Each sentence has– a unique semantics (represented by a situation vector)– a probability of occurrence (higher for shorter sentences)

A model of the comprehension process

• A simple recurrent network (SRN) maps microlanguage sentences onto the vectors of the corresponding situations

• Displays semantic systematicity (in the sense of Fodor & Pylyshyn, 1988; Hadley, 1994)

input (40 units)words

hidden (120 units)word sequences

output (150 units)situation vectors

Simulated word-reading time

• No sense of processing a word over time in the standard SRN

• Addition: output vector update is a dynamical process, expressed by a differential equation (Frank, in press)

• This yields a processing time for each word: simulated reading times

• Word-processing times compared to formal measures of the amount of information conveyed by each word

Word information and reading time

• Assumption: human linguistic competence is captured by probabilistic language models

• Such models give rise to formal measures of the amount of word-information content

• The more information is conveyed by a word, the more cognitive effort is involved in processing it

• This leads to longer reading time on the word

highly expected word less expected word

Word information and expectation

1a) It is raining cats and1b) She is training cats and

dogsdogs

These expectations arise from knowledge of linguistic forms

Word information and expectation

• Syntactic surprisal (Hale, 2001; Levy 2008)

• formalization of a word’s unexpectedness• measure of word information• follows from word’s probability given the sentence so far:

−log P(wi+1|w1,…,wi),under a particular probabilistic language model

• Any reasonably accurate language model estimates surprisal values that predict word-reading times (Demberg & Keller, 2008; Smith & Levy, 2008; Frank, 2009; Wu et al., 2010)

low uncertainty

Word information and uncertainty about the rest of the sentence

2a) It is raining

high uncertainty

cats high uncertainty reduction

high uncertainty

Word information and uncertainty about the rest of the sentence

2a) It is raining2b) She is training

high uncertainty

catscats

high uncertainty reductionlow uncertainty reduction

These uncertainties arise from knowledge of linguistic forms

Word information and uncertainty about the rest of the sentence• Syntactic entropy

– formalization of the amount of uncertainty about the rest of the sentence

– can be computed from a probabilistic language model• Entropy reduction is an alternative measure of the

amount of information the word conveys (Hale, 2003, 2006)

• Predicts word-reading times independently from surprisal (Frank, 2010)

high semantic surprisallow semantic surprisal

World knowledge and word expectation

acceptedaccepted

These expectations arise from knowledge of the world

3a) The brilliant paper was immediately3b) The terrible paper was immediately

Traxler et al. (2000): words take longer to read if they are less expected given the situation described so far

high semantic entropy

low semantic entropy

World knowledge and uncertainty about the rest of the sentence

accepted/rejectedaccepted/rejected

4a) The brilliant paper was immediately4b) The mediocre paper was immediately

low semantic entropy

World knowledge and uncertainty about the rest of the sentence

accepted/rejectedaccepted/rejected

4a) The brilliant paper was immediately4b) The mediocre paper was immediately

low semantic entropy reduction

high semantic entropy reduction

These uncertainties arise from knowledge of the world

Syntactic versus semantic word information

Syntactic information

Semantic information

Source of knowledge

Language The world

Probabilities of Word sequences States of the worldCognitive task Sentence

recognitionSimulation of described situation

Word-information measures in the sentence-comprehension model

For each word of each microlanguage sentence, four information values can be computed• Syntactic surprisal and syntactic entropy reduction:

follow directly from the microlanguage sentence’s occurrence probabilities

• Semantic surprisal and semantic entropy reduction: follow from probabilities of situations described by the sentences (estimated by situation vectors)

Computing semantic surprisal

sentence so far w1,…,wi

complete sentences

described situations

situation vectors

vector for disjunction of situations

w1,…,wi,… w1,…,wi,… w1,…,wi,… w1,…,wi,…

sit1 sit2 sit3 sit4

sit1 sit2 sit3 sit4

sit1 sit2 sit3 sit4

Computing semantic surprisal

sentence so far w1,…,wi

complete sentences w1,…,wi,…

described situations

situation vectors

vector for disjunction of situations

w1,…,wi,… w1,…,wi,… w1,…,wi,…

sit1

sit1

sit2

sit2

sit3

sit3

sit4

sit4

sit1 sit2 sit3 sit4

w1,…,wi+1

w1,…,wi+1,… w1,…,wi+1,…

sit2 sit4

sit2 sit4

sit2 sit4

Computing semantic surprisal

vector for disjunction of situations sit1 sit2 sit3 sit4 sit2 sit4

conditional probability estimate

P(sit2 sit4|sit1sit2sit3sit4)

semantic surprisal of word wi+1

−log P(sit2sit4|sit1sit2sit3sit4)

Computing semantic entropy reduction is more tricky, but also possible

ResultsNested linear regression

Predictor Coefficient R2

Semantic surprisal 0.04 .310Semantic entropy reduction 0.64 .082Syntactic surprisal 0.12 .026Word position 0.08 .011Syntactic entropy reduction 0.20 .001

all p < 10−8

ConclusionsMental simulation, word information, and processing time

• Semantic word information, formalized with respect to world knowledge, provides a more formal basis for the notion of mental simulation

• The sentence-comprehension model correctly predicts slower processing of more informative words

• Irrespective of information source (syntax/semantics) and information measure (surprisal/entropy red.)

More conclusionsLearning syntax

• Words that convey more syntactic information take longer to process: The SRN is sensitive to sentence probabilities

• But sentence probabilities are irrelevant to the network’s task of mapping sentences to situations

• No part of the model is meant to learn anything about syntax. It is not a probabilistic language model.

• Merely learning the sentence-situation mapping, can result in the acquisition of useful syntactic knowledge