Emergence of Semantics from Experience

Jay McClelland

Department of Psychology andCenter for Mind, Brain, and Computation

Stanford University

1

• Representation is a pattern of activation distributed over neurons within and across brain areas.

• Bidirectional propagation of activation underlies the ability to bring these representations to mind from given inputs.

• The knowledge underlying propagation of activation is in the connections.

• Experience affects our knowledge representations through a gradual connection adjustment process

language

The Parallel Distributed Processing Approach to Semantic Cognition

2

Distributed Representations:and Overlapping Patterns for Related

Concepts

dog goat hammer

dog goat hammer

3

Emergence of Meaning in Learned Distributed Representations

• Learned distributed representations that capture important aspects of meaning emerge through a gradual learning process in simple connectionist networks

• The progression of learning captures several aspects of cognitive development:– Differentiation of Concepts– Illusory Correlations– Overgeneralization– And many other things

4

5

The Rumelhart Model

6

The Training Data:

All propositions true of items at the bottom levelof the tree, e.g.:

Robin can {grow, move, fly}

7

Target output for ‘robin can’ input

8

aj

ai

wij

neti=ajwij

wki

Forward Propagation of Activation

9

k ~ (tk-ak)

wij

i ~ kwki

wki

aj

Back Propagation of Error ()

Error-correcting learning:

At the output layer: wki = kai

At the prior layer: wij = jaj

…

ai

10

11

12

Experience

Early

Later

LaterStill

13

Why Does the Model Show Progressive Differentiation?

• Learning is sensitive to patterns of coherent covariation

• Coherent Covariation:

– The tendency for properties of objects to co-vary in clusters

• Figure shows attribute loadings on the principal dimensions of covariation. These capture:

– 1. Plants vs. animals– 2. Birds vs. fish– 3. Trees vs. flowers

• Same color = features that covary

• Diff color = anti-covarying features

14

Trajectories of Concept Representations During Differentiation

15

Illusory Correlations

16

A typical property thata particular object lackse.g., pine has leaves

An infrequent,atypical property

17

Overgeneralization of Frequent Names to Similar Objects

“dog”

“goat”“tree”

18

19

Other Applications of the Model

• Expertise effects

• Conceptual reorganization

• Effects of language and culture

• Effects of brain damage:

– Loss of differentiation– Overgeneralization in

object naming– Illusory correlations

camel swan 20

Conclusion

• We represent objects using patterns of activity over neuron-like processing units

• These patterns depend on connection weights learned through experience

• Differences in experience lead to differences in conceptual representations.

21