the emergent structure of semantic knowledge
DESCRIPTION
The Emergent Structure of Semantic Knowledge. Jay McClelland Department of Psychology and Center for Mind, Brain, and Computation Stanford University. language. The Parallel Distributed Processing Approach to Semantic Cognition. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/1.jpg)
The Emergent Structure of Semantic Knowledge
Jay McClelland
Department of Psychology andCenter for Mind, Brain, and Computation
Stanford University
![Page 2: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/2.jpg)
• Representation is a pattern of activation distributed over neurons within and across brain areas.
• Bidirectional propagation of activation underlies the ability to bring these representations to mind from given inputs.
• The knowledge underlying propagation of activation is in the connections.
• Experience affects our knowledge representations through a gradual connection adjustment process
language
The Parallel Distributed Processing Approach to Semantic Cognition
![Page 3: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/3.jpg)
Distributed Representations:and Overlapping Patterns for Related
Concepts
dog goat hammer
dog goat hammer
![Page 4: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/4.jpg)
Kiani et al, J Neurophysiol 97: 4296–4309, 2007.
![Page 5: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/5.jpg)
Emergence of Meaning and Metaphor
• Learned distributed representations that capture important aspects of meaning emerge through a gradual learning process in simple connectionist networks
• Metaphor arises naturally as a byproduct of learning information in homologous domains in models of this type
![Page 6: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/6.jpg)
Emergence of Meaning: Differentiation, Reorganization, and Context-Sensitivity
![Page 7: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/7.jpg)
![Page 8: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/8.jpg)
The Rumelhart Model
![Page 9: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/9.jpg)
The Training Data:
All propositions true of items at the bottom levelof the tree, e.g.:
Robin can {grow, move, fly}
![Page 10: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/10.jpg)
Target output for ‘robin can’ input
![Page 11: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/11.jpg)
aj
ai
wij
neti=ajwij
wki
Forward Propagation of Activation
![Page 12: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/12.jpg)
k ~ (tk-ak)
wij
i ~ kwki
wki
aj
Back Propagation of Error ()
Error-correcting learning:
At the output layer: wki = kai
At the prior layer: wij = jaj
…
ai
![Page 13: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/13.jpg)
![Page 14: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/14.jpg)
![Page 15: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/15.jpg)
Experience
Early
Later
LaterStill
![Page 16: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/16.jpg)
![Page 17: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/17.jpg)
• Waves of differentiation reflect coherent covariation of properties across items.
• Patterns of coherent covariation are reflected in the principal components of the property covariance matrix.
• Figure shows attribute loadings on the first three principal components:
– 1. Plants vs. animals– 2. Birds vs. fish– 3. Trees vs. flowers
• Same color = features covary in
component• Diff color = anti-covarying
features
What Drives Progressive
Differentiation?
![Page 18: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/18.jpg)
Sensitivity to Coherence Requires Convergence
A
A
A
![Page 19: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/19.jpg)
Conceptual Reorganization (Carey, 1985)
• Carey demonstrated that young children ‘discover’ the unity of plants and animals as living things with many shared properties only around the age of 10.
• She suggested that the coalescence of the concept of living thing depends on learning about diverse aspects of plants and animals including– Nature of life sustaining processes– What it means to be dead vs. alive– Reproductive properties
• Can reorganization occur in a connectionist net?
![Page 20: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/20.jpg)
Conceptual Reorganization in the Model
• Suppose superficial appearance information, which is not coherent with much else, is always available…
• And there is a pattern of coherent covariation across information that is contingently available in different contexts.
• The model forms initial representations based on superficial appearances.
• Later, it discovers the shared structure that cuts across the different contexts, reorganizing its representations.
![Page 21: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/21.jpg)
![Page 22: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/22.jpg)
Organization of Conceptual Knowledge Early and Late in Development
![Page 23: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/23.jpg)
![Page 24: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/24.jpg)
Overall Structure Extracted by a Structured Statistical Model
![Page 25: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/25.jpg)
![Page 26: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/26.jpg)
Sensitivity to Context
Context-sensitive representation
Context-general representation
![Page 27: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/27.jpg)
Relation-specificrepresentations
• IS Representations (top) reflect idiosyncratic appearance properties.
• HAS representations are similar to the context-general representations (middle).
• Can representations collapse differences between plants, since there is little that plants can do.
• The fish are all the same, because there’s no difference in what they can do.
![Page 28: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/28.jpg)
Ongoing Work
• Can the representations learned in the distributed connectionist model capture different patterns of generalization of different kinds of properties?
• Simulations already show context-specific patterns of property generalization.
• We are currently collecting detailed data from a new data set to explore the sufficiency of the model to explain experimental data on context specific patterns of generalization.
![Page 29: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/29.jpg)
Generalization of different property types
• At different points in training, the network is taught one of:– Maple can queem– Maple is queem– Maple has queem
• Only weights from hidden to output are allowed to change.
• Network is then tested to see how strongly ‘queem’ is activated then same relation is paired with other items.
queem
![Page 30: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/30.jpg)
Generalization to other concepts after training with can, has, or is queem
![Page 31: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/31.jpg)
Ongoing Work
• Can the representations learned in the distributed connectionist model capture different patterns of generalization of different kinds of properties?
• Our simulations already show context-specific patterns of property generalization.
We are currently conducting new experiments to gather experimental data on context specific patterns of generalization that we will use to test an extended version of the model trained with a much larger training set.
![Page 32: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/32.jpg)
Metaphor in Connectionist Models of Semantics
• By metaphor I mean:
the application of a relation learned in one domain to a novel situation in another
![Page 33: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/33.jpg)
Hinton’s Family Tree Network
Person 1 Relation
Person 2
Training data: Colin’s Father is James … Alfonso’s Father is Marco …
![Page 34: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/34.jpg)
English Tree Recovered Italian Tree Recovered
![Page 35: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/35.jpg)
Understanding Via Metaphor in the Family Trees Network
Marco’s father is Pierro.Who is James’s father?
![Page 36: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/36.jpg)
Future Work: Metaphors We Live By
• In Hinton’s model, neither domain is the base – each influences the other equally
• But research suggests that some domains serve as a base that influences other domains– Lakoff – physical structure as a base for the structure
of an intellectual argument– Boroditsky – space as a base for time
• In connectionist networks, primacy and frequency both influence performance
• This allows the models to simulate how early and pervasive experience may allow one domain to serve as the base for others experienced later or less frequently
• Influences can still run in both directions, but to different extents
![Page 37: The Emergent Structure of Semantic Knowledge](https://reader030.vdocument.in/reader030/viewer/2022032612/56812dab550346895d92d27e/html5/thumbnails/37.jpg)
Emergence of Meaning and Metaphor
• Learned distributed representations that capture important aspects of meaning emerge through a gradual learning process in simple connectionist networks
• Metaphor arises naturally as a byproduct of learning information in homologous domains in models of this type