Situation Models and Embodied Language Processes

Franz Schmalhofer

University of Osnabrück / Germany

1) Memory and Situation Models

2) Computational Modeling of Inferences

3) What Memory and Language are for

4) Neural Correlates

5) Integration of Behavioral Experiments and Neural Correlates (ERP; fMRI) by Formal Models

Cognition and Knowledge

• Traditional:

– Cognition = Computation

– Representation by propositions

– Propositions are abstract relations

• Embodiment of Meaning

– Cognition is serving perception and actions

– Representation = Patterns of possible bodily interactions with the world (lawfully related to the world)

– What an object, event, sentence means for you, is what you can do with the object, event, sentence.

Embodiment of Memory (Glenberg, 1997)

• Projectable properties: information available through the senses

• Non-projectable properties: information available through other sources (e.g. memory)

• Conceptualization: Combination (mesh) of projectable and non-projectable properties

Primary function of memory is to mesh the embodied conceptualizations of projectable properties of the environment with embodied experiences that provide non-projectable properties

Embodiment of Memory

• Evidence for embodiment...• ...and affect:

– Forced frowning or smiling influences affective judgments (Berkowitz & Trocolli 1990, Berkowitz et. al 1993)

• ...and imagery:– Actually rotating facilitates orientation opposed to imagining

rotating(Montello & Presson 1993, Rieser etl al. 1994).

• ...and memory:– Retrieval of memorized spatial layouts depends on position

on body axis (Bryant 1992).

Embodied Memory

• Combination of patterns of possible action

= meshing.

• Meshing of patterns of action derived from:– projectable properties of the environment

– non-projectable properties

• as (spatial-functional) constraint satisfaction.

Function of Memory

• Meshing projectable & non-projectable properties:

• Meshing is important for:– perception

– imagination

– Comprehension

• Projectable properties as well as non-projectable properties can be meshed with each-other.

Interaction of perceptions and memory

• Environment has to be primary

• Clamping projectable properties keeps the system reality-oriented

• Experiences stay individuated

Updating memory

• Experiences are shifts between conceptualizations

• Trajectories from one pattern of action to another

• Trajectories more often used become reinforced

Prediction and Planning

• Hypothetical conceptualizations by using trajectories from memory

• But:– Simulating action does not change the environment

– Clamped projectable properties provide wrong constraints for prediction

– Environment has to be suppressed (effortfull process)

• Suppression loosens tie to reality

Two modes of memory (like the distinction between implicit/explicit)

• Automatic

– Meshing of projectable and nonprojectable properties

– Causes conceptual priming: based on non-projectable properties, (therefore semantic)

• Effortful

– Suppression of projectable properties, conceptualization by trajectories from memory

Cognitive Meshing

• Imagine a ball

• Now Imagine that it has yellow and white stripes

• Now Imagine that it is deflated

• Mutual modification of mashed pattern: not only the ball but also the stripes become deformed when the ball is deflated.

• Patterns of actions to the same spatio-functional constraints.

What memory and language are for

• The primary function of memory “is to mesh

the embodied conceptualizations of projectable

properties of the environment with embodied

experiences that provide nonprojectable

properties…This meshed conceptualization,

the meaning, is in the service of control of

action in a three-dimensional environment“

(Glenberg 1997)

• “Language is a surrogate for experience” (Taylor and Tversky, 1992):

Summary: Embodied Representations & Symbol Grounding

• Embodied Meaning – action based coding of objects and situations

• Embodied representations are lawfully and analogically related to properties of the world (Harnard, 1990, 1993)

• Notion of “mesh”: mutual modification of patterns of action (Glenberg, 1997)

• Meaning of a situation is a meshed pattern of possible actions = embodied conceptualization.

Overview of the methods of cognitive neuropsychology/science (I)

• Advances in science by technology:– Invention of the telescope in 1608 changed astronomers‘

observational methods

– If well-formulated questions are not asked, even the most powerful tools will not provide sensible answers

• Cognitive Psychology / Computer Modeling

• Neuroanatomy– Gross Neuroanatomy (general structures and connections)

– Fine Neuroanatomy (components of individual neurons)

Overview of the methods of cognitive neuropsychology/science (II)

• Neurophysiology (experimental methods used with animals)– Electrical stimulation

– Single-cell recording

– Lesions

– Genetic manipulations

• Neurology– Structural imaging and neurological damage

– Causes of neurological disorders (vascular disorders,tumors, degenerous and infectious disorders, traumata, epilepsies)

– Functional neuro-surgery

Overview of the methods of cognitive neuropsychology/science (III)

• Converging methods– Cognitive deficits following brain damage

– Virtual lesions: Transcranial magnetic stimulation (TMS)

– Functional imaging• Electrical and magnetic signals in the brain (EEG, MEG)

• Metabolic signals– Positron emission tomography (PET): regional cerebral blood flow;

– fMRI: blood oxygenation level dependent effect or BOLD effect

Brain functions (1810-1819)

• Do parts of the brain working independently enable the mind? (componential hypothesis)

• Franz Joseph Gall and J. G. Spurzheim– 35 specific brain functions– Language, color perception,

hope, self-esteem– With practice, areas grow,

causing a bump in the overlying skull

– Anatomical personology– phrenology

• Does the whole brain work in concert?(wholistic hypothesis)

• Pierre Flourens (1794-1867)– All sensations, all perceptions

and all volitions occupy the same seat in these cerebral organs.

– The faculty of sensation, percept and volition is then esssentially one faculty.

– Empirical evidence: no matter where he leasoned a bird brain, the bird recovered

Language Areas

• Broca • Wernicke

Neuroanatomist Korbinian Brodman (1909)

• Analyzed cellular organization of the cortex

• Tissue stains to visualize different brain regions

• To a large extent cytoarchitecturally described brain areas do indeed represent functionally distinct brain regions

Componential or wholistic? Again the question

• Camillo Golgi – Developed stain that

impregnated individual neurons

– Believed the whole brain to be a continuous mass of tissue that shares a common cytoplasm

• Cajal– Used Golgi stains

– Identified the unitary nature of neurons

– Transmittion of information by electricity

How does the nervous system work (20-th century)

• Understand how single neurons behave and interact

• Knowing all the elements, can we figure out the system?

• Billions of neurons

• Brain-damaged humans show lack of typical symptoms

• Impossible to localize „higher cognitive functions“

Jackson: Lesion might well affect other structures in the brain because the lesion might have damaged neurons connected to other regions; diaschisis: damage of one part can create problems for another.

Gestaltist view: The whole is different from the sum of its parts

Summary

• Localists– Wrong, in that they tried to map behaviors and perceptions into single

locations in the cortex

– Any behavior is produced by many areas

– Complex functions

• Globalists– A function can be achieved in numerous different ways (in this sense

the globalists were right)

• But – simple processes that are recruited to exercise an ability are localized

Event-Related Brain Potentials

What are event-related brain potentials (ERPs)

• Like EEG, but related to an event (a task, e.g. making a decision, reading a word, etc.)

• The ERP (a few µV is small in relation to the EEG (about 50µV)

• The international 10-20 system (Jasper 1958) allows for between-laboratory and between-experiment comparisons

EEG profiles obtained during various states of consciousness

After Penfield and Jasper (1954)

How ERPs are obtained from EEG-data

Schematic representation of ERP-Procedure

ERP-components

ERP-Components

• Usually labeled by polarity and latency, P300, P3 (ordinal latency of the component)

• Scalp locations, e.g. frontal P300• Psychological or experimental conditions

– Novelty P3

– Readiness potential

– Mismatch negativity, MMN

• Sensory or exogeneous • Interaction subject – response (task requirements)

endogenous

From the brain to the scalp

• Distant manifestations of activations of populations of neurons (recorded on surface of skull)

• Requirements– Neurons must act synchronously– Electric fields must be oriented so that they cumulate– Therefore only a subset of neural activity is visible– Open field organizations (dentritic trees are ordered), neurons

are organized in layers, most of cortex, parts of thalamus, cerebellum and others

• Presynaptic potentials (spikes) high frequency• Postsynaptic potentials (slower), summation thereof

Forward and backward solutions – inverse dipole modeling

Conclusions from ERPs

• Just a sample of neuronal activity– If you find the same effect in different experimental

conditions

– If you do not find an effect

From the scalp to the brain: Inferring the sources of ERPs

• Observations are: voltages differences between scalp electrodes and a reference electrode.

• Identify neural generators of ERPs: – indefinite number of unknown parameters

– No unique solution

– Head is not a homogeneous medium

– Difficult to compute

• Non-invasive and invasive techniques– Dense electrode arrays and source monitoring

– Neurophysiological knowledge, other imaging techniques

• Invasive techniques– Implanting electrodes, lesion studies with animals

The concept of components

• Voltage x time x location function. • Segments of the ERP waveform to covary in response

to a specific experimental condition – Positive, negative

– Aspects of the ERP waveform

– In terms of neural structures that generate them

• But a peak may be the sum of several functionally and structurally distinct components

Quantification of ERP components

• Artifacts: eyeballs, eyelids, muscles of the head. • Signal to noise ratio

– ERPs are constant over trials

– Noise is random

– ERPs are independent of the background noise

• Peak measurements• Covariation measures (e.g. covary with condition)• Source-activity measures (algorithms for dipole, Loreta,

Baillet & Garnero, 1997) spatio-temporal dipole model; distributed source models

Problems in Component Measurements

• Are components identical. • Revisions of component classification • Components overlap• Principle component analysis (application statistics,

linear algebra)• Subtraction procedure, only amplitude, not latency vary

across conditions

Experimental logic

• Discovery: establishing functional significance • Components antecedents (I.e. Experimental

manipulations) • Consequences of variation• Speculations about the psychological or

neuropsychological function it manifests

Psycho physiological inference

• Conditions are different • Conditions differ at a particular time • Conditions differ with respect to the latency of some

process• Conditions differ with respect to the degree to which

some process occurs

Some ERP-findings

• Movement-related potentials– Lateralized readiness potential (LRP)

– Contingent negative variation (CNV)

– Error-related negativity (ERN)

• Sensory components– The early negatives (ERPs and locus of selective attention)

– The middle latency cognitive components (mismatch negativity of MMN)

– N200s (or N2)

• The late cognitive ERPs: – P300, elicited by deviant stimuli

– The “frontal” P3, elicited by novel stimuli, novelty P3 (no memory template is available

Some ERP-findings (continued)

• ERP effects associated with subsequent memory– Distinctive word (van Restoff, character change, large P300s; recalled

ones show larger P300s as compared to not recalled ones.– Same-different task (Sanquist et al., 1980) larger amplitude P300s were

better recognized in subsequent recognition test.– Two-process model of recognition (large P300 when explicit recollection

as opposed to “just know”

• N 400 (language-related)– More prolonged over the right rather than the left hemisphere– N400 may be generated by the parahippocampal anterior fusiform gyrus– A distinctively semantic process– Inversely related to the subject’s expectancy (cloze probability)– Semantically related to sentence completion produce smaller N 400. “The

pizza was too hot to drink / cry”.

Brain Imaging

Brain Imaging

• Magnetic resonance imaging (MRI)• Angelo Mosso (1846-1910): correlation blood flow –

neuronal activity • Seymour Kety (1915-2000), Lou Sokoloff quantified

relation (middle of the 20-th century)• Position Emission tomography (PET)• 1980s: Michael Posner, Steve Peterson: Study human

cognition by PET; Marcus Raichle• Donders (1868) method of subtraction

Subtraction Method

Brain imaging (fMRI)

• Behavior of hydrogen atoms or protons in a magnetic field

• Paul Lauterbur: MRI (Nobel-prize, 2003)• Seiji Ogawa: functional states of the brain (fMRI);

– Amount of oxygen carried by hemoglobin chances the degree to which hemoglobin disturbs a magnetic field

– Tracking blood flow

– BOLD-signal: blood oxygen level dependent

Steps in analysis: cortex-segmentation

Computational Modeling and fMRI

A prominent cognitive architecture

ACT-R

(Anderson & Lebiere, 1998)

How to map time predictions to the BOLD-signal

Experimental Predictions by the ACT-R model

Comparison of model

predictions and observations by

a measure of proportionality

Summary

• Embodied memory representations

• New methods of cognitive science

• ERP

• fMRI

• Modeling