the time dimension for scene analysis deliang wang perception & neurodynamics lab the ohio state...
TRANSCRIPT
![Page 1: The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA](https://reader034.vdocument.in/reader034/viewer/2022051316/5697bf9c1a28abf838c930ad/html5/thumbnails/1.jpg)
The Time Dimension for Scene Analysis
DeLiang Wang
Perception & Neurodynamics LabThe Ohio State University, USA
![Page 2: The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA](https://reader034.vdocument.in/reader034/viewer/2022051316/5697bf9c1a28abf838c930ad/html5/thumbnails/2.jpg)
Presentation outline
Introduction Scene analysis and temporal correlation theory
Oscillatory Correlation LEGION network
Oscillatory Correlation Approach to Scene Analysis Image segmentation Object selection Cocktail party problem
Concluding remarks
![Page 3: The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA](https://reader034.vdocument.in/reader034/viewer/2022051316/5697bf9c1a28abf838c930ad/html5/thumbnails/3.jpg)
Scene analysis problem
![Page 4: The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA](https://reader034.vdocument.in/reader034/viewer/2022051316/5697bf9c1a28abf838c930ad/html5/thumbnails/4.jpg)
Binding problem
Feature binding (integration) is a fundamental problem in neuroscience and perception (and perceptrons)
Binding problemin Rosenblatt’sperceptrons
![Page 5: The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA](https://reader034.vdocument.in/reader034/viewer/2022051316/5697bf9c1a28abf838c930ad/html5/thumbnails/5.jpg)
Temporal correlation theory
Temporal correlation theory proposes a solution to the nervous integration problem (von der Malsburg’81; also Milnor’74)
Application to cocktail party processing (von der Malsburg & Schneider’86)
![Page 6: The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA](https://reader034.vdocument.in/reader034/viewer/2022051316/5697bf9c1a28abf838c930ad/html5/thumbnails/6.jpg)
Physiological evidence (Gray et al.’89)
![Page 7: The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA](https://reader034.vdocument.in/reader034/viewer/2022051316/5697bf9c1a28abf838c930ad/html5/thumbnails/7.jpg)
Oscillatory correlation theory
• Oscillators represent feature detectors
• Binding is encoded by synchrony within an oscillator assembly and desynchrony between different assemblies
![Page 8: The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA](https://reader034.vdocument.in/reader034/viewer/2022051316/5697bf9c1a28abf838c930ad/html5/thumbnails/8.jpg)
Computational requirements Need to synchronize locally coupled oscillator
population Need to desynchronize different populations, when
facing multiple objects Synchrony and desynchrony
must be achieved rapidly
![Page 9: The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA](https://reader034.vdocument.in/reader034/viewer/2022051316/5697bf9c1a28abf838c930ad/html5/thumbnails/9.jpg)
LEGION architecture
LEGION - Locally Excitatory Globally Inhibitory Oscillator Network (Terman & Wang’95)
![Page 10: The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA](https://reader034.vdocument.in/reader034/viewer/2022051316/5697bf9c1a28abf838c930ad/html5/thumbnails/10.jpg)
Relaxation oscillator as building block
Typical x trace (membrane potential)
With stimulus
Without stimulus
![Page 11: The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA](https://reader034.vdocument.in/reader034/viewer/2022051316/5697bf9c1a28abf838c930ad/html5/thumbnails/11.jpg)
Analytical results
Theorem 1. (Synchronization). The oscillators in a connected block synchronize at an exponential rate
Theorem 2. (Multiple patterns) If at the beginning all the oscillators of the same block synchronize and different blocks desynchronize, then synchrony within each block and the ordering of activations among different blocks are maintained
Theorem 3. (Desynchronization) If at the beginning all the oscillators of the system lie not too far away from each other, then the condition of Theorem 2 will be satisfied after some time. Moreover, the time it takes to satisfy the condition is no greater than N cycles, where N is the number of blocks
![Page 12: The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA](https://reader034.vdocument.in/reader034/viewer/2022051316/5697bf9c1a28abf838c930ad/html5/thumbnails/12.jpg)
Connectedness problem
Minsky-Papert connectedness problem is a long-standing problem in perceptron learning
The problem exposes fundamental limitations of supervised learning, and illustrates the importance of proper representations
![Page 13: The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA](https://reader034.vdocument.in/reader034/viewer/2022051316/5697bf9c1a28abf838c930ad/html5/thumbnails/13.jpg)
Connectedness problem: LEGION solution Basic idea:
Synchronization within a connected pattern and desynchronization between different ones
![Page 14: The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA](https://reader034.vdocument.in/reader034/viewer/2022051316/5697bf9c1a28abf838c930ad/html5/thumbnails/14.jpg)
Presentation outline
Introduction Scene analysis and temporal correlation theory
Oscillatory Correlation LEGION network
Oscillatory Correlation Approach to Scene Analysis Image segmentation Object selection Cocktail party problem
Concluding remarks
![Page 15: The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA](https://reader034.vdocument.in/reader034/viewer/2022051316/5697bf9c1a28abf838c930ad/html5/thumbnails/15.jpg)
Oscillatory correlation approach to scene segmentation
Feature extraction first takes place An visual feature can be pixel intensity, depth, local image patch,
texture element, optic flow, etc. An auditory feature can be a pure tone, amplitude and frequency
modulation, onset, harmonicity, etc.
Connection weights between neighboring oscillators are set to be proportional to feature similarity
Global inhibitor controls granularity of segmentation Larger inhibition results in more and smaller regions
Segments pop out from LEGION in time
![Page 16: The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA](https://reader034.vdocument.in/reader034/viewer/2022051316/5697bf9c1a28abf838c930ad/html5/thumbnails/16.jpg)
Image segmentation example: Demo
Input image
![Page 17: The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA](https://reader034.vdocument.in/reader034/viewer/2022051316/5697bf9c1a28abf838c930ad/html5/thumbnails/17.jpg)
Image segmentation example
Input image Segmentation result
![Page 18: The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA](https://reader034.vdocument.in/reader034/viewer/2022051316/5697bf9c1a28abf838c930ad/html5/thumbnails/18.jpg)
Object selection The slow inhibitor keeps
trace of each pattern, which can be overcome by only more salient (larger) patterns
Unlike traditional winner-take-all dynamics, selection (competition) takes place at the object level Consistent with object-
based attention theory Binding precedes attention,
rather than attention precedes binding (Treisman & Gelade’80)
![Page 19: The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA](https://reader034.vdocument.in/reader034/viewer/2022051316/5697bf9c1a28abf838c930ad/html5/thumbnails/19.jpg)
Results of object selection
Input image LEGION output Selection outputInput LEGION segmentation Selection
![Page 20: The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA](https://reader034.vdocument.in/reader034/viewer/2022051316/5697bf9c1a28abf838c930ad/html5/thumbnails/20.jpg)
Cocktail party problem
• In a natural environment, target speech is usually corrupted by acoustic interference, creating a speech segregation problem Popularly known as cocktail-party problem (Cherry’53); also
ball-room problem (Helmholtz, 1863)
• Human listeners organize sound in a perceptual process called auditory scene analysis (Bregman’90)
Auditory scene analysis (ASA) takes place in two conceptual stages: Segmentation. Decompose the acoustic signal into ‘sensory
elements’ (segments) Grouping. Combine segments into groups, so that segments in
the same group likely originate from the same sound source
![Page 21: The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA](https://reader034.vdocument.in/reader034/viewer/2022051316/5697bf9c1a28abf838c930ad/html5/thumbnails/21.jpg)
NeuralOscillatorNetwork
Correlogram
Cross-channelCorrelation
ResynthesisHairCells
CochlearFiltering
Speechand Noise
ResynthesizedSpeech
ResynthesizedNoise
Correlogram (detail)
TimeLag
Neural Oscillator Network (detail)
GroupingLayer
SegmentationLayer
GlobalInhibitor
TimeFrequency
Oscillatory correlation for ASA (Wang & Brown’99)
Fre
quen
cy
![Page 22: The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA](https://reader034.vdocument.in/reader034/viewer/2022051316/5697bf9c1a28abf838c930ad/html5/thumbnails/22.jpg)
Auditory periphery: Cochleagram
Cochleagram representation of the utterance: “Why were you all weary?” mixed with phone ringing
Time (seconds)0.0 1.5
5000
2741
1457
729
315
80
![Page 23: The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA](https://reader034.vdocument.in/reader034/viewer/2022051316/5697bf9c1a28abf838c930ad/html5/thumbnails/23.jpg)
Grouping layer: Example
Two streams emerge from the group layer Foreground: left (original mixture ) Background: right
More recent results (Hu & Wang’04):
Time (seconds)0.0 1.5
5000
2741
1457
729
315
80
Time (seconds)0.0 1.5
5000
2741
1457
729
315
80
![Page 24: The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA](https://reader034.vdocument.in/reader034/viewer/2022051316/5697bf9c1a28abf838c930ad/html5/thumbnails/24.jpg)
Back to physiology
Chattering cells recorded by Gray & McCormick’96
Burst oscillations are best modeled by relaxation oscillators
![Page 25: The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA](https://reader034.vdocument.in/reader034/viewer/2022051316/5697bf9c1a28abf838c930ad/html5/thumbnails/25.jpg)
Versatility and time dimension
The principle of universality: “Give me a concrete problem and I will devise a network that solves it.” (von der Malsburg’99) It characterizes artificial intelligence
The principle of versatility: “Given the network, learn to cope with situations and problems as they arise.” (von der Malsburg’99) It characterizes natural intelligence
Time dimension is necessary for versatility Flexible and infinitely extensible Irreplaceable by spatial organization
![Page 26: The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA](https://reader034.vdocument.in/reader034/viewer/2022051316/5697bf9c1a28abf838c930ad/html5/thumbnails/26.jpg)
Conclusion
Advances in dynamical analysis overcome computational obstacles of oscillatory correlation theory
Major progress is made towards solving the scene analysis problem
From Hebb’s cell assemblies to von der Malsburg’s correlation theory, time is an indispensable dimension for scene analysis