on natural scenes analysis, sparsity and coding efficiency redwood center for theoretical...

On Natural Scenes Analysis, Sparsity

and Coding Efficiency

Redwood Center forTheoretical NeuroscienceUniversity of California, Berkeley

Mind, Brain & ComputationStanford University

Vivienne Ming

Adapted by J. McClelland for PDP class, March 1, 2013

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Two Proposals

Natural Scene AnalysisNeural/cognitive computation can only

be fully understood in “naturalistic” contexts

Efficient (Sparse) Coding TheoryNeural computation should follow

information theoretic principles

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Classical Physiology

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Classical Physiology

+

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Classical Physiology

+

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Reverse Correlation

Jones and Palmer (1987)

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Limits of Classical Physiology

Assumes units (neurons) are linear so known nonlinearities are "added on" to the models

Contrast sensitivity “Non-classical receptive fields” Two-tone inhibition ETC.

Assumes that units operate independently activity of one cell doesn't depend on the activity of others i.e., characterizing cell-by-cell equivalent to characterizing the whole

population of evolution and development, drifting gratings

and white noise are very "unnatural“ Is it possible that our sensory systems are functionally adapted to the

statistics of “natural” (evolutionarily relevant) signals? Would this adaptation affect our characterization of cells?

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Response to Natural MovieClassical Receptive

Field Response

Response in“Context”

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Limits of Classical Physiology

Assumes units (neurons) are linear so known nonlinearities are "added on" to the models

Contrast sensitivity “Non-classical receptive fields” Two-tone inhibition ETC.

Assumes that units operate independently activity of one cell doesn't depend on the activity of others i.e., characterizing cell-by-cell equivalent to characterizing the whole

population

Finally, in terms of evolution and development, drifting gratings and white noise seem very "unnatural“ Is it possible that our sensory systems are functionally adapted to the

statistics of “natural” (evolutionarily relevant) signals? Would this adaptation affect our characterization of cells? How can we test this?

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Efficient Coding Theory Barlow (1961); Attneave (1954)

Natural images are redundantStatistical dependencies amongst pixel

values in space and time An efficient visual system should

reduce redundancyRemoving statistical dependencies

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Information TheoryShannon (1949)

Optimally efficient codes reflect the statistics of target signals

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Natural Scenes Analysis:First-Order StatisticsN

aïve

Mod

els

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Natural Scenes Analysis:First-Order Statistics

Histogram Equalization

Intensity Histogram

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Natural Scenes Analysis:Second-Order Statistics

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Natural Scenes Analysis:Second-Order Statistics

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Natural Scenes Analysis:Second-Order Statistics

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Spatial Correlations

Compare intensityat this pixel

To the intensityat this neighbor

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Spatial Correlations

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

The Ubiquitous .

Flat (White) PowerSpectrum

f1

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Example: synthetic 1/f signals

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Natural Scenes Analysis:Principal Components Analysis

PCA Rotation Whitening

Information theory saysthis is an ideal code.

No redundancy

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

PCA vs. Center Surround

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Natural Scenes Analysis:Higher-Order Statistics

PCA Rotation Whitening

Principle dimensions of variation don’t align with data’s intrinsic structure

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Natural Scenes Analysis:Higher-Order Statistics

Need a more powerful learning algorithmIndependent Component Analysis (ICA)

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Which are the independent components in the scene

below?

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

M

mmm tstx

1

)()(

+_______

+=

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

The Model

x = s + n Overcomplete: #(s) >> #(x) Factorial: p(s) = i p(si) Sparse: p(si) = exp(g(si))

Where g(.) is some non-Gaussian distribution e.g., Laplacian: g(s) = −|s| e.g., Cauchy: g(s) = −log(2 + s2)

The noise is assumed to be additive Gaussian n ~ N(0, 2I)

Goal: find dictionary of functions, , such that coefficients, s, are as sparse and statistically independent as possible

Information Theorydemands sparseness

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Learning

log likelihood L() = <log p(x|)> Learning rule:

Basically the delta rule:

D = (x − s)sT

Impose constraint to encourage the variances of each s to be approximately equal to prevent trivial solutions

Usually whiten the inputs before learning Forces network to find structure beyond second-order Increases stability

L

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Sparsity

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

?

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Efficient Auditory CodingSmith & Lewicki (2006)

Extend Olshausen (2002) to deal with time-varying signalse.g., sounds or movies

Train the network on “Natural” soundsEnvironmental TransientsEnvironmental AmbientsAnimal Vocalizations

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Cat ANF Revcor Filters

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Efficient Kernels

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Population Coding

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Population Coding

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Population Coding

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Population Coding

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Population Coding

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Speech

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Speech

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Speech

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Efficient Coding Literature

Empirical Weliky, Fiser, Hunt & Wagner (2003) Vinje & Gallant (2002) DeWeese, Wehr & Zador (2003) Laurent (2002) Theunissen (2003)

Theoretical Field (1987) van Hateren (1992) Simoncelli & Olshausen (2001) Olshausen & Field (1996) Bell & Sejnowski (1997) Hyvarinen & Hoyer (2000) Smith & Lewicki (2006) Doi & Lewicki (2006)

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Hierarchical Structure?

Can we identify interesting structure in the world by looking at higher order statistics of the activations of the linear features discovered by the first-order model? Karklin and Lewicki (2005) looked for patterns

at the level of the variances of the linear features.

Karklin and Lewicki (2009) looked for patterns at the level of the covariances of the linear features.

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Looking at Hierarchical Structure

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Looking at Hierarchical Structure

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Looking at Hierarchical Structure

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Looking at Hierarchical Structure

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Looking at Hierarchical Structure

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Generalizing the standard ICA model

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Generalizing the standard ICA model

Instead of:

we now have units u and v such that

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Independent density components

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Karklin & Lewicki (2009)

The model tries to find the values of the yj’s that lead to a combined covariance matrixC that matches the covariance of the data represented by activities across first-level filters.

The learning process involves a search for vectors bk and weights wjk that allow themodel to fit the data while keeping the yj’s sparse and independent.

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Responses of Cell to Gratings

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Efficient Coding Summary

Statistic Computation

AlgorithmExample

Biological Example

Reference

1st-orderContrast

gain control

Histogram equalizatio

n

Retina or H1

adaptation

Fairhall et al. (2001)

2nd-order Whitening PCARetinal/

Thalamic coding

Atick (1992)

Higher-order

Sparse Coding

ICA / Sparsenet V1 coding Olshausen &

Field (1996)

Time-varying

Shift-invariance

Efficient Spike

Coding

Cochlear coding

Smith & Lewicki 2006

Hierarchical

Conditional Independenc

e

Hierarchical coding

?Karklin & Lewicki ’05,’09