![Page 1: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/1.jpg)
On Natural Scenes Analysis, Sparsity
and Coding Efficiency
Redwood Center forTheoretical NeuroscienceUniversity of California, Berkeley
Mind, Brain & ComputationStanford University
Vivienne Ming
Adapted by J. McClelland for PDP class, March 1, 2013
![Page 2: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/2.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Two Proposals Natural Scene Analysis
Neural/cognitive computation can only be fully understood in “naturalistic” contexts
Efficient (Sparse) Coding TheoryNeural computation should follow
information theoretic principles
![Page 3: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/3.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Classical Physiology
![Page 4: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/4.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Classical Physiology
+
![Page 5: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/5.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Classical Physiology
+
![Page 6: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/6.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Reverse Correlation
Jones and Palmer (1987)
![Page 7: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/7.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Limits of Classical Physiology
Assumes units (neurons) are linear so known nonlinearities are "added on" to the models
Contrast sensitivity “Non-classical receptive fields” Two-tone inhibition ETC.
Assumes that units operate independently activity of one cell doesn't depend on the activity of others i.e., characterizing cell-by-cell equivalent to characterizing the whole
population of evolution and development, drifting gratings
and white noise are very "unnatural“ Is it possible that our sensory systems are functionally adapted to the
statistics of “natural” (evolutionarily relevant) signals? Would this adaptation affect our characterization of cells?
![Page 8: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/8.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Response to Natural MovieClassical Receptive
Field Response
Response in“Context”
![Page 9: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/9.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Limits of Classical Physiology
Assumes units (neurons) are linear so known nonlinearities are "added on" to the models
Contrast sensitivity “Non-classical receptive fields” Two-tone inhibition ETC.
Assumes that units operate independently activity of one cell doesn't depend on the activity of others i.e., characterizing cell-by-cell equivalent to characterizing the whole
population
Finally, in terms of evolution and development, drifting gratings and white noise seem very "unnatural“ Is it possible that our sensory systems are functionally adapted to the
statistics of “natural” (evolutionarily relevant) signals? Would this adaptation affect our characterization of cells? How can we test this?
![Page 10: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/10.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Efficient Coding Theory Barlow (1961); Attneave (1954)
Natural images are redundantStatistical dependencies amongst pixel
values in space and time An efficient visual system should
reduce redundancyRemoving statistical dependencies
![Page 11: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/11.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Information TheoryShannon (1949)
Optimally efficient codes reflect the statistics of target signals
![Page 12: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/12.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Natural Scenes Analysis:First-Order StatisticsN
aïve
Mod
els
![Page 13: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/13.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Natural Scenes Analysis:First-Order Statistics
Histogram Equalization
Intensity Histogram
![Page 14: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/14.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Natural Scenes Analysis:Second-Order Statistics
![Page 15: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/15.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Natural Scenes Analysis:Second-Order Statistics
![Page 16: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/16.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Natural Scenes Analysis:Second-Order Statistics
![Page 17: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/17.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Spatial CorrelationsCompare intensityat this pixel
To the intensityat this neighbor
![Page 18: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/18.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Spatial Correlations
![Page 19: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/19.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
The Ubiquitous .
Flat (White) PowerSpectrum
f1
![Page 20: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/20.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Example: synthetic 1/f signals
![Page 21: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/21.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Natural Scenes Analysis:Principal Components Analysis
PCA Rotation Whitening
Information theory saysthis is an ideal code.
No redundancy
![Page 22: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/22.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
PCA vs. Center Surround
![Page 23: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/23.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Natural Scenes Analysis:Higher-Order Statistics
PCA Rotation Whitening
Principle dimensions of variation don’t align with data’s intrinsic structure
![Page 24: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/24.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Natural Scenes Analysis:Higher-Order Statistics
Need a more powerful learning algorithmIndependent Component Analysis (ICA)
![Page 25: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/25.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Which are the independent components in the scene
below?
![Page 26: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/26.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
M
mmm tstx
1
)()(
+_______
+=
![Page 27: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/27.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
The Modelx = s + n
Overcomplete: #(s) >> #(x) Factorial: p(s) = i p(si) Sparse: p(si) = exp(g(si))
Where g(.) is some non-Gaussian distribution e.g., Laplacian: g(s) = −|s| e.g., Cauchy: g(s) = −log(2 + s2)
The noise is assumed to be additive Gaussian n ~ N(0, 2I)
Goal: find dictionary of functions, , such that coefficients, s, are as sparse and statistically independent as possible
Information Theorydemands sparseness
![Page 28: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/28.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Learning log likelihood L() = <log p(x|)> Learning rule:
Basically the delta rule:
D = (x − s)sT
Impose constraint to encourage the variances of each s to be approximately equal to prevent trivial solutions
Usually whiten the inputs before learning Forces network to find structure beyond second-order Increases stability
DL
![Page 29: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/29.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Sparsity
![Page 30: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/30.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
?
![Page 31: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/31.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Efficient Auditory CodingSmith & Lewicki (2006)
Extend Olshausen (2002) to deal with time-varying signalse.g., sounds or movies
Train the network on “Natural” soundsEnvironmental TransientsEnvironmental AmbientsAnimal Vocalizations
![Page 32: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/32.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
![Page 33: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/33.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Cat ANF Revcor Filters
![Page 34: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/34.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Efficient Kernels
![Page 35: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/35.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Population Coding
![Page 36: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/36.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Population Coding
![Page 37: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/37.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Population Coding
![Page 38: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/38.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Population Coding
![Page 39: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/39.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Population Coding
![Page 40: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/40.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Speech
![Page 41: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/41.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Speech
![Page 42: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/42.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Speech
![Page 43: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/43.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Efficient Coding Literature Empirical
Weliky, Fiser, Hunt & Wagner (2003) Vinje & Gallant (2002) DeWeese, Wehr & Zador (2003) Laurent (2002) Theunissen (2003)
Theoretical Field (1987) van Hateren (1992) Simoncelli & Olshausen (2001) Olshausen & Field (1996) Bell & Sejnowski (1997) Hyvarinen & Hoyer (2000) Smith & Lewicki (2006) Doi & Lewicki (2006)
![Page 44: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/44.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Hierarchical Structure? Can we identify interesting structure in the
world by looking at higher order statistics of the activations of the linear features discovered by the first-order model? Karklin and Lewicki (2005) looked for patterns
at the level of the variances of the linear features.
Karklin and Lewicki (2009) looked for patterns at the level of the covariances of the linear features.
![Page 45: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/45.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Looking at Hierarchical Structure
![Page 46: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/46.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Looking at Hierarchical Structure
![Page 47: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/47.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Looking at Hierarchical Structure
![Page 48: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/48.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Looking at Hierarchical Structure
![Page 49: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/49.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Looking at Hierarchical Structure
![Page 50: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/50.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Generalizing the standard ICA model
![Page 51: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/51.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Generalizing the standard ICA model
Instead of:
we now have units u and v such that
![Page 52: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/52.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Independent density components
![Page 53: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/53.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Karklin & Lewicki (2009)
The model tries to find the values of the yj’s that lead to a combined covariance matrixC that matches the covariance of the data represented by activities across first-level filters.
The learning process involves a search for vectors bk and weights wjk that allow themodel to fit the data while keeping the yj’s sparse and independent.
![Page 54: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/54.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
![Page 55: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/55.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
![Page 56: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/56.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
![Page 57: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/57.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Responses of Cell to Gratings
![Page 58: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/58.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
![Page 59: On Natural Scenes Analysis, Sparsity and Coding Efficiency](https://reader036.vdocument.in/reader036/viewer/2022081507/5681666d550346895dda0b09/html5/thumbnails/59.jpg)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Efficient Coding SummaryStatistic Computatio
nAlgorithmExample
Biological Example Reference
1st-orderContrast
gain control
Histogram equalizatio
n
Retina or H1
adaptationFairhall et al.
(2001)
2nd-order Whitening PCARetinal/
Thalamic coding
Atick (1992)
Higher-order
Sparse Coding
ICA / Sparsenet V1 coding Olshausen &
Field (1996)
Time-varying
Shift-invariance
Efficient Spike
CodingCochlear coding Smith & Lew
icki 2006
Hierarchical
Conditional Independenc
eHierarchical coding ?
Karklin & Lewicki ’05,’09