bnp5spatial cvpr2012 applied bayesian nonparametrics
Post on 05-Apr-2018
225 Views
Preview:
TRANSCRIPT
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 1/48
Applied Bayesian
Nonparametrics5. Spatial Models via Gaussian
Processes, not MRFsTutorial at CVPR 2012
Erik SudderthBrown University
NIPS 2008:E. Sudderth & M. Jordan, Shared Segmentationof Natural Scenes using Dependent Pitman-Yor Processes.
CVPR 2012:S. Ghosh & E. Sudderth, Nonparametric Learning for Layered Segmentation of Natural Images.
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 2/48
Human Image Segmentation
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 3/48
BNP Image Segmentation
•! How many regions does this image contain?
•! What are the sizes of these regions?
Segmentation as Partitioning
•! Huge variability in segmentations across images
•! Want multiple interpretations, ranked by probability
Why Bayesian Nonparametrics?
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 4/48
BNP Image Segmentation
Inference
!! Stochastic search &expectation propagation
Model!! Dependent Pitman-Yor processes
!! Spatial coupling via Gaussian processes
Results
!! Multiple segmentations of natural images
Learning
!! Conditional covariance
calibration
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 5/48
Feature Extraction
•! Partition image into ~1,000 superpixels
•! Compute texture and color features:Texton Histograms (VQ 13-channel filter bank) Hue-Saturation-Value (HSV) Color Histograms
•! Around 100 bins for each histogram
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 6/48
Pitman-Yor Mixture Model
Observed features(color & texture)
Assign features
to segments
PY segment size prior
Visual segment
appearance model
Color:
Texture:
π
z1z2
z3z4
x1
x2
x3x4x
c
i∼Mult(θc
zi)
xs
i∼Mult(θs
zi)
zi ∼Mult(π)
πk = vk
k−1
=1
(1− v)
vk ∼ Beta(1− a, b+ ka)
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 7/48
Dependent DP&PY Mixtures
Observed features(color & texture)
Visual segment
appearance model
Color:
Texture:
z1z2
z3z4
x1
x2
x3x4x
c
i∼Mult(θc
zi)
xs
i∼Mult(θs
zi)
π1π2
π3
π4
Assign features
to segments
zi ∼Mult(πi)
Some dependent
prior with DP/PY
“like” marginals
Kernel/logistic/probit
stick-breaking process,
order-based DDP,
!
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 8/48
Example: Logistic of Gaussians
•! Pass set of Gaussian processes through softmax to get probabilities of independent segment assignments
•! Nonparametric analogs have similar properties
Figueiredo et. al., 2005, 2007
Fernandez & Green, 2002 Woolrich & Behrens, 2006
Blei & Lafferty, 2006
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 9/48
Discrete Markov Random Fields
Ising and Potts Models
•! Interactive foreground segmentation
•! Supervised training for known categories
Previous Applications
!but learning is challenging, and little
success at unsupervised segmentation.
GrabCut:Rother,Kolmogorov, & Blake 2004
Verbeek & Triggs, 2007
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 10/48
Region Classification with
Markov Field Aspect Models
Local:74%
MRF:78%
Verbeek & Triggs, CVPR 2007
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 11/48
10-State Potts Samples
States sorted by size: largest in blue, smallest in red
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 12/48
number of edges on whichstates take same value
1996 IEEE DSP Workshop
edge strength
Even within the phasetransition region, samples
lack the size distribution
and spatial coherence of
real image segments
naturalimages
giantcluster
verynoisy
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 13/48
Geman & Geman, 1984
200 Iterations
128 x128 grid8 nearest neighbor edges
K = 5 statesPotts potentials:
10,000 Iterations
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 14/48
Product of Potts and DP? Orbanz & Buhmann 2006
Potts Potentials DP Bias:
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 15/48
Spatially Dependent Pitman-Yor
•! Cut random surfaces (samples from a GP)
with thresholds(as in Level Set Methods )
•! Assign each pixel tothe first surface which
exceeds threshold(as in Layered Models )
Duan, Guindani, & Gelfand,
Generalized Spatial DP , 2007
π
z1z2
z3z4
x1
x2
x3x4
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 16/48
Spatially Dependent Pitman-Yor
•! Cut random surfaces (samples from a GP)
with thresholds(as in Level Set Methods )
•! Assign each pixel tothe first surface which
exceeds threshold(as in Layered Models )
Duan, Guindani, & Gelfand,
Generalized Spatial DP , 2007
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 17/48
Spatially Dependent Pitman-Yor
•! Cut random surfaces (samples from a GP)
with thresholds(as in Level Set Methods )
•! Assign each pixel tothe first surface which
exceeds threshold(as in Layered Models )
•! Retains Pitman-Yor marginals while jointly
modeling rich spatial
dependencies (as in Copula Models )
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 18/48
Stick-Breaking Revisited
0 1
Multinomial Sampler: Sequential Binary Sampler:
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 19/48
PY Gaussian Thresholds
Sequential Binary Sampler:Gaussian Sampler:
NormalCDF
because
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 20/48
PY Gaussian Thresholds
Sequential Binary Sampler:Gaussian Sampler:
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 21/48
Spatially Dependent Pitman-Yor Non-Markov
GaussianProcesses:
PY prior:
Segment size
Feature
Assignments
Normal
CDF
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 22/48
Preservation of PY MarginalsWhy Ordered Layer Assignments?
Stick Size Prior Random Thresholds
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 23/48
Samples from PY Spatial Prior
Comparison: Potts Markov Random Field
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 24/48
Outline
Inference
!! Stochastic search &expectation propagation
Model!! Dependent Pitman-Yor processes
!! Spatial coupling via Gaussian processes
Results
!! Multiple segmentations of natural images
Learning
!! Conditional covariance
calibration
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 25/48
Mean Field for Dependent PY
K
K
Factorized Gaussian Posteriors
Sufficient Statistics
Allows closed form update of via
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 26/48
Mean Field for Dependent PY
K
K
Updating Layered PartitionsEvaluation of beta normalization constants:
Jointly optimize each layer’s threshold
and Gaussian assignment surface, fixing
all other layers, via backtracking
conjugate gradient with line search
Reducing Local Optima
Place factorized posterior on eigenfunctions
of Gaussian process, not single features
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 27/48
Robustness and Initialization
Log-likelihood bounds versus iteration, for many random
initializations of mean field variational inference on a single image.
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 28/48
Alternative: Inference by Search
Consider hard
assignments of
superpixels to
layers (partitions) Integrate
likelihood
parametersanalytically
(conjugacy)
Marginalize layer
support functions
via expectation
propagation (EP):
approximate but
very accurate
No need for a finite, conservative model truncation!
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 29/48
Maximization ExpectationEM Algorithm
!! E-step: Marginalize latent variables (approximate)
! M-step: Maximize likelihood bound given model parameters
ME Algorithm
!! M-step: Maximize likelihood given latent assignments
! E-step: Marginalize random parameters (exact)
Kurihara & Welling, 2009
Why Maximization-Expectation?
!! Parameter marginalization allows Bayesian “model selection”
!! Hard assignments allow efficient algorithms, data structures
!! Hard assignments consistent with clustering objectives
!! No need for finite truncation of nonparametric models
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 30/48
Discrete Search Moves
!! Merge: Combine a pair of regions into a single region
!! Split: Break a single regioninto a pair of regions (for
diversity, a few proposals)
!! Shift: Sequentially move
single superpixels to themost probable region
!! Permute: Swap the position
of two layers in the order
Stochastic proposals, accepted if and only if they improve our EP
estimate of marginal likelihood:
Marginalization of continuous variables simplifies these moves!
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 31/48
Inferring Ordered Layers
Order A: Front, Middle, Back Order B: Front, Middle, Back
!! Which is preferred by a diagonal covariance?
!! Which is preferred by a spatial covariance?
Order B
Order A
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 32/48
Inference Across Initializations
Mean Field Variational EP Stochastic Search
Best Worst Best Worst
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 33/48
BSDS: Spatial PY Inference
S p a t i a l P Y ( E P )
S p a t i a l P Y ( M F )
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 34/48
Outline
Inference
!! Stochastic search &expectation propagation
Model!! Dependent Pitman-Yor processes
!! Spatial coupling via Gaussian processes
Results
!! Multiple segmentations of natural images
Learning
!! Conditional covariance
calibration
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 35/48
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 36/48
Learning from Human Segments
!! Data unavailable to learn models of all the categories we’reinterested in: We want to discover new categories!
! Use logistic regression, and basis expansion of image cues,
to learn binary “are we in the same segment” predictors:
!! Generative: Distance only
!!Conditional: Distance, intervening contours,
!
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 37/48
From Probability to Correlation
There is an injectivemapping between
covariance and the
probability that two
superpixels are in
the same segment.
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 38/48
Low-Rank Covariance Projection
!! The pseudo-covariance constructed by considering eachsuperpixel pair independently may not be positive definite
!! Projected gradient method finds low rank (factor analysis),
unit diagonal covariance close to target estimates
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 39/48
Prediction of Test Partitions
Heuristic versus Learned
Image Partition Probabilities
Learned Probability versus
Rand index measure
of partition overlap
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 40/48
Comparing Spatial PY Models
Image PY Learned PY Heuristic
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 41/48
Outline
Inference
!! Stochastic search &expectation propagation
Model!! Dependent Pitman-Yor processes
!! Spatial coupling via Gaussian processes
Results
!! Multiple segmentations of natural images
Learning
!! Conditional covariance
calibration
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 42/48
Other Segmentation Methods
FH Graph Mean Shift NCuts gPb+UCM Spatial PY
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 43/48
Quantitative Comparisons
Berkeley Segmentation LabelMe Scenes
!! On BSDS, similar or better than all methods except gPb
! On LabelMe, performance of Spatial PY is better than gPb
!! Implementation efficiency and search run-time
!! Histogram likelihoods discard too much information
!!Most probable segmentation does not minimize Bayes risk
Room for Improvement:
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 44/48
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 45/48
Multiple Spatial PY Modes
Most Probable
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 46/48
Spatial PY Segmentations
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 47/48
7/31/2019 BNP5spatial Cvpr2012 Applied Bayesian Nonparametrics
http://slidepdf.com/reader/full/bnp5spatial-cvpr2012-applied-bayesian-nonparametrics 48/48
Conclusions
!! Conventional MCMC & variationallearning prone to local optima, hard to
scale to large datasets.
But better methods on the way!
!! Literature remains fairly technical.But growing number of tutorials!
!but bravery is required
top related