parallel gibbs sampling from colored fields to thin junction trees
DESCRIPTION
Parallel Gibbs Sampling From Colored Fields to Thin Junction Trees. Joseph Gonzalez. Yucheng Low. Arthur Gretton. Carlos Guestrin. Gibbs Sampling [ Geman & Geman , 1984]. Sequentially for each variable in the model Select variable Construct conditional given adjacent assignments - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Parallel Gibbs Sampling From Colored Fields to Thin Junction Trees](https://reader033.vdocument.in/reader033/viewer/2022051402/568162bc550346895dd347e2/html5/thumbnails/1.jpg)
Parallel Gibbs SamplingFrom Colored Fields to Thin Junction Trees
Yucheng Low
Arthur Gretton
Carlos Guestrin
Joseph Gonzalez
![Page 2: Parallel Gibbs Sampling From Colored Fields to Thin Junction Trees](https://reader033.vdocument.in/reader033/viewer/2022051402/568162bc550346895dd347e2/html5/thumbnails/2.jpg)
2
Gibbs Sampling [Geman & Geman, 1984]
Sequentially for each variable in the modelSelect variableConstruct conditional given adjacent assignments Flip coin and updateassignment to variable
Initi
al A
ssig
nmen
t
![Page 3: Parallel Gibbs Sampling From Colored Fields to Thin Junction Trees](https://reader033.vdocument.in/reader033/viewer/2022051402/568162bc550346895dd347e2/html5/thumbnails/3.jpg)
3
From the original paper on Gibbs Sampling:
“…the MRF can be divided into collections of [variables] with each collection assigned to an independently running asynchronous processor.”
Converges to the wrong distribution!
-- Stuart and Donald Geman, 1984.
![Page 4: Parallel Gibbs Sampling From Colored Fields to Thin Junction Trees](https://reader033.vdocument.in/reader033/viewer/2022051402/568162bc550346895dd347e2/html5/thumbnails/4.jpg)
4
The problem with Synchronous Gibbs
Adjacent variables cannot be sampled simultaneously.
Strong PositiveCorrelation
t=0
Parallel Execution
t=2 t=3
Strong PositiveCorrelation
t=1
Sequential
Execution
Strong NegativeCorrelation
![Page 5: Parallel Gibbs Sampling From Colored Fields to Thin Junction Trees](https://reader033.vdocument.in/reader033/viewer/2022051402/568162bc550346895dd347e2/html5/thumbnails/5.jpg)
5
Time
Chromatic Sampler
Compute a k-coloring of the graphical modelSample all variables with same color in parallel
Sequential Consistency:
![Page 6: Parallel Gibbs Sampling From Colored Fields to Thin Junction Trees](https://reader033.vdocument.in/reader033/viewer/2022051402/568162bc550346895dd347e2/html5/thumbnails/6.jpg)
6
Properties of the Chromatic Sampler
Converges to the correct distribution
Quantifiable acceleration in mixing
Time to updateall variables once
# Variables# Colors# Processors
![Page 7: Parallel Gibbs Sampling From Colored Fields to Thin Junction Trees](https://reader033.vdocument.in/reader033/viewer/2022051402/568162bc550346895dd347e2/html5/thumbnails/7.jpg)
9
Theoretical Contributions on 2-colorable models
Stationary distribution of Synchronous Gibbs
Corollary: Synchronous Gibbs sampler is correct for single variable marginals.
Variables in Color 1
Variables in Color 2
![Page 8: Parallel Gibbs Sampling From Colored Fields to Thin Junction Trees](https://reader033.vdocument.in/reader033/viewer/2022051402/568162bc550346895dd347e2/html5/thumbnails/8.jpg)
10
Models With Strong Dependencies
Single variable Gibbs updates tend to mix slowly:
Ideally we would like to draw joint samples.Blocking
Strong Dependencies
X1
X2
![Page 9: Parallel Gibbs Sampling From Colored Fields to Thin Junction Trees](https://reader033.vdocument.in/reader033/viewer/2022051402/568162bc550346895dd347e2/html5/thumbnails/9.jpg)
Carnegie Mellon
An asynchronous Gibbs Sampler that adaptively addresses strong dependencies.
Splash Gibbs Sampler
11
![Page 10: Parallel Gibbs Sampling From Colored Fields to Thin Junction Trees](https://reader033.vdocument.in/reader033/viewer/2022051402/568162bc550346895dd347e2/html5/thumbnails/10.jpg)
12
Splash Gibbs Sampler
Step 1: Grow multiple Splashes in parallel:
ConditionallyIndependent
![Page 11: Parallel Gibbs Sampling From Colored Fields to Thin Junction Trees](https://reader033.vdocument.in/reader033/viewer/2022051402/568162bc550346895dd347e2/html5/thumbnails/11.jpg)
13
Splash Gibbs Sampler
Step 2: Calibrate the trees in parallel
![Page 12: Parallel Gibbs Sampling From Colored Fields to Thin Junction Trees](https://reader033.vdocument.in/reader033/viewer/2022051402/568162bc550346895dd347e2/html5/thumbnails/12.jpg)
14
Splash Gibbs Sampler
Step 3: Sample trees in parallel
![Page 13: Parallel Gibbs Sampling From Colored Fields to Thin Junction Trees](https://reader033.vdocument.in/reader033/viewer/2022051402/568162bc550346895dd347e2/html5/thumbnails/13.jpg)
15
Adaptively Prioritized Splashes
Adapt the shape of the Splash to span strongly coupled variables:
Converges to the correct distributionRequires vanishing adaptation
Noisy Image BFS Splashes Adaptive Splashes
![Page 14: Parallel Gibbs Sampling From Colored Fields to Thin Junction Trees](https://reader033.vdocument.in/reader033/viewer/2022051402/568162bc550346895dd347e2/html5/thumbnails/14.jpg)
16
Experimental ResultsMarkov logic network with strong dependencies
10K Variables 28K Factors
The Splash sampler outperforms the Chromatic sampler on models with strong dependencies
Likelihood Final Sample
Bette
r
Splash
Chromatic
“Mixing”
BetterSplash
Chromatic
Speedup in Sample Generation
Bette
r
Splash
Chromatic
![Page 15: Parallel Gibbs Sampling From Colored Fields to Thin Junction Trees](https://reader033.vdocument.in/reader033/viewer/2022051402/568162bc550346895dd347e2/html5/thumbnails/15.jpg)
17
Conclusions
Chromatic Gibbs sampler for models with weak dependencies
Converges to the correct distributionQuantifiable improvement in mixing
Theoretical analysis of the Synchronous Gibbs sampler on 2-colorable models
Proved marginal convergence on 2-colorable models
Splash Gibbs sampler for models with strong dependencies
Adaptive asynchronous tree constructionExperimental evaluation demonstrates an improvement in mixing