distant kin in the em family david a. van dyk department of statistics university of california,...
TRANSCRIPT
![Page 1: Distant Kin in the EM Family David A. van Dyk Department of Statistics University of California, Irvine. (Joint work with Xiao Li Meng and Taeyong Park.)](https://reader035.vdocument.in/reader035/viewer/2022070307/551abfab55034656628b5884/html5/thumbnails/1.jpg)
Distant Kin in the EM Family
David A. van Dyk
Department of Statistics
University of California, Irvine.
(Joint work with Xiao Li Meng and Taeyong Park.)
![Page 2: Distant Kin in the EM Family David A. van Dyk Department of Statistics University of California, Irvine. (Joint work with Xiao Li Meng and Taeyong Park.)](https://reader035.vdocument.in/reader035/viewer/2022070307/551abfab55034656628b5884/html5/thumbnails/2.jpg)
Outline The EM Family Tree The Stochastic Cousins Some Odd Relations
o Nested EM and The Partially Blocked Gibbs Sampler
A Newly Found Kinsmano A Stochastic ECME/AECM Sampler:
The Partially Collapsed Gibbs Sampler
![Page 3: Distant Kin in the EM Family David A. van Dyk Department of Statistics University of California, Irvine. (Joint work with Xiao Li Meng and Taeyong Park.)](https://reader035.vdocument.in/reader035/viewer/2022070307/551abfab55034656628b5884/html5/thumbnails/3.jpg)
The EM Family Tree
EM Algorithm
StochasticSimulation
Variance Calculations
Gauss-Seidel
Efficient DA
Monte CarloIntegration
ECM
ECME
SEMDA
samplerMCEM
PXEM
Efficient DAEM
NEM
AECM
SECM
Algorithms
Methods
1977 (Dempster, Laird, & Rubin)1987 (Tanner & Wong)1990 (Wie & Tanner)1998 (Liu, Rubin, & Wu)1995 (van Dyk, Meng, & Rubin)1991 (Meng & Rubin)1993 (Meng & Rubin)2000 (van Dyk)1994 (Liu & Rubin)1997 (Meng & van Dyk)
![Page 4: Distant Kin in the EM Family David A. van Dyk Department of Statistics University of California, Irvine. (Joint work with Xiao Li Meng and Taeyong Park.)](https://reader035.vdocument.in/reader035/viewer/2022070307/551abfab55034656628b5884/html5/thumbnails/4.jpg)
Stochastic Cousins
EM Algorithm
Gauss-Seidel
Efficient DA
Monte CarloIntegration
ECM
ECME
MCEM
PXEM
Efficient DAEM
NEM
AECM
DA Sampler
Gibbs Sampler
Marginal DAPX-DA
Partially BlockedGibbs Sampler
???
![Page 5: Distant Kin in the EM Family David A. van Dyk Department of Statistics University of California, Irvine. (Joint work with Xiao Li Meng and Taeyong Park.)](https://reader035.vdocument.in/reader035/viewer/2022070307/551abfab55034656628b5884/html5/thumbnails/5.jpg)
The EM and DA Algorithms
p(M|)
p(|M)
p(M|)
p(|M)
EM Algorithm DA Sampler
ExpectationStep
MaximizationStep
RandomDraw
BACK
![Page 6: Distant Kin in the EM Family David A. van Dyk Department of Statistics University of California, Irvine. (Joint work with Xiao Li Meng and Taeyong Park.)](https://reader035.vdocument.in/reader035/viewer/2022070307/551abfab55034656628b5884/html5/thumbnails/6.jpg)
An NEM Algorithmwith a Monte Carlo E-step
NEM Algorithm The Stochastic Version
E-Step
M-Step
Draw
p(M1|)
p(|M1)
p(M|)
p(, M2|M1)
p(M1|M2,) p(M2|M1,) p(M1|M2,) p(M2|M1,)
p(M2|M1,) p(|M) p(M2|M1,) p(|M)
![Page 7: Distant Kin in the EM Family David A. van Dyk Department of Statistics University of California, Irvine. (Joint work with Xiao Li Meng and Taeyong Park.)](https://reader035.vdocument.in/reader035/viewer/2022070307/551abfab55034656628b5884/html5/thumbnails/7.jpg)
A Partially-Blocked SamplerNEM Algorithm Partially-Blocked Sampler
E-Step
M-Step
Draw
p(M1|)
p(|M1)
p(M1|, M2)
p(, M2|M1)
p(M1|M2,) p(M2|M1,)
p(M2|M1,) p(|M) p(M2|M1,) p(|M)
BACK
![Page 8: Distant Kin in the EM Family David A. van Dyk Department of Statistics University of California, Irvine. (Joint work with Xiao Li Meng and Taeyong Park.)](https://reader035.vdocument.in/reader035/viewer/2022070307/551abfab55034656628b5884/html5/thumbnails/8.jpg)
Ordering CM-steps in ECMEECME Algorithm
E-Step
M-Step
Draw
p(M|)
p(2|1)p(1|M,2)
Monotone Convergence
ECME Algorithm
p(M|)
p(1|M, 2)p(2|1)
NO Monotone Convergence
Reducing conditioning
Speed upconvergence!
But BE CAREFUL!
Step orderMatters!
![Page 9: Distant Kin in the EM Family David A. van Dyk Department of Statistics University of California, Irvine. (Joint work with Xiao Li Meng and Taeyong Park.)](https://reader035.vdocument.in/reader035/viewer/2022070307/551abfab55034656628b5884/html5/thumbnails/9.jpg)
A Stochastic Version of ECMEECME Algorithm
E-Step
M-Step
Draw
p(M|)
p(2|1)p(1|M,2)
p(M|)
p(2|1)p(1|M, 2)
Incompatible Conditional Distributions
What is the stationary distribution
of this chain??
![Page 10: Distant Kin in the EM Family David A. van Dyk Department of Statistics University of California, Irvine. (Joint work with Xiao Li Meng and Taeyong Park.)](https://reader035.vdocument.in/reader035/viewer/2022070307/551abfab55034656628b5884/html5/thumbnails/10.jpg)
AECM and Partially Collapsed Samplers
E
M
D
p(M|)
p(M,2|1)p(1|M,2)
p(M|)
p(1|M,2) p(2|1)
p(M,2|1)
p(1|M,2)
p(M|)
p(1|M,2) p(2|M1,1)
p(M|)
p(2|M1,1)p(1|M,2)
ECME
PartiallyCollapsed
Blocked Sampler
AECM
PartiallyCollapsed
Incompatibledraws
Stationary distributionmust be verified!
![Page 11: Distant Kin in the EM Family David A. van Dyk Department of Statistics University of California, Irvine. (Joint work with Xiao Li Meng and Taeyong Park.)](https://reader035.vdocument.in/reader035/viewer/2022070307/551abfab55034656628b5884/html5/thumbnails/11.jpg)
Completely Collapsed Samplers
E
M
D
p(M|)
p(M,2|1)p(M,1|2)
p(M|)
p(1|M,2) p(2|1)
p(2|1)p(1|2)
ECME Collapsed Sampler
CompleteCollapse
Blocking (ECME) is a special case of Partial Collapse (AECM). We expect Collapsed Samplers (CM) to perform better than Partially
Collapsed Samplers (AECM). And we expect Collapsing (CM) to perform better than Blocking (ECME).
Many of these relationships are known, I emphasize the connections between EM-type DA-type algorithms.
![Page 12: Distant Kin in the EM Family David A. van Dyk Department of Statistics University of California, Irvine. (Joint work with Xiao Li Meng and Taeyong Park.)](https://reader035.vdocument.in/reader035/viewer/2022070307/551abfab55034656628b5884/html5/thumbnails/12.jpg)
Reducing Conditioning in Gibbs: The Simplest ExampleConsider a two-step Gibbs Sampler:
The Markov Chain
has stationary dist’n With target margins butWithout the correlation of the target distributionAND converges quickly!
Iteration t
Iteration t+1/2
We regain the target distribution with a one-step shifted chain.
![Page 13: Distant Kin in the EM Family David A. van Dyk Department of Statistics University of California, Irvine. (Joint work with Xiao Li Meng and Taeyong Park.)](https://reader035.vdocument.in/reader035/viewer/2022070307/551abfab55034656628b5884/html5/thumbnails/13.jpg)
Heads Up!!
Reducing the conditioning within Gibbs involves new challenges:
The order of the draws may effect the stationary distribution of the chain.
The conditional distributions may no be compatible with any joint distribution.
The steps sometimes can be blocked to form an ordinary Gibbs sampler with fewer steps.
![Page 14: Distant Kin in the EM Family David A. van Dyk Department of Statistics University of California, Irvine. (Joint work with Xiao Li Meng and Taeyong Park.)](https://reader035.vdocument.in/reader035/viewer/2022070307/551abfab55034656628b5884/html5/thumbnails/14.jpg)
An Example from Astronomy
Parameterized Latent Poisson Process
Underlying Poisson intensity is a mixture of a broad feature and several narrow features.
The “line location” and mixture indicator are highly correlated.
Photon energy
Line location
Emission Line
Spectral Model for Photon Counts
![Page 15: Distant Kin in the EM Family David A. van Dyk Department of Statistics University of California, Irvine. (Joint work with Xiao Li Meng and Taeyong Park.)](https://reader035.vdocument.in/reader035/viewer/2022070307/551abfab55034656628b5884/html5/thumbnails/15.jpg)
An Example from Astronomy
Standard sampler simulates
Photon energy
Line location
Emission Line
Spectral Model for Photon Counts
which may converge very slowly or not at all.
![Page 16: Distant Kin in the EM Family David A. van Dyk Department of Statistics University of California, Irvine. (Joint work with Xiao Li Meng and Taeyong Park.)](https://reader035.vdocument.in/reader035/viewer/2022070307/551abfab55034656628b5884/html5/thumbnails/16.jpg)
An Incompatible Gibbs Sampler
![Page 17: Distant Kin in the EM Family David A. van Dyk Department of Statistics University of California, Irvine. (Joint work with Xiao Li Meng and Taeyong Park.)](https://reader035.vdocument.in/reader035/viewer/2022070307/551abfab55034656628b5884/html5/thumbnails/17.jpg)
Computational Gains
![Page 18: Distant Kin in the EM Family David A. van Dyk Department of Statistics University of California, Irvine. (Joint work with Xiao Li Meng and Taeyong Park.)](https://reader035.vdocument.in/reader035/viewer/2022070307/551abfab55034656628b5884/html5/thumbnails/18.jpg)
Verifying the Stationary Distribution of Sampler 2
![Page 19: Distant Kin in the EM Family David A. van Dyk Department of Statistics University of California, Irvine. (Joint work with Xiao Li Meng and Taeyong Park.)](https://reader035.vdocument.in/reader035/viewer/2022070307/551abfab55034656628b5884/html5/thumbnails/19.jpg)
The General Strategy
BACK