approximate decoding: icm, block methods, alpha-beta swap ... · approximate decoding: icm, block...
TRANSCRIPT
![Page 1: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/1.jpg)
Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion
Julieta Martinez
University of British Columbia
August 25, 2015
![Page 2: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/2.jpg)
https://www.youtube.com/watch?v=kp3ik5f3-2c&t=18m36s
![Page 3: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/3.jpg)
Outline
• Motivation
• Approximate decoding
• ICM: Iterated conditional modes
• Block methods
• Alpha-beta swap & alpha-expansion
![Page 4: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/4.jpg)
Motivation
![Page 5: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/5.jpg)
![Page 6: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/6.jpg)
![Page 8: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/8.jpg)
Tung, Frederick, and James J. Little. "Collageparsing: Nonparametric scene parsing by adaptive overlapping windows." Computer Vision–ECCV 2014. Springer International Publishing, 2014. 511-525.
![Page 9: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/9.jpg)
![Page 10: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/10.jpg)
![Page 11: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/11.jpg)
![Page 12: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/12.jpg)
![Page 13: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/13.jpg)
Motivation
• We often have more than 2 states for each node
• Often the states are not directly comparable
• Exact decoding is NP-hard — but we still want real-time predictions
![Page 14: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/14.jpg)
ICM: Iterated Conditional Modes
![Page 15: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/15.jpg)
• Until convergence
![Page 16: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/16.jpg)
• Until convergence
• For xi 2 X
![Page 17: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/17.jpg)
• Until convergence
• For
• Assume that all other nodes are fixed
xi 2 X
xj 2 X, j 6= i
![Page 18: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/18.jpg)
• Until convergence
• For
• Assume that all other nodes are fixed
• Solve for
xi 2 X
xj 2 X, j 6= i
xi 2 X
![Page 19: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/19.jpg)
• Until convergence
• For
• Assume that all other nodes are fixed
• Solve for
xi 2 X
xj 2 X, j 6= i
xi 2 X
![Page 20: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/20.jpg)
• Until convergence
• For
• Assume that all other nodes are fixed
• Solve for
xi 2 X
xj 2 X, j 6= i
xi 2 X
![Page 21: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/21.jpg)
• Until convergence
• For
• Assume that all other nodes are fixed
• Solve for
xi 2 X
xj 2 X, j 6= i
xi 2 X
![Page 22: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/22.jpg)
• Until convergence
• For
• Assume that all other nodes are fixed
• Solve for
xi 2 X
xj 2 X, j 6= i
xi 2 X
![Page 23: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/23.jpg)
• Until convergence
• For
• Assume that all other nodes are fixed
• Solve for
xi 2 X
xj 2 X, j 6= i
xi 2 X
![Page 24: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/24.jpg)
• Until convergence
• For
• Assume that all other nodes are fixed
• Solve for
xi 2 X
xj 2 X, j 6= i
xi 2 X
…and so on
![Page 25: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/25.jpg)
• Until convergence
• For
• Assume that all other nodes are fixed
• Solve for
xi 2 X
xj 2 X, j 6= i
xi 2 X
…and so on
iterated
![Page 26: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/26.jpg)
• Until convergence
• For
• Assume that all other nodes are fixed
• Solve for
xi 2 X
xj 2 X, j 6= i
xi 2 X
…and so on
conditional
iterated
![Page 27: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/27.jpg)
• Until convergence
• For
• Assume that all other nodes are fixed
• Solve for
xi 2 X
xj 2 X, j 6= i
xi 2 X
…and so on
conditional
iterated
modes
![Page 28: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/28.jpg)
• Pros
• Super fast
• Super easy to implement
• Cons
• Greedy — will get stuck on local minima
• In practice, local minima will not be very good
Pros and cons of ICM
![Page 29: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/29.jpg)
Improving ICM
• Restart with different initializations
• Look at all the nodes, and update only the one that gives the best improvement
Guaranteed to find global minimum if done enough times
![Page 30: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/30.jpg)
Improving ICM
ICM with best improvement
Vanilla ICM
ICM with restarts
![Page 31: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/31.jpg)
Other variants of local search
• Local search is an area of research on its own
• Usually a top performer on SAT, TSP, scheduling, and other NP-hard problems
• Read the book of my supervisor ☺
![Page 32: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/32.jpg)
Block methods
![Page 33: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/33.jpg)
Block ICM
• In ICM we updated one node at a time
• Simple extension: update more than one node at a time
• In fact, we now have a toolbox to do exact decoding in large graphs as long as they have
a nice graph structure
![Page 34: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/34.jpg)
Block ICM
![Page 35: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/35.jpg)
Block ICM
decode optimally
![Page 36: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/36.jpg)
Block ICM
decode optimally too
![Page 37: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/37.jpg)
Other block methods• Many methods have this 1-variable, >1-variables
generalization
• Inference: mean field updates the marginal of 1 variable at a time — we can update the marginals of >1 variables at a time
• Sampling: Gibbs sampling samples 1 variable at a time — we can sample a block of >1 variables at a time
Mark will cover this next Monday
Jason will cover this tomorrow
![Page 38: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/38.jpg)
Alpha-beta swap & alpha-expansion
![Page 39: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/39.jpg)
An overview
original labelling
alpha-beta swap
ICM
alpha-expansion
Boykov, Yuri, Olga Veksler, and Ramin Zabih. "Fast approximate energy minimization via graph cuts." Pattern Analysis and Machine Intelligence, IEEE Transactions on 23.11 (2001): 1222-1239.
![Page 40: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/40.jpg)
Remember graph cuts• Assume a binary label set (e.g.
black and white)
• Connect nodes to “source” and “sink”
• If the submodular property is satisfied ( ) then we can decode this exactly solving a max-flow problem
✓01 + ✓10 > ✓00 + ✓11
• Problem: most images are not binary
http://www.caam.rice.edu/~wy1/ParaMaxFlow/2007/06/binary-code.html
![Page 41: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/41.jpg)
Alpha-beta swap
• All pixels not labelled alpha or beta stay fixed
• Do a graph cut on alpha and beta
• Some alpha nodes become beta and vice-versaBoykov, Yuri, Olga Veksler, and Ramin Zabih. "Fast approximate energy minimization via graph cuts." Pattern Analysis and Machine
Intelligence, IEEE Transactions on 23.11 (2001): 1222-1239.
![Page 42: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/42.jpg)
Alpha-expansion
• All pixels not alpha may change to alpha
• Some pixels become alpha
Boykov, Yuri, Olga Veksler, and Ramin Zabih. "Fast approximate energy minimization via graph cuts." Pattern Analysis and Machine Intelligence, IEEE Transactions on 23.11 (2001): 1222-1239.
![Page 43: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/43.jpg)
Alpha-expansion
• The required constraints are stronger than those for beta-swap:for any alpha, beta and gamma triplet
✓(↵,↵) + ✓(�, �) < ✓(↵, �) + ✓(�,↵)
Boykov, Yuri, Olga Veksler, and Ramin Zabih. "Fast approximate energy minimization via graph cuts." Pattern Analysis and Machine Intelligence, IEEE Transactions on 23.11 (2001): 1222-1239.
![Page 44: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/44.jpg)
Performance on the Tsukuba image
Boykov, Yuri, Olga Veksler, and Ramin Zabih. "Fast approximate energy minimization via graph cuts." Pattern Analysis and Machine Intelligence, IEEE Transactions on 23.11 (2001): 1222-1239.
![Page 45: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/45.jpg)
• If the minimum cut severs edge t^p_i, assign label i to p
Exact multi-label optimization
• There is one case when multi-label optimization is possible
• We have to assume that the energy is
E(f) =X
p2PDpfp +
X
(p,q)2N
�pq|fp � fq|
Boykov, Yuri, and Olga Veksler. "Graph cuts in vision and graphics: Theories and applications." Handbook of mathematical models in computer vision. Springer US, 2006. 79-96.
![Page 46: Approximate decoding: ICM, block methods, alpha-beta swap ... · Approximate decoding: ICM, block methods, alpha-beta swap & alpha-expansion Julieta Martinez University of British](https://reader033.vdocument.in/reader033/viewer/2022060319/5f0caedf7e708231d4369e33/html5/thumbnails/46.jpg)
Cool applications
• Graphcut textures (SIGGRAPH 03): https://www.youtube.com/watch?v=Ya6BshBH6G4
• Grabcut demo (SIGGRAPH 04) using OpenCV: https://youtu.be/kAwxLTDDAwU?t=19s