graphcut -based optimisation for computer vision
DESCRIPTION
GraphCut -based Optimisation for Computer Vision. Ľubor Ladický. Overview. Motivation Min-Cut / Max-Flow (Graph Cut) Algorithm Markov and Conditional Random Fields Random Field Optimisation using Graph Cuts Submodular vs. Non-Submodular Problems Pairwise vs. Higher Order Problems - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/1.jpg)
GraphCut-based OptimisationGraphCut-based Optimisationfor Computer Visionfor Computer Vision
Ľubor Ladický
![Page 2: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/2.jpg)
Overview
• Motivation
• Min-Cut / Max-Flow (Graph Cut) Algorithm
• Markov and Conditional Random Fields
• Random Field Optimisation using Graph Cuts
• Submodular vs. Non-Submodular Problems
• Pairwise vs. Higher Order Problems
• 2-Label vs. Multi-Label Problems
• Recent Advances in Random Field Optimisation
• Conclusions
![Page 3: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/3.jpg)
Overview
• Motivation
• Min-Cut / Max-Flow (Graph Cut) Algorithm
• Markov and Conditional Random Fields
• Random Field Optimisation using Graph Cuts
• Submodular vs. Non-Submodular Problems
• Pairwise vs. Higher Order Problems
• 2-Label vs. Multi-Label Problems
• Recent Advances in Random Field Optimisation
• Conclusions
![Page 4: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/4.jpg)
Image Labelling Problems
Image Denoising Geometry Estimation Object Segmentation
Assign a label to each image pixelAssign a label to each image pixel
Building
Sky
Tree Grass
Depth Estimation
![Page 5: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/5.jpg)
Image Labelling Problems
• Labellings highly structured
Possible labelling Unprobable labelling Impossible labelling
![Page 6: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/6.jpg)
Image Labelling Problems
• Labellings highly structured
• Labels highly correlated with very complex dependencies
• Neighbouring pixels tend to take the same label
• Low number of connected components
• Classes present may be seen in one image
• Geometric / Location consistency
• Planarity in depth estimation
• … many others (task dependent)
![Page 7: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/7.jpg)
Image Labelling Problems
• Labelling highly structured
• Labels highly correlated with very complex dependencies
• Independent label estimation too hard
![Page 8: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/8.jpg)
Image Labelling Problems
• Labelling highly structured
• Labels highly correlated with very complex dependencies
• Independent label estimation too hard
• Whole labelling should be formulated as one optimisation problem
![Page 9: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/9.jpg)
Image Labelling Problems
• Labelling highly structured
• Labels highly correlated with very complex dependencies
• Independent label estimation too hard
• Whole labelling should be formulated as one optimisation problem
• Number of pixels up to millions
• Hard to train complex dependencies
• Optimisation problem is hard to infer
![Page 10: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/10.jpg)
Image Labelling Problems
• Labelling highly structured
• Labels highly correlated with very complex dependencies
• Independent label estimation too hard
• Whole labelling should be formulated as one optimisation problem
• Number of pixels up to millions
• Hard to train complex dependencies
• Optimisation problem is hard to infer
Vision is hard !
![Page 11: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/11.jpg)
Image Labelling Problems
• You can
either
• Change the subject from the Computer Vision to the History of Renaissance Art
Vision is hard !
![Page 12: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/12.jpg)
Image Labelling Problems
• You can
either
• Change the subject from the Computer Vision to the History of Renaissance Art
or
• Learn everything about Random Fields and Graph Cuts
Vision is hard !
![Page 13: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/13.jpg)
Foreground / Background Estimation
Rother et al. SIGGRAPH04
![Page 14: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/14.jpg)
Foreground / Background Estimation
Data term Smoothness term
Data term
Estimated using FG / BG colour models
Smoothness term
where
Intensity dependent smoothness
![Page 15: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/15.jpg)
Foreground / Background Estimation
Data term Smoothness term
![Page 16: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/16.jpg)
Foreground / Background Estimation
Data term Smoothness term
How to solve this optimisation problem?
![Page 17: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/17.jpg)
Foreground / Background Estimation
Data term Smoothness term
How to solve this optimisation problem?
• Transform into min-cut / max-flow problem
• Solve it using min-cut / max-flow algorithm
![Page 18: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/18.jpg)
Overview
• Motivation
• Min-Cut / Max-Flow (Graph Cut) Algorithm
• Markov and Conditional Random Fields
• Random Field Optimisation using Graph Cuts
• Submodular vs. Non-Submodular Problems
• Pairwise vs. Higher Order Problems
• 2-Label vs. Multi-Label Problems
• Applications
• Conclusions
![Page 19: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/19.jpg)
Max-Flow Problem
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
Task :
Maximize the flow from the sink to the source such that
1) The flow it conserved for each node
2) The flow for each pipe does not exceed the capacity
![Page 20: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/20.jpg)
Max-Flow Problem
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
![Page 21: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/21.jpg)
Max-Flow Problem
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
flow from the source
capacity
flow from node i to node j
conservation of flow
set of edges
set of nodes
![Page 22: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/22.jpg)
Max-Flow Problem
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
![Page 23: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/23.jpg)
Max-Flow Problem
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
![Page 24: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/24.jpg)
Max-Flow Problem
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 3
![Page 25: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/25.jpg)
Max-Flow Problem
source
sink
9
5
6-3
8-3
42+
32
2
5-3
3-3
5
5
311
6
2
3
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 3
+3
![Page 26: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/26.jpg)
Max-Flow Problem
source
sink
9
5
3
5
45
2
2
2
5
5
311
6
2
3
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 3
3
![Page 27: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/27.jpg)
Max-Flow Problem
source
sink
9
5
3
5
45
2
2
2
5
5
311
6
2
3
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 3
3
![Page 28: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/28.jpg)
Max-Flow Problem
source
sink
9
5
3
5
45
2
2
2
5
5
311
6
2
3
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 6
3
![Page 29: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/29.jpg)
Max-Flow Problem
source
sink
9-3
5
3
5-3
45
2
2
2
5
5
3-311
6
2
3-3
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 6
3
+3
+3
![Page 30: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/30.jpg)
Max-Flow Problem
source
sink
6
5
3
2
45
2
2
2
5
5
11
6
2
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 6
3
3
3
![Page 31: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/31.jpg)
Max-Flow Problem
source
sink
6
5
3
2
45
2
2
2
5
5
11
6
2
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 6
3
3
3
![Page 32: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/32.jpg)
Max-Flow Problem
source
sink
6
5
3
2
45
2
2
2
5
5
11
6
2
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 11
3
3
3
![Page 33: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/33.jpg)
Max-Flow Problem
source
sink
6-5
5-5
3
2
45
2
2
2
5-5
5-5
1+51
6
2+5
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 11
3
3
3
![Page 34: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/34.jpg)
Max-Flow Problem
source
sink
1
3
2
45
2
2
2
61
6
7
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 11
3
33
![Page 35: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/35.jpg)
Max-Flow Problem
source
sink
1
3
2
45
2
2
2
61
6
7
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 11
3
33
![Page 36: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/36.jpg)
Max-Flow Problem
source
sink
1
3
2
45
2
2
2
61
6
7
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 13
3
33
![Page 37: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/37.jpg)
Max-Flow Problem
source
sink
1
3
2-2
45
2-2
2-2
2-2
61
6
7
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 13
3
33
+2
+2
![Page 38: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/38.jpg)
Max-Flow Problem
source
sink
1
3
45
61
6
7
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 13
3
33
2
2
![Page 39: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/39.jpg)
Max-Flow Problem
source
sink
1
3
45
61
6
7
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 13
3
33
2
2
![Page 40: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/40.jpg)
Max-Flow Problem
source
sink
1
3
45
61
6
7
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 15
3
33
2
2
![Page 41: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/41.jpg)
Max-Flow Problem
source
sink
1
3-2
4-25
61
6-2
7
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 15
3
33+
2
2
2-2
+2
![Page 42: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/42.jpg)
Max-Flow Problem
source
sink
1
1
5
61
4
7
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 15
3
35
2
2
2
![Page 43: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/43.jpg)
Max-Flow Problem
source
sink
1
1
5
61
4
7
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 15
3
35
2
2
2
![Page 44: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/44.jpg)
Max-Flow Problem
source
sink
1
1
5
61
4
7
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 15
3
35
2
2
2
![Page 45: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/45.jpg)
Max-Flow Problem
source
sink
1
1
5
61
4
7
Ford & Fulkerson algorithm (1956)
Why is the solution globally optimal ?
flow = 15
3
35
2
2
2
![Page 46: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/46.jpg)
Max-Flow Problem
source
sink
1
1
5
61
4
7
Ford & Fulkerson algorithm (1956)
Why is the solution globally optimal ?
1. Let S be the set of reachable nodes in the
residual graph
flow = 15
3
35
2
2
2
S
![Page 47: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/47.jpg)
Max-Flow Problem
Ford & Fulkerson algorithm (1956)
Why is the solution globally optimal ?
1. Let S be the set of reachable nodes in the
residual graph
2. The flow from S to V - S equals to the sum
of capacities from S to V – S
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
S
![Page 48: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/48.jpg)
Max-Flow Problem
Ford & Fulkerson algorithm (1956)
Why is the solution globally optimal ?
1. Let S be the set of reachable nodes in the residual graph
2. The flow from S to V - S equals to the sum of capacities from S to V – S
3. The flow from any A to V - A is upper bounded by the sum of capacities from A to V – A
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
A
![Page 49: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/49.jpg)
Max-Flow Problem
flow = 15
Ford & Fulkerson algorithm (1956)
Why is the solution globally optimal ?
1. Let S be the set of reachable nodes in the residual graph
2. The flow from S to V - S equals to the sum of capacities from S to V – S
3. The flow from any A to V - A is upper bounded by the sum of capacities from A to V – A
4. The solution is globally optimal
source
sink
8/9
5/5
5/6
8/8
2/4
0/2
2/2
0/2
5/5
3/3
5/5
5/5
3/30/10/1
2/6
0/2
3/3
Individual flows obtained by summing up all paths
![Page 50: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/50.jpg)
Max-Flow Problem
flow = 15
Ford & Fulkerson algorithm (1956)
Why is the solution globally optimal ?
1. Let S be the set of reachable nodes in the residual graph
2. The flow from S to V - S equals to the sum of capacities from S to V – S
3. The flow from any A to V - A is upper bounded by the sum of capacities from A to V – A
4. The solution is globally optimal
source
sink
8/9
5/5
5/6
8/8
2/4
0/2
2/2
0/2
5/5
3/3
5/5
5/5
3/30/10/1
2/6
0/2
3/3
![Page 51: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/51.jpg)
Max-Flow Problem
source
sink
1000
1000
Ford & Fulkerson algorithm (1956)
Order does matter
1000
1000
1
![Page 52: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/52.jpg)
Max-Flow Problem
source
sink
1000
1000
Ford & Fulkerson algorithm (1956)
Order does matter
1000
1000
1
![Page 53: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/53.jpg)
Max-Flow Problem
source
sink
999
999
Ford & Fulkerson algorithm (1956)
Order does matter
1000
1000
1
![Page 54: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/54.jpg)
Max-Flow Problem
source
sink
999
999
Ford & Fulkerson algorithm (1956)
Order does matter
1000
1000
1
![Page 55: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/55.jpg)
Max-Flow Problem
source
sink
999
999
Ford & Fulkerson algorithm (1956)
Order does matter
999
999
1
![Page 56: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/56.jpg)
Max-Flow Problem
source
sink
999
999
Ford & Fulkerson algorithm (1956)
Order does matter
• Standard algorithm not polynomial
• Breath first leads to O(VE2)
• Path found in O(E)
• At least one edge gets saturated
• The saturated edge distance to the source has to increase and is at most V leading to O(VE)
(Edmonds & Karp, Dinic)
999
999
1
![Page 57: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/57.jpg)
Max-Flow Problem
source
sink
999
999
Ford & Fulkerson algorithm (1956)
Order does matter
• Standard algorithm not polynomial
• Breath first leads to O(VE2)
(Edmonds & Karp)
• Various methods use different algorithm to find the path and vary in complexity
999
999
1
![Page 58: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/58.jpg)
Min-Cut Problem
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
Task :
Minimize the cost of the cut
1) Each node is either assigned to the source S or sink T
2) The cost of the edge (i, j) is taken if (iS) and (jT)
![Page 59: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/59.jpg)
Min-Cut Problem
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
![Page 60: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/60.jpg)
Min-Cut Problem
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
source set sink set
edge costs
![Page 61: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/61.jpg)
Min-Cut Problem
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
source set sink set
edge costs
S
T
cost = 18
![Page 62: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/62.jpg)
Min-Cut Problem
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
source set sink set
edge costs
S
T
cost = 25
![Page 63: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/63.jpg)
Min-Cut Problem
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
source set sink set
edge costs
S
T
cost = 23
![Page 64: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/64.jpg)
Min-Cut Problem
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
x1x2
x3
x4
x5
x6
![Page 65: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/65.jpg)
Min-Cut Problem
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
x1x2
x3
x4
x5
x6
transformable into Linear program (LP)
![Page 66: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/66.jpg)
Min-Cut Problem
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
x1x2
x3
x4
x5
x6
Dual to max-flow problem
transformable into Linear program (LP)
![Page 67: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/67.jpg)
Min-Cut Problem
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
x1x2
x3
x4
x5
x6
After the substitution of the constraints :
![Page 68: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/68.jpg)
Min-Cut Problem
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
xi = 0 xi S xi = 1 xi T
x1x2
x3
x4
x5
x6
source edges sink edges
Edges between variables
![Page 69: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/69.jpg)
Min-Cut Problem
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
C(x) = 5x1 + 9x2 + 4x3 + 3x3(1-x1) + 2x1(1-x3)
+ 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1)
+ 1x5(1-x1) + 6x5(1-x3) + 5x6(1-x3) + 1x3(1-x6)
+ 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)
+ 8(1-x5) + 5(1-x6)
x1x2
x3
x4
x5
x6
![Page 70: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/70.jpg)
Min-Cut Problem
C(x) = 5x1 + 9x2 + 4x3 + 3x3(1-x1) + 2x1(1-x3)
+ 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1)
+ 1x5(1-x1) + 6x5(1-x3) + 5x6(1-x3) + 1x3(1-x6)
+ 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)
+ 8(1-x5) + 5(1-x6)
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
x1x2
x3
x4
x5
x6
![Page 71: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/71.jpg)
Min-Cut Problem
C(x) = 2x1 + 9x2 + 4x3 + 2x1(1-x3)
+ 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1)
+ 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6)
+ 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)
+ 5(1-x5) + 5(1-x6)
+ 3x1 + 3x3(1-x1) + 3x5(1-x3) + 3(1-x5)
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
x1x2
x3
x4
x5
x6
![Page 72: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/72.jpg)
Min-Cut Problem
C(x) = 2x1 + 9x2 + 4x3 + 2x1(1-x3)
+ 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1)
+ 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6)
+ 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)
+ 5(1-x5) + 5(1-x6)
+ 3x1 + 3x3(1-x1) + 3x5(1-x3) + 3(1-x5)
3x1 + 3x3(1-x1) + 3x5(1-x3) + 3(1-x5)
=
3 + 3x1(1-x3) + 3x3(1-x5)
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
x1x2
x3
x4
x5
x6
![Page 73: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/73.jpg)
Min-Cut Problem
C(x) = 3 + 2x1 + 9x2 + 4x3 + 5x1(1-x3)
+ 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1)
+ 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6)
+ 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)
+ 5(1-x5) + 5(1-x6) + 3x3(1-x5)
source
sink
9
5
3
5
45
2
2
2
5
5
311
6
2
33
x1x2
x3
x4
x5
x6
![Page 74: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/74.jpg)
Min-Cut Problem
C(x) = 3 + 2x1 + 9x2 + 4x3 + 5x1(1-x3)
+ 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1)
+ 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6)
+ 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)
+ 5(1-x5) + 5(1-x6) + 3x3(1-x5)
source
sink
9
5
3
5
45
2
2
2
5
5
311
6
2
33
x1x2
x3
x4
x5
x6
![Page 75: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/75.jpg)
Min-Cut Problem
C(x) = 3 + 2x1 + 6x2 + 4x3 + 5x1(1-x3)
+ 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1)
+ 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6)
+ 2x5(1-x4) + 6(1-x4)
+ 2(1-x5) + 5(1-x6) + 3x3(1-x5)
+ 3x2 + 3x6(1-x2) + 3x5(1-x6) + 3(1-x5)
3x2 + 3x6(1-x2) + 3x5(1-x6) + 3(1-x5)
=
3 + 3x2(1-x6) + 3x6(1-x5)
source
sink
9
5
3
5
45
2
2
2
5
5
311
6
2
33
x1x2
x3
x4
x5
x6
![Page 76: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/76.jpg)
Min-Cut Problem
C(x) = 6 + 2x1 + 6x2 + 4x3 + 5x1(1-x3)
+ 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1)
+ 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6)
+ 2x5(1-x4) + 6(1-x4) + 3x2(1-x6) + 3x6(1-x5)
+ 2(1-x5) + 5(1-x6) + 3x3(1-x5)
source
sink
6
5
3
2
45
2
2
2
5
5
11
6
2
3
3
3
x1x2
x3
x4
x5
x6
![Page 77: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/77.jpg)
Min-Cut Problem
C(x) = 6 + 2x1 + 6x2 + 4x3 + 5x1(1-x3)
+ 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1)
+ 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6)
+ 2x5(1-x4) + 6(1-x4) + 3x2(1-x6) + 3x6(1-x5)
+ 2(1-x5) + 5(1-x6) + 3x3(1-x5)
source
sink
6
5
3
2
45
2
2
2
5
5
11
6
2
3
3
3
x1x2
x3
x4
x5
x6
![Page 78: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/78.jpg)
Min-Cut Problem
C(x) = 11 + 2x1 + 1x2 + 4x3 + 5x1(1-x3)
+ 3x3(1-x1) + 7x2(1-x3) + 2x4(1-x1)
+ 1x5(1-x1) + 3x5(1-x3) + 6x3(1-x6)
+ 2x5(1-x4) + 6(1-x4) + 3x2(1-x6) + 3x6(1-x5)
+ 2(1-x5) + 3x3(1-x5)
source
sink
1
3
2
45
2
2
2
61
6
7
3
33
x1x2
x3
x4
x5
x6
![Page 79: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/79.jpg)
Min-Cut Problem
C(x) = 11 + 2x1 + 1x2 + 4x3 + 5x1(1-x3)
+ 3x3(1-x1) + 7x2(1-x3) + 2x4(1-x1)
+ 1x5(1-x1) + 3x5(1-x3) + 6x3(1-x6)
+ 2x5(1-x4) + 6(1-x4) + 3x2(1-x6) + 3x6(1-x5)
+ 2(1-x5) + 3x3(1-x5)
source
sink
1
3
2
45
2
2
2
61
6
7
3
33
x1x2
x3
x4
x5
x6
![Page 80: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/80.jpg)
Min-Cut Problem
C(x) = 13 + 1x2 + 4x3 + 5x1(1-x3)
+ 3x3(1-x1) + 7x2(1-x3) + 2x1(1-x4)
+ 1x5(1-x1) + 3x5(1-x3) + 6x3(1-x6)
+ 2x4(1-x5) + 6(1-x4) + 3x2(1-x6) + 3x6(1-x5)
+ 3x3(1-x5)
source
sink
1
3
45
61
6
7
3
33
2
2
x1x2
x3
x4
x5
x6
![Page 81: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/81.jpg)
Min-Cut Problem
C(x) = 13 + 1x2 + 2x3 + 5x1(1-x3)
+ 3x3(1-x1) + 7x2(1-x3) + 2x1(1-x4)
+ 1x5(1-x1) + 3x5(1-x3) + 6x3(1-x6)
+ 2x4(1-x5) + 6(1-x4) + 3x2(1-x6) + 3x6(1-x5)
+ 3x3(1-x5)
source
sink
1
3
45
61
6
7
3
33
2
2
x1x2
x3
x4
x5
x6
![Page 82: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/82.jpg)
Min-Cut Problem
C(x) = 15 + 1x2 + 4x3 + 5x1(1-x3)
+ 3x3(1-x1) + 7x2(1-x3) + 2x1(1-x4)
+ 1x5(1-x1) + 6x3(1-x6) + 6x3(1-x5)
+ 2x5(1-x4) + 4(1-x4) + 3x2(1-x6) + 3x6(1-x5)
source
sink
1
1
5
61
4
7
3
35
2
2
2x1
x2
x3
x4
x5
x6
![Page 83: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/83.jpg)
Min-Cut Problem
C(x) = 15 + 1x2 + 4x3 + 5x1(1-x3)
+ 3x3(1-x1) + 7x2(1-x3) + 2x1(1-x4)
+ 1x5(1-x1) + 6x3(1-x6) + 6x3(1-x5)
+ 2x5(1-x4) + 4(1-x4) + 3x2(1-x6) + 3x6(1-x5)
source
sink
1
1
5
61
4
7
3
35
2
2
2x1
x2
x3
x4
x5
x6
![Page 84: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/84.jpg)
Min-Cut Problem
C(x) = 15 + 1x2 + 4x3 + 5x1(1-x3)
+ 3x3(1-x1) + 7x2(1-x3) + 2x1(1-x4)
+ 1x5(1-x1) + 6x3(1-x6) + 6x3(1-x5)
+ 2x5(1-x4) + 4(1-x4) + 3x2(1-x6) + 3x6(1-x5)
source
sink
1
1
5
61
4
7
3
35
2
2
2x1
x2
x3
x4
x5
x6 cost = 0
min cut
S
T
cost = 0
• All coefficients positive
• Must be global minimum
S – set of reachable nodes from s
![Page 85: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/85.jpg)
Min-Cut Problem
C(x) = 15 + 1x2 + 4x3 + 5x1(1-x3)
+ 3x3(1-x1) + 7x2(1-x3) + 2x1(1-x4)
+ 1x5(1-x1) + 6x3(1-x6) + 6x3(1-x5)
+ 2x5(1-x4) + 4(1-x4) + 3x2(1-x6) + 3x6(1-x5)
source
sink
1
1
5
61
4
7
3
35
2
2
2x1
x2
x3
x4
x5
x6 cost = 0
min cut
S
T• All coefficients positive
• Must be global minimum
T – set of nodes that can reach t
(not necesarly the same)
![Page 86: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/86.jpg)
Overview
• Motivation
• Min-Cut / Max-Flow (Graph Cut) Algorithm
• Markov and Conditional Random Fields
• Random Field Optimisation using Graph Cuts
• Submodular vs. Non-Submodular Problems
• Pairwise vs. Higher Order Problems
• 2-Label vs. Multi-Label Problems
• Recent Advances in Random Field Optimisation
• Conclusions
![Page 87: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/87.jpg)
Markov and Conditional RF
• Markov / Conditional Random fields model probabilistic dependencies of the set of random variables
![Page 88: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/88.jpg)
Markov and Conditional RF
• Markov / Conditional Random fields model conditional dependencies between random variables
• Each variable is conditionally independent of all other variables given its neighbours
![Page 89: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/89.jpg)
Markov and Conditional RF
• Markov / Conditional Random fields model conditional dependencies between random variables
• Each variable is conditionally independent of all other variables given its neighbours
• Posterior probability of the labelling x given data D is :
partition function potential functionscliques
![Page 90: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/90.jpg)
Markov and Conditional RF
• Markov / Conditional Random fields model conditional dependencies between random variables
• Each variable is conditionally independent of all other variables given its neighbours
• Posterior probability of the labelling x given data D is :
• Energy of the labelling is defined as :
partition function potential functionscliques
![Page 91: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/91.jpg)
Markov and Conditional RF
• The most probable (Max a Posteriori (MAP)) labelling is defined as:
![Page 92: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/92.jpg)
Markov and Conditional RF
• The most probable (Max a Posteriori (MAP)) labelling is defined as:
• The only distinction (MRF vs. CRF) is that in the CRF the conditional dependencies between variables depend also on the data
![Page 93: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/93.jpg)
Markov and Conditional RF
• The most probable (Max a Posteriori (MAP)) labelling is defined as:
• The only distinction (MRF vs. CRF) is that in the CRF the conditional dependencies between variables depend also on the data
• Typically we define an energy first and then pretend there is an underlying probabilistic distribution there, but there isn’t really (Pssssst, don’t tell anyone)
![Page 94: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/94.jpg)
Overview
• Motivation
• Min-Cut / Max-Flow (Graph Cut) Algorithm
• Markov and Conditional Random Fields
• Random Field Optimisation using Graph Cuts
• Submodular vs. Non-Submodular Problems
• Pairwise vs. Higher Order Problems
• 2-Label vs. Multi-Label Problems
• Recent Advances in Random Field Optimisation
• Conclusions
![Page 95: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/95.jpg)
Standard CRF Energy
Pairwise CRF models
Data term Smoothness term
![Page 96: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/96.jpg)
Graph Cut based Inference
The energy / potential is called submodular (only) if for every pair of variables :
![Page 97: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/97.jpg)
Graph Cut based Inference
For 2-label problems x{0, 1} :
![Page 98: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/98.jpg)
Graph Cut based Inference
For 2-label problems x{0, 1} :
Pairwise potential can be transformed into :
where
![Page 99: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/99.jpg)
Graph Cut based Inference
For 2-label problems x{0, 1} :
Pairwise potential can be transformed into :
where
![Page 100: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/100.jpg)
Graph Cut based Inference
For 2-label problems x{0, 1} :
Pairwise potential can be transformed into :
where
![Page 101: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/101.jpg)
Graph Cut based Inference
After summing up :
![Page 102: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/102.jpg)
Graph Cut based Inference
After summing up :
Let :
![Page 103: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/103.jpg)
Graph Cut based Inference
After summing up :
Let : Then :
![Page 104: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/104.jpg)
Graph Cut based Inference
After summing up :
Let : Then :
Equivalent st-mincut problem is :
![Page 105: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/105.jpg)
Foreground / Background Estimation
Data term Smoothness term
Data term
Estimated using FG / BG colour models
Smoothness term
where
Intensity dependent smoothness
![Page 106: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/106.jpg)
Foreground / Background Estimation
source
sink
Rother et al. SIGGRAPH04
![Page 107: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/107.jpg)
Extendable to (some) Multi-label CRFs
• each state of multi-label variable encoded using multiple binary variables
• the cost of every possible cut must be the same as the associated energy
• the solution obtained by inverting the encoding
Graph Cut based Inference
ObtainsolutionGraph Cut
BuildGraph
Encoding
multi-label energy solution
![Page 108: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/108.jpg)
Dense Stereo Estimation
Left Camera Image Right Camera Image Dense Stereo Result
• For each pixel assigns a disparity label : z• Disparities from the discrete set {0, 1, .. D}
![Page 109: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/109.jpg)
Dense Stereo Estimation
Data term
Left image feature
Shifted right image feature
Left Camera Image Right Camera Image
![Page 110: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/110.jpg)
Dense Stereo Estimation
Data term
Left image feature
Shifted right image feature
Smoothness term
![Page 111: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/111.jpg)
Dense Stereo Estimation
Ishikawa PAMI03
Encodingsource
xi0
xiD
xk0
xkD
..
.
xk1 xk
2
source
sink
.
![Page 112: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/112.jpg)
Dense Stereo Estimation
Ishikawa PAMI03
Unary Potential
source
xi0
xiD
xk0
xkD
..
.
xk1 xk
2∞
∞
∞
∞
∞
∞
∞ ∞
source
sink
.
cost = +
The cost of every cut should beequal to the corresponding energy under the encoding
![Page 113: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/113.jpg)
Dense Stereo Estimation
Ishikawa PAMI03
Unary Potential
source
xi0
xiD
xk0
xkD
..
.
xk1 xk
2∞
∞
∞
∞
∞
∞
∞ ∞
source
sink
.
cost = +
The cost of every cut should beequal to the corresponding energy under the encoding
![Page 114: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/114.jpg)
Dense Stereo Estimation
Ishikawa PAMI03
Unary Potential
source
xi0
xiD
xk0
xkD
..
.
xk1 xk
2∞
∞
∞
∞
∞
∞
∞ ∞
source
sink
.
cost = ∞
The cost of every cut should beequal to the corresponding energy under the encoding
![Page 115: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/115.jpg)
Dense Stereo Estimation
Ishikawa PAMI03
Pairwise Potential
source
xi0
xiD
xk0
xkD
..
.
xk1 xk
2∞
∞
∞
∞
∞
∞
∞ ∞
K
K
K
K
source
sink
.
cost = + + K
The cost of every cut should beequal to the corresponding energy under the encoding
![Page 116: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/116.jpg)
Dense Stereo Estimation
Ishikawa PAMI03
source
xi0
xiD
xk0
xkD
..
.
xk1 xk
2∞
∞
∞
∞
∞
∞
∞ ∞
K
K
K
K
source
sink
.
cost = + + D K
Pairwise Potential
The cost of every cut should beequal to the corresponding energy under the encoding
![Page 117: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/117.jpg)
Dense Stereo Estimation
See [Ishikawa PAMI03] for more details
source
xi0
xiD
xk0
xkD
..
.
xk1 xk
2∞
∞
∞
∞
∞
∞
∞ ∞
source
sink
.
Extendable to any convex cost
Convex function
![Page 118: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/118.jpg)
Graph Cut based Inference
Move making algorithms
• Original problem decomposed into a series of subproblems solvable with graph cut
• In each subproblem we find the optimal move from the current solution in a restricted search space
UpdatesolutionGraph Cut
BuildGraph
Encoding
Propose move
Initial solutionsolution
![Page 119: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/119.jpg)
Graph Cut based Inference
Example : [Boykov et al. 01]
α-expansion– each variable either keeps its old label or changes to α– all α-moves are iteratively performed till convergence
Transformation function :
α-swap– each variable taking label α or can change its label to α or – all α-moves are iteratively performed till convergence
![Page 120: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/120.jpg)
Graph Cut based Inference
Sufficient conditions for move making algorithms :(all possible moves are submodular)
α-swap : semi-metricity
Proof:
![Page 121: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/121.jpg)
Graph Cut based Inference
Sufficient conditions for move making algorithms :(all possible moves are submodular)
α-expansion : metricity
Proof:
![Page 122: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/122.jpg)
Object-class Segmentation
Data term Smoothness termData term
Smoothness term
where
Discriminatively trained classifier
![Page 123: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/123.jpg)
Object-class Segmentation
Original Image Initial solution
grass
![Page 124: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/124.jpg)
Object-class Segmentation
Original Image Initial solution Building expansion
grass
grass
building
![Page 125: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/125.jpg)
Object-class Segmentation
Original Image Initial solution Building expansion
Sky expansion
grass
grass
grass
building
building
sky
![Page 126: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/126.jpg)
Object-class Segmentation
Original Image Initial solution Building expansion
Sky expansion Tree expansion
grass
grass
grass grass
building
building building
sky sky
tree
![Page 127: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/127.jpg)
Object-class Segmentation
Original Image Initial solution Building expansion
Sky expansion Tree expansion Final Solution
grass
grass
grass grass grass
building
building buildingbuilding
sky sky sky
tree treeaeroplane
![Page 128: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/128.jpg)
Graph Cut based Inference
Range-expansion [Kumar, Veksler & Torr 1]– Expansion version of range swap moves– Each varibale can change its label to any label in a convex
range or keep its old label
Range-(swap) moves [Veksler 07]– Each variable in the convex range can change its label to any
other label in the convex range – all range-moves are iteratively performed till convergence
![Page 129: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/129.jpg)
Dense Stereo Reconstruction
Data term
Smoothness term
Same as before
Left Camera Image Right Camera Image Dense Stereo Result
![Page 130: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/130.jpg)
Dense Stereo Reconstruction
Data term
Smoothness term
Same as before
Left Camera Image Right Camera Image Dense Stereo Result
Convex range Truncation
![Page 131: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/131.jpg)
Graph Cut based Inference
Original Image Initial Solution
![Page 132: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/132.jpg)
Graph Cut based Inference
Original Image Initial Solution After 1st expansion
![Page 133: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/133.jpg)
Graph Cut based Inference
Original Image Initial Solution
After 2nd expansion
After 1st expansion
![Page 134: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/134.jpg)
Graph Cut based Inference
Original Image Initial Solution
After 2nd expansion
After 1st expansion
After 3rd expansion
![Page 135: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/135.jpg)
Graph Cut based Inference
Original Image Initial Solution
After 2nd expansion
After 1st expansion
After 3rd expansion Final solution
![Page 136: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/136.jpg)
Graph Cut based Inference
• Extendible to certain classes of binary higher order energies
• Higher order terms have to be transformable to the pairwise submodular energy with binary auxiliary variables z
C
Higher order term Pairwise term
UpdatesolutionGraph Cut
Transf. to pairwise subm.
Encoding
Propose move
Initial solutionsolution
![Page 137: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/137.jpg)
Graph Cut based Inference
• Extendible to certain classes of binary higher order energies
• Higher order terms have to be transformable to the pairwise submodular energy with binary auxiliary variables z
C
Higher order term Pairwise term
Example :
Kolmogorov ECCV06, Ramalingam et al. DAM12
![Page 138: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/138.jpg)
Graph Cut based Inference
Enforces label consistency of the clique as a weak constraint
Input image Pairwise CRF Higher order CRF
Kohli et al. CVPR07
![Page 139: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/139.jpg)
Graph Cut based Inference
UpdatesolutionGraph Cut
Transf. to pairwise subm.
Encoding
Propose move
Initial solution solution
If the energy is not submodular / graph-representable
– Overestimation by a submodular energy
– Quadratic Pseudo-Boolean Optimisation (QPBO)
![Page 140: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/140.jpg)
Graph Cut based Inference
UpdatesolutionGraph Cut
Overestim. by pairwise subm.
Encoding
Propose move
Initial solutionsolution
Overestimation by a submodular energy (convex / concave)
– We find an energy E’(t) s.t.
– it is tight in the current solution E’(t0) = E(t0)
– Overestimates E(t) for all t E’(t) ≥ E(t)
– We replace E(t) by E’(t)
– The moves are not optimal, but quaranteed to converge
– The tighter over-estimation, the better
Example :
![Page 141: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/141.jpg)
Graph Cut based Inference
Quadratic Pseudo-Boolean Optimisation– Each binary variable is encoded using two binary
variables xi and xi s.t. xi = 1 - xi _
_
![Page 142: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/142.jpg)
Graph Cut based Inference
Quadratic Pseudo-Boolean Optimisation
source
sink
xixj
xi xj
_
_
– Energy submodular in xi and xi
– Solved by dropping the constraint xi = 1 - xi
– If xi = 1 - xi the solution for xi is optimal
_
_
_
![Page 143: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/143.jpg)
Overview
• Motivation
• Min-Cut / Max-Flow (Graph Cut) Algorithm
• Markov and Conditional Random Fields
• Random Field Optimisation using Graph Cuts
• Submodular vs. Non-Submodular Problems
• Pairwise vs. Higher Order Problems
• 2-Label vs. Multi-Label Problems
• Recent Advances in Random Field Optimisation
• Conclusions
![Page 144: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/144.jpg)
Motivation
• To propose formulations enforcing certain structure properties of the output useful in computer vision
– Label Consistency in segments as a weak constraint
– Label agreement over different scales
– Label-set consistency
– Multiple domains consistency
• To propose feasible Graph-Cut based inference method
![Page 145: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/145.jpg)
Structures in CRF
• Taskar et al. 02 – associative potentials
• Woodford et al. 08 – planarity constraint
• Vicente et al. 08 – connectivity constraint
• Nowozin & Lampert 09 – connectivity constraint
• Roth & Black 09 – field of experts
• Woodford et al. 09 – marginal probability
• Delong et al. 10 – label occurrence costs
![Page 146: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/146.jpg)
Object-class Segmentation Problem
• Aims to assign a class label for each pixel of an image
• Classifier trained on the training set
• Evaluated on never seen test images
![Page 147: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/147.jpg)
Pairwise CRF models
Standard CRF Energy for Object Segmentation
Cannot encode global consistency of labels!!
Local context
Ladický, Russell, Kohli, Torr ECCV10
![Page 148: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/148.jpg)
• Thing – Thing • Stuff - Stuff • Stuff - Thing
[ Images from Rabinovich et al. 07 ]
Encoding Co-occurrence
Co-occurrence is a powerful cue [Heitz et al. '08][Rabinovich et al. ‘07]
Ladický, Russell, Kohli, Torr ECCV10
![Page 149: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/149.jpg)
• Thing – Thing • Stuff - Stuff • Stuff - Thing
Encoding Co-occurrence
Co-occurrence is a powerful cue [Heitz et al. '08][Rabinovich et al. ‘07]
Proposed solutions :
1. Csurka et al. 08 - Hard decision for label estimation2. Torralba et al. 03 - GIST based unary potential3. Rabinovich et al. 07 - Full-connected CRF
[ Images from Rabinovich et al. 07 ] Ladický, Russell, Kohli, Torr ECCV10
![Page 150: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/150.jpg)
So...
What properties should these global co-occurence potentials have ?
Ladický, Russell, Kohli, Torr ECCV10
![Page 151: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/151.jpg)
Desired properties
1. No hard decisions
Ladický, Russell, Kohli, Torr ECCV10
![Page 152: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/152.jpg)
Desired properties
1. No hard decisions
Incorporation in probabilistic framework
Unlikely possibilities are not completely ruled out
Ladický, Russell, Kohli, Torr ECCV10
![Page 153: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/153.jpg)
Desired properties
1. No hard decisions2. Invariance to region size
Ladický, Russell, Kohli, Torr ECCV10
![Page 154: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/154.jpg)
Desired properties
1. No hard decisions2. Invariance to region size
Cost for occurrence of {people, house, road etc .. } invariant to image area
Ladický, Russell, Kohli, Torr ECCV10
![Page 155: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/155.jpg)
Desired properties
1. No hard decisions2. Invariance to region size
The only possible solution :
Local context Global contextCost defined over the
assigned labels L(x)
L(x)={ , , }
Ladický, Russell, Kohli, Torr ECCV10
![Page 156: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/156.jpg)
Desired properties
1. No hard decisions2. Invariance to region size3. Parsimony – simple solutions preferred
L(x)={ building, tree, grass, sky }
L(x)={ aeroplane, tree, flower, building, boat, grass, sky }
Ladický, Russell, Kohli, Torr ECCV10
![Page 157: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/157.jpg)
Desired properties
1. No hard decisions2. Invariance to region size3. Parsimony – simple solutions preferred4. Efficiency
Ladický, Russell, Kohli, Torr ECCV10
![Page 158: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/158.jpg)
Desired properties
1. No hard decisions2. Invariance to region size3. Parsimony – simple solutions preferred4. Efficiency
a) Memory requirements as O(n) with the image size and number or labels
b) Inference tractable
Ladický, Russell, Kohli, Torr ECCV10
![Page 159: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/159.jpg)
• Torralba et al.(2003) – Gist-based unary potentials
• Rabinovich et al.(2007) - complete pairwise graphs
• Csurka et al.(2008) - hard estimation of labels present
Previous work
Ladický, Russell, Kohli, Torr ECCV10
![Page 160: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/160.jpg)
• Zhu & Yuille 1996 – MDL prior• Bleyer et al. 2010 – Surface Stereo MDL prior• Hoiem et al. 2007 – 3D Layout CRF MDL Prior
• Delong et al. 2010 – label occurence cost
Related work
C(x) = K |L(x)|
C(x) = ΣLKLδL(x)
Ladický, Russell, Kohli, Torr ECCV10
![Page 161: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/161.jpg)
• Zhu & Yuille 1996 – MDL prior• Bleyer et al. 2010 – Surface Stereo MDL prior• Hoiem et al. 2007 – 3D Layout CRF MDL Prior
• Delong et al. 2010 – label occurence cost
Related work
C(x) = K |L(x)|
C(x) = ΣLKLδL(x)
All special cases of our model
Ladický, Russell, Kohli, Torr ECCV10
![Page 162: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/162.jpg)
Inference for Co-occurence
Label indicator functions
Co-occurence representation
Ladický, Russell, Kohli, Torr ECCV10
![Page 163: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/163.jpg)
Inference for Co-occurence
Move Energy Cost of current label set
Ladický, Russell, Kohli, Torr ECCV10
![Page 164: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/164.jpg)
Inference for Co-occurence
Move Energy
Decomposition to α-dependent and α-independent part
α-independent α-dependent
Ladický, Russell, Kohli, Torr ECCV10
![Page 165: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/165.jpg)
Inference for Co-occurence
Move Energy
Decomposition to α-dependent and α-independent part
Either α or all labels in the image after the move
Ladický, Russell, Kohli, Torr ECCV10
![Page 166: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/166.jpg)
Inference for Co-occurence
Move Energy
submodular non-submodular
Ladický, Russell, Kohli, Torr ECCV10
![Page 167: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/167.jpg)
Inference for Co-occurence
Move Energy
non-submodular
Non-submodular energy overestimated by E'(t)• E'(t) = E(t) for current solution• E'(t) E(t) for any other labelling
Ladický, Russell, Kohli, Torr ECCV10
![Page 168: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/168.jpg)
Inference for Co-occurence
Move Energy
non-submodular
Non-submodular energy overestimated by E'(t)• E'(t) = E(t) for current solution• E'(t) E(t) for any other labelling
Occurrence - tight
Ladický, Russell, Kohli, Torr ECCV10
![Page 169: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/169.jpg)
Inference for Co-occurence
Move Energy
non-submodular
Non-submodular energy overestimated by E'(t)• E'(t) = E(t) for current solution• E'(t) E(t) for any other labelling
Co-occurrence overestimation
Ladický, Russell, Kohli, Torr ECCV10
![Page 170: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/170.jpg)
Inference for Co-occurence
Move Energy
non-submodular
Non-submodular energy overestimated by E'(t)• E'(t) = E(t) for current solution• E'(t) E(t) for any other labelling
General case [See the paper]
Ladický, Russell, Kohli, Torr ECCV10
![Page 171: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/171.jpg)
Inference for Co-occurence
Move Energy
non-submodular
Non-submodular energy overestimated by E'(t)• E'(t) = E(t) for current solution• E'(t) E(t) for any other labelling
Quadratic representation
Ladický, Russell, Kohli, Torr ECCV10
![Page 172: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/172.jpg)
Potential Training for Co-occurence
Ladický, Russell, Kohli, Torr ICCV09
Generatively trained Co-occurence potential
Approximated by 2nd order representation
![Page 173: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/173.jpg)
Results on MSRC data set
Input Image
Comparisons of results with and without co-occurence
Input ImageWithoutCo-occurence
WithCo-occurence
WithoutCo-occurence
WithCo-occurence
Ladický, Russell, Kohli, Torr ICCV09
![Page 174: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/174.jpg)
Pixel CRF Energy for Object-class Segmentation
Pairwise CRF models
Data term Smoothness term
Input Image Pairwise CRF Result
Shotton et al. ECCV06
![Page 175: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/175.jpg)
Pixel CRF Energy for Object-class Segmentation
Pairwise CRF models
Data term Smoothness term
• Lacks long range interactions
• Results oversmoothed
• Data term information may be insufficient
![Page 176: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/176.jpg)
Segment CRF Energy for Object-class Segmentation
Pairwise CRF models
Data term Smoothness term
Input Image UnsupervisedSegmentation
Shi, Malik PAMI2000,Comaniciu, Meer PAMI2002,
Felzenschwalb, Huttenlocher, IJCV2004,
![Page 177: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/177.jpg)
Segment CRF Energy for Object-class Segmentation
Pairwise CRF models
Data term Smoothness term
Input Image Pairwise CRF ResultBatra et al. CVPR08, Yang et al. CVPR07, Zitnick et al. CVPR08,
Rabinovich et al. ICCV07, Boix et al. CVPR10
![Page 178: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/178.jpg)
Segment CRF Energy for Object-class Segmentation
Pairwise CRF models
Data term Smoothness term
• Allows long range interactions
• Does not distinguish between instances of objects
• Cannot recover from incorrect segmentation
![Page 179: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/179.jpg)
Segment CRF Energy for Object-class Segmentation
Pairwise CRF models
Data term Smoothness term
• Allows long range interactions
• Does not distinguish between instances of objects
• Cannot recover from incorrect segmentation
![Page 180: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/180.jpg)
Segment CRF Energy for Object-class Segmentation
Robust PN approach
Pairwise CRF Segment consistency
term
Input ImageKohli, Ladický, Torr CVPR08
Higher order CRF
![Page 181: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/181.jpg)
Segment CRF Energy for Object-class Segmentation
Robust PN approach
Pairwise CRF Segment consistency
term
Kohli, Ladický, Torr CVPR08
![Page 182: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/182.jpg)
Segment CRF Energy for Object-class Segmentation
Robust PN approach
Pairwise CRF Segment consistency
term
Kohli, Ladický, Torr CVPR08
No dominant label
![Page 183: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/183.jpg)
Segment CRF Energy for Object-class Segmentation
Robust PN approach
Pairwise CRF Segment consistency
term
Kohli, Ladický, Torr CVPR08
Dominant label l
![Page 184: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/184.jpg)
Segment CRF Energy for Object-class Segmentation
Robust PN approach
Pairwise CRF Segment consistency
term
Kohli, Ladický, Torr CVPR08
• Enforces consistency as a weak constraint
• Can recover from incorrect segmentation
• Can combine multiple segmentations
• Does not use features at segment level
![Page 185: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/185.jpg)
Segment CRF Energy for Object-class Segmentation
Robust PN approach
Pairwise CRF Segment consistency
term
Kohli, Ladický, Torr CVPR08
Transformable to pairwise graph with auxiliary variable
![Page 186: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/186.jpg)
Segment CRF Energy for Object-class Segmentation
Robust PN approach
Pairwise CRF Segment consistency
term
Ladický, Russell, Kohli, Torr ICCV09
where
![Page 187: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/187.jpg)
Segment CRF Energy for Object-class Segmentation
Robust PN approach
Pairwise CRF Segment consistency
term
Input ImageLadický, Russell, Kohli, Torr ICCV09
Higher order CRF
![Page 188: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/188.jpg)
Associative Hierarchical CRFs
Ladický, Russell, Kohli, Torr ICCV09
Can be generalized to Hierarchical model
• Allows unary potentials for region variables
• Allows pairwise potentials for region variables
• Allows multiple layers and multiple hierarchies
![Page 189: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/189.jpg)
Associative Hierarchical CRFs
Ladický, Russell, Kohli, Torr ICCV09
AH CRF Energy for Object-class Segmentation
Pairwise CRF Higher order term
![Page 190: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/190.jpg)
Associative Hierarchical CRFs
Ladický, Russell, Kohli, Torr ICCV09
AH CRF Energy for Object-class Segmentation
Pairwise CRF Higher order term
Higher order term recursively defined as
Segment unary term
Segment pairwise term
Segment higher order term
![Page 191: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/191.jpg)
Associative Hierarchical CRFs
Ladický, Russell, Kohli, Torr ICCV09
AH CRF Energy for Object-class Segmentation
Pairwise CRF Higher order term
Higher order term recursively defined as
Segment unary term
Segment pairwise term
Segment higher order term
Why is this generalisation useful?
![Page 192: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/192.jpg)
Associative Hierarchical CRFs
Ladický, Russell, Kohli, Torr ICCV09
Let us analyze the case with
• one segmentation
• potentials only over segment level
![Page 193: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/193.jpg)
Associative Hierarchical CRFs
Ladický, Russell, Kohli, Torr ICCV09
Let us analyze the case with
• one segmentation
• potentials only over segment level
1) Minimum is segment-consistent
![Page 194: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/194.jpg)
Associative Hierarchical CRFs
Ladický, Russell, Kohli, Torr ICCV09
Let us analyze the case with
• one segmentation
• potentials only over segment level
1) Minimum is segment-consistent
2) Cost of every segment-consistent CRF equal to the cost of pairwise CRF over segments
![Page 195: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/195.jpg)
Associative Hierarchical CRFs
Ladický, Russell, Kohli, Torr ICCV09
Let us analyze the case with
• one segmentation
• potentials only over segment level
Equivalent to pairwise CRF over segments!
![Page 196: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/196.jpg)
Associative Hierarchical CRFs
Ladický, Russell, Kohli, Torr ICCV09
• Merges information over multiple scales
• Easy to train potentials and learn parameters
• Allows multiple segmentations and hierarchies
• Allows long range interactions
• Limited (?) to associative interlayer connections
![Page 197: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/197.jpg)
Inference for AH-CRF
Graph Cut based move making algorithms [Boykov et al. 01]α-expansion transformation function for base layer
Ladický (thesis) 2011
![Page 198: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/198.jpg)
Inference for AH-CRF
Graph Cut based move making algorithms [Boykov et al. 01]α-expansion transformation function for base layer
Ladický (thesis) 2011
α-expansion transformation function for auxiliary layer
(2 binary variables per node)
![Page 199: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/199.jpg)
Inference for AH-CRF
Ladický (thesis) 2011
Interlayer connection between the base layer and the first auxiliary layer:
![Page 200: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/200.jpg)
Inference for AH-CRF
Ladický (thesis) 2011
Interlayer connection between the base layer and the first auxiliary layer:
The move energy is:
where
![Page 201: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/201.jpg)
Inference for AH-CRF
Ladický (thesis) 2011
The move energy for interlayer connection between the base and auxiliary layer is:
Can be transformed to binary submodular pairwise function:
![Page 202: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/202.jpg)
Inference for AH-CRF
Ladický (thesis) 2011
The move energy for interlayer connection between auxiliary layers is:
Can be transformed to binary submodular pairwise function:
![Page 203: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/203.jpg)
Inference for AH-CRF
Ladický (thesis) 2011
Graph constructions
![Page 204: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/204.jpg)
Inference for AH-CRF
Ladický (thesis) 2011
Pairwise potential between auxiliary variables :
![Page 205: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/205.jpg)
Inference for AH-CRF
Ladický (thesis) 2011
Move energy if x = x : c d
Can be transformed to binary submodular pairwise function:
![Page 206: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/206.jpg)
Inference for AH-CRF
Ladický (thesis) 2011
Move energy if x ≠ x : c d
Can be transformed to binary submodular pairwise function:
![Page 207: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/207.jpg)
Inference for AH-CRF
Ladický (thesis) 2011
Graph constructions
![Page 208: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/208.jpg)
Potential training for Object class Segmentation
Pixel unary potential
Unary likelihoods based on spatial configuration (Shotton et al. ECCV06)
Classifier trained using boosting
Ladický, Russell, Kohli, Torr ICCV09
![Page 209: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/209.jpg)
Potential training for Object class Segmentation
Segment unary potential
Ladický, Russell, Kohli, Torr ICCV09
Classifier trained using boosting (resp. Kernel SVMs)
![Page 210: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/210.jpg)
Results on MSRC dataset
![Page 211: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/211.jpg)
Results on Corel dataset
![Page 212: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/212.jpg)
Results on VOC 2010 dataset
Input Image Result Input Image Result
![Page 213: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/213.jpg)
VOC 2010-Qualitative
Input Image Result Input Image Result
![Page 214: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/214.jpg)
CamVid-Qualitative
Our Result
Input Image
GroundTruth
Brostow et al.
![Page 215: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/215.jpg)
Quantitative comparison
MSRC dataset
Corel dataset
VOC2009 dataset
![Page 216: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/216.jpg)
Quantitative comparison
Ladický, Russell, Kohli, Torr ICCV09
MSRC dataset
VOC2009 dataset
![Page 217: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/217.jpg)
Dense Stereo Reconstruction
Unary Potential
Unary Cost dependent on the similarity of patches, e.g.cross correlation
Disparity = 0
![Page 218: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/218.jpg)
Dense Stereo Reconstruction
Unary Cost dependent on the similarity of patches, e.g.cross correlation
Unary Potential
Disparity = 5
![Page 219: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/219.jpg)
Dense Stereo Reconstruction
Unary Cost dependent on the similarity of patches, e.g.cross correlation
Unary Potential
Disparity = 10
![Page 220: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/220.jpg)
Dense Stereo Reconstruction
• Encourages label consistency in adjacent pixels
• Cost based on the distance of labels
Linear Truncated Quadratic Truncated
Pairwise Potential
![Page 221: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/221.jpg)
Dense Stereo Reconstruction
Does not work for Road Scenes !
Original Image Dense Stereo Reconstruction
![Page 222: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/222.jpg)
Dense Stereo Reconstruction
Does not work for Road Scenes !
Patches can be matched to any other patch for flat surfices
Different brightness in cameras
![Page 223: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/223.jpg)
Dense Stereo Reconstruction
Does not work for Road Scenes !
Patches can be matched to any other patch for flat surfices
Different brightness in cameras
Could object recognition for road scenes help?
Recognition of road scenes is relatively easy
![Page 224: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/224.jpg)
Joint Dense Stereo Reconstructionand Object class Segmentation
sky
• Object class and 3D location are mutually informative
• Sky always in infinity (disparity = 0)
![Page 225: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/225.jpg)
Joint Dense Stereo Reconstructionand Object class Segmentation
building carsky
• Object class and 3D location are mutually informative
• Sky always in infinity (disparity = 0)
• Cars, buses & pedestrians have their typical height
![Page 226: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/226.jpg)
Joint Dense Stereo Reconstructionand Object class Segmentation
building carsky
• Object class and 3D location are mutually informative
• Sky always in infinity (disparity = 0)
• Cars, buses & pedestrians have their typical height
• Road and pavement on the ground plane
road
![Page 227: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/227.jpg)
Joint Dense Stereo Reconstructionand Object class Segmentation
building carsky
• Object class and 3D location are mutually informative
• Sky always in infinity (disparity = 0)
• Cars, buses & pedestrians have their typical height
• Road and pavement on the ground plane
• Buildings and pavement on the sides
road
![Page 228: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/228.jpg)
Joint Dense Stereo Reconstructionand Object class Segmentation
building carsky
• Object class and 3D location are mutually informative
• Sky always in infinity (disparity = 0)
• Cars, buses & pedestrians have their typical height
• Road and pavement on the ground plane
• Buildings and pavement on the sides
• Both problems formulated as CRF
• Joint approach possible? road
![Page 229: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/229.jpg)
Joint Formulation
• Each pixels takes label zi = [ xi yi ] L1 L2
• Dependency of xi and yi encoded as a unary and pairwise potential, e.g.
• strong correlation between x = road, y = near ground plane
• strong correlation between x = sky, y = 0
• Correlation of edge in object class and disparity domain
![Page 230: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/230.jpg)
Joint Formulation
• Weighted sum of object class, depth and joint potential
• Joint unary potential based on histograms of height
Object layer
Disparity layer
Joint unary links
Unary Potential
![Page 231: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/231.jpg)
Joint Formulation
Pairwise Potential
• Object class and depth edges correlated
• Transitions in depth occur often at the object boundaries
Object layer
Disparity layer
Joint pairwise links
![Page 232: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/232.jpg)
Joint Formulation
![Page 233: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/233.jpg)
Inference
• Standard α-expansion• Each node in each expansion move keeps its old label
or takes a new label [xL1, yL2],
• Possible in case of metric pairwise potentials
![Page 234: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/234.jpg)
Inference
• Standard α-expansion• Each node in each expansion move keeps its old label
or takes a new label [xL1, yL2],
• Possible in case of metric pairwise potentials
Too many moves! ( |L1| |L2| ) Impractical !
![Page 235: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/235.jpg)
Inference
• Projected move for product label space• One / Some of the label components remain(s)
constant after the move
• Set of projected moves• α-expansion in the object class projection• Range-expansion in the depth projection
![Page 236: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/236.jpg)
Dataset
• Leuven Road Scene dataset• Contained
• 3 sequences• 643 pairs of images
• We labelled• 50 training + 20 test images• Object class (7 labels)• Disparity (100 labels)
• Publicly available• http://cms.brookes.ac.uk/research/visiongroup/files/Leuven.zip
Left camera Right camera Object GT Disparity GT
![Page 237: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/237.jpg)
Qualitative Results
• Large improvement for dense stereo estimation
• Minor improvement in object class segmentation
![Page 238: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/238.jpg)
Quantitative Results
Dependency of the ratio of correctly labelled pixels within the maximum allowed error delta
![Page 239: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/239.jpg)
• We proposed :• New models enforcing higher order structure
• Label-consistency in segments• Consistency between scale• Label-set consistency• Consistency between different domains
• Graph-Cut based inference methods• Source code http://cms.brookes.ac.uk/staff/PhilipTorr/ale.htm
There is more to come !
Summary
![Page 240: GraphCut -based Optimisation for Computer Vision](https://reader038.vdocument.in/reader038/viewer/2022103023/56812bc0550346895d90011a/html5/thumbnails/240.jpg)
Questions?
Thank you