![Page 1: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/1.jpg)
Jianxiong XIAO, Jingdong WANG,
Ping TAN, Long QUAN
Department of Computer Science & Engineering
The Hong Kong University of Science & Technology
Joint Affinity Propagation
for
Multiple View Segmentation
ICCV 2007Eleventh IEEE International Conference on Computer Vision
Rio de Janeiro, Brazil, October 14-20, 2007
![Page 2: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/2.jpg)
2
Outline
Part 1: Introduction
Part 2: Our Approach
– Formulation
– Optimization:
• Hierarchical Sparse Affinity Propagation
• Semi-supervised Contraction
Part 3: Experiment Results
Part 4: Conclusion
![Page 3: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/3.jpg)
3
Outline
Part 1: Introduction
Part 2: Our Approach
– Formulation
– Optimization:
• Hierarchical Sparse Affinity Propagation
• Semi-supervised Contraction
Part 3: Experiment Results
Part 4: Conclusion
![Page 4: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/4.jpg)
4
• Get 3D points and camera positions from 2D
images (geometry computation)
• Get 3D objects from unstructured 3D points
(objects reconstruction)
recovered 3D points recovered object modelsinput images
Image-based modeling
Two Steps Methods:
![Page 5: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/5.jpg)
5
Structure from motion
![Page 6: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/6.jpg)
6
Data segmentation
• Pure 2D segmentation & 3D clustering is hard!
– J. Shi and J. Malik. Normalized Cuts and Image Segmentation
– etc.
• Multiple view joint segmentation
– Simultaneously segment 3D points and 2D images
– Jointly utilize both 2D and 3D information
2D?
3D?
![Page 7: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/7.jpg)
7
Our work
• Explore for multiple view joint segmentation by simultaneously utilizing 2D and 3D data.
• The availability of both 2D and 3D data can bring complementary information for segmentation.
• Propose two practical algorithms for joint segmentation:
– Hierarchical Sparse Affinity Propagation
– Semi-supervised Contraction
![Page 8: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/8.jpg)
8
Outline
Part 1: Introduction
Part 2: Our Approach
– Formulation
– Optimization:
• Hierarchical Sparse Affinity Propagation
• Semi-supervised Contraction
Part 3: Experiment Results
Part 4: Conclusion
![Page 9: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/9.jpg)
9
Outline
Part 1: Introduction
Part 2: Our Approach
– Formulation
– Optimization:
• Hierarchical Sparse Affinity Propagation
• Semi-supervised Contraction
Part 3: Experiment Results
Part 4: Conclusion
![Page 10: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/10.jpg)
10
Problem formulation
iII The set of images
The set of regions
A joint point
A set of labels
Set of visibilities
Set of joint points
kki PI ,u
nn PPzyx ,,,,,,, 11 uux
klL
jV v
jX x
We now want to get the inference of L, given X,
V and I.
![Page 11: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/11.jpg)
11
graph model
Graph based segmentation
Graph G = { V, E }:
V: 3D points recovered from SFM
E: each point connected to its K-nearest neighbors, and two end points of each edge both visible at least in one view
![Page 12: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/12.jpg)
12
Joint similarity
• 3D coordinates
• 3D normal
• Color
• Contour
• Patch
jisjis
jisjisjis
tic
c
,,
,,, 3
![Page 13: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/13.jpg)
13
3D similarity
jisjisjis
jis
jis
nd
n
ji
n
d
ji
d
,,,
2,
2,
333
2
3
2
3
2
3
2
3
nn
pp ip jp
in jn
![Page 14: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/14.jpg)
14
2D color similarity
2
2
2,
c
ji
c
EEjis
cc
2
,
2
maxmed,
ic
vvjitv
ic
tgjis vv
.p
.q
p q
d2d(p,q)
= gradient of i-th image ig
![Page 15: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/15.jpg)
15
Utilizing the texture information
• Hyper Graph?
• Higher Order Prior Smoothness?
• …
![Page 16: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/16.jpg)
16
Competitive region growing
• Associate patches with each 3D point.
![Page 17: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/17.jpg)
17
Patch filtering
• A small error around the object boundary may result in a large color difference.
![Page 18: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/18.jpg)
18
Patch histogram similarity
jki
k
t
k
ji
t hhdt
hhdjis ,1
,,1
0
For each joint point
• Collect all its patches
• Build an average color histogram
• Down-sample the patches t-1 times
• A vector of histograms 10 ,, thhh
nP
0h
where d (·, ·) is the dissimilarity measures for histograms.
![Page 19: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/19.jpg)
19
Learning
• The concept of segmentation is obviously subjective.
• Hence, some user assistant information will greatly improve the segmentation.
![Page 20: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/20.jpg)
20
Handle the ambiguity
• To improve robustness and handle the ambiguity of the projections near the boundary
![Page 21: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/21.jpg)
21
Outline
Part 1: Introduction
Part 2: Our Approach
– Formulation
– Optimization:
• Hierarchical Sparse Affinity Propagation
• Semi-supervised Contraction
Part 3: Experiment Results
Part 4: Conclusion
![Page 22: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/22.jpg)
22
Affinity propagation [Frey & Dueck 2007]
• find several exemplars such that the sum of the similarities between the data points and the corresponding exemplars is maximized.
• i.e. searching over valid configurations of the labels so as to minimize the energy
• i.e. maximizing the net similarity
Ncc ,,1 c
N
i
icisE1
,c
N
k
kES1
ccc
![Page 23: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/23.jpg)
23
Responsibility
• The responsibility sent from data point to candidate exemplar point , reflects the accumulated evidence for how well-suited point is to serve as the exemplar for point , taking into account other potential exemplars for point .
kir , ik
i
i
Responsibility
i k
k
![Page 24: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/24.jpg)
24
Availability
• The availability , sent from the candidate exemplar point to point , reflects the accumulated evidence for how appropriate it would be for point to choose point as its exemplar, taking into account the support from other points that point should be an exemplar.
kia ,ik
i k
ik
Availability
k
![Page 25: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/25.jpg)
25
Responsibility & Availability
Responsibility
i k
Availability
kii
kk
kirkkrkia
kiskiakiskir
,'
'
,',0max,,0min,
',',max,,
![Page 26: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/26.jpg)
26
Outline
Part 1: Introduction
Part 2: Our Approach
– Formulation
– Optimization:
• Hierarchical Sparse Affinity Propagation
• Semi-supervised Contraction
Part 3: Experiment Results
Part 4: Conclusion
![Page 27: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/27.jpg)
27
Sparse affinity propagation
• Affinity propagation on a sparse graph, called sparse affinity propagation, is more efficient as pointed in [Brendan Frey, Delbert Dueck 2007].
• Then sparse affinity propagation runs in O(T|E|) time with T the number of the iterations and |E| the number of the edges.
• Here, the time complexity is O(Tn) since |E| = O(n).
![Page 28: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/28.jpg)
28
Original sparse AP
• The number of the data points that have the same exemplar i is at most degree(i), where degree(i) is the number of nodes connecting i.
This will result in
unexpectedly too
many fragments.
![Page 29: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/29.jpg)
29
Hierarchical sparse AP
G’=G(V,E);
while (true)
{
[Exemplars, Label] = Sparse Affinity Propagation (G’);
G’= (V’=Exemplars, E’);
if ( Satisfy Stopping Condition ) break;
}
ji
ji cqExemplarcpExemplar
EqpVqpccE
)(,)(
,',,',,'
![Page 30: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/30.jpg)
30
Hierarchical sparse AP
L=1L=2 L=5 L=8
L=14 L=17L=11
![Page 31: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/31.jpg)
31
Outline
Part 1: Introduction
Part 2: Our Approach
– Formulation
– Optimization:
• Hierarchical Sparse Affinity Propagation
• Semi-supervised Contraction
Part 3: Experiment Results
Part 4: Conclusion
![Page 32: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/32.jpg)
32
Semi-supervised contraction
0,, qqspps
![Page 33: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/33.jpg)
33
Semi-supervised contraction
![Page 34: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/34.jpg)
34
Semi-supervised contraction
![Page 35: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/35.jpg)
35
Semi-supervised contraction
• Finally, when the algorithm converged, availabilities and responsibilities are combined to identify exemplars.
• For point , its corresponding label is obtained as
kirkiak
qpk,,max arg
,
*
i
![Page 36: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/36.jpg)
36
Semi-supervised contraction
![Page 37: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/37.jpg)
37
Outline
Part 1: Introduction
Part 2: Our Approach
– Formulation
– Optimization:
• Hierarchical Sparse Affinity Propagation
• Semi-supervised Contraction
Part 3: Experiment Results
Part 4: Conclusion
![Page 38: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/38.jpg)
38
Results
![Page 39: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/39.jpg)
39
Results
![Page 40: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/40.jpg)
40
Results
![Page 41: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/41.jpg)
41
Outline
Part 1: Introduction
Part 2: Our Approach
– Formulation
– Optimization:
• Hierarchical Sparse Affinity Propagation
• Semi-supervised Contraction
Part 3: Experiment Results
Part 4: Conclusion
![Page 42: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/42.jpg)
42
Conclusion
![Page 43: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/43.jpg)
43
Thank you!
Questions?
Contact: Jianxiong XIAO [email protected]
ICCV 2007Eleventh IEEE International Conference on Computer Vision
Rio de Janeiro, Brazil, October 14-20, 2007
Joint Affinity Propagation for Multiple View Segmentation
![Page 44: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/44.jpg)
44
2D color similarity
• Contour based similarity
![Page 45: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/45.jpg)
45
Time complexity
• Compared with the spectral clustering approach in [Quan 2007], the hierarchical sparse affinity propagation is more efficient, running in O(TLn) with T the number of the iterations and L the number of the hierarchies, and more effective.
![Page 46: Joint Affinity Propagation for Multiple View Segmentationvision.princeton.edu/projects/2007/ICCV/presentation.pdf · Original sparse AP • The number of the data points that have](https://reader033.vdocument.in/reader033/viewer/2022051921/600e857d2eadb37bf0767524/html5/thumbnails/46.jpg)
46
Segmentation process pipeline
Automatic segmentation Assisted
segmentation