robust spectral 3d-bodypart segmentation along time fabio cuzzolin, diana mateus, edmond boyer, radu...
TRANSCRIPT
Robust spectral 3D-bodypart segmentation
along time
Fabio Cuzzolin, Diana Mateus, Edmond Boyer, Radu Horaud
Perception project meeting24/4/2007
Submitted to ICCV’07
Robust spectral segmentation
Consistent bodypart segmentation in sequences Why clustering in the embedding space K-wise clustering Branch detection Seed propagation Merging-splitting clusters Algorithm Results Influence of d and K Topology changes Comparison with EM clustering Comparison with ISOMAP clustering
Problem
Segmenting bodyparts of moving articulated bodies along sequences, in a consistent way in an unsupervised fashion robustly, with respect to changes of
the topology of the moving body as a bulding block of a wider motion
analysis and capture framework
Clustering in the embedding space
Locally Linear Embedding: preserves the local Locally Linear Embedding: preserves the local structure of the datasetstructure of the dataset generates a lower-dim embedded cloud less sensitive to topology changes than other methods shape of the embedded cloud fairly stable under AND less computationally expensive then ISOMAP
40 50 60 70
60
8010
20
30
40
50
60
70
80
3D shape
0.05
0
0.05
0.060.04
0.020
0.02
0.05
0
0.05
0.1
LLE space
50
40
30
20
10
010
20
30
40
40
30
20
10
0
10
20
30
40
50
ISOMAP space
Pose invariance with LLE To ensure consistent To ensure consistent
segmentation the stability of the segmentation the stability of the embedded cloud is necessaryembedded cloud is necessary
rigid part
rigid part
moving joint area
unaffected neighborhoods
unaffected neighborhoods
affected neighborhoods
010
20
0
10
0
5
10
15
20
25
30
35
40
45
010
2030
05
10
5
10
15
20
25
30
35
40
45
50
0.080.06
0.040.02
00.02
0. 2
0. 1
0
0.1
0.2
0.15
0. 1
0.05
0
0.05
0.1
0.06 0.04 0.02 0 0.02 0.04 0.06 0.08
0. 2
0. 1
0
0.1
0.2
0. 1
0.05
0
0.05
0.1
0.15 LLE works with local LLE works with local neighborhoods -> neighborhoods -> stable under stable under articulated motionarticulated motion
Algorithm
0.060.0200.040
0.05
0.15
0. 1
0.05
0
0.05
0
10
20
0
10
20
0
5
10
15
20
25
30
35
40
45
50
Original dataset
0
10
20
0
10
20
0
5
10
15
20
25
30
35
40
45
50
Clustering in original spaceBranch terminations
Segmentation in the embedding space
a) b) c)
Pictorial illustration of the overall algorithm
K-wise clustering LLE maps the 3D shape to a lower-dimensional LLE maps the 3D shape to a lower-dimensional
shapeshape Idea: clustering collinear points togetherIdea: clustering collinear points together
K-wise clustering:K-wise clustering:
a hypergraph H is built by measuring the affinity of all triads a weighted graph G which approximates H is constructed by constrained
linear least square optimization the approximating graph is partitioned by spectral clustering (n-cut)
Branch detection and number of clusters
Branches can be detected easilyBranches can be detected easily
Branch termination not detected Branch termination detected
An embedded point is a termination if its projection on the line interpolating its An embedded point is a termination if its projection on the line interpolating its neighborhood is an extremumneighborhood is an extremum
Seed propagation along time
To ensure time To ensure time consistency clusters’ consistency clusters’ seeds have to be seeds have to be propagated along propagated along timetime
Old positions of Old positions of clusters in 3D are clusters in 3D are added to new cloud added to new cloud and embeddedand embedded
Result: new seedsResult: new seeds
7080
9050
6070
80
10
20
30
40
50
60
70
80
90X(t)
7580
8590
50
60
70
80
10
20
30
40
50
60
70
80
90X(t+1)
c1(t)
LLE(t) LLE(t+1)
Y(t) Y(t+1)
xi4(t)
xi1(t)
xi2(t)
xi3(t)
c2(t)
c3(t)
c4(t) c'2(t)
c'1(t)
c'3(t)
c'4(t)
Merging/splitting clusters
1. At each t all branch terminations of Y(t) are detected;
2. if t=0 they are used as seeds for k-wise clustering;
3. otherwise (t>0) standard k-means is performed on Y(t) using branch terminations as seeds, yielding a rough partition of the embedded cloud into distinct branches;
4. propagated seeds in the same partition are merged;
5. for each partition of Y(t) not containing any old seed a new seed is defined as the related branch termination.
Results - 1
40 50 60 70 80 90
40
50
60
70
80
10
20
30
40
50
60
70
80
90
Clustering in original space
5060
7080
50
60
70
20
30
40
50
60
70
80
90
Cluster centroids trajectories
Results - 2
0 5 10 15 20 25 30
0
10
20
0
5
10
15
20
25
30
35
40
45
50
Clustering in original space
510
1520
25
5
10
15
20
25
5
10
15
20
25
30
35
40
45
Cluster centroids trajectories
Estimating k
The number of neighbors can be estimated from the data sequenceThe number of neighbors can be estimated from the data sequence Admissible k: yields neighborhoods which do not span different
bodyparts
0 20 40 60 80 1000
1
2
3
4
5
6
7
05
1015
20
05
1015
0
5
10
15
20
25
30
35
40
45
50
0
10
20
0
10
0
5
10
15
20
25
30
35
40
45
50
0 5 10 15 20 25 300
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Influence of dimension
Choosing a larger dimension for the embedding space Choosing a larger dimension for the embedding space improves the resolution of the segmentationimproves the resolution of the segmentation
010
205
1015
5
10
15
20
25
30
35
40
45
50
3D Segmentation
0.04 0.02
00.02
0.040.06
0. 2
0. 1
0
0.1
0.20. 2
0. 1
0
0.1
0.2
Clustering in embedding space
d=3 0510152025
510
15
5
10
15
20
25
30
35
40
45
50
0.060.04
0.020
0.020.04
0. 2
0. 1
0
0.1
0.20. 2
0.15
0. 1
0.05
0
0.05
0.1
0.15
0.060.04
0.020
0.020.04
0. 2
0. 1
0
0.1
0.20. 1
0.05
0
0.05
0.1
0.15
0.060.04
0.020
0.020.04
0. 2
0. 1
0
0.1
0.20. 1
0.05
0
0.05
0.1
0.15
0. 20.15
0. 10.05
00.05
0.1
0. 2
0. 1
0
0.1
0.20. 1
0.05
0
0.05
0.1
0.15
d=4
Change of topology When topology changes, clusters merge or When topology changes, clusters merge or
split to adaptsplit to adapt
05101520
0
5
10
15
0
5
10
15
20
25
30
35
40
45
danceuse, t = 16
05
1015
20
0
10
5
10
15
20
25
30
35
40
45
50
danceuse, t = 18
05
1015
20
0
5
10
15
0
5
10
15
20
25
30
35
40
45
50
danceuse, t = 20
0.06 0.04 0.02 0 0.020.02
0
0.02
0.04
0.06
0.08
0.1
0.12
0.15
0. 1
0.05
0
0.020
0.02
0.02
0
0.02
0.04
0.06
0.02
0
0.02
0.04
0.06
0.08
0.1
0.12
0.04 0.02 0 0.02 0.04
0.12
0. 1
0.08
0.06
0.04
0.02
0
0.02
0.02
0
0.02
0.04
0.06
0.08
0.1
0.12
Performance of EM clustering EM clustering fits a multi-Gaussian distribution EM clustering fits a multi-Gaussian distribution
to the data through the EM algorithmto the data through the EM algorithm
0 5 10 15 20 25 30
2468101214
5
10
15
20
25
30
35
40
45
50
05
1015
20
5
10
15
20
25
5
10
15
20
25
30
35
40
45
50
1015
2025
5
10
15
20
25
5
10
15
20
25
30
35
40
45
Cluster centroids trajectoriest = 40 t = 55
Performance of ISOMAP
The same The same propagation propagation scheme can scheme can be applied in be applied in the ISOMAP the ISOMAP spacespace
extremely extremely sensitivesensitive to to topology topology changeschanges
0
10
20
05
1015
5
10
15
20
25
30
35
40
45
20 15 10 5 0 5 10 15 20 25
25
20
15
10
5
0
5
10
05
1015
20
0
10
5
10
15
20
25
30
35
40
45
25 20 15 10 5 0 5 10 15 20 25
25
20
15
10
5
0
5
10
15
0
10
20
0
5
10
15
5
10
15
20
25
30
35
40
45
50
25 20 15 10 5 0 5 10 15 20 2525
20
15
10
5
0
5
10
15
Segmentation in original space
Clustering in ISOMAP space
t = 16 t = 18
Conclusions Unsupervised bodypart segmentation algorithm Unsupervised bodypart segmentation algorithm
which ensure consistency along timewhich ensure consistency along time Spectral method: clustering is performed in the
embedding space (in particular after LLE) as shape becomes lower-dim and different bodyparts are widely separated
Seeds are propagated along time and merged/splitted according to topology variations
Compares favorably with other techniques First step of motion analysis (matching, action
recognition, etc.)