learning jigsaws for clustering appearance and shape john winn, anitha kannan and carsten rother...
TRANSCRIPT
Learning Jigsawsfor clustering appearance and shape
John Winn, Anitha Kannan and Carsten Rother
NIPS 2006
Learning jigsawsAim: Cluster regions in images with similar appearance and shape.
Examples of clusters (jigsaw pieces)
EyeNoses Cheek Eyebrows
Road map
Clustering image patches
The Jigsaw model
Results on toy and real images
Learning jigsaw pieces
Discussion and conclusions
Clustering image patches
Patches
Clusters
[Leibe & Schiele, BMVC 2003]
Clustering image patches
Cluster?
Patch wrong shape
Clustering image patches
Cluster?
Patch wrong shape
Clustering image patches
Cluster?
Part is occluded
Clustering image patches
Cluster?
Need to adapt the patch shape depending on the image.
Road map
Clustering image patches
The Jigsaw model
Results on toy and real images
Learning jigsaw pieces
Discussion and conclusions
Aims of jigsaw model
Learn clusters (jigsaw pieces) so that:
1. Clustered patches have similar shape and appearance
2. Patches are as large as possible
3. Every image pixel belongs to exactly one patch (i.e. the images are segmented into patches)
The Jigsaw model
Jigsaw J
Image I1
...Image I2 Image INOffset map L2 Offset map LNOffset map L1
Region of constant offset
The Jigsaw model
Jigsaw J
Offset map prior (Potts model)
Appearance model
JigsawMean μ(z) and inverse variance λ(z) for each jigsaw pixel z.
Image I Offset map L
offset at pixel i
cost of patch boundary
Road map
Clustering image patches
The Jigsaw model
Results on toy and real images
Learning jigsaw pieces
Discussion and conclusions
Toy example
Learned by iteratively maximising joint probability w.r.t. jigsaw and offset maps
(see paper for details)
Image with segmentation Jigsaw
Mean Variance
Comparison: Mixture of Gaussians
fixed patch shape
Cluster centres
Comparison: Epitome
[Jojic et al., ICCV 2003]
fixed patch shape translation invariant
Epitome
Comparison: Jigsaw
learned patch shape translation invariant non-overlapping patches
Jigsaw
Comparison: all methodsOriginal
JigsawEpitome
Error = 0.054Error = 0.071
MoG
Error = 0.103
Faces example
Source: Olivetti face database
Face images with segmentations Jigsaw
128128 mean
Road map
Clustering image patches
The Jigsaw model
Results on toy and real images
Learning jigsaw pieces
Discussion and conclusions
Learning the jigsaw pieces
Jigsaw J
...Image I1 Image I2 Image INOffset map L2 Offset map LNOffset map L1
Learning the jigsaw pieces
Jigsaw J
...Image I1 Image I2 Image INOffset map L2 Offset map LNOffset map L1
Learning the jigsaw pieces
Jigsaw J
...Image I1 Image I2 Image INOffset map L2 Offset map LNOffset map L1
Shape clustering on faces
Jigsaw showing piecesCommonly used pieces
Road map
Clustering image patches
The Jigsaw model
Results on toy and real images
Learning jigsaw pieces
Discussion and conclusions
Jigsaw applications
Can be used as ‘plug-and-play’ replacement for fixed-shape patch model in existing systems.
Applications include: Object recognition/detection Object segmentation Stereo matching Texture synthesis Super-resolution Motion segmentation Image/video compression
Future work
Allow rotation/scaling/deformationof the patches.
Incorporate shape clustering into the probabilistic model
Incorporate additional invariances e.g. to illumination
Apply to other domains: audio, biology
Conclusions
Jigsaw model allows learning the shape and appearance of recurring regions in images.
Jigsaw performs unsupervised discovery of object parts.
Thank you
Jigsaw paper (compressed)
http://johnwinn.org
Comparison: Epitome
[Jojic et al., ICCV 2003]
fixed patch shape translation invariant overlapping patches
Epitome
Patch averaging
Error = 0.071 Error = 0.054
EpitomeMoG