Unsupervised Learning of Categories from Sets of Partially Matching Image Features

Kristen GraumanTrevor DarrellMIT

Spectrum of supervisionMoreLess

Costs of supervisionSacrifice scalability: practical limit on number of classes, number of training examples per classBiases possible:human labeling could hinder potentialperformance

Related workFeature/part selection given category [Weber et al. ECCV 2000, Fergus et al. CVPR 2003, Berg et al. CVPR 2005,]

Leveraging previously seen categories or images[Fei-Fei et al. ICCV 2003, Murphy et al. NIPS 2003, Holub et al. 2005,]

Unsupervised category learning with probabilistic Latent Semantic Analysis [Hofmann 1999][Fergus et al., Quelhas et al., Sivic et al., ICCV 2005]

GoalsAutomatically recover categories from an unlabeled collection of images, and form predictive classifiers to label new images

Tolerate clutter, occlusion, common transformationsAllow optional, variable amount of supervisionEfficiency

Sets of local features

Partially matching setsoptimal partial matching

Clustering with a partial matching

Clustering with a partial matching

Clustering with a partial matching

Clustering with a partial matching

Computing the partial matchingOptimal matchingGreedy matchingPyramid match for sets with features[Grauman and Darrell, ICCV 2005]

Review: Pyramid match

Review: Pyramid matchEfficient time for pyramids and matchingOrders of magnitude faster than optimal match in practiceAccurateProduces rankings that are highly correlated with optimal matchUseful as kernel in discriminative classifier: 50% accuracy on Caltech101 with 15 training examples per class (58% with 30)Bounded expected cost error[ICCV 2005, JMLR (to appear)]

Pyramid match graphBuild graph over image collection, with edges weighted by pyramid match similarity values

Optional semi-supervisionAdjust pyramid match graph when pair-wise constraints are available. should groupshould not group

Graph partitioningEfficiently identify initial clusters with spectral clustering and normalized cuts criterion of [Shi & Malik]

Limitation of partial match graph partitionBackground feature matches

Extracting correspondencesExtend pyramid match to return approximate feature correspondences

Extracting correspondencesExtend pyramid match to return approximate feature correspondences

Extracting correspondences

Limitation of partial match graph partitionBackground feature matches

Inferring feature maskscontribution to match

Inferring feature masks

Refining intra-cluster matches

Refining intra-cluster matchesweighted feature mask

Refining intra-cluster matches

Refining intra-cluster matches

Refining intra-cluster matchesweighted feature mask

Selecting category prototypes13245

Selecting category prototypes

Inferred feature masksHarris-Affine detector [Mikolajczyk and Schmid]SIFT descriptors [Lowe]

Unsupervised recovery of category prototypes40 runs with 400 randomly selected images

Top percentile of prototypesPrototype accuracy / categoryCaltech-4 data set

Semi-supervised category labelingRecover categories and SVM classifiers from 400 unlabeled imagesClassify 2788 unseen examples40 runs with random cluster/test set/supervision selections

Caltech-4 data setRecognition rate / classAmount of supervisory information(number of must-group pairs)

Recent work:Vocabulary-guided pyramid matchUniform binsExtracting correspondences can be slow, scores inaccurate in high dimensions with uniform binsA vocabulary-guided pyramid match tunes pyramid partitions to the feature distributionAccurate for d > 100

[See our recent CSAIL tech report]

ContributionsEfficient unsupervised / semi-supervised category learning from sets of local featuresAutomatic recovery of per-image feature masks without class labelsExtension to pyramid match for explicit correspondences

Future workEnforce geometry, contiguous spatial regions for matching feature maskExplore exemplar-based classifiersAutomatic selection of number of categoriesIterative cluster refinement / mask inferenceOptimizing semi-supervision with a user