Dog Breed Classification Using Part Localization

Jiongxin Liu1, Angjoo Kanazawa2, David Jacobs2, and Peter Belhumeur1

1 Columbia University 2 University of Maryland

Fine-grained classification[Nilsback and Zisserman ’08]

[Parkhi et al ’12]

[Kumar et al ‘12]

[Branson et al ‘10]

Related work• Dense feature extraction:– Mine discriminative region with random forests [Yao et al

’11]– Multiple Kernel Learning [Nilsback and Zisserman ’08]– Post-segmentation [Parkhi and Zisserman ’12]

• Pose-normalized appearance:– Birdlets [Farrell et al ’11]

Related work• Dense feature extraction:– Mine discriminative region with random forests [Yao et al

’11]– Multiple Kernel Learning [Nilsback and Zisserman ’08]– Post-Segmentation [Parkhi and Zisserman ’12]

• Pose-normalized appearance:– Birdlets [Farrell et al ’11]

Generic sampling of features contains more noise than useful

information for fine-grained classification!

Same breed or not?Entlebucher Mountain Dog Greater Swiss Mountain Dog

NO!!

Key insight: Differences in common parts are more informative

Entlebucher Mountain Dog Greater Swiss Mountain Dog

Localize parts based on a non-parameteric method by [Belhumeur et al ‘11]

“Columbia dogs with parts” dataset133 breeds, 8351 images

Low inter-breed variationNorfolk Terrier or Cairn Terrier?

High intra-breed variationBoth labrador retriever

Innumerable Poses

Diverse Appearances

Varying geometry of parts

Overview of the system1. Face Detection 2. Part Detection 3. Feature Extraction and ear localization

4. One vs All classification

Pipeline 1: Dog Face Detection

Keep the 5 highest scoring windows

Pipeline 2: Localize Parts

Idea: From the “fit” to K most similar exemplars weighted by the

detector output, take the most probable part

location

Detector responsesPart locations

Review: Consensus of Exemplars

Local Part Detectors Part LocalizationExemplar Selection

...

Slide from Neeraj Kumar

RANSAC-like Exemplar Selection1. Repeat r times:

a. Choose random exemplar kb. Choose 2 random modes of local detector outputs D={di} on queryc. Find similarity transform t that aligns exemplar to these pointsd. Evaluate match of all i face parts for this (k,t) pair:

e. Add (k,t) pair to list of possible exemplars, ranked by score

2. Take top M (k,t) pairs for determining global configuration

Part detector probabilityat this (aligned) location

Probability of thisconfiguration givendetector outputs

Slide from Neeraj Kumar

Final Part LocalizationFor each face part i:

a. Compute distribution of this part from all M aligned exemplarsb. For each of the top M aligned exemplars [(k,t) pairs]:

Multiply normalized local detector outputs with global distribution of part computed from exemplars to get scores at each pixel location

c. Add all scores together to get final scores at each pixel and choose max

Slide from Neeraj Kumar

Pipeline 2: Localize Parts

From K most similar exemplars and the detector output, take the most probable part location

Detector responses

Difference between current part location and that of exemplar

Part locations

Pipeline 3: Infer ears using detected parts

With r(=10) exemplars from each breed

Pipeline 3: Infer ears using detected parts

With r(=10) exemplars from each breed

Pipeline 4: Classification

Extract SIFT at part locations for each breed+color histogram one vs all linear SVM classifier

Qualitative Results: Successful

Qualitative Results: Failures

Results: ROC curves

Available in iTunes now

Take a Picture

By tapping the nose

Get the breed!

Browse Dog Breeds

Thank you!!