The Layout Consistent Random Field for Recognizing and Segmenting

Partially Occluded Objects

By John Winn & Jamie ShottonCVPR 2006

presented by Tomasz Malisiewicz for CMU’s Misc-Read

April 26, 2006

Talk Overview

ObjectiveCRF HRF LayoutCRFLayoutCRF PotentialsLearningInferenceResultsSummary

LayoutCRF Objectives

To detect and segment partially occluded objects of a known category

To detect multiple object instances which possibly occlude each other

To define a part labeling which densely covers the object of interest

To model various types of occlusions (FG/BG, BG/FG, FG/FG)

Conditional Random Field (Lafferty ‘01)

A random field globally conditioned on the observation X

Discriminative framework where we model P(Y|X) and do not explicitly model the marginal P(X)

Hidden Random Field (Szumer ‘05)

Extension to CRF with hidden layer of variables

The hidden variables represent object ‘parts’ in this work

Deterministic MappingDeterministic Mapping

LayoutCRF

An HRF with asymmetric pair-wise potentials, extended with a set of discrete valued instance transformations {T1,…,TM}

M foreground object instances

LayoutCRF

*only one non-background class is considered at a time

M+1 instance labels: yi \in {0,1,…,M}

Each object instance has a separate set of H part labels hi \in {0,1,…,H x M}

LayoutCRF

Each transformation T represents the translation and left/right flip of an object instance by indexing all possible integer pixel translations for each flip orientation

Each T is linked to every hi

LayoutCRF Potentials

Unary Potentials: Use local information to infer part labels (randomized decision trees)

Asymmetric Pair-wise Potentials: Measure local part compatibilities

Instance Potentials: Encourage correct long-range spatial layout of parts for each object instance

LayoutCRF Potentials: Unary

A set of decision trees; each trained on a random subset of the data (improves generalization and efficiency)

Each DT returns a distribution over part labels; K DTs are averaged

Each non-terminal node in the DT evaluates an intensity difference or absolute intensity difference between a learned pair of pixels and compares this to a learned threshold

Window of sizeD around pixel i

Layout Consistency (for pair-wise potentials)

Neighboring pixels whose labels are not layout consistent are not part of thesame object

Colors represent part labels

A label is layout-consistent with itself, and with those labels that are adjacentin the grid ordering above

Distinguished Transitions

1. Background: hi and hj are BG labels2. Consistent FG: hi and hj are layout-consistent FG labels3. Object edge: one label is BG, the other is part label lying on object edge4. Class occlusion: one label is interior FG label, the other is a BG label5. Instance occlusion: both are FG labels, but not layout-consistent (at least one label is object edge)6. Inconsistent Interior FG: both labels are interior FG labels, but not layout-consistent (rare)

LayoutCRF Potentials: Pair-wise

The value of the pair-wise potential varies according to the transition type

eij is image-based edge cost which encourages object edges to align with image boundaries

Contrast term estimated for each image

LayoutCRF Potentials: Instance

Look-up tables (histograms)

Encourage the correct spatial layout of parts for each object instance by gravitating parts towards their expected positions, given transformation of the instance

Weighs strength of potential

Returns position i inverse-transformedby the transformation Tm

LayoutCRF: What comes next?

We just defined the LayoutCRF and its potentials

First we need to learn the parameters of the LayoutCRF from labeled training data

Then we apply the model to a new image (inference) to obtain a detection and segmentation

Learning (the model parameters)

Supervised Algorithm requires foreground / background segmentation, but not part labels

Unary Potential and Part Labeling

Part labeling for the training images is initialized based on a dense regular grid that fits the object bounding box

Unary classifiers are learned, then new labeling is inferred

*Two iterations are sufficient

Dense grid is spatially quantized such that a unique part covers several pixels (on average 8x8)

Learning Pair-wise Potentials

Parameters are learned via cross-validation by a search over a sensible range of positive values

Gradient-based ML learning too slow; (future work: more efficient means of learning these parameters)

Learning Instance Potentials

Deformed part labelings of all training images are aligned on their segmentation mask centroids

A bounding box is placed relative to the centroid around the part labelings

For each pixel within the bounding box, the distribution over part labels is learned by histogramming the deformed training image labels

Empirical Distribution over parts h given position w

Inference (on a novel image)

Initially, we don’t know the number of object instances and their locations

Step1: collapse part labels across instances, merge instance labels together, and remove transformations. MAP inference is performed to obtain part labeling image h*

Inference (on a novel image)

Step2: determine number of layout-consistent regions in h* using connected component analysis; pixels are connected if they are layout-consistent

This gives us an estimate of M (number of object instances) and initial instance labeling

estimate T separately for each instance label

Inference (on a novel image)

Step3: re-run MAP inference with full model to get full h, which now distinguishes between instances

Approximate MAP inference via Annealed Expansion Move Algorithm

Alternating regular grid expansions at random offset and standard alpha expansions (for changing to BG label)

Annealing schedule weakens pair-wise potential during early stages by raising to a power less than one

Results on Cars

*Training on imagesthat contain onlyone visible carinstance

False Positive

Segmentation Accuracy on Cars

Evaluated segmentation accuracy on 20 randomly chosen images of cars, containing 34 car instances

Segmentation Accuracy per instance: ratio of intersection to the union of the detected and ground-truth segmentations = .67

Results on Faces

Multi-class LayoutCRF (Future Work)

Summary

LayoutCRF used to detect multiple instances of an object of a given class

Deformed-grid part labeling densely covers the object

Simultaneous detection and segmentation

Questions?

References

J. Lafferty, A. McCallum, and F. Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In International Conference on Machine Learning, 2001.

M. Szummer. Learning diagram parts with hidden random fields. In International Conference on Document Analysis and Recognition, 2005.

J. Winn and J. Shotton. The Layout Consistent Random Field for Recognizing and Segmenting Partially Occluded Objects. In CVPR 2006.