Foreground ModelingThe Shape of Things that

Came

Nathan JacobsAdvisor: Robert Pless

Computer ScienceWashington University in St. Louis

Nathan Jacobs 2

Visual Surveillance

• Observe people and vehicles– Where are they?– Where have they been?– Where are they going?

• Answering these questions requires object tracking.

Nathan Jacobs 3

Probabilistic Tracking

• Tracking is commonly cast in a Bayesian framework to estimate object shape and location– Initial estimate = combination of image data likelihood and

initialization prior– Updated estimate = combination of image data likelihood and

state prediction prior

• Likelihood functions are the focus of most tracking work– Color histograms, templates

• Our focus is on the prior terms

Nathan Jacobs 4

Quotes from yesterday

• “Initialization of tracking is important but not addressed here.”

• “Our object model assumes a well calibrated camera and a flat-ground plane.”

• “The prior term is a tricky thing to design.”

Nathan Jacobs 5

Passive Vision : The Big Picture

• Learn strong scene-specific priors by watching the same scene for a long time– Made easier because the cameras are static– Should be learned online

• Priors can be used to improve anomaly detection and tracking algorithms

Nathan Jacobs 6

Scene-specific Motion Priors

Nathan Jacobs 7

Unusual Traffic Motion

Video segment with anomalous motion (an ambulance using the median to pass stopped cars).

False color sequence highlighting anomalous motions.

Nathan Jacobs 8

Online Prior Learning for Tracking

Online learning and use of motion priors:

• reduces the number of particles needed

• increases the number of objects that can be tracked. Frames

Obj

ects

Nathan Jacobs 9

What else can we model?

• Watching for a long time allows us to build models of– Pixel intensity– Image derivatives– Image motion patterns

• We now transition to features based on the shape of foreground objects

Nathan Jacobs 10

An Example Video

Nathan Jacobs 11

Generating Examples Shapes

Current FrameForeground Mask Shape Descriptor

Background Image

For a long time:

1. Detect foreground objects

2. Generate a shape descriptor for each object

3. Add shape descriptor to training set

Nathan Jacobs 12

Shape Descriptor

• Currently using a simple shape model– A 20-dimensional feature vector– Each dimension is the distance from center to edge

of object

• Other shape models are possible

Nathan Jacobs 13

Two Shape Model Types

• Both models are PCA subspaces

• Global model– Subspace is location independent – Distribution estimate is location dependent

• Local model– Subspace and distribution are both location

dependent

Nathan Jacobs 14

A location-independent Shape Basis

Generate shape subspace using PCA on all shapes in training set.

First Principle Component

(~size)

Second Principle Component

(~orientation)

Nathan Jacobs 15

Location-dependent Coefficients

First Principle Component

(~size)

Second Principle Component

(~orientation)

Nathan Jacobs 16

Location-specific Shape Subspaces

Generate a shape subspace using shapes found in a small region of the image.

Location-specific mean shapes

Nathan Jacobs 17

Location-specific Shape Subspaces

Shape subspaces are location dependent.

Much smaller variations in some regions.

First PC Variations Second PC Variations

Nathan Jacobs 18

Shapes in (Shape) Space

Nathan Jacobs 19

An Example from PETS

First Principle Component

(~size)

Second Principle Component(~aspect)

Nathan Jacobs 20

Location-dependent Mean Shapes

Mean Shapes

Nathan Jacobs 21

Location-dependent Subspaces

First PC Variations Second PC Variations

Nathan Jacobs 22

Object Initialization for Tracking

• Object initialization is a crucial step of any tracking algorithm

• Use shape priors to determine object boundaries– Combines image information and shape prior– Penalize unlikely shapes– More accurate than image information alone

• Major point: strong priors make simple methods work

Nathan Jacobs 23

Object Boundary Detection

• Goal is to determine object boundaries to improve tracking initialization

• Algorithm– Find candidates using background subtraction– Initialize each candidate with a location-

specific mean shape– Optimize shape by gradient descent in PCA

shape subspace (penalize object overlap)• Image data term: sum of per-pixel foreground

probability inside shape• Shape prior term: sum of absolute value of PCA

coefficients

Nathan Jacobs 24

Segmentation Results

Subspace only Subspace and Prior

Global shape model

Local shape model

Nathan Jacobs 25

Summary

1. Static cameras give strong priors.

2. Unsupervised training of a localized shape prior is possible.

3. Localized shape priors can be used to improve object initialization for tracking.

Background

Nathan Jacobs 27

Efficient Segmentation

• Use gradient descent in low-dimensional shape subspace

• Gradient estimation– For each underlying

shape parameter• Sum along two edges

of polygon– For PC components and

object location• Weighted combination

of polygon edge scores

Nathan Jacobs 28

Choice of Support Region

Nathan Jacobs 29

Choosing Constants for Updating Prior Models

Threshold

.99999

0.9999

0.999

0.99

0.9

The best learning rate depends on scene,

application, time-of-day, weather, image location.

Slow update Fast updateCurrent Frame

VSSN 2006

Nathan Jacobs 30

Segmentation Energy Function

Minimize

Penalty on size

Per-pixel foreground likelihood

Shape penalty based on prior (sum of PCA coefficients)