prakash chockalingam clemson university non-rigid multi-modal object tracking using gaussian mixture...

Prakash ChockalingamPrakash Chockalingam

Clemson UniversityClemson University

Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models

Committee MembersCommittee Members

Dr Stan Birchfield (chair)Dr Stan Birchfield (chair)Dr Robert SchalkoffDr Robert Schalkoff

Dr Brian DeanDr Brian Dean

Tracking OverviewTracking Overview

Tracker Tasks

Feature Descriptors

Object Model

Update / LearningMechanism

TrackingFramework

Color

Gradients

Texture

Shape

Motion

Template

Contour

Active Appearance

ProbabilityDensities

Mean Shift

Pixel-wiseClassification

Optical Flow

Filtering techniques

No Update

Adaboost

Expectation Maximization

Re-weightingStrategy

Object Detection

Manual

Segmentation

Feature Points

ApproachApproach

• Tracking Framework: Target and background is modeled as a mixture of Gaussians in a joint feature-spatial space. A strength map is computed indicating the probability of each pixel belonging to the foreground.

• Contour Extraction: Contour is extracted using a discrete implementation of level sets

• Image Segmentation: Each Gaussian (fragment) is adapted to the image data by segmenting the image.

• Update Mechanism: The parameters of all the Gaussians are updated based on tracked data

• Results

Tracking FrameworkTracking Framework

Bayesian Formulation:Bayesian Formulation:

0: 0: 1 0: 1

arg

( | , ) ( | ) ( | ) ( | )t t t t t t t t t

t et background shape

p I p I p I p

Image data of all frames

Contour at time t Previously seen contours

*

**( | ) ( | )t t t

y R

p I p y

Assuming conditional independence among pixels,

Feature vector

* { , }

Object ModelingObject Modeling

f1

f2

*( | )tp y ?

Gaussian Mixture Model (GMM):

*

*1

( | ) ( | , )k

t tj

prior likelihood

p j p y j

( | )tp j * * 1 **

1( | , ) exp{ ( ) ( ) ( )}

2T

t j j jp y j y y

Strength Image:

( )( ) log

( )

p xx

p x

>0 for Foreground<0 for Background

y

Strength ImageStrength Image

GMM Linear Classifier Single Gaussian

Strength Image (contd…)Strength Image (contd…)

…

Linear ClassifierSingle Gaussian

Individual Fragments

Final Strength Strength Without Spatial Information

TopicsTopics

• Tracking Framework: Target and background is modeled as a mixture of Gaussians in a joint feature-spatial space. A strength map is computed indicating the probability of each pixel belonging to the foreground.

• Contour Extraction: Contour is extracted using a discrete implementation of level sets

• Image Segmentation: Each Gaussian (fragment) is adapted to the image data by segmenting the image.

• Update Mechanism: The parameters of all the Gaussians are updated based on tracked data

• Results

Contour ExtractionContour Extraction

Implicit representation of growing region

Likelihood term(Strength image) Regularization term

Energy Functional:

(strength image) (frontier)

( ) ( ) * ( )x x

E x x G x

> 0 Inside< 0 Outside

Contour Extraction (contd…)Contour Extraction (contd…)

( ) 0, ( ) 0y y

( ) 0, ( ) 0y y

( ) 0, ( ) 0y y

( ) 0, ( ) 0y y

(Region to be shrunk)

(Region already grown)

(Region to be grown)

(Region that need not be considered)

Contour Extraction (contd…)Contour Extraction (contd…)

4{ ' : ' ( ), ( ') 0, ( ') 0}gx x N x x x

4{ : ' ( )x x N x ( ) 0, ( ') 0}gx x such that

x x’

xx’

4{ ' : ' ( ), ( ') 0, ( ') 0}gx x N x x x

4{ : ' ( )x x N x ( ) 0, ( ') 0}g gx x such that

Dilation

Contraction

Contour Extraction (contd…)Contour Extraction (contd…)

Expand

Remove interior points

Contract

Remove exterior points

Contour Extraction (contd…)Contour Extraction (contd…)

Likelihood

Final Region

TopicsTopics

• Tracking Framework: Target and background is modeled as a mixture of Gaussians in a joint feature-spatial space. A strength map is computed indicating the probability of each pixel belonging to the foreground.

• Contour Extraction: Contour is extracted using a discrete implementation of level sets

• Image Segmentation: Each Gaussian (fragment) is adapted to the image data by segmenting the image.

• Update Mechanism: The parameters of all the Gaussians are updated based on tracked data

• Results

Region SegmentationRegion Segmentation

Mode-seeking region growing algorithm:

do {

• Pick a seed point that is not associated to any fragment

• Grow the fragment from the seed point based on the similarity of the pixel and its neighbor’s appearance

• Stop growing the fragment if no more similar pixels are present in the neighborhood of the fragment

} until all pixels are assigned

Seed point:

3

1

( ) ( )ii

x x

Eigen values of 3x3 RGB covariance matrix

1,..., nS where ( , , )i i i ix y

Region Segmentation (contd…)Region Segmentation (contd…)

• Pick the minimum element in S. Create a region to hold the pixel and add the neighbors in a fixed window.

• Compute Mean μj and Covariance Σj of the region.

• Likelihood:

• Grow the region as before with two additional steps: Update μj, and Σj, as a new pixel is added Remove the corresponding element in S if a pixel is added

• Continue above steps if S is not empty.

( ) ( ( ), ( , ))j j jx MD f x

Initial region

Mahalanobis distance

Configurable parameter

Region Segmentation (contd…)Region Segmentation (contd…)

Region Growing Graph-Based Mean-Shift

Region Segmentation (contd…)Region Segmentation (contd…)

Region Growing Graph-Based Mean-Shift

TopicsTopics

• Tracking Framework: Target and background is modeled as a mixture of Gaussians in a joint feature-spatial space. A strength map is computed indicating the probability of each pixel belonging to the foreground.

• Contour Extraction: Contour is extracted using a discrete implementation of level sets

• Image Segmentation: Each Gaussian (fragment) is adapted to the image data by segmenting the image.

• Update Mechanism: The parameters of all the Gaussians are updated based on tracked data

• Results

Update MechanismUpdate Mechanism

f1

f2

* *, ,,j t j t

• Update parameters of existing fragments

• Detect fragment occlusion

• Find new fragments

Initial Frame Initial Model Fragment Association

Update Mechanism (contd…)Update Mechanism (contd…)

* * * * *, ,0: ,0(1 )j t j j t j j

( ) *,

* 0,0:

( )

0

tt

j

j t tt

e

e

Initial Model

(function of past and current values)

Weight computed by comparing Mahalanobis distance

Updating parameters of existing fragments:

Update Mechanism (contd…)Update Mechanism (contd…)

Occluded fragments:

If a fragment is associated with less than 0.2% of the image pixels, then the fragment is declared as occluded.

Finding new fragments:

( ){ : log 0}

( )

p xT x

p x

Helps in handling self-occlusion

Spatial AlignmentSpatial Alignment

The spatial parameters are updated using the motion vectors from Joint Lucas-Kanade approach

Lucas-Kanade Joint Lucas-Kanade

Algorithm summaryAlgorithm summaryInitial frame:

• The user marks the object to be tracked.

• The target object and background scene are segmented based on their appearance similarity.

• The target object and background scene are modeled using a mixture of Gaussians where each Gaussian correspond to a fragment in the joint feature-spatial space

Subsequent frames:

• Update the spatial parameters of GMM using the motion vectors of Joint Lucas-Kanade

• Each pixel is classified into either foreground or background by generating a strength map using the Gaussian mixture model (GMM) of the object and background.

• The strength map is integrated into a discrete level set formulation to obtain accurate contour of the object.

• Using the tracked data, the appearance parameters of the GMM are updated.

TopicsTopics

• Tracking Framework: Target and background is modeled as a mixture of Gaussians in a joint feature-spatial space. A strength map is computed indicating the probability of each pixel belonging to the foreground.

• Contour Extraction: Extract contour using a discrete implementation of level sets

• Image Segmentation: Each Gaussian (fragment) is adapted to the image data by segmenting the image.

• Update Mechanism: The parameters of all the Gaussians are updated based on tracked data

• Results

Experimental ResultsExperimental Results

Elmo Sequence Monkey Sequence

Experimental Results (Contd…)Experimental Results (Contd…)

Person Sequence Fish Sequence

Experimental Results: Self-Occlusion Experimental Results: Self-Occlusion

Without Self-Occlusion Module With Self-Occlusion Module

ConclusionConclusion

• A tracking framework based on modeling the object as mixture of Gaussians is proposed

• An efficient discrete implementation of level sets is employed to extract contour.

• A mode-seeking region growing algorithm is used to segment the image.

• A simple re-weighting strategy is proposed to update the parameters of Gaussians.

Future Directions:

• Incorporate shape priors.

• Utilize the extracted shapes to learn more robust priors.

• An offline or online evaluation mechanism during the initialization phase.

• Adding global information into the region segmentation process.

• Automating the object detection and initialization.

Questions ?Questions ?

Thank you !Thank you !