an object tacking paradigm with active appearance models for augmented reality
DESCRIPTION
An Object Tacking Paradigm with Active Appearance Models for Augmented Reality. Presented by Pat Chan Pik Wah 28/04/2005 Qualifying Examination. Outline. Research Objective Introduction Augmented Reality Object Tracking Active Appearance Models (AAMs) Proposed Object Tracking Paradigm - PowerPoint PPT PresentationTRANSCRIPT
An Object Tacking Paradigm with Active Appearance Models for Augmented Reality
Presented by Pat Chan Pik Wah
28/04/2005
Qualifying Examination
Outline
Research Objective Introduction
Augmented Reality Object Tracking Active Appearance Models (AAMs)
Proposed Object Tracking Paradigm Paradigm Architecture Experiments
Research Issues Conclusion
Research Objective
Object tracking is an essential component for Augmented Reality.
There is a lack of good object tracking paradigm. Active Appearance Models is promising. Propose a new object tracking paradigm with AAMs
in order to provide a real-time and accurate registration for Augmented Reality.
Nature of the paradigm: Effective Accurate Robust
Augmented Reality
An Augmented Reality system supplements the real world with virtual objects that appear to coexist in the same space as the real world
Properties : Combine real and virtual objects in a real environment Runs interactively, and in real time Registers (aligns) real and virtual objects with each other
Augmented Reality
Projects related to AR
Augmented Reality
Display Presenting virtual objects on real environment
Tracking Following user’s and virtual object’s movements
by means of a special device or techniques 3D Modeling
Forming virtual object Registration
Blending real and virtual objects
Object Tracking
Visual content can be modeled as a hierarchy of abstractions.
At the first level are the raw pixels with color or brightness information.
Further processing yields features such as edges, corners, lines, curves, and color regions.
A higher abstraction layer may combine and interpret these features as objects and their attributes.
Pixels
edges, corners, lines, curves, and color regions
Object
Object Tracking
Accurately tracking the user’s position is crucial for AR registration
The objective is to obtain an accurate estimate of the position (x,y) of the object tracked
Tracking = correspondence + constraints + estimation Based on reference image of the object, or properties
of the objects. Two main stages for tracking object in video:
Isolation of objects from background in each frames Association of objects in successive frames in order to trace
them
Object Tracking
Object Tracking can be briefly divides into following stages: Input (object and camera) Detecting the Objects Motion Estimation Corrective Feedback Occlusion Detection
Object Tracking
Expectation Maximization Find the local maximum likelihood solution Some variables are hidden or incomplete
Kalman Filter Optimal linear predict the state of a model
Condensation Combines factored sampling with learned dynam
ical models propagate an entire probability of object position
and shape
Object Tracking
Pervious Work : Marker-based Tracking Feature-based Tracking Template-based object tracking Correlation-based tracking Change-based tracking 2D layer tracking tracking of articulated objects
Pervious Work
Marker-based Tracking Marker-less based Tracking Feature-based Tracking
Shape-based approaches Color-based approaches
Pervious Work
Template-based object tracking Fixed template matching
Image subtraction Correlation
Deformable template matching
Pervious Work
Object tracking using motion information Motion-based approaches Model-based approaches Boundary-based approaches
Snakes Geodesic active contour models
Region-based approaches
Active Appearance Models
The Active Appearance Model (AAM) algorithm is a powerful tool for modeling images of deformable objects.
AAM combines a subspace-based deformable model of an object’s appearance
Fit the model to a previously unseen image.
Timeline for development of AAMs and ASMs
Active Appearance Models (AAMs)
2D linear shape is defined by 2D triangulated mesh and in particular the vertex locations of the mesh.
Shape s can be expressed as a base shape s0.
pi are the shape parameter. s0 is the mean shape and the matrices si are the eigenvectors corresp
onding to the m largest eigenvalues
Active Appearance Models (AAMs)
The appearance of an independent AAM is defined within the base mesh s0. A(u) defined over the pixels u ∈ s0
A(u) can be expressed as a base appearance A0(u) plus a linear combination of l appearance
Coefficients λi are the appearance parameters.
A0(u) A1(u) A2(u) A3(u)
Active Appearance Models (AAMs)
The AAM model instance with shape parameters p and appearance parameters λ is then created by warping the appearance A from the base mesh s0 to the model shape s.
Piecewise affine warp W(u; p):(1) for any pixel u in s0 find out which triangle it lies in,(2) warp u with the affine warp for that triangle.
M(W(u;p))
Fitting AAMs
Minimize the error between I (u) and M(W(u; p)) = A(u).
If u is a pixel in s0, then the corresponding pixel in the input image I is W(u; p).
At pixel u the AAM has the appearance
At pixel W(u; p), the input image has the intensity I (W(u; p)).
Minimize the sum of squares of the difference between these two quantities:
uu u u
DEMO Video – 2D AAMs
DEMO Video – 2D AAMs
Recent Work for Improving AAMs
Combine 2D+3D AAMs
Combined 2D + 3D AAMs
At time t, we have 2D AAM shape vector in all N images into a matrix:
Represent as a 3D linear shape modes W = MB =
Compute the 3D Model
AAM shapes
AAM appearance
First three 3D shapes modes
Constraining an AAM with 3D Shape
Constraints on the 2D AAM shape parameters p = (p1, … , pm) that force the AAM to only move in a way that is consistent with the 3D shape modes:
and the 2D shape variation of the 3D shape modes over all imaging condition is:
Legitimate values of P and p such that the 2D projected 3D shape equals the 2D shape of AAM. The constraint is written as:
An Object Tacking Paradigm with Active Appearance Models
Proposed Object Tracking Paradigm
Training Active Appearance Model
Training Images
1. Shape Model2. Appearance Model Initialization Motion Modeling Kalman Filter
Occlusion Detection
VideoVideo
Paradigm Architecture
Steps in Object Tracking Paradigm
Preporcessing Training the Active Appearance Model. Get the shape model and the appearance model for the object to
be tracked. Initialization
Locating the object position in the video. In our scheme, we make use of AAMs.
Motion Modeling Estimate the motion of the object Modeling the AAMs as a problem in the Kalman filter to perform t
he prediction. Occlusion Detection
Preventing the lost of position of the object by occluding of other objects.
Enhancing Active Appearance Models
Shape
Appearance
Combine the shape and the appearance parameters for optimization
In video, shape and appearance may not enough, there are many characteristics and features, such as lightering, brightness, etc…
L=[L1, L2, ……, Lm]T
Iterative Search for Fitting Active Appearance Model
Iterative Search for Fitting Active Appearance Model
Can be improved by:1. Prediction matrix2. Searching space
Initialization for AAMs
Motion Modeling
Initial estimate in a frame should be better predicted than just the adaptation from the previous frame.
Can be achieved by motion estimation AAMs can do the modeling part Kalman filter can do the prediction part
Kalman Filter
Adaptive filter Model the state of a discrete dynamic
system. Originally developed in 1960 Filter out noise in electronic signals.
Kalman Filter
Formally, we have the model
For our tracking system,
1
1
1 1
1
( )
( )
( )
k
k
n
n n
n
k
k k k
k
k
k k
k
S
x A u
c
S
x B A u w
c
S
z H A u v
c
1
1
1
1
1
'
'
1 0 0
0 1 0
0 0 1 0 '
0 0 0 1 '
1 0 0 0
0 1 0 0
'0 0 1 0
'0 0 0 1
k
k
k
k
n
nn
n
n
k k
k
kk k
k
k
x
yx
x
y
xtyt
x wx
y
x
yz v
x
y
11
1
1 0
0 1
1 0
0 1
nn
n
kk k
k
kk k
k
xx
y
xx w
y
xz v
y
Kalman Filter
Occlusion Detection WHY?WHY?
Positioning of objects To perform cropping When a real object overlays a virtual one, the
virtual object should be cropped before the overlay HOW?HOW?
High resolution and sharp object boundaries Right occluding boundaries of objects Camera matrix for video capturing
Proposed Object Tracking Paradigm
Training Active Appearance Model
Training Images
1. Shape Model2. Appearance Model Initialization
Active Appearance Model Fitting
Kalman Filter
Occlusion Detection
VideoVideo
Paradigm Architecture
Experimental Setup
AAM-api from DTU OpenCV Pentium 4 CPU 2.00GHz and 512MB
RAM
Experiment on AAMs (1)
Training Image
Experiment on AAMs (1)
Shape Texture
Experiment on AAMs (1)
Initialization After optimized
Demo Video
Demo Video
Demo Video
Demo Video
Experiment on AAMs (2)
Training Images
Experiment on AAMs
Shape Texture
Experiment on AAMs
Trapped in local minimum
Initialization After optimized
Experiment on AAMs
Experiment on AAMs
Fit to the face
Initialization After optimized
Experiment on AAMs
Object Tracking with AAMs
Experiment on Kalman Filter
Demo Video
Experiment on Kalman Filter
Demo Video
Research Issues
AAMs tracking is accurate Very slow Cannot perform real-time tracking
Kalman filter help is to increase the speed in prediction Modeling the problem from AAMs to Kalman Filter
Improving the fitting algorithm in the AAMs Occlusion detection
Important to object tracking Preventing the lost of the position
Conclusion
We have done a survey on object tracking and active appearance model is done
We proposed a paradigm on video object tracking with active appearance models
Goal: Robust Real-time Good performance
We have done some initial experiments: Experiments on AAMs Experiments on Kalman filter for object tracking
Q & A