an object tacking paradigm with active appearance models for augmented reality

61
An Object Tacking Paradigm with Active Appearance Models for Augmented Reality Presented by Pat Chan Pik Wah 28/04/2005 Qualifying Examination

Upload: rufus

Post on 22-Jan-2016

30 views

Category:

Documents


0 download

DESCRIPTION

An Object Tacking Paradigm with Active Appearance Models for Augmented Reality. Presented by Pat Chan Pik Wah 28/04/2005 Qualifying Examination. Outline. Research Objective Introduction Augmented Reality Object Tracking Active Appearance Models (AAMs) Proposed Object Tracking Paradigm - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Presented by Pat Chan Pik Wah

28/04/2005

Qualifying Examination

Page 2: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Outline

Research Objective Introduction

Augmented Reality Object Tracking Active Appearance Models (AAMs)

Proposed Object Tracking Paradigm Paradigm Architecture Experiments

Research Issues Conclusion

Page 3: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Research Objective

Object tracking is an essential component for Augmented Reality.

There is a lack of good object tracking paradigm. Active Appearance Models is promising. Propose a new object tracking paradigm with AAMs

in order to provide a real-time and accurate registration for Augmented Reality.

Nature of the paradigm: Effective Accurate Robust

Page 4: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Augmented Reality

An Augmented Reality system supplements the real world with virtual objects that appear to coexist in the same space as the real world

Properties : Combine real and virtual objects in a real environment Runs interactively, and in real time Registers (aligns) real and virtual objects with each other

Page 5: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Augmented Reality

Projects related to AR

Page 6: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Augmented Reality

Display Presenting virtual objects on real environment

Tracking Following user’s and virtual object’s movements

by means of a special device or techniques 3D Modeling

Forming virtual object Registration

Blending real and virtual objects

Page 7: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Object Tracking

Visual content can be modeled as a hierarchy of abstractions.

At the first level are the raw pixels with color or brightness information.

Further processing yields features such as edges, corners, lines, curves, and color regions.

A higher abstraction layer may combine and interpret these features as objects and their attributes.

Pixels

edges, corners, lines, curves, and color regions

Object

Page 8: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Object Tracking

Accurately tracking the user’s position is crucial for AR registration

The objective is to obtain an accurate estimate of the position (x,y) of the object tracked

Tracking = correspondence + constraints + estimation Based on reference image of the object, or properties

of the objects. Two main stages for tracking object in video:

Isolation of objects from background in each frames Association of objects in successive frames in order to trace

them

Page 9: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Object Tracking

Object Tracking can be briefly divides into following stages: Input (object and camera) Detecting the Objects Motion Estimation Corrective Feedback Occlusion Detection

Page 10: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Object Tracking

Expectation Maximization Find the local maximum likelihood solution Some variables are hidden or incomplete

Kalman Filter Optimal linear predict the state of a model

Condensation Combines factored sampling with learned dynam

ical models propagate an entire probability of object position

and shape

Page 11: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Object Tracking

Pervious Work : Marker-based Tracking Feature-based Tracking Template-based object tracking Correlation-based tracking Change-based tracking 2D layer tracking tracking of articulated objects

Page 12: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Pervious Work

Marker-based Tracking Marker-less based Tracking Feature-based Tracking

Shape-based approaches Color-based approaches

Page 13: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Pervious Work

Template-based object tracking Fixed template matching

Image subtraction Correlation

Deformable template matching

Page 14: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Pervious Work

Object tracking using motion information Motion-based approaches Model-based approaches Boundary-based approaches

Snakes Geodesic active contour models

Region-based approaches

Page 15: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Active Appearance Models

The Active Appearance Model (AAM) algorithm is a powerful tool for modeling images of deformable objects.

AAM combines a subspace-based deformable model of an object’s appearance

Fit the model to a previously unseen image.

Page 16: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Timeline for development of AAMs and ASMs

Page 17: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Active Appearance Models (AAMs)

2D linear shape is defined by 2D triangulated mesh and in particular the vertex locations of the mesh.

Shape s can be expressed as a base shape s0.

pi are the shape parameter. s0 is the mean shape and the matrices si are the eigenvectors corresp

onding to the m largest eigenvalues

Page 18: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Active Appearance Models (AAMs)

The appearance of an independent AAM is defined within the base mesh s0. A(u) defined over the pixels u ∈ s0

A(u) can be expressed as a base appearance A0(u) plus a linear combination of l appearance

Coefficients λi are the appearance parameters.

A0(u) A1(u) A2(u) A3(u)

Page 19: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Active Appearance Models (AAMs)

The AAM model instance with shape parameters p and appearance parameters λ is then created by warping the appearance A from the base mesh s0 to the model shape s.

Piecewise affine warp W(u; p):(1) for any pixel u in s0 find out which triangle it lies in,(2) warp u with the affine warp for that triangle.

M(W(u;p))

Page 20: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Fitting AAMs

Minimize the error between I (u) and M(W(u; p)) = A(u).

If u is a pixel in s0, then the corresponding pixel in the input image I is W(u; p).

At pixel u the AAM has the appearance

At pixel W(u; p), the input image has the intensity I (W(u; p)).

Minimize the sum of squares of the difference between these two quantities:

uu u u

Page 21: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

DEMO Video – 2D AAMs

Page 22: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

DEMO Video – 2D AAMs

Page 23: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Recent Work for Improving AAMs

Combine 2D+3D AAMs

Page 24: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Combined 2D + 3D AAMs

At time t, we have 2D AAM shape vector in all N images into a matrix:

Represent as a 3D linear shape modes W = MB =

Page 25: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Compute the 3D Model

AAM shapes

AAM appearance

First three 3D shapes modes

Page 26: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Constraining an AAM with 3D Shape

Constraints on the 2D AAM shape parameters p = (p1, … , pm) that force the AAM to only move in a way that is consistent with the 3D shape modes:

and the 2D shape variation of the 3D shape modes over all imaging condition is:

Legitimate values of P and p such that the 2D projected 3D shape equals the 2D shape of AAM. The constraint is written as:

Page 27: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

An Object Tacking Paradigm with Active Appearance Models

Page 28: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Proposed Object Tracking Paradigm

Training Active Appearance Model

Training Images

1. Shape Model2. Appearance Model Initialization Motion Modeling Kalman Filter

Occlusion Detection

VideoVideo

Paradigm Architecture

Page 29: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Steps in Object Tracking Paradigm

Preporcessing Training the Active Appearance Model. Get the shape model and the appearance model for the object to

be tracked. Initialization

Locating the object position in the video. In our scheme, we make use of AAMs.

Motion Modeling Estimate the motion of the object Modeling the AAMs as a problem in the Kalman filter to perform t

he prediction. Occlusion Detection

Preventing the lost of position of the object by occluding of other objects.

Page 30: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Enhancing Active Appearance Models

Shape

Appearance

Combine the shape and the appearance parameters for optimization

In video, shape and appearance may not enough, there are many characteristics and features, such as lightering, brightness, etc…

L=[L1, L2, ……, Lm]T

Page 31: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Iterative Search for Fitting Active Appearance Model

Page 32: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Iterative Search for Fitting Active Appearance Model

Can be improved by:1. Prediction matrix2. Searching space

Page 33: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Initialization for AAMs

Page 34: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Motion Modeling

Initial estimate in a frame should be better predicted than just the adaptation from the previous frame.

Can be achieved by motion estimation AAMs can do the modeling part Kalman filter can do the prediction part

Page 35: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Kalman Filter

Adaptive filter Model the state of a discrete dynamic

system. Originally developed in 1960 Filter out noise in electronic signals.

Page 36: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Kalman Filter

Formally, we have the model

For our tracking system,

1

1

1 1

1

( )

( )

( )

k

k

n

n n

n

k

k k k

k

k

k k

k

S

x A u

c

S

x B A u w

c

S

z H A u v

c

1

1

1

1

1

'

'

1 0 0

0 1 0

0 0 1 0 '

0 0 0 1 '

1 0 0 0

0 1 0 0

'0 0 1 0

'0 0 0 1

k

k

k

k

n

nn

n

n

k k

k

kk k

k

k

x

yx

x

y

xtyt

x wx

y

x

yz v

x

y

11

1

1 0

0 1

1 0

0 1

nn

n

kk k

k

kk k

k

xx

y

xx w

y

xz v

y

Page 37: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Kalman Filter

Page 38: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Occlusion Detection WHY?WHY?

Positioning of objects To perform cropping When a real object overlays a virtual one, the

virtual object should be cropped before the overlay HOW?HOW?

High resolution and sharp object boundaries Right occluding boundaries of objects Camera matrix for video capturing

Page 39: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Proposed Object Tracking Paradigm

Training Active Appearance Model

Training Images

1. Shape Model2. Appearance Model Initialization

Active Appearance Model Fitting

Kalman Filter

Occlusion Detection

VideoVideo

Paradigm Architecture

Page 40: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Experimental Setup

AAM-api from DTU OpenCV Pentium 4 CPU 2.00GHz and 512MB

RAM

Page 41: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Experiment on AAMs (1)

Training Image

Page 42: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Experiment on AAMs (1)

Shape Texture

Page 43: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Experiment on AAMs (1)

Initialization After optimized

Page 44: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Demo Video

Page 45: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Demo Video

Page 46: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Demo Video

Page 47: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Demo Video

Page 48: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Experiment on AAMs (2)

Training Images

Page 49: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Experiment on AAMs

Shape Texture

Page 50: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Experiment on AAMs

Trapped in local minimum

Initialization After optimized

Page 51: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Experiment on AAMs

Page 52: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Experiment on AAMs

Fit to the face

Initialization After optimized

Page 53: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Experiment on AAMs

Page 54: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Object Tracking with AAMs

Page 55: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Experiment on Kalman Filter

Page 56: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Demo Video

Page 57: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Experiment on Kalman Filter

Page 58: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Demo Video

Page 59: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Research Issues

AAMs tracking is accurate Very slow Cannot perform real-time tracking

Kalman filter help is to increase the speed in prediction Modeling the problem from AAMs to Kalman Filter

Improving the fitting algorithm in the AAMs Occlusion detection

Important to object tracking Preventing the lost of the position

Page 60: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Conclusion

We have done a survey on object tracking and active appearance model is done

We proposed a paradigm on video object tracking with active appearance models

Goal: Robust Real-time Good performance

We have done some initial experiments: Experiments on AAMs Experiments on Kalman filter for object tracking

Page 61: An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Q & A