ferns

Some Methods for Augmented Reality ApplicationsVincent Lepetit, Mustafa Ozuysal, Julien Pilet, Pascal Lagger, Pascal Fua

2

QuickTime™ and a decompressor

are needed to see this picture.

QuickTime™ and a decompressor

are needed to see this picture.

QuickTime™ and a decompressor

are needed to see this picture.

Vision-Based 3D Tracking

3

QuickTime™ and a decompressor

are needed to see this picture.QuickTime™ and a

decompressorare needed to see this picture.

Recursive Tracking

t = 1t = 0 t = 2 ...

Real-Time 3D Object Detection

Runs at 15 Hz

QuickTime™ and a decompressor

are needed to see this picture.

5

Keypoint Recognition

Pre-processingMake the actual classification easier

Nearest neighbor classification

One class per keypoint: the set of the keypoint’s possible appearances under various perspective, lighting, noise...

The general approach [Lowe, Schmid, Mikolajczyk, Matas] is a particular case of classification:

Search in the Database

6

Used at run-time to recognize the keypoints

Training phase Classifier

7

A New Classifier: Ferns

Presentation on an Example

Ferns: TrainingThe tests compare the intensities of two pixels around the keypoint:

Invariant to light change by any raising function.

Posterior probabilities:

Ferns: Training

6

1

5

0

1

1

1

0

0

1

0

1

Ferns: Training

Ferns: Training Results

Ferns: Recognition

14

Compromise:

proportional to

but complete representation of the joint distribution infeasible.

Naive Bayesian ignores the correlation:

ie probabilities stored by the leaves.

We are looking for

Justification

It Really Works

QuickTime™ and a decompressor

are needed to see this picture.

16

QuickTime™ and a decompressor

are needed to see this picture.

17

Ferns Implementation

17

1: for(int i = 0; i < H; i++) P[i ] = 0.; 2: for(int k = 0; k < M; k++) { 3: int index = 0, * d = D + k * 2 * S; 4: for(int j = 0; j < S; j++) { 5: index <<= 1; 6: if (*(K + d[0]) < *(K + d[1])) 7: index++; 8: d += 2; } 9: p = PF + k * shift2 + index * shift1;10: for(int i = 0; i < H; i++) P[i] += p[i]; }

Very simple to implement;No need for orientation nor perspective correction;No parameters to tune;Very fast.

1818

Fast, easy to implement;No parameters to tune.

Takes a lot of memory:

floating point values to store.

19

Feature Harvesting:

Estimating the Posterior Probabilities from Video

Sequences

Feature Harvesting

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

Estimate the posterior probabilities from a training video sequence:

Feature Harvesting

Update Classifier

Detect Object in Current Frame

Training examplesMatches

With the ferns, we can easily:- add a class;- remove a class;- add samples of a class to refine the classifier. Incremental learning

No need to store image patches; We can select the keypoints the classifier can recognize.

Feature Harvesting Steps

Feature Harvesting

Test Sequence

QuickTime™ and a decompressor

are needed to see this picture.

Application to Deformable Objects

26

Detecting a Deformable Object

15 frames/sec.

QuickTime™ and a decompressor

are needed to see this picture.

27

Energy Minimization

),(),(),( YXYXYX DDC ελεε +=

Wide Baseline Matching

Input Image Model Image

Regularization Term

Realistic Rendering

Realistic Rendering

QuickTime™ and a decompressor

are needed to see this picture.

31

QuickTime™ and a decompressor

are needed to see this picture.

Application to Geometric and Photometric Calibration

Application to Geometric and Photometric Calibration

QuickTime™ and a decompressor

are needed to see this picture.

Calibration

• Internal parameters;

Calibration

• Internal parameters;• Relative motions;• Chain the relative motions to get the absolute poses;• Bundle adjustment.

Photometric Calibration

Make use of the same calibration object.

Two methods :• Dynamic, with Lambertian rendering only;• Static, allowing shadows and specular reflections.

Using the Geometric Calibration Target to Measure Light

Solve for gain, bias of each camera, and irradiance for observed normals.

ntalbedo(m)I(camera, t, m)

m

Radiance Map Computation

INTERPOLATION

Lambertian material rendering;Updated in real-time.

40

Using and Updating the Radiance Map

QuickTime™ and a decompressor

are needed to see this picture.

Point Light Sources for AR

DECONVOLUTION

Retrieve point light sources for:– Cast shadows;– Specular reflections.

Not real-time anymore.

QuickTime™ and aMS-MPEG4 v2 decompressor

are needed to see this picture.

4343

You can download the source code (under GPL) for • Keypoint recognition,• Planar object detection, and• Multi-camera geometric and photometric calibration,at http://cvlab.epfl.ch/software/bazar

Installation on Windows, Linux, and Mac OSX should be easy.

Thank you

44

45

ReferencesV. Lepetit and P. Fua. Keypoint recognition using randomized trees. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(9):1465-1479, September 2006.

M. Ozuysal, V. Lepetit, F. Fleuret and P. Fua. Feature Harvesting for Tracking-by-Detection. In European Conference on Computer Vision, Vol. 3, pp. 592--605, 2006.

J. Pilet, V. Lepetit and P. Fua. Fast Non-Rigid Surface Detection, Registration, and Realistic Augmentation. Accepted to International Journal of Computer Vision, 2006.

M. Salzmann, S. Ilic and P. Fua. Physically Valid Shape Parameterization for Monocular 3-D Deformable Surface Tracking. British Machine Vision Conference, 2005.

46

Compromise:

Justification

proportional to

but complete representation of the joint distribution infeasible.

Naive Bayesian ignores the correlation:

We are looking for

59

Comparison with SIFTRecognition rate

FERNS

SIFT

Frame Index

Number of Inliers

60

Comparison with SIFTComputation time

• SIFT: 1 ms to compute the descriptor of a keypoint (without including convolution);

• FERNS: 13.5 10-3 ms to classify one keypoint into 200 classes.

Naive Feature Tracking

Build Classifier

Track Keypoints by Frame-to-Frame Matching

Training Examples

Does not work very well: Prone to drift & tracking failures;

How to select “good” keypoints? Feature harvesting.

Handling Light Changes

QuickTime™ and a decompressor

are needed to see this picture.

63

3D Shape Recovery from Monocular Video

M. Salzmann, S. Ilic and P. Fua. Physically Valid Shape Parameterization for Monocular 3-D Deformable Surface Tracking. BMVC05.

64

Bending modes

Extension modes

Database of Feasible Shapes

PCA

65

Vision for the America's Cup

CTI Project

66

Stretchable Material

67

Robust Bundle Adjustment

• One pixel reprojection

error

Error evolution

68

Same Camera, 2 Sets of Images

•Random initialization;•Successive minimizations with lower andlower values for r:

r0 = 1000 r = 1 ou 2 pixels

Optimization

QuickTime™ and a decompressor

are needed to see this picture.

Internal Parameters Calibration

0

0

Homography

External Parameter Calibration

0

0

A1

A2

A3

Relative 3D Pose

World Coordinate System

Absolute 3D Pose

P1

P3P4

P2

Bundle Adjustment on: P1, P2, P3, P4, A1, A2 and A3

Using the Geometric Calibration Target to Measure Light

Solve for gain, bias of each camera, and irradiance for observed normals.

ntalbedo(m)I(camera, t, m)

m