ferns

61

Click here to load reader

Upload: anon-430581

Post on 16-Aug-2014

235 views

Category:

Documents


11 download

TRANSCRIPT

Page 1: Ferns

Some Methods for Augmented Reality ApplicationsVincent Lepetit, Mustafa Ozuysal, Julien Pilet, Pascal Lagger, Pascal Fua

Page 2: Ferns

2

QuickTime™ and a decompressor

are needed to see this picture.

QuickTime™ and a decompressor

are needed to see this picture.

QuickTime™ and a decompressor

are needed to see this picture.

Vision-Based 3D Tracking

Page 3: Ferns

3

QuickTime™ and a decompressor

are needed to see this picture.QuickTime™ and a

decompressorare needed to see this picture.

Recursive Tracking

t = 1t = 0 t = 2 ...

Page 4: Ferns

Real-Time 3D Object Detection

Runs at 15 Hz

QuickTime™ and a decompressor

are needed to see this picture.

Page 5: Ferns

5

Keypoint Recognition

Pre-processingMake the actual classification easier

Nearest neighbor classification

One class per keypoint: the set of the keypoint’s possible appearances under various perspective, lighting, noise...

The general approach [Lowe, Schmid, Mikolajczyk, Matas] is a particular case of classification:

Search in the Database

Page 6: Ferns

6

Used at run-time to recognize the keypoints

Training phase Classifier

Page 7: Ferns

7

A New Classifier: Ferns

Page 8: Ferns

Presentation on an Example

Page 9: Ferns

Ferns: TrainingThe tests compare the intensities of two pixels around the keypoint:

Invariant to light change by any raising function.

Posterior probabilities:

Page 10: Ferns

Ferns: Training

6

1

5

0

1

1

1

0

0

1

0

1

Page 11: Ferns

Ferns: Training

Page 12: Ferns

Ferns: Training Results

Page 13: Ferns

Ferns: Recognition

Page 14: Ferns

14

Compromise:

proportional to

but complete representation of the joint distribution infeasible.

Naive Bayesian ignores the correlation:

ie probabilities stored by the leaves.

We are looking for

Justification

Page 15: Ferns

It Really Works

QuickTime™ and a decompressor

are needed to see this picture.

Page 16: Ferns

16

QuickTime™ and a decompressor

are needed to see this picture.

Page 17: Ferns

17

Ferns Implementation

17

1: for(int i = 0; i < H; i++) P[i ] = 0.; 2: for(int k = 0; k < M; k++) { 3: int index = 0, * d = D + k * 2 * S; 4: for(int j = 0; j < S; j++) { 5: index <<= 1; 6: if (*(K + d[0]) < *(K + d[1])) 7: index++; 8: d += 2; } 9: p = PF + k * shift2 + index * shift1;10: for(int i = 0; i < H; i++) P[i] += p[i]; }

Very simple to implement;No need for orientation nor perspective correction;No parameters to tune;Very fast.

Page 18: Ferns

1818

Fast, easy to implement;No parameters to tune.

Takes a lot of memory:

floating point values to store.

Page 19: Ferns

19

Feature Harvesting:

Estimating the Posterior Probabilities from Video

Sequences

Page 20: Ferns

Feature Harvesting

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

Estimate the posterior probabilities from a training video sequence:

Page 21: Ferns

Feature Harvesting

Update Classifier

Detect Object in Current Frame

Training examplesMatches

With the ferns, we can easily:- add a class;- remove a class;- add samples of a class to refine the classifier. Incremental learning

No need to store image patches; We can select the keypoints the classifier can recognize.

Page 22: Ferns

Feature Harvesting Steps

Page 23: Ferns

Feature Harvesting

Page 24: Ferns

Test Sequence

QuickTime™ and a decompressor

are needed to see this picture.

Page 25: Ferns

Application to Deformable Objects

Page 26: Ferns

26

Detecting a Deformable Object

15 frames/sec.

QuickTime™ and a decompressor

are needed to see this picture.

Page 27: Ferns

27

Energy Minimization

),(),(),( YXYXYX DDC ελεε +=

Wide Baseline Matching

Input Image Model Image

Regularization Term

Page 28: Ferns

Realistic Rendering

Page 29: Ferns

Realistic Rendering

Page 30: Ferns

QuickTime™ and a decompressor

are needed to see this picture.

Page 31: Ferns

31

QuickTime™ and a decompressor

are needed to see this picture.

Page 32: Ferns

Application to Geometric and Photometric Calibration

Page 33: Ferns

Application to Geometric and Photometric Calibration

Page 34: Ferns

QuickTime™ and a decompressor

are needed to see this picture.

Page 35: Ferns

Calibration

• Internal parameters;

Page 36: Ferns

Calibration

• Internal parameters;• Relative motions;• Chain the relative motions to get the absolute poses;• Bundle adjustment.

Page 37: Ferns

Photometric Calibration

Make use of the same calibration object.

Two methods :• Dynamic, with Lambertian rendering only;• Static, allowing shadows and specular reflections.

Page 38: Ferns

Using the Geometric Calibration Target to Measure Light

Solve for gain, bias of each camera, and irradiance for observed normals.

ntalbedo(m)I(camera, t, m)

m

Page 39: Ferns

Radiance Map Computation

INTERPOLATION

Lambertian material rendering;Updated in real-time.

Page 40: Ferns

40

Using and Updating the Radiance Map

QuickTime™ and a decompressor

are needed to see this picture.

Page 41: Ferns

Point Light Sources for AR

DECONVOLUTION

Retrieve point light sources for:– Cast shadows;– Specular reflections.

Not real-time anymore.

Page 42: Ferns

QuickTime™ and aMS-MPEG4 v2 decompressor

are needed to see this picture.

Page 43: Ferns

4343

You can download the source code (under GPL) for • Keypoint recognition,• Planar object detection, and• Multi-camera geometric and photometric calibration,at http://cvlab.epfl.ch/software/bazar

Installation on Windows, Linux, and Mac OSX should be easy.

Thank you

Page 44: Ferns

44

Page 45: Ferns

45

ReferencesV. Lepetit and P. Fua. Keypoint recognition using randomized trees. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(9):1465-1479, September 2006.

M. Ozuysal, V. Lepetit, F. Fleuret and P. Fua. Feature Harvesting for Tracking-by-Detection. In European Conference on Computer Vision, Vol. 3, pp. 592--605, 2006.

J. Pilet, V. Lepetit and P. Fua. Fast Non-Rigid Surface Detection, Registration, and Realistic Augmentation. Accepted to International Journal of Computer Vision, 2006.

M. Salzmann, S. Ilic and P. Fua. Physically Valid Shape Parameterization for Monocular 3-D Deformable Surface Tracking. British Machine Vision Conference, 2005.

Page 46: Ferns

46

Compromise:

Justification

proportional to

but complete representation of the joint distribution infeasible.

Naive Bayesian ignores the correlation:

We are looking for

Page 47: Ferns

59

Comparison with SIFTRecognition rate

FERNS

SIFT

Frame Index

Number of Inliers

Page 48: Ferns

60

Comparison with SIFTComputation time

• SIFT: 1 ms to compute the descriptor of a keypoint (without including convolution);

• FERNS: 13.5 10-3 ms to classify one keypoint into 200 classes.

Page 49: Ferns

Naive Feature Tracking

Build Classifier

Track Keypoints by Frame-to-Frame Matching

Training Examples

Does not work very well: Prone to drift & tracking failures;

How to select “good” keypoints? Feature harvesting.

Page 50: Ferns

Handling Light Changes

QuickTime™ and a decompressor

are needed to see this picture.

Page 51: Ferns

63

3D Shape Recovery from Monocular Video

M. Salzmann, S. Ilic and P. Fua. Physically Valid Shape Parameterization for Monocular 3-D Deformable Surface Tracking. BMVC05.

Page 52: Ferns

64

Bending modes

Extension modes

Database of Feasible Shapes

PCA

Page 53: Ferns

65

Vision for the America's Cup

CTI Project

Page 54: Ferns

66

Stretchable Material

Page 55: Ferns

67

Robust Bundle Adjustment

• One pixel reprojection

error

Error evolution

Page 56: Ferns

68

Same Camera, 2 Sets of Images

Page 57: Ferns

•Random initialization;•Successive minimizations with lower andlower values for r:

r0 = 1000 r = 1 ou 2 pixels

Optimization

Page 58: Ferns

QuickTime™ and a decompressor

are needed to see this picture.

Page 59: Ferns

Internal Parameters Calibration

0

0

Homography

Page 60: Ferns

External Parameter Calibration

0

0

A1

A2

A3

Relative 3D Pose

World Coordinate System

Absolute 3D Pose

P1

P3P4

P2

Bundle Adjustment on: P1, P2, P3, P4, A1, A2 and A3

Page 61: Ferns

Using the Geometric Calibration Target to Measure Light

Solve for gain, bias of each camera, and irradiance for observed normals.

ntalbedo(m)I(camera, t, m)

m