real-time projector tracking on complex geometry using ordinary imagery tyler johnson and henry...

Real-Time Projector Tracking on Complex

Geometry Using Ordinary Imagery

Tyler Johnson and Henry FuchsUniversity of North Carolina – Chapel Hill

ProCams June 18, 2007 - Minneapolis, MN

2 Real-Time Projector Tracking

Multi-Projector Display

3 Real-Time Projector Tracking

Dynamic Projector Repositioning

Make new portions of the scene visible

4 Real-Time Projector Tracking

Dynamic Projector Repositioning (2)

Increase spatial resolution or field-of-view

5 Real-Time Projector Tracking

Dynamic Projector Repositioning

Accidental projector bumping

6 Real-Time Projector Tracking

Goal

Given a pre-calibrated projector display, automatically compensate for changes in projector pose while the system is being used

7 Real-Time Projector Tracking

Previous Work

Class Active Passive

Technique Embedded Imperceptible Structured Light

Unmodified Imagery, Fixed Fiducials

References

Cotting04-05 Raskar03, Yang01

Online Projector Display Calibration Techniques

8 Real-Time Projector Tracking

Our Approach

Projector pose on complex geometry from unmodified user imagery without fixed fiducials

Rely on feature matches between projector and stationary camera.

9 Real-Time Projector Tracking

Overview

UpfrontCamera/projector calibrationDisplay surface estimation

At run-time in independent threadMatch features between projector and cameraUse RANSAC to identify false correspondencesUse feature matches to compute projector posePropagate new pose to the rendering

10 Real-Time Projector Tracking

Projector Pose Computation

Display Surface

Camera

Projector

11 Real-Time Projector Tracking

Difficulties

Projector and camera images are difficult to match

Radiometric differences, large baselines etc.

No guarantee of correct matchesNo guarantee of numerous strong features

12 Real-Time Projector Tracking

Feature Matching

Projector Image

Camera Image

P

13 Real-Time Projector Tracking

Feature Matching SolutionPredictive Rendering

Projector Image

Prediction Image Camera Image

14 Real-Time Projector Tracking

Predictive Rendering

Account for the followingProjector transfer functionCamera transfer functionProjector spatial intensity variation• How the brightness of the projector varies with FOV

Camera response to the three projector primaries

CalibrationProject a number of uniform white/color images• see paper for details

15 Real-Time Projector Tracking

Predictive Rendering Steps

Two steps: Geometric Prediction• Warp projector image to correspond with the

camera’s view of the imagery

Radiometric Prediction• Calculate the intensity that the camera will

observe at each pixel

16 Real-Time Projector Tracking

Step 1: Geometric Prediction

Two-Pass RenderingCamera takes place of viewer

Display Surface

Camera

Projector

17 Real-Time Projector Tracking

Step 2: Radiometric Prediction

Pixels of the projector image have been warped to their corresponding location in the camera image.Now, transform the corresponding projected intensity at each camera pixel to take into account radiometry.

18 Real-Time Projector Tracking

Radiometric Prediction (2)

Projector Intensity

(r,g,b)

Predicted Camera Intensity

(i)

Projector Response Projector IntensitySurface

Orientation/DistanceCamera Response

Spatial Intensity Scaling

θ

Proj. COP

r

2

cos

r

bgr III ,,

Projector Image

Prediction Image

19 Real-Time Projector Tracking

Prediction Results

Captured Camera Image Predicted Camera Image

20 Real-Time Projector Tracking

Prediction Results (2)

Errormean - 15.1 intensity levelsstd - 3.3 intensity levels

Contrast Enhanced Difference Image

21 Real-Time Projector Tracking

Video

22 Real-Time Projector Tracking

Implementation

Predictive RenderingGPU pixel shader

Feature detectionOpenCV

Feature matchingOpenCV implementation of Pyramidal KLT Tracking

Pose calculationNon-linear least-squares • [Haralick and Shapiro, Computer and Robot Vision, Vol.

2]• Strictly co-planar correspondences are not degenerate

23 Real-Time Projector Tracking

Matching Performance

Performance using geometric and radiometric prediction

Performance using only geometric prediction

Matching performance over 1000 frames for different types of imagery

Max. 200 feature detected per frame

24 Real-Time Projector Tracking

Tracking Performance

Pose estimation at 27 HzCommodity laptop• 2.13 GHz Pentium M• NVidia GeForce 7800 GTX GO

640x480 greyscale cameraMax. 75 feature matches/frame

Implement in separate thread to guarantee rendering performance

25 Real-Time Projector Tracking

Contribution

New projector display technique allowing rapid and automatic compensation for changes in projector pose

Does not rely on fixed fiducials or modifications to user imageryFeature-based, with predictive rendering used to improve matching reliabilityRobust against false stereo correspondencesApplicable to synthetic imagery with fewer strong features

26 Real-Time Projector Tracking

Limitations

Camera cannot be movedTracking can be lost due to

Insufficient featuresRapid projector motion

Affected by changes in environmental lighting conditionsRequires uniform surface

27 Real-Time Projector Tracking

Future Work

Extension to multi-projector displayWhich features belong to which projector?

Extension to intelligent projector modules

Cameras move with projector

Benefits of global illumination simulation in predictive rendering

[Bimber VR 2006]

28 Real-Time Projector Tracking

Thank You

Funding support: ONR N00014-03-1-0589Funding support: ONR N00014-03-1-0589DARWARS Training Superiority program DARWARS Training Superiority program

VIRTE – Virtual Technologies and Environments programVIRTE – Virtual Technologies and Environments program