online tracking of outdoor lighting variations for augmented reality with moving cameras yanli liu...
Post on 28-Dec-2015
214 Views
Preview:
TRANSCRIPT
Online Tracking of Outdoor Lighting Variations for Augmented Reality
with Moving Cameras
Yanli Liu1,2 and Xavier Granier2,3,4
1: College of Computer Science, Sichuan University, P.R.China
2: INRIA Bordeaux Sud-Ouest, France
3:LP2N (CNRS, Univ. Bordeaux, Institut d'Optique)
4:LaBRI (CNRS, University of Bordeaux)
MotivationTwo consistency
Geometric consistency Devices Camera position
GPS, UWB, Omnisense WiFi, cell information
Camera pose Linear accelerometers
Tracking via computer vision [Cornelis et al. LNCS 2001, zhang et al. CVPR 2007, Xu et al. image
and Vision Computing 2008]
Illumination consistency outdoor lighting is largely dependent on
weather and time
MotivationTwo problems
Online process first step toward real-time solutions
Moving viewpoints Handhold
camera jitter
Previous WorkMarkers or lighting probes [Debevec Siggraph’ 98,
Agusanto ISMAR’03, Kanbara ICPR’04, Hensley I3D’07]
too dense sampling our method does not require any
supplemental devices
Debevec Siggraph’ 98
Previous WorkThree components of shading
BRDF geometry lighting
Fix other one or two components [Wang PG’02, Li ICCV’03, Hara PAMI’05, Andersen ICPR’06, Sun ICCV’09]
3D reconstruction controlled environment (indoor or lab)
[Wang PG’02]original image rendered image
Previous WorkTime-lapse outdoor video analysis [Sunkavalli
Siggraph’07, Sunkavalli CVPR 08]
take whole video sequence as input Post-processing
[Sunkavalli Siggraph’07]
Previous WorkLearning based outdoor illumination
estimation [Liu TVC’09, Liu CAVW’10, Xing C&G’11]
offline stage learning fixed viewpoint
Liu CAVW’10
moving viewpoints
Our MethodKey ideas
Tracking illumination variation by tracking feature points
Feature points tracking is error prone.
Select reliable feature points using global illumination constraint and spatial-temporal coherency.
Outdoor lighting [Sunkavalli SIG’07, Sunkavalli CVPR 08, Madsen InTech 2010]
the sunlight directional light colored intensity sun direction
the skylight ambient light colored intensity
( )sun tL
( )sky tL
( )l t
Illumination and BRDF model
Illumination and BRDF modelNeutral reflection model [Lee PAMI’90, Montoliu
LNCS’05, Eibenberger ICIP 2010, ICCV 2011]
the color of the specular reflection is the same as the color of the incident lighting.
Phong-like model
, ( )( ) [ , ( ) ] ( )
( )
pm sunp pp pp pp
skyp
n h tt k n l t tsI L
tL
System Initialization Tracking illumination variation by tracking
feature points 3D geometry vs normals planar feature points
KLT feature-points
clustered feature-points
first frame
plane segmentation[Hoiem IJCV’07]
mean-shift color segmentation
threshold-based Shadow detection
System InitializationBRDF initialization
pixels difference at in sun lit regions depend on specular parameters and :
Assuming piecewise constant , and
Spatially varying diffuse
p
pk pm
,
2
( , )i j
diffdiffpi i p
j p
E k m I I
p
2, 1,
(2) (1)
, ,[ ]p p
p p
diffp pp
diff m msunp p pp
I I I
n h n hI k L
pk pm 1sunL
2 2
, 1 ( ) ( )( 1) ( 1)sun skysmooth sun sky
t t t L t Lt tE L L
Energy function
Outdoor lighting is nearly constant during time intervals less than 1/5 second.
control the smooth degree of skylight
Alignment-based weight
, 1( ) (1 ) ( )data smootht tE t tE E
,
2
, ,,
( )i j
datappi j p
i j p
t I IE
2( )/
, ,,
1= ijpI I
i j pi j
ez
Tracking Lighting Variation with Reliable Feature Points
Tracking Reliable Features and Their AttributesFeature points labeling
Three attributes: Normal (plane, homography matrix) BRDF parameters Shadow situation
Spatial & temporal coherency
Tracking Reliable Features and Their AttributesFeature points labeling
t -1 t
paired point is labeled in 1t
current point is not in shadow
22
, ,( ) (1 )i j p i p i jpd q H p I I
, ,i ik m
ppq
compute lighting
,i ik m
tL
p
Results and DiscussionQuantitative results
PC: Intel i7 2.67GHz and 6GB RAMMATLABVideo resolution 640 480
Average fps and average number of feature points estimated on 1,000 frames
Results and discussionQuantitative results
Average percentage of different steps in total computational cost
ConclusionFully image-based pipeline
online tracking of lighting variations of outdoor videos.
Manages lighting changes and misalignment of feature points
Ensure a stable estimation on a sparse set feature points.
Limitations and Future WorkRough shadow detection
3D reconstruction vs shadow detection Sun-lit features
Initialization automatic initialization: easy but may fail in
some cases manual initialization: may be tedious for a
non-expert user. Semi-assisted initialization
Limitations and Future WorkTracking independently on R, G, and B
channels priori model of outdoor illumination color or
spectra difficult to optimization
The first step of a long march to a seamless and real-time AR solution for videos with moving viewpoints.
top related