![Page 1: Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras](https://reader035.vdocument.in/reader035/viewer/2022062322/568145b0550346895db2ae69/html5/thumbnails/1.jpg)
Online Tracking of Outdoor Lighting Variations for Augmented Reality
with Moving Cameras
Yanli Liu1,2 and Xavier Granier2,3,4
1: College of Computer Science, Sichuan University, P.R.China
2: INRIA Bordeaux Sud-Ouest, France
3:LP2N (CNRS, Univ. Bordeaux, Institut d'Optique)
4:LaBRI (CNRS, University of Bordeaux)
![Page 2: Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras](https://reader035.vdocument.in/reader035/viewer/2022062322/568145b0550346895db2ae69/html5/thumbnails/2.jpg)
Augmented reality mobility
Motivation
![Page 3: Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras](https://reader035.vdocument.in/reader035/viewer/2022062322/568145b0550346895db2ae69/html5/thumbnails/3.jpg)
MotivationTwo consistency
Geometric consistency Devices Camera position
GPS, UWB, Omnisense WiFi, cell information
Camera pose Linear accelerometers
Tracking via computer vision [Cornelis et al. LNCS 2001, zhang et al. CVPR 2007, Xu et al. image
and Vision Computing 2008]
Illumination consistency outdoor lighting is largely dependent on
weather and time
![Page 4: Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras](https://reader035.vdocument.in/reader035/viewer/2022062322/568145b0550346895db2ae69/html5/thumbnails/4.jpg)
MotivationTwo problems
Online process first step toward real-time solutions
Moving viewpoints Handhold
camera jitter
![Page 5: Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras](https://reader035.vdocument.in/reader035/viewer/2022062322/568145b0550346895db2ae69/html5/thumbnails/5.jpg)
Previous WorkMarkers or lighting probes [Debevec Siggraph’ 98,
Agusanto ISMAR’03, Kanbara ICPR’04, Hensley I3D’07]
too dense sampling our method does not require any
supplemental devices
Debevec Siggraph’ 98
![Page 6: Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras](https://reader035.vdocument.in/reader035/viewer/2022062322/568145b0550346895db2ae69/html5/thumbnails/6.jpg)
Previous WorkThree components of shading
BRDF geometry lighting
Fix other one or two components [Wang PG’02, Li ICCV’03, Hara PAMI’05, Andersen ICPR’06, Sun ICCV’09]
3D reconstruction controlled environment (indoor or lab)
[Wang PG’02]original image rendered image
![Page 7: Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras](https://reader035.vdocument.in/reader035/viewer/2022062322/568145b0550346895db2ae69/html5/thumbnails/7.jpg)
Previous WorkTime-lapse outdoor video analysis [Sunkavalli
Siggraph’07, Sunkavalli CVPR 08]
take whole video sequence as input Post-processing
[Sunkavalli Siggraph’07]
![Page 8: Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras](https://reader035.vdocument.in/reader035/viewer/2022062322/568145b0550346895db2ae69/html5/thumbnails/8.jpg)
Previous WorkLearning based outdoor illumination
estimation [Liu TVC’09, Liu CAVW’10, Xing C&G’11]
offline stage learning fixed viewpoint
Liu CAVW’10
moving viewpoints
![Page 9: Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras](https://reader035.vdocument.in/reader035/viewer/2022062322/568145b0550346895db2ae69/html5/thumbnails/9.jpg)
Our MethodKey ideas
Tracking illumination variation by tracking feature points
Feature points tracking is error prone.
Select reliable feature points using global illumination constraint and spatial-temporal coherency.
![Page 10: Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras](https://reader035.vdocument.in/reader035/viewer/2022062322/568145b0550346895db2ae69/html5/thumbnails/10.jpg)
Outdoor lighting [Sunkavalli SIG’07, Sunkavalli CVPR 08, Madsen InTech 2010]
the sunlight directional light colored intensity sun direction
the skylight ambient light colored intensity
( )sun tL
( )sky tL
( )l t
Illumination and BRDF model
![Page 11: Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras](https://reader035.vdocument.in/reader035/viewer/2022062322/568145b0550346895db2ae69/html5/thumbnails/11.jpg)
Illumination and BRDF modelNeutral reflection model [Lee PAMI’90, Montoliu
LNCS’05, Eibenberger ICIP 2010, ICCV 2011]
the color of the specular reflection is the same as the color of the incident lighting.
Phong-like model
, ( )( ) [ , ( ) ] ( )
( )
pm sunp pp pp pp
skyp
n h tt k n l t tsI L
tL
![Page 12: Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras](https://reader035.vdocument.in/reader035/viewer/2022062322/568145b0550346895db2ae69/html5/thumbnails/12.jpg)
System Initialization Tracking illumination variation by tracking
feature points 3D geometry vs normals planar feature points
KLT feature-points
clustered feature-points
first frame
plane segmentation[Hoiem IJCV’07]
mean-shift color segmentation
threshold-based Shadow detection
![Page 13: Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras](https://reader035.vdocument.in/reader035/viewer/2022062322/568145b0550346895db2ae69/html5/thumbnails/13.jpg)
System InitializationBRDF initialization
pixels difference at in sun lit regions depend on specular parameters and :
Assuming piecewise constant , and
Spatially varying diffuse
p
pk pm
,
2
( , )i j
diffdiffpi i p
j p
E k m I I
p
2, 1,
(2) (1)
, ,[ ]p p
p p
diffp pp
diff m msunp p pp
I I I
n h n hI k L
pk pm 1sunL
![Page 14: Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras](https://reader035.vdocument.in/reader035/viewer/2022062322/568145b0550346895db2ae69/html5/thumbnails/14.jpg)
2 2
, 1 ( ) ( )( 1) ( 1)sun skysmooth sun sky
t t t L t Lt tE L L
Energy function
Outdoor lighting is nearly constant during time intervals less than 1/5 second.
control the smooth degree of skylight
Alignment-based weight
, 1( ) (1 ) ( )data smootht tE t tE E
,
2
, ,,
( )i j
datappi j p
i j p
t I IE
2( )/
, ,,
1= ijpI I
i j pi j
ez
Tracking Lighting Variation with Reliable Feature Points
![Page 15: Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras](https://reader035.vdocument.in/reader035/viewer/2022062322/568145b0550346895db2ae69/html5/thumbnails/15.jpg)
Tracking Reliable Features and Their AttributesFeature points labeling
Three attributes: Normal (plane, homography matrix) BRDF parameters Shadow situation
Spatial & temporal coherency
![Page 16: Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras](https://reader035.vdocument.in/reader035/viewer/2022062322/568145b0550346895db2ae69/html5/thumbnails/16.jpg)
Tracking Reliable Features and Their AttributesFeature points labeling
t -1 t
paired point is labeled in 1t
current point is not in shadow
22
, ,( ) (1 )i j p i p i jpd q H p I I
, ,i ik m
ppq
compute lighting
,i ik m
tL
p
![Page 17: Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras](https://reader035.vdocument.in/reader035/viewer/2022062322/568145b0550346895db2ae69/html5/thumbnails/17.jpg)
Results and DiscussionQuantitative results
PC: Intel i7 2.67GHz and 6GB RAMMATLABVideo resolution 640 480
Average fps and average number of feature points estimated on 1,000 frames
![Page 18: Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras](https://reader035.vdocument.in/reader035/viewer/2022062322/568145b0550346895db2ae69/html5/thumbnails/18.jpg)
Results and discussionQuantitative results
Average percentage of different steps in total computational cost
![Page 19: Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras](https://reader035.vdocument.in/reader035/viewer/2022062322/568145b0550346895db2ae69/html5/thumbnails/19.jpg)
Results and DiscussionVisual results
Building scene Wall scene
![Page 20: Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras](https://reader035.vdocument.in/reader035/viewer/2022062322/568145b0550346895db2ae69/html5/thumbnails/20.jpg)
ConclusionFully image-based pipeline
online tracking of lighting variations of outdoor videos.
Manages lighting changes and misalignment of feature points
Ensure a stable estimation on a sparse set feature points.
![Page 21: Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras](https://reader035.vdocument.in/reader035/viewer/2022062322/568145b0550346895db2ae69/html5/thumbnails/21.jpg)
Limitations and Future WorkRough shadow detection
3D reconstruction vs shadow detection Sun-lit features
Initialization automatic initialization: easy but may fail in
some cases manual initialization: may be tedious for a
non-expert user. Semi-assisted initialization
![Page 22: Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras](https://reader035.vdocument.in/reader035/viewer/2022062322/568145b0550346895db2ae69/html5/thumbnails/22.jpg)
Limitations and Future WorkTracking independently on R, G, and B
channels priori model of outdoor illumination color or
spectra difficult to optimization
The first step of a long march to a seamless and real-time AR solution for videos with moving viewpoints.
![Page 23: Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras](https://reader035.vdocument.in/reader035/viewer/2022062322/568145b0550346895db2ae69/html5/thumbnails/23.jpg)
Thanks for your attention!
Questions?