Interactive Virtual Relighting and Interactive Virtual Relighting and Remodelling of Real ScenesRemodelling of Real Scenes

C. LoscosC. Loscos11, MC. Frasson, MC. Frasson1,21,2,G. Drettakis,G. Drettakis11,,

B. WalterB. Walter11, X. Granier, X. Granier11, P. Poulin, P. Poulin22

(1)(1) iMAGIS* - GRAVIR/IMAG - INRIA Rhône-Alpes iMAGIS* - GRAVIR/IMAG - INRIA Rhône-Alpes

* iMAGIS is a joint project of CNRS/INRIA/UJF/INPG* iMAGIS is a joint project of CNRS/INRIA/UJF/INPG

(2)(2) Département d ’informatique et de recherche opérationnelle, Département d ’informatique et de recherche opérationnelle,

Université de MontréalUniversité de Montréal

MAGISMAGISii

MotivationMotivation

Interior design

Geometric modificationChanges in lighting

MAGISMAGISii

MotivationMotivation

Goal: interactive systemGoal: interactive system simple capture processsimple capture process interactive ( ~ 1 sec. per frame)interactive ( ~ 1 sec. per frame) modification of lightingmodification of lighting modification of geometrymodification of geometry

MAGISMAGISii

MotivationMotivation

We have to:We have to: create a simple model of the real scenecreate a simple model of the real scene

geometrygeometry approximate reflectanceapproximate reflectance

represent real global illuminationrepresent real global illumination develop interactive methods for modificationsdevelop interactive methods for modifications

Goal is to be convincing, not highly accurateGoal is to be convincing, not highly accurate

MAGISMAGISii

Previous WorkPrevious Work

Geometric reconstructionGeometric reconstruction vision methods [vision methods [Faugeras et al. 97, ...Faugeras et al. 97, ...] (] (RealiseRealise)) constraint-based systems constraint-based systems

[[Debevec et al. 96Debevec et al. 96, , Poulin et al. 98Poulin et al. 98] ] software: Photomodeler, etc.software: Photomodeler, etc.

Reflectance recoveryReflectance recovery e.g., [e.g., [Sato et al. 97, Ward92, Debevec98, Sato et al. 97, Ward92, Debevec98,

Yu et al. 98, etcYu et al. 98, etc].].

MAGISMAGISii

Previous WorkPrevious Work

Real-time direct shadowsReal-time direct shadows real point light source [real point light source [State et al. 96State et al. 96]]

Common global illuminationCommon global illumination non-interactivenon-interactive

[[Nakamae et al. 86, Fournier et al. 93, Nakamae et al. 86, Fournier et al. 93, Jancène et al. 95, Debevec 98, Jancène et al. 95, Debevec 98, Yu et al. 98, Yu et al. 99Yu et al. 98, Yu et al. 99]]

interactiveinteractive [[Drettakis et al. 97, Loscos et al. 98Drettakis et al. 97, Loscos et al. 98]]

MAGISMAGISii

Algorithm OverviewAlgorithm Overview

InputInput

Pre-processPre-process

Interactive modificationInteractive modification

MAGISMAGISii

Algorithm Overview - AssumptionsAlgorithm Overview - Assumptions

Single viewpointSingle viewpoint

Diffuse assumptionDiffuse assumption

Lighting:Lighting:

direct lighting: ray castingdirect lighting: ray casting indirect lighting: hierarchical radiosityindirect lighting: hierarchical radiosity

radiosity radiosity == reflectance reflectance xx ( direct light ( direct light ++ indirect light ) indirect light )

MAGISMAGISii

Simple Input ProcessSimple Input Process

Geometric reconstructionGeometric reconstruction several (4-5) images from several (4-5) images from differentdifferent viewpoints viewpoints geometric modelling using “geometric modelling using “Rekon”Rekon” [ [Poulin et al. 98Poulin et al. 98]]

Reflectance reconstruction Reflectance reconstruction several (5-7) images from a several (5-7) images from a singlesingle viewpoint viewpoint different lighting conditions: different lighting conditions:

single light source at different positionssingle light source at different positions ““radiance images”radiance images”

MAGISMAGISii

InputInput

Radiance images from single viewpointRadiance images from single viewpoint combining multiple images reduces artefacts of combining multiple images reduces artefacts of

estimationestimation

different lighting conditions

MAGISMAGISii

Pre-processPre-process

Computation of approximate diffuse reflectance Computation of approximate diffuse reflectance pixel by pixelpixel by pixel

compute individual reflectance imagescompute individual reflectance images merge reflectance images using confidence values merge reflectance images using confidence values

Initialise lighting systemInitialise lighting system data structuredata structure hierarchical radiosity systemhierarchical radiosity system

MAGISMAGISii

Reflectance ComputationReflectance Computation

For each radiance imageFor each radiance image

photograph reflectance

reflectance = radiosity / ( direct light + indirect light )

MAGISMAGISii

Confidence ImagesConfidence Images

Estimate confidenceEstimate confidence confidence ~ quality of reflectance estimateconfidence ~ quality of reflectance estimate create a confidence image per light source create a confidence image per light source

positionposition

Begin with confidence = VisibilityBegin with confidence = Visibility low in shadow regions low in shadow regions

Filtering process to remove unwanted effectsFiltering process to remove unwanted effects low for outliers (specular effects, light tripod)low for outliers (specular effects, light tripod)

MAGISMAGISii

Merged Reflectance ComputationMerged Reflectance Computation

x

x

avg.

reflectance confidence

merged reflectance

MAGISMAGISii

Interactive Modification: Shadow ReprojectionInteractive Modification: Shadow Reprojection

Direct illumination: pixel by pixelDirect illumination: pixel by pixel

Indirect illumination: optimised radiosity solutionIndirect illumination: optimised radiosity solution

pixel

Reflectance

Indirect lighting

Direct lighting

MAGISMAGISii

Shadow Re-projectionShadow Re-projection

photographphotograph simulated

MAGISMAGISii

Add/move/remove object (virtual or real)Add/move/remove object (virtual or real)

Visible surface changes: pixel by pixel local updateVisible surface changes: pixel by pixel local update project bounding box of dynamic objectproject bounding box of dynamic object

localise directly affected pixels localise directly affected pixels

original

object insertion

MAGISMAGISii

Add/move/remove object (virtual or real)Add/move/remove object (virtual or real)

Direct lighting updates: shaft structureDirect lighting updates: shaft structure localisation of visibility changes (shadows)localisation of visibility changes (shadows) accelerate visibility computation (blocker lists)accelerate visibility computation (blocker lists)

original

object insertion

MAGISMAGISii

Add/move/remove object (virtual or real)Add/move/remove object (virtual or real)

Indirect illumination computed by a radiosity Indirect illumination computed by a radiosity solution (optimised by the shaft structure)solution (optimised by the shaft structure)

Example: moving objectExample: moving object

Position 1 Position 2

MAGISMAGISii

Real Object RemovalReal Object Removal

MAGISMAGISii

Removing Real ObjectsRemoving Real Objects

Use of the reflectance image (lighting effects Use of the reflectance image (lighting effects removed) to generate new texturesremoved) to generate new textures

reflectance images

MAGISMAGISii

Light Source ModificationLight Source Modification

Insertion of a virtual light sourceInsertion of a virtual light source computation for every pixelcomputation for every pixel

new form-factorsnew form-factors new visibilitynew visibility

Indirect illumination: radiosity solutionIndirect illumination: radiosity solution

MAGISMAGISii

Lighting ModificationLighting Modification

Insertion of virtual lightInsertion of virtual lightOriginal virtual lightingOriginal virtual lighting

MAGISMAGISii

VideoVideo

MAGISMAGISii

ConclusionConclusion

Input Input data simple to acquiredata simple to acquire

Pre-processPre-process data structures optimised for fast updatesdata structures optimised for fast updates

Interactive modificationInteractive modification add and move virtual objectsadd and move virtual objects remove real objects remove real objects relightingrelighting

MAGISMAGISii

Future WorkFuture Work

Improve reflectance computationImprove reflectance computation use of high dynamic range images use of high dynamic range images

(instead of RGB)(instead of RGB) better control of indirect illuminationbetter control of indirect illumination

Allow motion of real objectsAllow motion of real objects

Faster: parallel computationFaster: parallel computation

MAGISMAGISii

Future WorkFuture Work

Remove restrictionsRemove restrictions diffuse reflectance [diffuse reflectance [Yu et al. 99Yu et al. 99]] fixed view-pointfixed view-point