realistic rendering of synthetic objects into real …cs4340/documents/realistic-rendering... ·...
TRANSCRIPT
CS4340 Digital Special Effects
Semester 2, 2011/2012
School of Computing National University of Singapore
Realistic Rendering of Synthetic Objects into
Real Scenes
Guest Lecture by Low Kok Lim
2
Goal
To put synthetic objects (computer rendered objects) into
pictures or video of real scenes such that results "look
right"
Need to match
Scale
Camera motion
intrinsic & extrinsic
parameters
Illumination
[Frank Vitz, 2003]
Mystique in X-Men 2
3
real computer
generated real computer
generated
Taken from http://www.virtualcinematography.org/publications/acrobat/BRDF-s2003.pdf
Mr. Smith
from
Matrix
Reloaded
4
Match Illumination
Old (labor-intensive) methods
Manually survey positions of light sources, and instantiate
similar virtual lights to light virtual objects
Photograph a neutral reference object in the scene, and
use it as a guide to manually configure a lighting
environment
Reflection mapping
Cannot easily simulate indirect illumination effects
between real and virtual objects
5
Image-Based Lighting (IBL)
Solves the problem by
"faithfully" recording the
scene radiance
In a High-Dynamic Range
Light Probe Image
Use the recorded scene
radiance to light the
synthetic objects
[Paul Debevec, 2002]
6
Light probe image
A frame of the short film "Rendering with Natural Light"
http://www.debevec.org/RNL/
7
Light probe image
[Debevec1998]
8
Overview of IBL Steps
1. Acquire background photographs or video
2. Acquire and assemble the light probe image
3. Construct light-based model
Map the light probe to an emissive surface surrounding
the scene
4. Identify local scene and model its geometry and
reflectance
5. Render the scene as illuminated by the IBL
environment
6. Postprocess, tone map and composite the renderings
9
Detour
We will come back to the details of the IBL steps later
Need to first understand
High-dynamic range imaging
For faithful recording of scene radiance
Global illumination
For realistic rendering of synthetic objects and part of real
scene
10
High Dynamic Range Imaging (HDRI)
11
Motivation
Ordinary cameras cannot record wide
range of scene radiance in one image
Typically only 8-11 stops (EV)
Solution: Take multiple images of different exposures
(different exposure times) and "combine" them
Multiple exposures HDR image
Tone-mapped image Images from http://www.cambridgeincolour.com/tutorials/high-dynamic-range.htm
12
Results
Combining the multiple exposures, we get
Irradiance at each pixel (unknown scale)
The HDR image
Camera response function
R, G, B channels are generally different
13
Example
Exposures from 30 sec to 1/1000 sec, at 1-stop
increment
[Devebec1997]
14
Example
Response functions of a Fuji 100 ASA negative film
15
Example
The HDR image (the
false colors show
relative radiance
values)
Dynamic range about
25,000:1 (>14 stops)
[Devebec1997]
16
Example
Tone-mapped
image
Input images
Images from http://en.wikipedia.org/wiki/Tone_mapping
17
Application of HDRI
Recovery of surface BRDF
Image processing and photography
Exposures after image acquisition
Images from http://en.wikipedia.org/wiki/High_dynamic_range_image
18
Application of HDRI
Blurring (e.g. simulating out-of-focus)
Motion Blur
Images from http://en.wikipedia.org/wiki/High_dynamic_range_image
19
Application of HDR Images
More realistic
rendering
HDR rendering
supported in
hardware
Images from http://en.wikipedia.org/wiki/High_dynamic_range_rendering
20
HDRI References
Books
High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting by Erik Reinhard, Greg Ward, Sumanta Pattanaik, and Paul Debevec, 2005
Software Tools
HDR Shop: http://gl.ict.usc.edu/HDRShop/
Photoshop CS2: http://www.adobe.com/products/photoshop/
Photomatix: http://www.hdrsoft.com/
HDR Image Formats
ILM OpenEXR (.exr): http://www.openexr.com/
RADIANCE RGBE (.hdr or .rgbe): http://radsite.lbl.gov/radiance/
Papers
[Devebec1997]
Paul Devebec et al., "Recovering High Dynamic Range Radiance Maps from Photographs," SIGGRAPH '97
[Mitsunaga1999]
Tomoo Mitsunaga et al., "Radiometric Self Calibration," CVPR '99
21
Global Illumination
22
Global Illumination
Evaluating light reflected from a point x by taking into
consideration all illumination that arrives at the point
Figure by Frédo Durand, MIT
23
The Rendering Equation
Mathematical formulation of global illumination
Integrate over the
hemisphere around x
x
24
The Rendering Equation
Cannot be evaluated analytically
In practice, send tons of random rays (Monte Carlo
methods)
It is recursive
To evaluate Lref (x, ref), we need to evaluate Lin (x', in),
and so on
25
Some Lighting Effects
Color bleeding caused by
diffuse-to-diffuse interactions
Caustics caused by
focusing of light
[Henrik Jensen]
26
Global Illumination Algorithms
Ray-tracing approach
Whitted ray tracing [Whitted1980]
Distributed ray tracing [Cook1984]
Path tracing [Kajiya1986]
Two-pass ray tracing [Arvo1986]
Photon mapping [Jensen1995]*
not a complete GI algorithm
Finite-element approach
Radiosity [Goral1984]
27
Path Tracing
For each pixel, shoot multiple random primary rays
At each intersection, only a secondary ray is shot
The secondary ray can be in any direction, not just sampled from
the specular lobe
Each primary ray from the eye and its subsequent secondary
rays form a light path
The ray tree has
branching factor of one
28
Path Tracing
Simulates complete global illumination
But at very high computational cost
Indirect illumination, such as caustics, exhibits high variance
10 paths / pixel
[Henrik Jensen]
29
Radiosity
Implements only diffuse-diffuse interactions
Scene is discretized into patches, and interaction
between patches are considered
Global illumination solution is computed by solving a set
of linear equations
Solution is view independent and consists of a constant
radiosity (W/m2) for every patch in the scene
Once solution is computed, it can be viewed from any view
30
Radiosity Images
[Cornell University Program of Computer Graphics]
The Cornell Box
31
Global Illumination References
Books
Advanced Global Illumination, Second Edition by Philip Dutré, Kavita Bala, Philippe Bekaert, 2006
Physically Based Rendering: From Theory to Implementation by Matt Pharr & Greg Humphreys, 2004
Realistic Ray Tracing, 2nd Edition by Peter Shirley & R. Keith Morley, 2003
Realistic Image Synthesis Using Photon Mapping by Henrik Wann Jensen, 2001
Principles of Digital Image Synthesis by Andrew S. Glassner, 1995
Radiosity and Realistic Image Synthesis by Michael F. Cohen & John R. Wallace, 1993
32
Global Illumination References
Non-Commercial Renderers
YafRay: http://www.yafray.org/
RADIANCE: http://radsite.lbl.gov/radiance/
PBRT (Physically-Based Raytracer): http://www.pbrt.org/
POV-Ray: http://www.povray.org/ (v3.6 does not support HDR IBL)
MegaPOV: http://megapov.inetart.net/
Indigo Renderer: http://www.indigorenderer.com/
Commercial Renderers
Mental Ray: http://www.mentalimages.com/
Pixar's RenderMan: https://renderman.pixar.com/
Maxwell Renderer: http://www.maxwellrender.com/
33
Global Illumination References
Papers
Rendering equation [Kajiya1986]
J. T. Kajiya, "The Rendering Equation," SIGGRAPH '86
Whitted ray tracing [Whitted1980]
T. Whitted, "An Improved Illumination Model for Shaded Display," Comm. ACM, 23(6):343-349, 1980
Distributed ray tracing [Cook1984]
R. Cook et al., "Distributed Ray Tracing," SIGGRAPH '84
Radiosity [Goral1984]
C. Goral et al., "Modeling the Interaction of Light Between Diffuse Surfaces," SIGGRAPH '84
Path tracing [Kajiya1986]
Two-pass ray tracing [Arvo1986]
J. Arvo, "Backwards Ray Tracing," Developments in Ray Tracing, SIGGRAPH '86 Course Notes #12
Photon mapping [Jensen1995]
H. W. Jensen et al., "Photn Maps in Bidirectional Monte Carlo Ray Tracing of Complex Objects," Computer & Graphics 19(2):215-224, 1995
34
Image-Based Lighting (IBL)
35
Overview of IBL Steps
1. Acquire background photographs or video
2. Acquire and assemble the light probe image
3. Construct light-based model
Map the light probe to an emissive surface surrounding
the scene
4. Identify local scene and model its geometry and
reflectance
5. Render the scene as illuminated by the IBL
environment
6. Postprocess, tone map and composite the renderings
36
Use this example
to demonstrate
the IBL steps
[Debevec1998]
Example
Synthetic objects
Background photo
37
1. Acquire Background Photograph
38
2. Acquire Light Probe Image (HDR)
Light probe image The pattern is for camera calibration.
39
3. Construct Light-Based Model
Need to have an
approximate 3D
model of the
environment
40
Separation of Scene
41
4. Identify Local Scene
Model its geometry and estimate its reflectance
References
Yizhou Yi et al., "Inverse Global Illumination: Recovering Reflectance Models of Real Scenes from Photographs," SIGGRAPH '99
Paul Devebec et al., "Estimating Surface Reflectance Properties of s Complex Scene Under Natural Illumination," ACM Transactions on Graphics, 2005
42
5. Render Local and Synthetic Scene
Using light-based model as lighting
43
6. Compositing
When estimate of local scene reflectance is accurate
Lfinal = Llocal+synthetic + (1) Lbackground
Lfinal
Lbackground
Llocal+synthetic
(Tone-mapped)
44
6. Compositing using Differential Rendering
When estimate of local scene reflectance is not accurate
Lfinal = Llocal+synthetic + (1) (Lbackground + Llocal+synthetic Llocal )
Llocal+synthetic
(Tone-mapped)
Llocal (Tone-mapped)
Lfinal
45
Other Examples
46
Other Examples
47
IBL References
Books and Notes
High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting by Erik Reinhard, Greg Ward, Sumanta Pattanaik, and Paul Debevec, 2005
HDRI and Image-Based Lighting by Paul Devebec et al., SIGGRAPH 2003 Course #19, http://www.debevec.org/IBL2003/
Software Tools
HDR Shop: http://gl.ict.usc.edu/HDRShop/
Renderers
As listed in the global illumination references
Papers
[Devebec1998]
Paul Devebec, “Rendering Synthetic Objects into Real Scenes: Bridging Traditional and Image-based Graphics with Global Illumination and High Dynamic Range Photography," SIGGRAPH '98
48
Semi-Automatic Approach
Semi-Automatic Approach
From a single LDR photo, semi-automatically estimate
Geometry
Camera parameters
Surface properties
Lighting info
49
System Overview
Scene synthesis
Object insertion Input image
Scene authoring
50
System Overview
Scene authoring
51
Bounding geometry
Occluding geometry
Supporting geometry
Light sources Spectral
matting[Levin et
al. ’09]
Manual input
Manual input
Spatial Layout
[Hedau et al. ’09]
52
Textured billboard
(with transparency) Bounding cuboid
Extruded polygon
Area lights
53
System Overview
Scene synthesis
54
Scene Synthesis
Textured billboard Bounding
cuboid
Extruded polygon
Area lights
Physical scene model Rendered scene
Auto-material estimation
&
Auto-lighting refinement
Match input image and rendered scene 55
Material Estimation
Input + geometry
Direct
Reflectance
Retinex-like
decomposition
56
Input image Physical model
Geometry w/
materials
Lights
Lighting Estimation
57
Input image Rendered (initial)
Lighting Estimation
Rendered (final)
58
Lighting Estimation
Result using initial
lights
Result using refined
lights
59
External Lighting
60
Shaft bounding box
Source bounding box
External Lighting
61
Shaft direction
External Lighting
Shadow matting via [Guo et al. ‘11]
62
63
64
System Overview
Object insertion
65
Inserting Objects
Load scene into 3D modeler
Insert objects, animations
Render with any physically based renderer
66
Final Composite
Additive differential technique [Debevec ‘98]
67
68
Results
69
70
71
72
73
74
75
References
Kevin Karsch, Varsha Hedau, David Forsyth, Derek Hoiem, "Rendering Synthetic Objects into Legacy Photographs," SIGGRAPH Asia 2011
http://kevinkarsch.com/publications/sa11.html
76
77
The End