Image-based Rendering of Real Objects Image-based Rendering of Real Objects with Complex BRDFswith Complex BRDFs
Intensity of One PixelIntensity of One Pixel
s1(, )
q
p
Consider the measured intensity at one pixelConsider the measured intensity at one pixel
I1(1, 1)
as the isotropic point source is moved over as the isotropic point source is moved over the surface.the surface.
I1(, )
Note similarity to Note similarity to
•(Levoy, Hanrahan, 1996) (Levoy, Hanrahan, 1996)
•(Gortler et al, 1996)(Gortler et al, 1996)
Phong Intensity of One Pixel: Phong Intensity of One Pixel: II11((, , ))
This is effectively a 2-D slice of a point’s BRDF except for This is effectively a 2-D slice of a point’s BRDF except for
• ShadowingShadowing
• 1/r1/r22 falloff from the source falloff from the source
Intensity of One Pixel: Intensity of One Pixel: II11((, , ))
This is effectively a 2-D slice of a point’s BRDF except for This is effectively a 2-D slice of a point’s BRDF except for
• ShadowingShadowing
• 1/r1/r22 falloff from the source falloff from the source
Image AcquisitionImage Acquisition
Intensity Over Second SurfaceIntensity Over Second Surface
s1(, )
s2(, )
p
Now, consider moving an isotropic point Now, consider moving an isotropic point source over a second surface and measuring source over a second surface and measuring the intensity of the same pixel: the intensity of the same pixel:
I2(, )I1(, )
I2(, )
II11((, , ) and I) and I22((, , ))
Inner
Sphere:
I1(, )
Outer
Sphere:
I2(, )
Relation Between Intensity MapsRelation Between Intensity Maps
When the surface point p, When the surface point p, s1() and s2() are collinear (in correspondence), the measured pixel intensities are simply related by the relative 1/r2 losses.
s1(, )
s2(, )
p
Depth EstimationDepth Estimation
s1(, )
s2(, )
p()
This correspondence can be expressed as a This correspondence can be expressed as a change of coordinates change of coordinates 22((; ; )) and and
22((; ; ) parameterized by depth ) parameterized by depth .
We can then estimateWe can then estimate by minimizing by minimizing:
O()= [I2(2(), 2 ()) - r2 I1(1,1) ]2d1d1
A Reconstructed Depth MapA Reconstructed Depth Map
143 Images on 143 Images on each surfaceeach surface
Rendering Synthetic Images: Point SourcesRendering Synthetic Images: Point Sources
New light position
Intersection with the sphere
• Intersect light ray throughIntersect light ray through P P with sphere.with sphere.
• Find triangle of light sources Find triangle of light sources containing containing PP..
• Interpolate pixel intensities of Interpolate pixel intensities of images corresponding to the images corresponding to the triangle vertices.triangle vertices.
• For a given image point, there For a given image point, there is a scene point: is a scene point: PP
PP
Rendered ImagesRendered Images
Rendered Image: A Sea ShellRendered Image: A Sea Shell
Isotropic point light source Isotropic point light source located between acquisition located between acquisition spheres.spheres.
Rendered Image: A PearRendered Image: A Pear
• Two light sourcesTwo light sources• Point source to the leftPoint source to the left• 3 by 5 cm area source3 by 5 cm area source to the rightto the right
Video Compositing of Real ObjectsVideo Compositing of Real Objects
Video Frame #567 Radiance Map Frame #567
Video CompositingVideo Compositing
Background Image #2313 Object Image #2313
Video CompositingVideo Compositing
Composite Frame #567 Composite Frame #567
Lighting Sensitive Displays
Shree Nayar Peter Belhumeur Terry Boult
Columbia Yale Lehigh
Computer Vision Laboratory
Columbia University
Sponsor: NSF ITR
Displays Everywhere
But, Displays are Passive
brightness
contrast
display
content
Lighting Sensitive Display (LSD)
• Senses the Environmental Illumination
• Modifies Displayed Content Accordingly
illumination
: Perception
: Reaction
State of the Art
brightness
contrastphotodetector
adjustment
Heijligers 62 ; Thomas 63; Gibson 64; Korda 65; Biggs 65; Szermy 68Newman 72; Constable 78; Fitzgibbon 82; Antwerp 85; Otenstein 93
Display’s Illumination Field
display
content
display
content
(s,t)(u,v)
L(s,t,u,v,
• Wide Range of Sources: Sunlight, Overcast, Halogen, Fluorescent ...
• Arbitrarily Complex : Point/Extended/Multiple Sources, Scene Radiance ...
(s,t)(u,v)
L(s,t,u,v,
Four-Dimensional Ray Manifold
Methods for Sensing the Illumination Field
photodetectorsoptical fibers
hemispherical camera
??
probe video
Compact Hemispherical Illumination Probe
compact wide angle optics
color video camera
neutral density filters
LSD Prototype
Sony 15” LCDFlat Display
HemisphericalProbe Camera
Matting
Wooden Frame
Content Modification : Rendering
• Power Efficiency
• Brighter in Sunlight
• Dimmer Indoors
• Compensation
• Spatially Varying Brightness
• Spatially Varying Color
• Photorealism
• Consistent Colors and Shadings
• Consistent Highlights and Shadows
All Modifications in Real-Time
Rendering Using Explicit Models: 2D+
v
s1
n
s2
O
viewer
source
source
display
rendered image
content: surface
Algorithms: Ray Tracing, Radiosity
Rendering Using Explicit Models: 3D
v
s1
n
s2
O
viewer
source
source
display
rendered image
content:shape, BRDF
Algorithms: Ray Tracing, Radiosity
Image based Rendering
probe cameracapture camera
Off-line Scene Capture
(with Kudelka and Swaminathan)
Efficient Representation and Rendering
Image Bases E
source directions
40
30
1616
Captured Images
i=1 i=4096
I
40
30
16
16
k=1 k=10
bloc
ks x
bas
is
3
0 x
40 x
10
Lighting Coefficient Vectors L
source directionsi=1 i=4096
SVD
Efficient Representation and Rendering
Compressed Coefficient Vectors
Coeff. Bases U
source directions
Coefficient Vectors L
i=1 i=4096
V
q=1 b=200
bloc
ks x
bas
is
3
0 x
40 x
10
SVD
source directionsi=1 i=4096
Real-Time Rendering
Compressed Coefficient Vector
Coeff Eigenvectors U
Illumination Field
Vs
Coefficient Vector
Image Eigenvectors E
Display
I
Compressed Coefficient Vectors V
s
X X X
UVs
Efficient Representation and RenderingCaptured Data
ComputeLocal Subspaces
Local Bases and Coefficients
source direction
Image Reconstruction
Display Illumination Field
Display
4 Gb
10 Mb 8 fps (laptop)
Efficient Representation and RenderingCaptured Data
ComputeLocal Subspaces
Local Bases and Coefficients
source direction
Image Reconstruction
Display Illumination Field
Display
4 Gb
10 Mb 8 fps (laptop)
Face
Still Life: Scene Capture
Still Life
Summary
• Lighting Sensitive Display:
• Senses Environmental Illumination
• Modifies Displayed Content
• Applications:
• Compensation: Computers, PDA’s, Televisions, Billboards
• Photorealism: Digital Art, E-Commerce, Future Homes
Capturing Scenes for Image based Rendering
probe cameracapture camera