a theory of locally low dimensional light transport dhruv mahajan (columbia university) ira...
Post on 21-Dec-2015
215 Views
Preview:
TRANSCRIPT
A Theory of Locally Low Dimensional Light Transport
Dhruv Mahajan (Columbia University)
Ira Kemelmacher-Shlizerman (Weizmann Institute)
Ravi Ramamoorthi (Columbia University)
Peter Belhumeur (Columbia University)
Image Relighting
Ng et al 2003
Relighting – Linear Combination
1l 2l Nl .......=
Images lit by directional light sources
Lighting Intensities
Nimeroff et al 94
Dorsey 95
Hallinan 94
Relighting – Matrix Vector Multiply
.......=
...
1l 2l Nl
1T 2T NT L
=
Input Lighting(Unfolded Cubemap)
Output ImageVector Transport Matrix
T L
1b
2b
Mb
...B
Light transport matrix dimensions
512 x 512 images
6 x 32 x 32 = 6144 cubemap lighting
Multiplication / Relighting cost
Approximately 1010 computations per frame
Multiplication intractable in real time
Need to compress the light transport
Light Transport – Computational Cost
Light Transport – SVD
1T 2T NTTransport Matrix
....... .......
....
12
3
KU S
12
3
K
....1w 2w 3w Kw
L
LightingVector
Relit Image
KP KK NKEigenvalues
1N
Hallinan 94
VT
VT
Basis Images
Projection Weights
Light Transport – SVD
- Global DimensionalityK
K
99
Large
....... .......
....
12
3
K
V
T
1T 2T NTTransport Matrix
KP KK NKEigenvalues
En
erg
y (i
n %
)
No. of Eigenvalues
Computation still intractable
Global Dimensionality
32326700 K6144
Locally Low Dimensional Light Transport
.......
p pixels
p rows
SVD
1 2 3 n....1w 2w 3w nw
Locally Low Dimensional Transport Npn ,4096p 6144N 50n Lighting Resolution
Dimensionality of the patchn
1T 2T NTTransport Matrix
Previous Work
Blockwise PCA – Nayar et al. 04 Image divided in to fixed size
square patches
Each patch compressed using PCA
Clustered PCA – Sloan et al. 03 Object divided in to fixed
number of clusters
Each cluster compressed using PCA
Previous Work
Surface light fields Nishino et al. 01 Chen et al. 02
General reflectance fields Matusik et al. 02 Garg et al. 06
Compression JPEG, MPEG
No Theoretical Analysis
Dimensionality vs Patch Size?
Dimensionality vs Material Properties?
Dimensionality vs Global Effects ?
Local Light Transport Dimensionality
Analysis of local light transport dimensionality
P
Dimensionality
Co
st
Patch Area1
Local Light Transport Dimensionality
Analysis of local light transport dimensionality
Dimensionality
Co
st
Patch Area
2 x 2
Local Light Transport Dimensionality
Analysis of local light transport dimensionality
Dimensionality
Co
st
Patch Area
Local Light Transport Dimensionality
Analysis of local light transport dimensionality
Dimensionality
Co
st
Patch Area
Local Light Transport Dimensionality
Analysis of local light transport dimensionality
Dimensionality
Co
st
Patch Area
Local Light Transport Dimensionality
Analysis of local light transport dimensionality
Dimensionality
Co
st
Patch Area
Rendering Cost
Theoretical analysis of rendering cost
Co
st
Patch Area
Overhead cost for rendering
Dimensionality
Overhead Cost
....1w 2w 3w nw
Global Lighting
Dimensionality cost = number of bases
Overhead Cost = Projection Weights
Co
st
Patch Area
Rendering Cost
Theoretical analysis of rendering cost
Co
st
Patch Area
Overhead cost for rendering
P
Rendering Cost
Theoretical analysis of rendering cost
Co
st
Patch Area
Overhead cost for rendering
Rendering Cost
Theoretical analysis of rendering cost
Co
st
Patch Area
Overhead cost for rendering
Rendering Cost
Theoretical analysis of rendering cost
Co
st
Patch Area
Overhead cost for rendering
Rendering Cost
Theoretical analysis of rendering cost
Co
st
Patch Area
Overhead cost for rendering
Rendering Cost
Theoretical analysis of rendering cost
Co
st
Patch Area
Overhead cost for rendering
Rendering Cost
Theoretical analysis of rendering cost
Co
st
Patch Area
Overhead cost for rendering
Patch SizeOptimal
Rendering cost = Dimensionality + Overhead
Contributions
Analysis of dimensionality of local light transport Change of dimensionality with size
Diffuse and glossy reflections Shadows
Analyzing rendering cost Analytical formula for optimal patch size
Practical Applications Fine tuning parameters of existing methods Scale images to very high resolutions Develop adaptive clustering algorithm
Local Light Transport Dimensionality
Analysis of local light transport dimensionality
Dimensionality
Co
st
Patch Area
Dimensionality vs. Patch Size
Large Area : linear relationship
slope = 1
slope - rate of change of dimensionality
Independent of material properties np,
log
(D
ime
nsi
ona
lity)
log (Patch Area)
pixels dimensionality
2,2 np 2,2 np
Diffuse/Specular BRDF
Dimensionality Patch Area
Dimensionality vs. Patch Size
Small Area : sub - linear relationship
np,
log
(D
ime
nsi
ona
lity)
log (Patch Area)
pixels dimensionality
slope < 1
np ,2np ,2
Diffuse/Specular BRDF
Mathematical Tools for Analysis
Convolution formula for glossy reflections and shadows
Ramamoorthi and Hanrahan 01
Basri and Jacobs 01
Ramamoorthi et al 04
Szego’s Eigenvalue Distribution Theorem
Eigenvalues of the light transport matrix of the patch
Fourier Scale and Convolution Theorems
Dimensionality as a function of patch size
Bandwidth of BRDF
Central Result
PatchDimensionality
PatchArea
ConstantBandwidth of BRDF
PatchDimensionality
PatchArea
Constant
Lighting
BRDF
low pass filter
Material property
Fourier
Transform
x
)(xf F
BRDF/ Material Properties
Bandwidth of BRDF
Central Result
PatchDimensionality
PatchArea
Constant
99% Energy
low frequency
highfrequency
Bandwidth
Central Result
log
Large Arealo
g (D
imen
sion
ality
)
log (Patch area)
log
Diffuse/Specular BRDF
Bandwidth of BRDF
PatchDimensionality
PatchArea ( ( )) Bandwidth
of BRDFPatchArea ConstantConstant
Large Area
log
(Dim
ensi
onal
ity)
log (Patch area)
Diffuse/Specular BRDF
log log Bandwidth of BRDF
PatchDimensionality
PatchArea ( () )
Large Area
Bandwidth of BRDF
)(loglo
g (D
imen
sion
ality
)
log (Patch area)
Diffuse/Specular BRDF
log logPatchDimensionality
PatchArea( ( ))
log log Bandwidth of BRDF
PatchDimensionality
PatchArea ( () )
Large Area
Bandwidth of BRDF
)(loglo
g (D
imen
sion
ality
)
log (Patch area)
Diffuse/Specular BRDF
linear relationship
slope = 1
log logPatchDimensionality
PatchArea( ( ))
Small Area
log
(Dim
ensi
onal
ity)
log (Patch area)
Diffuse/Specular BRDF
slope < 1
sublinear relationship
Bandwidth of BRDF
)(loglog logPatchDimensionality
PatchArea( ( ))
Contributions
Analysis of dimensionality of local light transport Change of dimensionality with size
Glossy reflections Shadows
Analyzing rendering cost Analytical formula for optimal patch size
Practical Applications Fine tuning parameters of existing methods Scale images to very high resolutions Develop adaptive clustering algorithm
Visibility Function
Blocker
Visibility Function = 0
Visibility Function = 1Visibility Function = 1
P
Lighting Directions
Shadows
Dimensionality changes slowly in presence of shadows
Diffuse and Specular BRDF Shadows
slope = .5
slope = 1lo
g (
Dim
en
sion
alit
y)
log (Patch area)
Light Transport = Visibility Function
Shadows – Step Blocker
x
y
z
1p2p
3p
Step Blocker
Dimensionality √Patch Area
Same Visibility Function
Dimensionality changes only along one dimension
log (Dimensionality) .5 log(Patch Area)
Different Visibility Function
Light Transport = Visibility Function
21, pp
x
z
3p
Contributions
Analysis of dimensionality of local light transport Change of dimensionality with size
Glossy reflections Shadows
Analyzing rendering cost Analytical formula for optimal patch size
Practical Applications Fine tuning parameters of existing methods Scale images to very high resolutions Develop adaptive clustering algorithm
Contributions
Analysis of dimensionality of local light transport Change of dimensionality with size
Glossy reflections Shadows
Analyzing rendering cost Analytical formula for optimal patch size
Practical Applications Fine tuning parameters of existing methods Scale images to very high resolutions Develop adaptive clustering algorithm
Overhead CostC
ost
Patch Area
Dimensionality
Overhead CostC
ost
Patch Area
P
Overhead
Dimensionality
Overhead CostC
ost
Patch Area
Dimensionality
Overhead
Overhead CostC
ost
Patch Area
Dimensionality
Overhead
Overhead CostC
ost
Patch Area
Dimensionality
Overhead
Overhead CostC
ost
Patch Area
Dimensionality
Overhead
Overhead CostC
ost
Patch Area
Dimensionality
Overhead
Rendering CostC
ost
Patch Area
Rendering Cost
Dimensionality
Overhead
Rendering Cost vs. Patch Size
Intermediate size :
Rate of increase in dimensionality
Rate of decrease in overhead=
Total cost minimum
Co
st
Patch Area
Rendering Cost
Dimensionality
Overhead
Minimum
Optimal Patch Size
1
12* KpOptimal Patch Size
- Global DimensionalityK
Optimal Patch Size
- Global DimensionalityK
1
12* KpOptimal Patch Size
- Function of slope of dimensionality curve
Dimensionality Curve
- From our theoretical analysis- Empirically from the given dataset
Optimal Patch Size – CPCA Example
7.168* p
1
12* KpOptimal Patch Size
Total cost
Face dataset across lighting
170~*p
110 220 330 440 550average cluster size
cost
per
pix
el
- Global DimensionalityK - Function of slope of
dimensionality curve
Glossy Reflections
1
12* KpOptimal Patch Size - Global DimensionalityK - Function of slope of
dimensionality curve
Kp ~*
Number of pixels in the patch increases with glossiness
Independent of material properties
Contributions
Analysis of dimensionality of local light transport Change of dimensionality with size
Glossy reflections Shadows
Analyzing rendering cost Analytical formula for optimal patch size
Practical Applications Fine tuning parameters of existing methods Scale images to very high resolutions Develop adaptive clustering algorithm
Setting Optimal Patch Size – CPCA
Setting Optimal Patch Size – CPCA
24000 vertices
57.Estimated 220 114.78
cost per pixel
]6.53[.
clusters
130-600 114.78-130
11 310.7large
6 X 32 X 32Cube Map45.0 Hz.
Contributions
Analysis of dimensionality of local light transport Change of dimensionality with size
Glossy reflections Shadows
Analyzing rendering cost Analytical formula for optimal patch size
Practical Applications Fine tuning parameters of existing methods Scale images to very high resolutions Develop adaptive clustering algorithm
Scaling of Cost With Resolution
Subdivide More
p
ppnew 4new resolutionIndependent of patch resolution
170~*p 140~*p
Optimal patch size same for both resolutions
1
12* Kp
- Global Dimensionality
- Function of slope of dimensionality curve
K
Scaling of Cost With Resolution
Sub-linear increase in cost with resolution
Increase in resolution - x0.4
24Increase in cost -
68.0
1.85
p
ppnew 4new resolution
Sublinear increase in cost with resolution
1024 1024 800 x 600
Scaling of Cost With Resolution
Scaling of Cost With Resolution
Summary
Analysis of dimensionality of local light transport
Diffuse and Glossy reflections, dimensionality area
Shadows, dimensionality √area
Analysis of rendering cost
Optimal patch size
Scaling of cost with resolution
Practical Applications
Setting optimal parameters in existing methods
Adaptive clustering algorithms
Future Work
More solid theoretical foundation High dimensional appearance compression
Representation
ECCV 2006, PAMI 2007
Analysis of light transport in frequency domain
TOG, Jan. 2007
Analysis of light transport in gradient domain
Siggraph 2007
Analysis of general local light transport for patches
top related