![Page 1: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/1.jpg)
Lambertian model of reflectance II: harmonic analysis
Ronen Basri
Weizmann Institute of Science
![Page 2: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/2.jpg)
Illumination cone
• What is the set of images of an object under different lighting, with any number of sources?
• Images are additive and non-negative
• This set, therefore, forms a convex cone in ℝ𝑝, 𝑝 number of pixels (Belhumeur & Kriegman)
= 0.5* +0.2* +0.3*
![Page 3: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/3.jpg)
Illumination cone
• Cone characterization is generic, holds also with specularities, shadows and inter-reflections
• Unfortunately, representing the cone is complicated (infinite degrees of freedom)
• Cone is “thin” for Lambertian objects; indeed the illumination cone of many objects can be represented with few PCA vectors (Yuille et al.)
![Page 4: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/4.jpg)
Illumination cone is often thin
Ball Face Phone Parrot
#1 48.2 53.7 67.9 42.8
#3 94.4 90.2 88.2 76.3
#5 97.9 93.5 94.1 84.7
#7 99.1 95.3 96.3 88.5
#9 99.5 96.3 97.2 90.7
(Yuille et al.)
![Page 5: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/5.jpg)
Lambertian reflectance is smooth
0 1 2 30
0.5
1
0 1 2 30
0.5
1
1.5
2
lighting
reflectance
(Basri & Jacobs; Ramamoorthi & Hanrahan)
![Page 6: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/6.jpg)
Reflectance obtained with convolution
+ + +
𝑅 𝑣 = 𝑘 𝑢, 𝑣 𝑙 𝑢 𝑑𝑢𝑆2
![Page 7: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/7.jpg)
Reflectance obtained with convolution
+ + +
𝑅 𝑣 = 𝑘 𝑢, 𝑣 𝑙 𝑢 𝑑𝑢𝑆2
![Page 8: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/8.jpg)
Spherical harmonics
𝑌𝑛𝑚 𝜃, ∅ =2𝑛 + 1
4𝜋
𝑛 − |𝑚| !
𝑛 + |𝑚| !𝑃𝑛𝑚(cos 𝜃)𝑒𝑖𝑚∅
𝑝𝑛𝑚 𝑧 =(1 − 𝑧2)𝑚/2
2𝑛𝑛!
𝑑𝑛+𝑚
𝑑𝑧𝑛+𝑚(𝑧2 − 1)𝑛
• Orthonormal basis for functions on the sphere
• n’th order harmonics have 2n+1 components
• Rotation = phase shift (same n, different m)
• In space coordinates: polynomials of degree n
• Funk-Hecke convolution theorem
![Page 9: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/9.jpg)
Spherical harmonics
1
Z Y X
23 1Z XZ YZ 22
YX XY
2 2 21X Y Z+ +
Positive values
Negative values
![Page 10: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/10.jpg)
Harmonic approximation
• Lighting, in terms of harmonics
ℓ 𝜃, 𝜙 = 𝑙𝑛𝑚𝑌𝑛𝑚(𝜃, 𝜙)
𝑛
𝑚=−𝑛
∞
𝑛=0
• Reflectance
𝑟 𝜃, 𝜙 = 𝑘 ∗ ℓ ≈ 𝑘𝑛𝑙𝑛𝑚𝑌𝑛𝑚(𝜃, 𝜙)
𝑛
𝑚=−𝑛
2
𝑛=0
• Approximation accuracy, 99.2% (Basri & Jacobs; Ramamoorthi & Hanrahan)
![Page 11: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/11.jpg)
Harmonic transform of kernel
1.023
0.495
-0.111
0.05
-0.029
0.886
-0.5
0
0.5
1
1.5
0 1 2 3 4 5 6 7 8
𝑘(𝜃) = max(cos 𝜃, 0) = 𝑘𝑛𝑌𝑛0
∞
𝑛=0
99.2%
![Page 12: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/12.jpg)
Subspace approximation
• Up to 2nd order:
– 9 basis images
– Accuracy: 99.2%
• Up to 1st order:
– 4 basis images: ambient + point source
– Accuracy: 87.5%
• In practice, due to self occlusions ~98% can be achieved with just 6 basis images (Ramamoorthi)
![Page 13: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/13.jpg)
Scope
• Harmonic representations handle convex, lambertian objects with multiple light sources (including attached shadows)
• Harmonic representations do not model cast shadows and inter-reflections
• Accuracy is maintained for fairly close light sources
• Representing specular objects may require a very large basis
![Page 14: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/14.jpg)
Applications
• We can use this theory to predict novel appearances under new lighting
• Harmonic lighting theory has led to applications in – Face recognition
– Photometric stereo
– 3D reconstruction with prior
– Motion analysis
![Page 15: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/15.jpg)
“Harmonic faces”
Positive values
Negative values
( , , )x y z
n n n n
ρ Albedo
n Surface normal
2(3 1)
zn
2 2( )
x yn n x y
n nx z
n n y zn n
zn x
n yn
![Page 16: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/16.jpg)
Non-negative light
• We can enforce in addition that light is non-negative, by projecting the illumination cone onto the harmonic space
• Closed-form constraints for 1st order approximation
• Sampling method, or Toeplitz matrix (Shirdhonkar & Jacobs) for higher orders
![Page 17: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/17.jpg)
Photometric stereo
L M
S
Image n
:
Image 1
Light n
:
Light 1
SVD recovers L and S up to an (𝑟 × 𝑟) ambiguity
nz
nz
ny
(3nz2-1)
(nx2-ny
2)
nxny
nxnz
nynz
(Basri, Jacobs & Kemelmacher)
![Page 18: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/18.jpg)
Photometric stereo
![Page 19: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/19.jpg)
Reconstruction with a prior
• Given just one image SFS is impractical • Reconstruction is possible when a prior is available • Energy
min𝑙,𝜌,𝑍
𝐷 + 𝑆Ω
• Data term 𝐷 = 𝐼 − 𝜌𝑙𝑇𝑌(𝑛 ) 2
• Regularization
𝑆 = 𝜆1 ∆ 𝑍 − 𝑍𝑝𝑟𝑖𝑜𝑟
2+ 𝜆2 ∆ 𝜌 − 𝜌𝑝𝑟𝑖𝑜𝑟
2
• Solve as a linear PDE
(Kemelmacher & Basri)
![Page 20: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/20.jpg)
Reconstruction with a prior
![Page 21: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/21.jpg)
More…
![Page 22: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/22.jpg)
Mooney faces
(Kemelmacher, Nadler & B, CVPR 2008)
![Page 23: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/23.jpg)
Motion + lighting
![Page 24: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/24.jpg)
Motion + lighting
• Given 2 images 𝐼 𝑝 = 𝜌𝑙𝑇𝑛 𝐽 𝑝′ = 𝜌𝑙𝑇𝑅𝑛
• Take ratio to eliminate albedo 𝐽(𝑝′)
𝐼(𝑝)=
𝑙𝑇𝑅𝑛
𝑙𝑇𝑛
• If motion is small we can represent 𝐽 𝑝′ using a Taylor expansion around 𝑝
(Basri & Frolova)
![Page 25: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/25.jpg)
Small motion
• We obtain a PDE that is quasi linear in 𝑧 𝑎𝑧𝑥 + 𝑏𝑧𝑦 = 𝑐
• Where 𝑎 𝑥, 𝑦, 𝑧 = 𝑙1 𝐼𝜃 − 𝑧𝐽𝑥 − 𝑙3𝐼 𝑏 𝑥, 𝑦, 𝑧 = 𝑙2 𝐼𝜃 − 𝑧𝐽𝑥 𝑐 𝑥, 𝑦, 𝑧 = −𝑙3 𝐼𝜃 − 𝑧𝐽𝑥 − 𝑙1𝐼
with
𝐼𝜃 =𝐽 − 𝐼
𝜃
• Can be solved with continuation (characteristics)
![Page 26: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/26.jpg)
Reconstruction
![Page 27: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/27.jpg)
More reconstructions
![Page 28: Lambertian model of reflectance II: harmonic analysis · harmonic analysis Ronen Basri Weizmann Institute of Science . ... •Rotation = phase shift (same n, different m) ... onto](https://reader036.vdocument.in/reader036/viewer/2022071006/5fc3d292ed91840c6a0fe284/html5/thumbnails/28.jpg)
Conclusion
• Understanding the effect of lighting on images is challenging, but can lead to better interpretation of images
• Harmonic analysis allows to model complex lighting in a linear model
• Various applications in recognition and reconstruction
• We only looked at Lambertian objects…