Download - Active Lighting for Appearance Decomposition
![Page 1: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/1.jpg)
Active Lighting for Appearance Decomposition
Todd ZicklerDEAS, Harvard University
![Page 2: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/2.jpg)
Appearance Decomposition
I = f (shape, reflectance)
Appearance
f -1( I ) = ?
illumination,
![Page 3: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/3.jpg)
Appearance Decomposition
Research Overview
APPEARANCE CAPTURE
COLOR IMAGE FILTERING
3D RECONSTRUCTION
PHOTOMETRIC INVARIANTS
![Page 4: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/4.jpg)
Appearance Decomposition
Getting 3D Shape:Image-based Reconstruction
I = f (shape, reflectance, illumination)
f -1( I ) = ?
![Page 5: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/5.jpg)
Appearance Decomposition
Reflectance: BRDF
n(µi ;Ái )
f r (µi ;Ái ;µo;Áo)
(µo;Áo)
Bi-directional Reflectance Distribution Function
![Page 6: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/6.jpg)
Appearance Decomposition
Conventional 3D Reconstruction:Restrictive Assumptions
LAMBERTIAN:IDEALLY DIFFUSE
![Page 7: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/7.jpg)
Appearance Decomposition
Example: Conventional Stereo
ASSUMPTION: Il = Ir
Il Ir
![Page 8: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/8.jpg)
Appearance Decomposition
Example: Conventional Stereo
Il Ir
ASSUMPTION: Il = Ir
![Page 9: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/9.jpg)
Appearance Decomposition
Conventional 3D Reconstruction:Restrictive Assumptions
Shape from shading[Tsai and Shaw, 1994]
Variational Stereo[Faugeras and Keriven, 1998]
Space Carving[Kutulakos and Seitz, 1998]
Multiple-window stereo[Fusiello et al., 1997]
![Page 10: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/10.jpg)
Appearance Decomposition
Reflectance: BRDF
)e,i(rf
ie
n
![Page 11: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/11.jpg)
Appearance Decomposition
Reflectance: BRDF
n
)e,i()e,i( 21 rr ff
![Page 12: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/12.jpg)
Appearance Decomposition
Helmholtz Reciprocity
ie
n
i e
[Helmholtz 1925; Minnaert 1941; Nicodemus et al. 1977]
)i,e()e,i( rr ff
n
![Page 13: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/13.jpg)
Appearance Decomposition
Stereo vs. Helmholtz Stereo
STEREO HELMHOLTZ STEREO
![Page 14: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/14.jpg)
Appearance Decomposition
Stereo vs. Helmholtz Stereo
STEREO HELMHOLTZ STEREO
![Page 15: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/15.jpg)
Appearance Decomposition
Stereo vs. Helmholtz Stereo
STEREO HELMHOLTZ STEREO
![Page 16: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/16.jpg)
Appearance Decomposition
Reciprocal Images
Specularities “fixed” to surface
Il Ir
Relation between Il and Ir independent of BRDF
![Page 17: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/17.jpg)
Appearance Decomposition
Reciprocity Constraint
2
r
rlrl
ˆˆ)ˆ,ˆ(
po
vnvv
rfI
n
vl^ vr
^
p
ol or
=
2
l
lrlr
ˆˆ)ˆ,ˆ(
po
vnvv
rfI
vl^ vr
^
p
ol or
n
![Page 18: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/18.jpg)
Appearance Decomposition
Reciprocity Constraint
2
r
rlrl
ˆˆ)ˆ,ˆ(
po
vnvv
rfI
n
vl^ vr
^
p
ol or
=
2
l
lrlr
ˆˆ)ˆ,ˆ(
po
vnvv
rfI
vl^ vr
^
p
ol or
n
0ˆ(ˆ(ˆ
2
r
rr2
l
ll
n
po
p)v
po
p)vII
Arbitrary reflectance Surface normal
![Page 19: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/19.jpg)
Appearance Decomposition
Reciprocal Acquisition
CAMERA
LIGHT SOURCE
![Page 20: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/20.jpg)
Appearance Decomposition
Recovered Normals
[Zickler et al. 2002]
![Page 21: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/21.jpg)
Appearance Decomposition
Recovered Surface
[Zickler et al., ECCV 2002]
![Page 22: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/22.jpg)
Appearance Decomposition
In Practice
1. Arbitrary Reflectance
2. Off-the-shelf components
3. Direct surface normals
4. Images aligned with recovered shape
5. Self-calibrating (coming…)
![Page 23: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/23.jpg)
Appearance Decomposition
Ongoing Work: Auto-calibration
[Zickler et al., CVPR 2003, CVPR 2006,…]
![Page 24: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/24.jpg)
Appearance Decomposition
Research Overview
APPEARANCE CAPTURE
COLOR IMAGE FILTERING
3D RECONSTRUCTION
PHOTOMETRIC INVARIANTS
![Page 25: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/25.jpg)
Appearance Decomposition
Reflectance Decomposition
DIFFUSE
= +
SPECULAR
[Phong 1975; Shafer, 1985]
![Page 26: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/26.jpg)
Appearance Decomposition
Reflectance Decomposition
, ,k R G B
[Shafer, 1985]
E
R
kC
Sk =Z
E (¸)Ck(¸)d
Dk =Z
E (¸)R(¸)Ck(¸)d
I R GB = ¾dD + ¾sSI k = Dkf d(i; e)n ¢i + Skf s (i; e)n ¢i
![Page 27: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/27.jpg)
Appearance Decomposition
Reflectance Decomposition:Simplifies the Vision Problem
= +LAMBERTIAN:IDEALLY DIFFUSE
I R GB = ¾dD + ¾sSI R GB = ¾dD + ¾sSI R GB = ¾dD + ¾sS= +
I R GB = f d(i; e)(n ¢i)D + f s (i; e)(n ¢i)S
I R GB = (n ¢i)f dD + f s (i; e)(n ¢i)S
![Page 28: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/28.jpg)
Appearance Decomposition
Reflectance Decomposition:A Difficult Inverse Problem
DIFFUSE
= +
SPECULAR
[Bajscy et al., 1996; Criminisi et al., 2005; Lee and Bajscy, 1992; Lin et al., 2002; Lin and Shum, 2001; Miyazaki et al., 2003; Nayar et al., 1997; Ragheb and Hancock, 2001; Sato and Ikeutchi, 1994; Tan and Ikeutchi, 2005; Wolfe and Boult, 1991,…]
I R GB = ¾dD + ¾sSI R GB = ¾dD + ¾sS I R GB = ¾dD + ¾sS= +
![Page 29: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/29.jpg)
Appearance Decomposition
Known Illuminant: Still Ill-posed
G
S
IRGB
B
R
D?
I R GB = ¾dD + ¾sS
![Page 30: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/30.jpg)
Appearance Decomposition
Known Illuminant: Still Ill-posed
G
S
IRGB
B
R
D?
I R GB = ¾dD + ¾sS
![Page 31: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/31.jpg)
Appearance Decomposition
Observation:Explicit Decomposition not Required
Gr2r1
S
IRGB
B
RJ
I R GB = ¾dD + ¾sS
J l =< I R GB ; r l >= ¾dr>l D
1. INVARIANT TOSPECULAR REFLECTIONS
2. BEHAVES ‘LAMBERTIAN’
![Page 32: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/32.jpg)
Appearance Decomposition
Observation:Explicit Decomposition not Required
Gr
2
r
1
S
IRGB
B
R J
I R GB = ¾dD + ¾sS
J l =< I R GB ; r l >= ¾dr>l D
IRGB || J ||
[Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005]
![Page 33: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/33.jpg)
Appearance Decomposition
Generalization: Mixed Illumination
Gr2r1
S
IRGB
B
RJ
r1
S1
IRGB
B
G
R
S2
J
SINGLE ILLUMINANT MIXED ILLUMINATION
[Zickler, Mallick, Kriegman, Belhumeur, CVPR 2006]
![Page 34: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/34.jpg)
Appearance Decomposition
Generalization: Mixed Illumination
![Page 35: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/35.jpg)
Appearance Decomposition
Example: Binocular Stereo
[Algorithm: Boykov, Veksler and Zabih, CVPR 1998]
Conventional Grayscale(R+G+B)/3
Specular Invariant, ||J||
(blue illuminant)
Specular Invariant, ||J|| (blue & yellow
illuminants)
One image from input
stereo pair
Reco
vere
d d
epth
![Page 36: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/36.jpg)
Appearance Decomposition
Example: Optical Flow
[Algorithm: Black and Anandan, 1993]
Conventi
onal
Gra
ysc
ale
(R-+
G+
B)/
3
Specu
lar
Invari
ant,
||J|
| (b
lue
illum
inant)
Specu
lar
Invari
ant,
||J
|| (b
lue &
yello
w
illum
inants
)
Ground truth flow
![Page 37: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/37.jpg)
Appearance Decomposition
Example: Photometric Stereo
[Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005]
J behaves ‘Lambertian’ Linear function of surface normal
J l =< I R GB ;r l >= ¾dr>l D = (n ¢i)f dr>
l D
![Page 38: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/38.jpg)
Appearance Decomposition
Example: Photometric Stereo
[Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005]
J behaves ‘Lambertian’ Linear function of surface normal
J l =< I R GB ;r l >= ¾dr>l D = (n ¢i)f dr>
l D
![Page 39: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/39.jpg)
Appearance Decomposition
Example: Photometric Stereo
[Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005]
J behaves ‘Lambertian’ Linear function of surface normal
J l =< I R GB ;r l >= ¾dr>l D = (n ¢i)f dr>
l D
![Page 40: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/40.jpg)
Appearance Decomposition
Example: Photometric Stereo
[Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005]
![Page 41: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/41.jpg)
Appearance Decomposition
Example: Photometric Stereo
[Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005]
![Page 42: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/42.jpg)
Appearance Decomposition
Generalized Hue
Gr2r1
S
IRGB
B
RJ
ψ
à = tan¡ 1(J 1=J 2) = tan¡ 1(r>1 D=r>
2 D)
J l =< I R GB ; r l >= ¾dr>l D
![Page 43: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/43.jpg)
Appearance Decomposition
Example: Material-based Segmentation
[Zickler, Mallick, Kriegman, Belhumeur, CVPR 2006]
Input image
Conventional Grayscale Specular Invariant ||J||
Conventional Hue Generalized Hue
![Page 44: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/44.jpg)
Appearance Decomposition
Active lighting can provide:1. Precise shape (surface normals) for a broad
class of (non-Lambertian) surfaces2. Specular and/or shading invariance
(e.g., optical flow, tracking, segmentation)3. Minimal hardware requirements
Active Lighting for Image-guided Surgery?
Endoscopic imagery:1. Illuminant(s) is/are controlled and known2. Non-Lambertian surfaces3. Lack of texture
![Page 45: Active Lighting for Appearance Decomposition](https://reader036.vdocument.in/reader036/viewer/2022070410/56814666550346895db387b0/html5/thumbnails/45.jpg)
Appearance Decomposition
Acknowledgements
Satya Mallick, UCSD
Peter Belhumeur, Columbia University
David Kriegman, UCSD
Sebastian Enrique, Columbia University
Ravi Ramamoorthi, Columbia University
[email protected]://www.eecs.harvard.edu/~zickler