active lighting for appearance decomposition
DESCRIPTION
Active Lighting for Appearance Decomposition. Todd Zickler DEAS, Harvard University. I = f ( shape,. reflectance ). illumination,. ?. f -1 ( I ) =. Appearance. Research Overview. COLOR IMAGE FILTERING. 3D RECONSTRUCTION. APPEARANCE CAPTURE. PHOTOMETRIC INVARIANTS. ?. - PowerPoint PPT PresentationTRANSCRIPT
Active Lighting for Appearance Decomposition
Todd ZicklerDEAS, Harvard University
Appearance Decomposition
I = f (shape, reflectance)
Appearance
f -1( I ) = ?
illumination,
Appearance Decomposition
Research Overview
APPEARANCE CAPTURE
COLOR IMAGE FILTERING
3D RECONSTRUCTION
PHOTOMETRIC INVARIANTS
Appearance Decomposition
Getting 3D Shape:Image-based Reconstruction
I = f (shape, reflectance, illumination)
f -1( I ) = ?
Appearance Decomposition
Reflectance: BRDF
n(µi ;Ái )
f r (µi ;Ái ;µo;Áo)
(µo;Áo)
Bi-directional Reflectance Distribution Function
Appearance Decomposition
Conventional 3D Reconstruction:Restrictive Assumptions
LAMBERTIAN:IDEALLY DIFFUSE
Appearance Decomposition
Example: Conventional Stereo
ASSUMPTION: Il = Ir
Il Ir
Appearance Decomposition
Example: Conventional Stereo
Il Ir
ASSUMPTION: Il = Ir
Appearance Decomposition
Conventional 3D Reconstruction:Restrictive Assumptions
Shape from shading[Tsai and Shaw, 1994]
Variational Stereo[Faugeras and Keriven, 1998]
Space Carving[Kutulakos and Seitz, 1998]
Multiple-window stereo[Fusiello et al., 1997]
Appearance Decomposition
Reflectance: BRDF
)e,i(rf
ie
n
Appearance Decomposition
Reflectance: BRDF
n
)e,i()e,i( 21 rr ff
Appearance Decomposition
Helmholtz Reciprocity
ie
n
i e
[Helmholtz 1925; Minnaert 1941; Nicodemus et al. 1977]
)i,e()e,i( rr ff
n
Appearance Decomposition
Stereo vs. Helmholtz Stereo
STEREO HELMHOLTZ STEREO
Appearance Decomposition
Stereo vs. Helmholtz Stereo
STEREO HELMHOLTZ STEREO
Appearance Decomposition
Stereo vs. Helmholtz Stereo
STEREO HELMHOLTZ STEREO
Appearance Decomposition
Reciprocal Images
Specularities “fixed” to surface
Il Ir
Relation between Il and Ir independent of BRDF
Appearance Decomposition
Reciprocity Constraint
2
r
rlrl
ˆˆ)ˆ,ˆ(
po
vnvv
rfI
n
vl^ vr
^
p
ol or
=
2
l
lrlr
ˆˆ)ˆ,ˆ(
po
vnvv
rfI
vl^ vr
^
p
ol or
n
Appearance Decomposition
Reciprocity Constraint
2
r
rlrl
ˆˆ)ˆ,ˆ(
po
vnvv
rfI
n
vl^ vr
^
p
ol or
=
2
l
lrlr
ˆˆ)ˆ,ˆ(
po
vnvv
rfI
vl^ vr
^
p
ol or
n
0ˆ(ˆ(ˆ
2
r
rr2
l
ll
n
po
p)v
po
p)vII
Arbitrary reflectance Surface normal
Appearance Decomposition
Reciprocal Acquisition
CAMERA
LIGHT SOURCE
Appearance Decomposition
Recovered Normals
[Zickler et al. 2002]
Appearance Decomposition
Recovered Surface
[Zickler et al., ECCV 2002]
Appearance Decomposition
In Practice
1. Arbitrary Reflectance
2. Off-the-shelf components
3. Direct surface normals
4. Images aligned with recovered shape
5. Self-calibrating (coming…)
Appearance Decomposition
Ongoing Work: Auto-calibration
[Zickler et al., CVPR 2003, CVPR 2006,…]
Appearance Decomposition
Research Overview
APPEARANCE CAPTURE
COLOR IMAGE FILTERING
3D RECONSTRUCTION
PHOTOMETRIC INVARIANTS
Appearance Decomposition
Reflectance Decomposition
DIFFUSE
= +
SPECULAR
[Phong 1975; Shafer, 1985]
Appearance Decomposition
Reflectance Decomposition
, ,k R G B
[Shafer, 1985]
E
R
kC
Sk =Z
E (¸)Ck(¸)d
Dk =Z
E (¸)R(¸)Ck(¸)d
I R GB = ¾dD + ¾sSI k = Dkf d(i; e)n ¢i + Skf s (i; e)n ¢i
Appearance Decomposition
Reflectance Decomposition:Simplifies the Vision Problem
= +LAMBERTIAN:IDEALLY DIFFUSE
I R GB = ¾dD + ¾sSI R GB = ¾dD + ¾sSI R GB = ¾dD + ¾sS= +
I R GB = f d(i; e)(n ¢i)D + f s (i; e)(n ¢i)S
I R GB = (n ¢i)f dD + f s (i; e)(n ¢i)S
Appearance Decomposition
Reflectance Decomposition:A Difficult Inverse Problem
DIFFUSE
= +
SPECULAR
[Bajscy et al., 1996; Criminisi et al., 2005; Lee and Bajscy, 1992; Lin et al., 2002; Lin and Shum, 2001; Miyazaki et al., 2003; Nayar et al., 1997; Ragheb and Hancock, 2001; Sato and Ikeutchi, 1994; Tan and Ikeutchi, 2005; Wolfe and Boult, 1991,…]
I R GB = ¾dD + ¾sSI R GB = ¾dD + ¾sS I R GB = ¾dD + ¾sS= +
Appearance Decomposition
Known Illuminant: Still Ill-posed
G
S
IRGB
B
R
D?
I R GB = ¾dD + ¾sS
Appearance Decomposition
Known Illuminant: Still Ill-posed
G
S
IRGB
B
R
D?
I R GB = ¾dD + ¾sS
Appearance Decomposition
Observation:Explicit Decomposition not Required
Gr2r1
S
IRGB
B
RJ
I R GB = ¾dD + ¾sS
J l =< I R GB ; r l >= ¾dr>l D
1. INVARIANT TOSPECULAR REFLECTIONS
2. BEHAVES ‘LAMBERTIAN’
Appearance Decomposition
Observation:Explicit Decomposition not Required
Gr
2
r
1
S
IRGB
B
R J
I R GB = ¾dD + ¾sS
J l =< I R GB ; r l >= ¾dr>l D
IRGB || J ||
[Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005]
Appearance Decomposition
Generalization: Mixed Illumination
Gr2r1
S
IRGB
B
RJ
r1
S1
IRGB
B
G
R
S2
J
SINGLE ILLUMINANT MIXED ILLUMINATION
[Zickler, Mallick, Kriegman, Belhumeur, CVPR 2006]
Appearance Decomposition
Generalization: Mixed Illumination
Appearance Decomposition
Example: Binocular Stereo
[Algorithm: Boykov, Veksler and Zabih, CVPR 1998]
Conventional Grayscale(R+G+B)/3
Specular Invariant, ||J||
(blue illuminant)
Specular Invariant, ||J|| (blue & yellow
illuminants)
One image from input
stereo pair
Reco
vere
d d
epth
Appearance Decomposition
Example: Optical Flow
[Algorithm: Black and Anandan, 1993]
Conventi
onal
Gra
ysc
ale
(R-+
G+
B)/
3
Specu
lar
Invari
ant,
||J|
| (b
lue
illum
inant)
Specu
lar
Invari
ant,
||J
|| (b
lue &
yello
w
illum
inants
)
Ground truth flow
Appearance Decomposition
Example: Photometric Stereo
[Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005]
J behaves ‘Lambertian’ Linear function of surface normal
J l =< I R GB ;r l >= ¾dr>l D = (n ¢i)f dr>
l D
Appearance Decomposition
Example: Photometric Stereo
[Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005]
J behaves ‘Lambertian’ Linear function of surface normal
J l =< I R GB ;r l >= ¾dr>l D = (n ¢i)f dr>
l D
Appearance Decomposition
Example: Photometric Stereo
[Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005]
J behaves ‘Lambertian’ Linear function of surface normal
J l =< I R GB ;r l >= ¾dr>l D = (n ¢i)f dr>
l D
Appearance Decomposition
Example: Photometric Stereo
[Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005]
Appearance Decomposition
Example: Photometric Stereo
[Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005]
Appearance Decomposition
Generalized Hue
Gr2r1
S
IRGB
B
RJ
ψ
à = tan¡ 1(J 1=J 2) = tan¡ 1(r>1 D=r>
2 D)
J l =< I R GB ; r l >= ¾dr>l D
Appearance Decomposition
Example: Material-based Segmentation
[Zickler, Mallick, Kriegman, Belhumeur, CVPR 2006]
Input image
Conventional Grayscale Specular Invariant ||J||
Conventional Hue Generalized Hue
Appearance Decomposition
Active lighting can provide:1. Precise shape (surface normals) for a broad
class of (non-Lambertian) surfaces2. Specular and/or shading invariance
(e.g., optical flow, tracking, segmentation)3. Minimal hardware requirements
Active Lighting for Image-guided Surgery?
Endoscopic imagery:1. Illuminant(s) is/are controlled and known2. Non-Lambertian surfaces3. Lack of texture
Appearance Decomposition
Acknowledgements
Satya Mallick, UCSD
Peter Belhumeur, Columbia University
David Kriegman, UCSD
Sebastian Enrique, Columbia University
Ravi Ramamoorthi, Columbia University
[email protected]://www.eecs.harvard.edu/~zickler