light and shadingf&p 2nd ed., sec. 2.2.4 surface recovered by integration f&p 2nd ed., sec....
TRANSCRIPT
![Page 1: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/1.jpg)
Light and shading
P. Claesz, Still Life with a Skull and a Writing Quill, 1628
![Page 2: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/2.jpg)
Outline• Small taste of radiometry• In-camera transformation of light• Reflectance properties of surfaces• Lambertian reflection model• Shape from shading• Estimating direction of light sources
![Page 3: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/3.jpg)
Image formationWhat determines the brightness of an image pixel?
Distribution and properties of light
sources
Surface shape and orientation
Surface reflectanceproperties
Optics
Sensor properties
Slide by L. Fei-Fei
Exposure
![Page 4: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/4.jpg)
Fundamental radiometric relationL: Radiance emitted from P toward P’
• Energy carried by a ray (Watts per sq. meter per steradian)E: Irradiance falling on P’ from the lens
• Energy arriving at a surface (Watts per sq. meter)
What is the relationship between E and L? Szeliski 2.2.3
P
P’
f z
d α
![Page 5: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/5.jpg)
Fundamental radiometric relation
• Image irradiance is linearly related to scene radiance• Irradiance is proportional to the area of the lens and
inversely proportional to the squared distance between the lens and the image plane
• The irradiance falls off as the angle between the viewing ray and the optical axis increases
LfdE
úúû
ù
êêë
é÷÷ø
öççè
æ= ap 4
2
cos4
Szeliski 2.2.3
P
P’
f z
d α
![Page 6: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/6.jpg)
Fundamental radiometric relation
LfdE
úúû
ù
êêë
é÷÷ø
öççè
æ= ap 4
2
cos4
S. B. Kang and R. Weiss. Can we calibrate a camera using an image of a flat, textureless Lambertian surface? ECCV 2000
![Page 7: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/7.jpg)
From light rays to pixel values
• Camera response function: the mapping f from irradiance to pixel values• Useful if we want to estimate material properties• Enables us to create high dynamic range (HDR) images• Classic reference: P. E. Debevec and J. Malik, Recovering High
Dynamic Range Radiance Maps from Photographs, SIGGRAPH 97
tEX D×=
( )tEfZ D×=LfdE
úúû
ù
êêë
é÷÷ø
öççè
æ= ap 4
2
cos4
![Page 8: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/8.jpg)
From light rays to pixel values
• For more information: • M. Brown, Understanding the In-Camera Image Processing Pipeline
for Computer Vision, CVPR 2016 Tutorial
![Page 9: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/9.jpg)
The interaction of light and surfacesWhat happens when a light ray hits a point on an object?
• Some of the light gets absorbed– converted to other forms of energy (e.g., heat)
• Some gets transmitted through the object– possibly bent, through refraction– or scattered inside the object (subsurface scattering)
• Some gets reflected– possibly in multiple directions at once
• Really complicated things can happen– fluorescence
Bidirectional reflectance distribution function (BRDF)
• How bright a surface appears when viewed from one direction when light falls on it from another
• Definition: ratio of the radiance in the emitted direction to irradiance in the incident direction
Source: Steve Seitz
![Page 10: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/10.jpg)
BRDFs can be incredibly complicated…
![Page 11: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/11.jpg)
Diffuse reflection
• Light is reflected equally in all directions• Dull, matte surfaces like chalk or latex paint• Microfacets scatter incoming light randomly
• Brightness of the surface depends on the incidence of illumination
brighter darker
![Page 12: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/12.jpg)
Diffuse reflection: Lambert’s law
B = ρ N ⋅S= ρ S cosθ
NS
B: radiosity (total power leaving the surface per unit area)ρ: albedo (fraction of incident irradiance reflected by the surface)N: unit normalS: source vector (magnitude proportional to intensity of the source)
θ
![Page 13: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/13.jpg)
Specular reflection• Radiation arriving along a source
direction leaves along the specular direction (source direction reflected about normal)
• On real surfaces, energy usually goes into a lobe of directions
• Phong model: reflected energy falls of with ( )dqncos
Moving the light source
Changing the exponent
![Page 14: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/14.jpg)
Specular reflection
Picture source
![Page 15: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/15.jpg)
Photometric stereo (shape from shading)• Can we reconstruct the shape of an object
based on shading cues?
Luca della Robbia,Cantoria, 1438
![Page 16: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/16.jpg)
Photometric stereoAssume:
• A Lambertian object• A local shading model (each point on a surface receives light
only from sources visible at that point)• A set of known light source directions• A set of pictures of an object, obtained in exactly the same
camera/object configuration but using different sources• Orthographic projection
Goal: reconstruct object shape and albedo
Sn
???S1
S2
F&P 2nd ed., sec. 2.2.4
![Page 17: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/17.jpg)
Example 1
Recovered albedo
Recovered normal field
F&P 2nd ed., sec. 2.2.4
Recovered surface model
![Page 18: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/18.jpg)
Example 2
…
Input
Recovered albedo Recovered normal field
x y z
Recovered surface model
![Page 19: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/19.jpg)
Image model• Known: source vectors Sjand pixel values Ij(x,y)• Unknown: surface normal N(x,y) and albedo ρ(x,y)
F&P 2nd ed., sec. 2.2.4
![Page 20: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/20.jpg)
( ) ( )( )( ) ( )( )
j
j
jj
yx
kyxyx
yxyxkyxI
VgSNSN
×=
×=
×=
),(
)(,,
,,),(
r
r
Image model• Known: source vectors Sjand pixel values Ij(x,y)• Unknown: surface normal N(x,y) and albedo ρ(x,y) • Assume that the response function of the camera
is a linear scaling by a factor of k• Lambert’s law:
F&P 2nd ed., sec. 2.2.4
![Page 21: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/21.jpg)
Least squares problem
• Obtain least-squares solution for g(x,y) (which we defined as N(x,y) r(x,y))
• Since N(x,y) is the unit normal, r(x,y) is given by the magnitude of g(x,y)
• Finally, N(x,y) = g(x,y) / r(x,y)
I1(x, y)I2 (x, y)!
In (x, y)
!
"
#####
$
%
&&&&&
=
V1T
V2T
!Vn
T
!
"
#####
$
%
&&&&&
g(x, y)
(n × 1)known known unknown
(n × 3) (3 × 1)
• For each pixel, set up a linear system:
F&P 2nd ed., sec. 2.2.4
![Page 22: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/22.jpg)
Synthetic example
Recovered albedo Recovered normal field
F&P 2nd ed., sec. 2.2.4
![Page 23: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/23.jpg)
Recall the surface is written as
This means the normal has the form:
Recovering a surface from normalsIf we write the estimated vector g as
Then we obtain values for the partial derivatives of the surface:
)),(,,( yxfyx
÷÷÷
ø
ö
ççç
è
æ
++=
111),(
22 y
x
yx
ff
ffyxN
÷÷÷
ø
ö
ççç
è
æ=
),(),(),(
),(
3
2
1
yxgyxgyxg
yxg
),(/),(),(),(/),(),(
32
31
yxgyxgyxfyxgyxgyxf
y
x
==
F&P 2nd ed., sec. 2.2.4
![Page 24: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/24.jpg)
Recovering a surface from normalsIntegrability: for the surface f to exist, the mixed second partial derivatives must be equal:
We can now recover the surface height at any point by integration along some path, e.g.
(for robustness, should take integrals over many different paths and average the results)
(in practice, they should at least be similar)
)),(/),((
)),(/),((
32
31
yxgyxgx
yxgyxgy
¶¶
=¶¶f (x, y) = fx (s, 0)ds
0
x
∫ +
fy (x, t)dt0
y
∫ +C
F&P 2nd ed., sec. 2.2.4
![Page 25: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/25.jpg)
Surface recovered by integration
F&P 2nd ed., sec. 2.2.4
![Page 26: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/26.jpg)
Limitations• Orthographic camera model• Simplistic reflectance and lighting model• No shadows• No interreflections• No missing data• Integration is tricky
![Page 27: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/27.jpg)
Finding the direction of the light sourceI(x,y) = N(x,y) ·S(x,y)
Full 3D case:
P. Nillius and J.-O. Eklundh, “Automatic estimation of the projected light source direction,” CVPR 2001
NS
![Page 28: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/28.jpg)
Finding the direction of the light source
Consider points on the occluding contour:
P. Nillius and J.-O. Eklundh, “Automatic estimation of the projected light source direction,” CVPR 2001
Image
Projection direction (z)
Nz positive
Nz = 0
Nz negative
![Page 29: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/29.jpg)
Finding the direction of the light sourceI(x,y) = N(x,y) ·S(x,y)
Full 3D case:
For points on the occluding contour, Nz = 0:
P. Nillius and J.-O. Eklundh, “Automatic estimation of the projected light source direction,” CVPR 2001
NS
![Page 30: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/30.jpg)
Finding the direction of the light source
P. Nillius and J.-O. Eklundh, “Automatic estimation of the projected light source direction,” CVPR 2001
![Page 31: Light and shadingF&P 2nd ed., sec. 2.2.4 Surface recovered by integration F&P 2nd ed., sec. 2.2.4 Limitations • Orthographic camera model • Simplistic reflectance and lighting](https://reader035.vdocument.in/reader035/viewer/2022071502/612148822d1cd11fa16ef59d/html5/thumbnails/31.jpg)
Application: Detecting composite photos
Fake photo
Real photo
M. K. Johnson and H. Farid, Exposing Digital Forgeries by Detecting Inconsistencies in Lighting, ACM Multimedia and Security Workshop, 2005.