5C-L1 Shape from shading
CS4495/6495 Introduction to Computer Vision
Thanks to Srinivasa Narasimhan, Shree Nayar, David Kreigman, Marc Pollefeys
Shape from shading/lighting
What is the relation between intensity and shape?
โข Need to look at the reflectance function
โข Reflectance Map
Surface normals: A bit of math
โข Letโs assume we have a surface ๐ง(๐ฅ, ๐ฆ)
โขWe can define the following:
โ๐๐ง
๐๐ฅ= ๐ โ
๐๐ง
๐๐ฆ= ๐
Surface Normal: A bit more math surface normal
๐ฅ ๐ฆ
z
๐ Suppose we have a point on the surface. We can define two tangents: ๐ก๐ฅ = 1,0,โ๐ ๐and ๐ก๐ฆ = 0,1,โ๐ ๐
2 2
| | |
1, ,1
| 1
x y
x y
Tt tN
Np q
p qt t
n
Surface Normal: Gradient space projection surface normal
๐ฅ ๐ฆ
z
๐
p
q (๐๐, ๐๐)
๐ง = 1 plane is called Gradient Space (๐๐ plane)
Gradient Space of Source and Normal z
๐ฅ ๐ฆ
๐
๐
1
๐ง = 1
๐ ๐
๐ง ๐ฌ
Unit normal vector:
๐ง =๐
|๐|=
๐, ๐, 1
๐2 + ๐2 + 1
Unit source vector:
๐ฌ =๐
|๐|=
๐๐, ๐๐, 1
๐๐2 + ๐๐
2 + 1
Normal vector
Unit normal vector
Source vector
Unit source vector
Gradient Space of Source and Normal z
๐ฅ ๐ฆ
๐๐ ๐
๐
1
๐ง = 1
๐ ๐
๐ง ๐ฌ
Unit normal vector:
๐ง =๐
|๐|=
๐, ๐, 1
๐2 + ๐2 + 1
Unit source vector:
๐ฌ =๐
|๐|=
๐๐, ๐๐, 1
๐๐2 + ๐๐
2 + 1
cos๐๐ = ๐ง โ ๐ฌ =๐๐๐ + ๐๐๐ + 1
๐2 + ๐2 + 1 ๐๐2 + ๐๐
2 + 1
Shape from shading: Problem definition
๐ผ(๐ฅ, ๐ฆ) ๐(๐ฅ, ๐ฆ)
Output: 3D shape of object Input: 1 or more images
Reflectance Map
Relates image brightness ๐ผ(๐ฅ, ๐ฆ) to surface orientation (๐, ๐) for given source direction and surface reflectance
Reflectance Map: Lambertian case
Terms: ๐: source brightness ๐: surface albedo (reflectance)
Image brightness:
๐ผ = ๐ โ ๐ โ cos๐๐ = ๐ โ ๐ (๐ง โ ๐ฌ)
Let ๐ โ ๐ = 1 then ๐ผ = cos๐๐ = ๐ง โ ๐ฌ
๐ผ(๐ฅ, ๐ฆ)
๐๐ ๐ฌ
๐ง
๐ฏ
Reflectance Map: Lambertian case
cone of constant ๐๐
Iso-brightness contour in pq space
๐ผ = cos๐๐ = ๐ง โ ๐ฌ =๐๐๐ + ๐๐๐ + 1
๐2 + ๐2 + 1 ๐๐2 + ๐๐
2 + 1
= ๐ ๐, ๐
Reflectance Map (Lambertian)
Note: ๐ ๐, ๐ is maximum when ๐, ๐ = ๐๐, ๐๐
๐๐ = 90โ ๐๐๐ + ๐๐๐ + 1 = 0
0.0
0.3
๐
๐ ๐, ๐ = 0.7
0.8
0.9
1.0
๐๐ , ๐๐
๐
Iso-brightness contours
Shape from a single image?
Given ๐ (๐, ๐) ((๐๐, ๐๐) and surface reflectance) can we determine (๐, ๐) uniquely for each image point?
NO ๐
๐
Shape from Shading
Need more information:
โขAdd more constraints: Shape-from-shading
โข Take more images: Photometric stereo
Given a single image of an object with known surface reflectance taken under a known light source, can we recover its shape?
Shape from shading
Given ๐ (๐, ๐) ((๐๐, ๐๐) and surface reflectance) can we determine (๐, ๐) uniquely for each image point?
โข Assume shape along the occluding boundary is known
โข Constraints on neighboring normals โ integrability
โข Smoothness
Shape from shading
Yes, a slightly ugly optimization
Shape from Shading: โRealโ Results
โข These single image methods work poorly in practice
โข Why? The assumptions are quite restrictive
Shape from Shading
Need more information:
โขAdd more constraints: Shape-from-shading
โข Take more images: Photometric stereo
Photometric stereo Output:
โข 3D shape of object
โข Albedo at (๐ฅ, ๐ฆ)
Input: Several images
โข Same object
โข Different lightings
โข Same pose
Photometric stereo Image brightness:
๐ง
๐ฌ๐
๐ฌ๐
๐ฏ
๐ฌ๐
๐ผ1 = ๐๐ง โ ๐ฌ๐
๐ผ2 = ๐๐ง โ ๐ฌ๐
๐ผ3 = ๐๐ง โ ๐ฌ๐
๐ผ = ๐ โ ๐ โ cos๐๐ = ๐๐ง โ ๐ฌ where k = 1
Write this as a matrix equation:
๐ผ1๐ผ2๐ผ3
= ๐๐ฌ๐
๐
๐ฌ๐ ๐
๐ฌ๐ ๐
๐ง
Solving the equations: Linear
๐ผ1๐ผ2๐ผ3
=๐ฌ๐
๐
๐ฌ๐ ๐
๐ฌ๐ ๐
๐๐ง
๐ ๐ ๐ง 3 ร 1 3 ร 3 3 ร 1
๐ง = ๐โ1๐
๐ = ๐ง
๐ง =๐ง
๐ง =๐ง
๐
๐ and ๐ are known
inverse
Adding more light sources
Get better results by using more (๐) lights:
๐ผ1โฎ๐ผ๐
=๐ฌ๐
๐
โฎ๐ฌ๐
๐
๐๐ง
Least squares solution:
๐ = ๐๐ง ๐๐๐ = ๐๐๐๐ง ๐ง = ๐๐๐ โ๐๐๐๐
Solve for๐, ๐ง as before
๐ ร 1 = ๐ ร 3 3 ร 1
Moore-Penrose pseudo inverse min ๐ โ ๐๐ง 2
2
Photometric stereo: pq space
๐
๐
๐๐ 1, ๐๐
1
๐๐ 2, ๐๐
2
๐๐ 3, ๐๐
3
Results: Lambertian sphere
Input Images
Estimated Albedo Estimated Surface Normals
Needles are projections of surface normal on image plane
Photometric stereo: Lambertian toy
Input Images
Estimated Surface Normals Estimated Albedo
Surface Normals
Albedo
Photometric stereo: Limitations
Big problems โข Doesnโt work for shiny things, semi-translucent things โข Shadows, inter-reflections Smaller problems โข Camera and lights have to be distant โข Calibration requirements
โ Measure light source directions, intensities โ Camera response function
Human shape from shading
Thomas R et al. J Vis 2010; 10:6
Also check Ramachandranโs work on Shape from Shading by Humans http://psy.ucsd.edu/chip/ramabio.html