5C-L1 Shape from shading

CS4495/6495 Introduction to Computer Vision

Thanks to Srinivasa Narasimhan, Shree Nayar, David Kreigman, Marc Pollefeys

Shape from shading

Shading as a cue for shape reconstruction

Shape from shading/lighting

What is the relation between intensity and shape?

• Need to look at the reflectance function

• Reflectance Map

Surface normals: A bit of math

• Let’s assume we have a surface 𝑧(𝑥, 𝑦)

•We can define the following:

−𝜕𝑧

𝜕𝑥= 𝑝 −

𝜕𝑧

𝜕𝑦= 𝑞

Surface Normal: A bit more math surface normal

𝑥 𝑦

z

𝐍 Suppose we have a point on the surface. We can define two tangents: 𝑡𝑥 = 1,0,−𝑝 𝑇and 𝑡𝑦 = 0,1,−𝑞 𝑇

2 2

| | |

1, ,1

| 1

x y

x y

Tt tN

Np q

p qt t

n

Surface Normal: Gaussian sphere surface normal

𝑥 𝑦

z

𝐍

Surface Normal: Gradient space projection surface normal

𝑥 𝑦

z

𝐍

Surface Normal: Gradient space projection surface normal

𝑥 𝑦

z

𝐍

p

q (𝒑𝟎, 𝒒𝟎)

𝑧 = 1 plane is called Gradient Space (𝑝𝑞 plane)

Gradient Space of Source and Normal z

𝑥 𝑦

𝑝

𝑞

1

𝑧 = 1

𝐒 𝐍

𝐧 𝐬

Unit normal vector:

𝐧 =𝐍

|𝐍|=

𝑝, 𝑞, 1

𝑝2 + 𝑞2 + 1

Unit source vector:

𝐬 =𝐒

|𝐒|=

𝑝𝑆, 𝑞𝑆, 1

𝑝𝑆2 + 𝑞𝑆

2 + 1

Normal vector

Unit normal vector

Source vector

Unit source vector

Gradient Space of Source and Normal z

𝑥 𝑦

𝜃𝑖 𝑝

𝑞

1

𝑧 = 1

𝐒 𝐍

𝐧 𝐬

Unit normal vector:

𝐧 =𝐍

|𝐍|=

𝑝, 𝑞, 1

𝑝2 + 𝑞2 + 1

Unit source vector:

𝐬 =𝐒

|𝐒|=

𝑝𝑆, 𝑞𝑆, 1

𝑝𝑆2 + 𝑞𝑆

2 + 1

cos𝜃𝑖 = 𝐧 ⋅ 𝐬 =𝑝𝑝𝑆 + 𝑞𝑞𝑆 + 1

𝑝2 + 𝑞2 + 1 𝑝𝑆2 + 𝑞𝑆

2 + 1

Shape from shading: Problem definition

𝐼(𝑥, 𝑦) 𝑍(𝑥, 𝑦)

Output: 3D shape of object Input: 1 or more images

Reflectance Map

Relates image brightness 𝐼(𝑥, 𝑦) to surface orientation (𝑝, 𝑞) for given source direction and surface reflectance

Reflectance Map: Lambertian case

Terms: 𝑘: source brightness 𝜌: surface albedo (reflectance)

Image brightness:

𝐼 = 𝜌 ⋅ 𝑘 ⋅ cos𝜃𝑖 = 𝜌 ⋅ 𝑘 (𝐧 ⋅ 𝐬)

Let 𝜌 ⋅ 𝑘 = 1 then 𝐼 = cos𝜃𝑖 = 𝐧 ⋅ 𝐬

𝐼(𝑥, 𝑦)

𝜃𝑖 𝐬

𝐧

𝐯

Reflectance Map: Lambertian case

cone of constant 𝜃𝑖

Iso-brightness contour in pq space

𝐼 = cos𝜃𝑖 = 𝐧 ⋅ 𝐬 =𝑝𝑝𝑠 + 𝑞𝑞𝑠 + 1

𝑝2 + 𝑞2 + 1 𝑝𝑆2 + 𝑞𝑆

2 + 1

= 𝑅 𝑝, 𝑞

Reflectance Map (Lambertian)

Note: 𝑅 𝑝, 𝑞 is maximum when 𝑝, 𝑞 = 𝑝𝑆, 𝑞𝑆

𝜃𝑖 = 90∘ 𝑝𝑝𝑆 + 𝑞𝑞𝑆 + 1 = 0

0.0

0.3

𝑝

𝑅 𝑝, 𝑞 = 0.7

0.8

0.9

1.0

𝑝𝑠, 𝑞𝑠

𝑞

Iso-brightness contours

Shape from a single image?

Given 𝑅(𝑝, 𝑞) ((𝑝𝑆, 𝑞𝑆) and surface reflectance) can we determine (𝑝, 𝑞) uniquely for each image point?

NO 𝑝

𝑞

Shape from Shading

Need more information:

•Add more constraints: Shape-from-shading

• Take more images: Photometric stereo

Given a single image of an object with known surface reflectance taken under a known light source, can we recover its shape?

Shape from shading

Given 𝑅(𝑝, 𝑞) ((𝑝𝑆, 𝑞𝑆) and surface reflectance) can we determine (𝑝, 𝑞) uniquely for each image point?

• Assume shape along the occluding boundary is known

• Constraints on neighboring normals — integrability

• Smoothness

Shape from shading

Yes, a slightly ugly optimization

Synthetic results

Shape from Shading: “Real” Results

• These single image methods work poorly in practice

• Why? The assumptions are quite restrictive

Shape from Shading

Need more information:

•Add more constraints: Shape-from-shading

• Take more images: Photometric stereo

Photometric stereo Output:

• 3D shape of object

• Albedo at (𝑥, 𝑦)

Input: Several images

• Same object

• Different lightings

• Same pose

Photometric stereo Image brightness:

𝐧

𝐬𝟏

𝐬𝟑

𝐯

𝐬𝟐

𝐼1 = 𝜌𝐧 ∙ 𝐬𝟏

𝐼2 = 𝜌𝐧 ∙ 𝐬𝟐

𝐼3 = 𝜌𝐧 ∙ 𝐬𝟑

𝐼 = 𝜌 ⋅ 𝑘 ⋅ cos𝜃𝑖 = 𝜌𝐧 ⋅ 𝐬 where k = 1

Write this as a matrix equation:

𝐼1𝐼2𝐼3

= 𝜌𝐬𝟏

𝑇

𝐬𝟐 𝑇

𝐬𝟑 𝑇

𝐧

Solving the equations: Linear

𝐼1𝐼2𝐼3

=𝐬𝟏

𝑇

𝐬𝟐 𝑇

𝐬𝟑 𝑇

𝜌𝐧

𝐈 𝐒 𝐧 3 × 1 3 × 3 3 × 1

𝐧 = 𝐒−1𝐈

𝜌 = 𝐧

𝐧 =𝐧

𝐧 =𝐧

𝜌

𝐈 and 𝐒 are known

inverse

Adding more light sources

Get better results by using more (𝑀) lights:

𝐼1⋮𝐼𝑀

=𝐬𝟏

𝑇

⋮𝐬𝑀

𝑇

𝜌𝐧

Least squares solution:

𝐈 = 𝐒𝐧 𝐒𝑇𝐈 = 𝐒𝑇𝐒𝐧 𝐧 = 𝐒𝑇𝐒 −𝟏𝐒𝑇𝐈

Solve for𝜌, 𝐧 as before

𝑀 × 1 = 𝑀 × 3 3 × 1

Moore-Penrose pseudo inverse min 𝐈 − 𝐒𝐧 2

2

Photometric stereo: pq space

𝑝

𝑞

𝑝𝑠1, 𝑞𝑠

1

𝑝𝑠2, 𝑞𝑠

2

𝑝𝑠3, 𝑞𝑠

3

Results: Lambertian sphere

Input Images

Estimated Albedo Estimated Surface Normals

Needles are projections of surface normal on image plane

Photometric stereo: Lambertian toy

Input Images

Estimated Surface Normals Estimated Albedo

Surface Normals

Albedo

And from GT: www.trimensional.com

Photometric stereo: Limitations

Big problems • Doesn’t work for shiny things, semi-translucent things • Shadows, inter-reflections Smaller problems • Camera and lights have to be distant • Calibration requirements

– Measure light source directions, intensities – Camera response function

A real application that works

A real application that works

Human shape from shading

Thomas R et al. J Vis 2010; 10:6

Also check Ramachandran’s work on Shape from Shading by Humans http://psy.ucsd.edu/chip/ramabio.html