1

CSE152, Spring 2020

Shape from Shadingand

Photometric Stereo

Introduction to Computer VisionCSE 152

Lecture 11

1

CSE152, Spring 2020

Announcements

• HW2 posted, due Friday 2/14– HW as modified (1.b extra credit)

• Presidents day (No Class), Mon 2/17• Midterm, Wed 2/19

– Sample questions are posted to the class web page.

2

2

CSE152, Spring 2020

MidtermWed, Feb. 19

• In class• Full period• Coverage – everything up to today’s lecture• The primary source of material to understand is

what was presented in class and on HW. If it only in the readings, then it’s very unlikely to be in the midterm.

• “Cheat sheet” – you can prepare a one-sided sheet of notes. It must be handwritten. (After the midterm, save your sheet since you can use the other side for the final).

• No calculators.

3

CSE152, Spring 2020

Incomplete list of topics covered…• Camera models• Factors in producing

images• Homogenous Coordinates, • Vanishing points• Lenses, Distortion• Sensors• Rigid transformation• Intrinsic and extrinsic

camera models

• Quantization/Resolution • Illumination• Reflectance

– BRDF– Lambertian– Specular– Phong

• Photometric Stereo– Lambertian– Normal estimation

4

3

CSE152, Spring 2020

Topics cont.• Noise

– Additive, Gaussian noise• Filtering, linear,

convolution with Kernel– Averaging/smoothing– correlation– Sharpening– Derivatives– Gaussian filter– Seperability– Median filers

• Corner detection• SIFT

• Stereo– Epipolar constraint– epipoles– Rectification– Reconstruction– Essential matrix– Fundamental matrix– 8 point algorithm– Calibration– Matching – trinocular

• Structure from Motion– RANSAC– Constraints– Bundle adjustment

5

CSE152, Spring 2020

A few more words on Photometric Image Formation

6

4

CSE152, Spring 2020

Last lecture: Photometric Image Formation

• Light gathering limitation of pinhole camera & lenses

• Thin lens – equation, focus, depth of field, circle of confusion,

• field of view• Aberations – sphereical, astymatism,

chromatic, distortion, vignetting• Light sources, • Color cameras• Radiometry – radiance, irradiance• BRDF7

CSE152, Spring 2020

Radiance• Power traveling at some point

in a specified direction, per unit area perpendicular to the direction of travel, per unit solid angle

• Units: watts per square meter per steradian : w/(m2sr1)

Irradiance• How much light is arriving at a

surface?• Irradiance E(x) – power per unit

area: W/cm2

• Total power arriving at the surface is given by adding irradiance over all incoming angles

xx

dA

q

dw

(q, f)

wq ddAPL)cos(

=

8

5

CSE152, Spring 2020

BRDF• Bi-directional Reflectance

Distribution Function r(qin, fin ; qout, fout)

• Function of– Incoming light direction:

qin , fin– Outgoing light direction:

qout , fout

• Ratio of incident irradiance to emitted radiance

n(qin,fin)

(qout,fout)

Li

Lo E

I

• Radiance Li from light source strikes surface and leads to radiance Lo which strikes image plane with Irradiance E causing pixel intensity I.

9

CSE152, Spring 2020where ωi=(θi, ϕi)

Emitted radiance Lr(x,ωr) in direction ωr for incoming radiances over the hemisphere

Lr (x,ωr ) = ρ(x,ωi ,ωr )H 2∫ Li (x,ωi )cosθidωi

10

6

CSE152, Spring 2020

Surface Reflectance Models

• Lambertian (1 parameter)• Phong (4 parameters)• Physics-based

– Specular [Blinn 1977], [Cook-Torrance 1982], [Ward 1992]

– Diffuse [Hanrahan, Kreuger 1993]

– Generalized Lambertian [Oren, Nayar 1995]

– Thoroughly Pitted Surfaces [Koenderink et al 1999]

– Hair, Skin, Cloth

Common Models Arbitrary Reflectance

• Phenomenological[Koenderink, Van Doorn 1996]

• Non-parametric models• Anisotropic• Non-uniform over surface• BRDF Measurement [Dana et

al, 1999], [Marschner ]

11

CSE152, Spring 2020

Lambertian Surface

The intensity (irradiance) I(u,v) of a pixel at (u,v) is:

• a(u,v) is the albedo of the surface projecting to (u,v)• n(u,v) is the direction of the surface normal• s0 is the light source intensity• s is the direction to the light source

ns

a

I(u,v)

^

I (u,v) = a(u,v)n(u,v)• s0s

12

7

CSE152, Spring 2020

Full Phong Reflectance Model

where

ia, im,d, im,s are brightness of ambient, diffuseand specular light sources.

ka, kd, ks,𝛼 parameterize Phong BRDF at a point

13

CSE152, Spring 2020

Shadows cast by a point source• A point that can’t see the source is in shadow• For point sources, two types of shadows: cast

shadows & attached shadows

Cast Shadow

Attached Shadow

14

8

CSE152, Spring 2020

Penumbra and Umbra(soft vs. hard shadow)

15

CSE152, Spring 2020

Hard vs soft shadows for different light source

Hard shadow(point light)

Soft shadow(small square light source)

Z. Dimitry, https://slideplayer.com/slide/5184420/

16

9

CSE152, Spring 2020

Hard vs Soft Shadow

17

CSE152, Spring 2020Figure from “Mutual Illumination,” by D.A. Forsyth and A.P. Zisserman, Proc. CVPR, 1989, copyright 1989 IEEE

At the top, geometry of a gutter with triangular cross-section; below, predicted radiosity solutions, scaled to lie on top of each other, for different albedos of the geometry. When albedo is close to zero, shading follows a local model; when it is close to one, there are substantial reflexes.

Inter-reflections

18

10

CSE152, Spring 2020

Shading reveals 3-D surface geometry

19

CSE152, Spring 2020

Shape-from-X• Where X is

– Shading– Photometric stereo– Stereo– Motion– Texture– Blur– Focus– Structured light– ….

20

11

CSE152, Spring 2020

Two Shape-from-X methods that use shading1. Shape from shading: Single Image, known

light direction, known BRDF.

2. Photometric stereo: Single viewpoint, multiple images under different lighting.

21

CSE152, Spring 2020

Multi-view stereo vs. Photometric Stereo

• Multi-view (binocular) Stereo – Multiple images– Dynamic scene– Multiple viewpoints– Fixed lighting– Correspondence Hard

• Photometric Stereo– Multiple images– Static scene– Fixed viewpoint– Multiple lighting conditions– Correspondence trivial

22

12

CSE152, Spring 2020

Photometric Stereo Rigs:Fixed camera, changing lighting

Because of single fixed viewpoint, a pixel location sees the same point of the scene across all images.

23

CSE152, Spring 2020

An example of photometric stereo

Input Images

24

13

CSE152, Spring 2020

Photometric Stereo Process

1. From a single viewpoint of a static scene, acquire k images under different lighting.

2. Independently, estimate the surface normal for each pixel location from k measurements.

3. Integrate the normals to estimate depth across the surface.

25

CSE152, Spring 2020

Assumptions

We generally make the following simplifying assumptions:1. Light source is distant2. BRDF is known and has a few parameters

26

14

CSE152, Spring 2020

BRDF

• Bi-directional Reflectance Distribution Function

r(qin, fin ; qout, fout)

• Function of– Incoming light direction:

qin , fin– Outgoing light direction:

qout , fout

• Ratio of incident irradiance to emitted radiance

n(qin,fin)

(qout,fout)

27

CSE152, Spring 2020

Coordinate system

x

y

f(x,y)

Surface: s(x,y) =(x,y, f(x,y))

(x,y)

∂s(x, y)∂x

= 1,0, ∂f∂x

⎛

⎝⎜

⎞

⎠⎟

∂s(x, y)∂y

= 0,1, ∂f∂y

⎛

⎝⎜

⎞

⎠⎟

Tangent vectors:

s(x,y)

Normal vector

n = ∂s∂x×∂s∂y

= −∂f∂x,−∂f∂y,1

⎛

⎝⎜

⎞

⎠⎟

Not unit length

𝜕𝑠𝜕𝑥

𝜕𝑠𝜕𝑦n

28

15

CSE152, Spring 2020

Representing Surface Normals• The surface normal is a direction (vector)

orthogonal to the tangent plane• Unit vector n where |n| = 1. • n lies on the unit sphere• We can also represent normal in Gradient

Space as slant 𝑝 = ()(*

and tilt q= ()(+

^^

^

29

CSE152, Spring 2020

Photometeric Stereo for Lambertian Surface with Distant Known

Lighting

30

16

CSE152, Spring 2020

Distant Light Source

s

e(u,v)

• When light source is “distant”, like the sun1. The direction ,𝒔 to the light source is constant over the

surface2. The distance from points on the surface to light location is

constant, and so brightness of light doesn’t vary.• So, we can treat lighting as a constant direction ,𝒔 and strength s0

31

CSE152, Spring 2020

Lambertian Surface

At image location (u,v), the intensity of a pixel x(u,v) is:

e(u,v) = [a(u,v) n(u,v)] · [s0s ] = b(u,v) · swhere

– a(u,v) is the albedo of the surface projecting to (u,v).– /𝒏(u,v) is the direction of the surface normal.– s0 is the light source intensity.– ,𝒔 is the direction to the light source.

And– 𝒔 = 𝒔𝟎,𝒔– 𝒃 𝑢, 𝑣 = 𝑎(𝑢, 𝑣)/𝒏(𝑢, 𝑣)

ns

^ ^

a

e(u,v)

32

17

CSE152, Spring 2020

Normal Field – Normals at every location

33

CSE152, Spring 2020

Lambertian Photometric stereo• If the light sources s1, s2, and s3 are known, then we can

recover b from as few as three images. [Silver 80, Woodham81]

• For a single pixel location in 3 gray scale images, we have

[e1 e2 e3 ] = bT[s1 s2 s3 ] where e1, e2, and e3 are measured pixel intensities and we know s1, s2, and s3. We can then compute b at the pixel location by solving a linear system.

• Normal is: = b/|b|, albedo is: a = |b|

[ ][ ] 1321-= 321

T sssb eeen

34

18

CSE152, Spring 2020

Computing b for every pixel• Note: S-1 = [s1 s2 s3 ]-1 only needs to be computed. Not per

pixel.• Slow way: Loop over every pixel and apply

• Better: For an image with n pixels,• Let

where ei,j is the measured image intensity at pixel i in image j, and row i of B is the surface normal scaled by the albedo (b) for pixel i

• So image formation equation is E = BS• Solving for B, we have: B = ES-1

E =

e1,1 e1,2 e1,3e2,1 e2,2 e2,3e3,1 e3,2 e3,3

!

⎡

⎣

⎢⎢⎢⎢⎢⎢

⎤

⎦

⎥⎥⎥⎥⎥⎥

, B =

b1,1 b1,2 b1,3b2,1 b2,2 b2,3b3,1 b3,2 b3,3

!

⎡

⎣

⎢⎢⎢⎢⎢⎢

⎤

⎦

⎥⎥⎥⎥⎥⎥

, S = s1 s2 s3⎡⎣⎢

⎤⎦⎥

[ ][ ] 1321-= 321

T sssb eee

35

CSE152, Spring 2020

What if we have more than 3 Images?Linear Least Squares

[e1 e2 e3…en] =bT[s1 s2 s3…sn]

Rewrite as e = Sb

wheree is n by 1b is 3 by 1S is n by 3

Let the residual ber = e - Sb

Squaring this: r2 = rTr = (e-Sb)T (e-Sb)

= eTe - 2bTSTe + bTSTSb

∇𝒃(r2) = 0 --zero gradient with respect to b is a necessary condition for a minimum, or-2STe+2STSb=0;

Solving for b yields

b= (STS)-1STe

36

19

CSE152, Spring 2020

Plastic Baby Doll: Normal Field

37

CSE152, Spring 2020

Next step:Go from normal field to surface

38

20

CSE152, Spring 2020

Depth recover as solution to partial differential equation

• From estimated n =(nx, ny, nz) at each pixel (x,y), compute the slant p(x,y) and tilt q(x,y)

p(x,y) = - nx(x,y)/nz(x,y)q(x,y) = - ny(x,y)/nz(x,y)

• But p and q are just just the partial derivatives of the depth function f(x,y) with respect to x and y.

• System of two first order partial differential equations (pde)

• Assume surface and derivatives are continuous and then solve pde for f(x,y)

∂f (x, y)∂x

= p(x, y)

∂f (x, y)∂y

= q(x, y)

39

CSE152, Spring 2020

Step 2: Recovering the surface f(x,y)Many methods: Simplest approach1. Integrate along the top row (x,0) to get f(x,0)

2. Then integrate along each column starting with f(x,0) of the first row

f(x,0)

∂f∂x

= p(x, y)

∂f∂y

= q(x, y)

f (x,0) = p(x ',0)dxx '=0

x

∫ ' or for a discrete image f(i,0) = p( j,0)j=1

i

∑ = f (i −1,0)+ p(i,0)

40

21

CSE152, Spring 2020

What might go wrong?

• Height z(x,y) is obtained by integration along a curve from (x0, y0).

• If one integrates the derivative field along any closed curve, on expects to get back to the starting value.

• Might not happen because of noisy estimates of (p,q)

ò ++=),(

),(00

00

)(),(),(yx

yx

qdypdxyxzyxz

[not responsible for this on exams]41

CSE152, Spring 2020

What might go wrong?

yf

xxf

y ¶¶

¶¶

=¶¶

¶¶

xq

yp

¶¶

=¶¶

Integrability. If f(x,y) is the height function, we expect that

In terms of estimated gradient (p,q), this means:

where p=-nx/nz, q=-ny/nz with n=[nx ny nz]

But n (and in turn p, q) are estimated independently at each pixel, equality is not going to exactly hold

[not responsible for this on exams]42

22

CSE152, Spring 2020

Horn’s Method[ “Robot Vision, B.K.P. Horn, 1986 ]

• Formulate estimation of surface height z(x,y) from gradient field by minimizing cost functional:

where (p,q) are estimated components of the gradient while zx and zy are partial derivatives of best fit surface

• Solved using calculus of variations – iterative updating

• z(x,y) can be discrete or represented in terms of basis functions.

• Integrability is naturally satisfied.

dxdyqzpz yx22

Image

)()( -+-òò

[not responsible for this on exams]43

CSE152, Spring 2020

Input Images

44

23

CSE152, Spring 2020

Recovered albedo

45

CSE152, Spring 2020

Recovered normal field(shown on recovered 3D structure)

46

24

CSE152, Spring 2020

Surface recovered by integration

47

CSE152, Spring 2020

Lambertian Photometric Stereo

Reconstruction with Albedo Map

48

25

CSE152, Spring 2020

Without the albedo map

49

CSE152, Spring 2020

Another person

50

26

CSE152, Spring 2020

No Albedo map

51

CSE152, Spring 2020

Artifacts from assuming Lambertian Model

52