university of cagliari - uniroma1.it · 2019-02-24 · lambert's cosine law ... at least 6...

University of CagliariDepartment of Mathematics and Computer Science

Photometric Stereo under unknown lights position

Anna Concas

Joint work with R. Dessì, C. Fenu, G. Rodriguez and M. Vanzi

Workshop"Due giorni di Algebra Lineare Numerica"

Roma, February 18-19 2019

A. Concas Photometric Stereo under unknown lights position

Shape from Shading

A classical problem in Computer Vision is the 3D reconstruction ofthe shape of an object, starting from a set of pictures.

There are two main optical 3D measurement techniques:

Stereo vision or Multivision: the images are taken fromdi�erent points of view under the same light conditions.


Shape from Shading

A classical problem in Computer Vision is the 3D reconstruction ofthe shape of an object, starting from a set of pictures.

There are two main optical 3D measurement techniques:

Photometric stereo: the pictures are taken from the sameviewpoint but under di�erent lighting conditions.


Photometric Stereo in Archaeology

Photometric Stereo method has a considerable application inarchaeology, expecially considering the fact that it overcomes thedisadvantages of other methods

low cost: the observation point is the same, so only onecamera is su�cient;

PS process is less invasive;

data can be processed in a more rapid and simple way.


Experimental setting

Working assumptions

the object is at the origin of a reference system in R3;

the z-axis coincides with the optical axis;

q pictures are available, with light source at `t (t = 1, . . . , q);

each image mt has resolution r × s, with p = rs pixels;

the pixels of each image are ordered lexicographically;

the surface of the object is represented by z = u(x , y).


Experimental setting

Theoretical assumptions

We assume that

the surface is Lambertian: the k − th pixel of the image obeysLambert's cosine law

ρknTk `t = mkt , k = 1, . . . , p, t = 1, . . . , q; (1)

the light sources are at in�nity;

there are no self-re�ections or black shades on the surface;

the optical point is su�ciently far from the object.


Poisson formulation

The Poisson method for a 3D reconstruction of a Lambertiansurface is divided into:

discretize the Lambert's law;

obtain an approximate discretization of the Laplace operator;

shape reconstruction by solving a Poisson partial di�erentialequation.

Lambert's law can be written as

DNTL = M

where

D = diag(ρ1, . . . , ρp) ∈ Rp×p

N = [n1, . . . ,np] ∈ R3×p

L = [`1, . . . , `q] ∈ R3×q

M = [m1, . . . ,mq] ∈ Rp×q


PS with known lights position

When the lights positions are known, is it possible to �rst compute

NT := DNT = ML†,

and then, the factorization ND = N follows by normalizing thecolumns of N.

It is required that q ≥ 3, to compute D and N.

The minimum number of pictures is 3.


Poisson formulation

Once N is known, numerically di�erentiating the �rst twocomponents of nk , k = 1, . . . , p, we get an approximation of the

Laplacian f (x , y) of u(x , y) at each point of the discretization.

Finally, the 3D pro�le of the object can be recovered by solving thePoisson partial di�erential equation

∆u(x , y) = f (x , y)

discretized with a centered �nite di�erences method (5 pointsscheme) and solving the resulting square linear system of sizep = rs.


Photometric Stereo under unknown lighting

The problem with unknown lights position is much more intricate.

The need for accurate information about the relative position of theligths and the object is a limitation of the method.

Referring to a 4-source Photometric Stereo, some papers conjecturethat the problem can be uniquely solved using only 4 images.

[Chen, Chen, ECCV 2006] suggest an approach based on the use oflow-order spherical harmonics for Lambertian objects.

[Basri et al., Int. Jour. Comp. V., 2007] propose a method basedon the decomposition of the ligth intensity into linear combinationsof spherical harmonics.


Photometric Stereo under unknown lighting (cont.)

The problem consists of determining the rank-3 factorization

NTL = M, (2)

where N = ND.

The solution is not unique: if (N, L) is a solution, then any matrixpair (A−T N,AL), with A ∈ R3×3 nonsingular, satis�es (2).

Lemma

The matrices D, N and L are determined up to a unitary

transformation, i.e. (QN,QL), is a solution for any orthogonal

matrix Q ∈ R3×3.


Issues related to the reconstruction

The previous lemma says that the original orientation of theobserved object cannot be determined without further information,which is reasonable since only the relative position between theobject and the camera is �xed.

This indetermination requires some care in the shapereconstruction:

the system object-camera may result rotated in thereconstruction ;

there is the possibility of axes re�ections.


Finding the lights position

Let the compact singular value decomposition of M be

M = UΣV T ,

with Σ = diag(σ1, . . . , σq), U ∈ Rp×q and V ∈ Rq×q.

In our case q � p, and the size of the factors only requires q ≥ 3.

When q is small, the SVD factorization may be computed e�centlyby the standard svd.


Finding the lights position (cont.)

Since it is expected that rank(M) = 3, we set

W = [σ1u1, σ2u2, σ3u3]T and Z = [v1, v2, v3]T ,

so thatW TZ ' M.

This produces the best rank-3 approximation to the data matrix M

with respect to the Euclidean norm and the Frobenius norm.


Finding the lights position (cont.)

Using the fact that the norms ‖`t‖ are proportional to the lightintensity, which has to be known up to a proportionality constant,it can be proved the following result.

Theorem

The normal vectors and the light sources position can be uniquely

determined from NTL = M, up to a unitary transformation, only if

at least 6 images taken in di�erent lighting conditions are available.

Since the solution is unique up to a unitary transformation Q, thesystem normals-lights may be rotated, possibly with axes re�ections.


Determine the right orientation of the surface

How to determine a valid rotation matrix Q:

good shooting procedure: order the pictures by distributing thelight positions counterclockwise all around the object;

restore the original orientation by changing the sign of thethird row of N and L;

the orthogonal matrix contained the direction of the axes

Q = [v1 v2 v3]

determines the sought rotation;

the matrices N and L are replaced by QN and QL.


A numerical experiment: synthetic data

u(x , y) = 12e

x sin(πx) sin(πy)



data set (101× 101 pixels)



1 2 3 4 5 6 710

-15

10-10

10-5

100

105

singular values of the data matrix M



-0.6

3

-0.4

4

0.5

-0.2

0

0.2

0

0.4

5

0

recovered light directions before orientation

-0.5

2

-0.5

6

1

7



2

0

1

0.2

0.4

0.4

0.50.2

recovered rotated light directions (error ~1e-16)

3

0.6

0

7

0-0.2

-0.4

4

-0.5-0.6

6

5



1

-0.5

0

0.5 1

reconstruction (error ~1e-04)

0.5

0

0-0.5

-1 -1


Synthetic data with lights at �xed distance

distance σ3/σ4 Elights Esurface

∞ 3.07e+15 9.89e-16 2.69e-041000 2.19e+03 1.95e-04 1.39e-03100 2.18e+02 1.95e-03 1.41e-0210 2.15e+01 1.95e-02 1.45e-011 1.77e+00 4.52e-01 3.89e+00

computer generated pictures

unit for distance is object width

σ3/σ4 represents �closeness to rank 3�

Elights = ‖L−L‖F‖L‖F , Esurface = ‖U−U‖F

‖U‖F


Experimental data set

the Shell data set



lights directions identi�ed by the reconstruction algorithm


Thanks for your attention


university of cagliari - uniroma1.it · 2019-02-24 · lambert's cosine law ... at least 6...

Documents