Shapelets Correlated with Surface Normals Produce Surfaces

Peter Kovesi

School of Computer Science & Software Engineering

The University of Western Australia

Shape from Shading and Shape from Texture attempt to

• Calculate the surface normal direction at each point in the image, and then

• Infer the surface shape from the surface normals.

Surface normals are usually calculated in terms of slant and tilt

N

slant

Slant is the angle between the viewing direction and the surface normal

surface

eye

Tilt is the angle of the projection of the surface normal on a plane perpendicular to the viewing direction

viewer plane perpendicularto viewing direction

N

slant

tilt

projection of surface normal

Tilt often has an ambiguity of Problem:

Tilt often has an ambiguity of Problem:

Problem:

Regularization schemes are often used to deal with noise. However, they typically involve the minimization of an Energy Term that has mixed units

€

E = α fx − ˆ f x∑2

+β ˆ f xx

2

+ γ I − ˆ n ⋅s2

+ ...

surface gradient error(highly nonlinear with surface angle)

bending energy shading error

Problem:

Regularization schemes are often used to deal with noise. However, they typically involve the minimization of an Energy Term that has mixed units

€

E = α fx − ˆ f x∑2

+β ˆ f xx

2

+ γ I − ˆ n ⋅s2

+ ...

surface gradient error(highly nonlinear with surface angle)

bending energy

• The weighting parameters have to act as unit conversion operators.• The appropriate values will be dependent on scale. • This can be problematic if the regularization is being applied within a

multi-scale framework. • Discontinuities have to be automatically detected and treated specially.

shading error

Alternative approach:

Project the surface gradients onto a set of integrable basis functions and reconstruct the surface from these.

Frankot and Chellappa [1988] first introduced this approach using Fourier basis functions.

Features of work presented here:• Localized, non-orthogonal basis functions are used.• Correlations with basis functions are performed with

respect to slant and tilt rather than in terms of gradient in x and y.

• Ambiguities of in tilt can be considered.• Low quality reconstructions are possible with slant

information alone.

Decomposing and Reconstructing Signals with Basis Functions

signal

filtered at scale 2

filtered at scale 4

filtered at scale 6

Reconstruction obtained by summing band-passed signals over 6 scales

band-pass filters/wavelets

We only have the gradient of the surface…

We only have the gradient of the surface…

Differentiation is linear

€

f ∗ ′ ′ g = ′ f ∗ ′ g

surface band-pass filter

surface gradient

We only have the gradient of the surface…

Differentiation is linear

A band-passed version of the signal can be obtained from a correlation performed between the gradient of the signal and integral of the band-pass filter.

€

f ∗ ′ ′ g = ′ f ∗ ′ g

surface band-pass filter Integral of band-pass filter(shapelet gradient)

surface gradient

Choice of Shapelet

• The gradient of shapelet function must satisfy the admissibility condition of zero mean.

• Must ensure preservation of phase information in the signal.

• The bank of shapelet filters should provide a uniform coverage of the signal spectrum so that it is faithfully reconstructed.

These requirements are satisfied by a Gaussian

Note that correlating the gradient of one shapelet filter with the gradient of the signal will correspond to extracting a band of frequencies from the signal gradient.

Note that correlating the gradient of one shapelet filter with the gradient of the signal will correspond to extracting a band of frequencies from the signal gradient.

The signal gradient spectrum differs from the original signal spectrum in that its amplitudes are scaled by the frequency.

Note that correlating the gradient of one shapelet filter with the gradient of the signal will correspond to extracting a band of frequencies from the signal gradient.

The signal gradient spectrum differs from the original signal spectrum in that its amplitudes are scaled by the frequency.

The sum of the shapelet gradient transfer functions should form a curve that is inversely proportional to frequency to counteract the scaling of the gradient spectrum. The net result will then be an even coverage of the original signal spectrum.

Shapelet bank formed by five Gaussians with height proportional to scale.

Transfer functions of the gradients of the shapelets above.

The sum forms a curve that is approximately inversely proportional to frequency.

A bank of shapelets will not form an orthogonal basis set

This does not matter if we have a strongly redundant basis set. (Achieved by using a continuous wavelet transform.)

A non-orthogonal basis set can be considered to be the sum of several orthogonal basis sets.- We get accurate reconstruction up to a scale factor

• The scaling of the reconstruction due to the redundancy of the basis set can be determined and corrected if required.

• Overcomplete basis sets have the advantage that small changes in local features result in correspondingly small changes in the basis coefficients.

• Overcompleteness also provides robustness to noise.

In Summary:

• Correlate the gradients/normals of the surface with the gradients/normals of a set of basis functions.

• These correlation results correspond to bandpassed versions of the original signal.

• Reconstruct surface by summing correlation results.

Perform correlation in terms of slant and tilt separately

- slant and tilt are calculated by different means.- slant and tilt have differing degrees of uncertainty.

Correlation of Slant

Slant lies in the range 0 - which only provides gradient magnitude information.

The slant correlation at scale i is formed by

With no tilt information this correlation measure matches positive and negative gradients equally. (No distinction between concave or convex shape)

)tan(=∇

€

C∇i = ∇ f ∗∇ si

gradient of surface

gradient of shapelet at scale i

Correlation of Tilt

• If gradient magnitudes match and tilts match correlation is +ve

• If gradient magnitudes match and tilts differ by correlation is -ve

• If tilts differ by /2 correlation is 0

Use the cosine of the tilt difference as a correlation measure

€

Cτi = cos(τ f − τ si)

tilt of surface tilt of shapelet at scale i

Overall Correlation

€

Ci = C∇i ⋅Cτi

Reconstruction

€

R = Ci

i

∑

synthetic test surface

needle diagram ofsurface normals

surface normals reconstruction over 2 scales

reconstruction over 4 scales reconstruction over 6 scales

Shapelet reconstruction using the gradients of the range data

Range data from OSU database

Slant and tilt with Gaussian noise of 0.3 radians standard deviation

Shapelet reconstruction

Terzopoulos multigrid reconstructionFrankot Chellappa reconstruction

Allowing for Tilt Ambiguity of

)(cos2sfC −=

When tilts are aligned value is 1When tilts are orthogonal value is 0When tilts are opposite value is 1

… results in rectification of the band-pass components of the surface

Reconstruction with a tilt ambiguity of

reconstruction oframps surface

bumps surface

reconstruction

Reconstruction with slant data alone

Slant-only reconstruction of bumps Slant-only reconstruction of ramps

Slant-only reconstruction of sincSinc function

Slant-only reconstruction of face …after segmentation and negation

Shapelet reconstruction

Real data: Surface normals from texture

Sand ridges: Slant set to /4 at all pointsTilt set to 0 for white regions, for black regions

Shapelet reconstruction

Shape from Occluding Contours

Manually marked contours

Surface normals at occluding contours are perpendicular to viewing direction.

This is a powerful cue to shape.

At all points slant is set to 0, except along contours where slant is set to , and tilt is set to be perpendicular to contour.

This gradient field is not integrable.

A crude approximation of the surface normals in the scene…

Shapelet reconstruction

Manually drawn contours

Conclusions

• Reconstructions from surface normals using shapelets is simple and the process is robust to noise.

• The use of basis functions implicitly imposes a continuity constraint in the reconstruction.

• Being able to treat slant and tilt separately permits data with tilt ambiguity, or no tilt data, to be considered.

• Much can be made from limited amounts of surface normal data. This allows some some aspects of structure to be deduced from line drawings.