RECONSTRUCTION OF NON-LAMBERTIAN SURFACES BY FUSION OF SHAPE FROMSHADING AND ACTIVE RANGE SCANNING

Steffen Herbort, Arne Grumpe, Christian Wohler

Image Analysis Group, Dortmund University of TechnologyOtto-Hahn-Str. 4, D-44227 Dortmund, Germany

ABSTRACT

In this paper, we present an algorithm for the fusion of sur-face normals estimated based on Shape from Shading withabsolute depth data under exploitation of the mutual advan-tages, regarding non-Lambertian surfaces with non-uniformalbedos. While photometric 3D reconstruction methods yielddense surface detail information which is reliable on smallscales, active range scanning provides absolute depth datawhich are typically noisy on small scales but reliable on largescales. The proposed algorithm applies an iterative refine-ment to the reconstructed surface in order to suppress errorsthat result from measurement uncertainties in the surface nor-mals and the absolute depth data by simultaneous minimiza-tion of a global error functional. The obtained surface is thebest fit to the observed image intensities and depth data. Weapply our framework to small-scale real-world objects and toregions of the lunar surface.

Index Terms— 3D surface reconstruction; Shape fromShading; range scanning; data fusion

1. INTRODUCTION

Depth estimation and 3D surface reconstruction based onphotometric data have been examined extensively sinceHorn’s introduction of Shape from Shading (SfS, [1]) andWoodham’s work on Photometric Stereo (PS, [2]). Algo-rithms like SfS and PS that determine surface gradients in-evitably need to compute the actual surface from the givengradient field, which is commonly prone to systematic errors.Horn initially solved that problem though his method of char-acteristic strip expansion [3], and later refined his approachby application of a reflectance map, see e.g. [4]. A directanalytical solution for the depth from gradient reconstructionproblem is proposed in [5], which mainly improves the effi-ciency of iterative approaches like [3]. Further methods forsurface shape reconstruction are assessed thoroughly in [6].

Improvements of the results obtained by current 3D sur-face reconstruction devices regarding accuracy, speed, price,and generalization of application can be achieved by using thephotometrically determined relative gradient data as a supple-ment for absolute depth data. General solutions for the case

of directly available gradient and absolute depth data are pro-posed e.g. in [7] and [8], where it is shown that replacing thenoisy high-frequency part of the absolute depth data with thehigh-frequency information from gradient based approachesresults in a highly detailed 3D surface reconstruction. Gener-alizations of the Lambertian case are examined e.g. in [9, 10]for metallic surfaces, furthermore for multi-view settings in[11, 12, 13] and for moving objects in [14].

2. 3D RECONSTRUCTION ALGORITHM

We use the following notations: The surface gradients aregiven by p and q, the surface by z, the image intensity byI , the reflectance by R, and the lateral coordinates by x andy. Subscript indices denote partial derivatives. For clarity, thedependence on x and y is omitted where possible. The simul-taneous recovery of both surface gradients and height fromimage intensities is achieved by Horn [3] through the varia-tional minimization of a global error functional composed ofthe intensity error EI =

∫∫(I − R)2dxdy and the integrabil-

ity error Eint =∫∫ [

(zx − p)2 + (zy − q)2]dxdy. We extend

Horn’s approach with the possibility to handle absolute depthdata in addition to gradient data by adding an error term withrespect to the corresponding digital elevation model (DEM),

EDEM =

∫∫ [fσDEM

(∂zDEM

∂x

)− fσDEM(p)

]2

+[fσDEM

(∂zDEM

∂y

)− fσDEM(q)

]2

dx dy

(1)

to Horn’s error functional E = EI + γ ·Eint. EDEM measuresthe deviation between the large-scale gradients of the depthdata and those of the optimized surface z. Depth data obtainedby active range scanning usually contains increased high fre-quency noise compared to the SfS gradient data. Therefore,we eliminate the noisy high-frequency part by introducing anadequate filter function fσDEM , which we implement as a Gaus-sian filter of width σDEM. The low-frequency active rangescanning data guarantee accuracy on large scales while thehigh-frequency information of the image is not affected.

The overall reconstruction error then amounts to EΣ =EI + γ · Eint + δ · EDEM with γ, δ ∈ R as weight factors.

For the minimization of EΣ, we use the calculus of variations[3]). This ultimately yields the iterative scheme

p(x, y)(n+1) = z(n)x +

1

γ

[I −R

(z(n)x , z(n)

y

)] ∂R∂p

∣∣∣∣z(n)x

+

δ

γ

∫∫ [fσDEM

(∂zDEM

∂x

)− fσDEM(z(n)

x )

]∂fσDEM(p)

∂p

∣∣∣∣z(n)x

dxdy

z(n+1) = z(n) − ε2

κ

[p(n+1)x + q(n+1)

y

]. (2)

The solution for q is obtained analogously. In Eq. (2), z isan average over the κ neighboring pixels while ε denotes theoffset between adjacent pixels.

Notably, the double integral in Eq. (2) can be imple-mented efficiently for the complete image as two subsequentlinear filtering operations as long as fσDEM is a linear filter.

Since the minimization of the intensity error is an essen-tial part of the optimization, the choice of the reflectancemodel is crucial for the success of the 3D reconstruction.Within our surface reconstruction scheme, we apply analyti-cal models to account for different surface reflectance behav-iors. For metallic surfaces, we use the reflectance model

R(ρ, ϑi, ϑe, α) = ρ

[cosϑi +

K∑k=1

Ak cosBk(ϑr)

]with cosϑr = 2 cosϑi cosϑe − cosα (3)

as described in [9]. The reflectance of the surface is thusbased on the incidence angle ϑi, the emission angle ϑe, thephase angle α (i.e. the angle between the illumination direc-tion and the viewing direction), and a locally varying propor-tionality factor ρ(x, y) loosely termed albedo. The angle ϑrdenotes the angle between the direction of specular reflectionand the viewing direction. Lambertian surfaces are modeledby Ak = 0 ∀ k. For metallic surfaces we use K ≥ 1 for mod-elling a broad specular lobe and, if necessary, a narrow spec-ular spike [9]. The parameters Ak and Bk are determinedempirically by gonioreflectometric examination of a sampleof the surface material.

For particulate planetary surfaces, an advanced analyti-cal reflectance model of the lunar surface is the Hapke model[15], which is an analytic approximation to the radiative trans-fer equation. Specifically, we use the isotropic multiple scat-tering (IMSA) approximation for our 3D reconstruction ap-proach. The analytic form is too long to be repeated here; cf.[15] for details. One important model parameter is the single-scattering albedow, which is the physical albedo of the grainsthat make up the surface, where the dependence of the HapkeIMSA reflectance on w is nonlinear. Here, the albedo ρ(x, y)corresponds to the spatially non-uniform physical albedo wand is recovered by our 3D reconstruction algorithm, whilethe other Hapke parameters are chosen according to [16].

In order to simultaneously reconstruct the surface and ap-proximate the locally varying albedo, we encapsulate the opti-mization according to Eq. (2) into a further iterative scheme.

In each iteration m the surface z(m) and the correspondinggradients p(m) and q(m) are used to determine the anglesϑ

(m)i and ϑ(m)

e . Afterwards, an approximation to the spatiallynon-uniform surface albedo ρ(x, y)(m) is computed by solv-ing R

(ρ(m)(x, y), ϑ

(m)i , ϑ

(m)e , α

)= I(x, y) pixel-wise with

respect to ρ(m)(x, y) and applying a Gaussian low-pass fil-ter g

σ(m)ρ

with kernel width σ(m)ρ to ρ(m)(x, y). The filtered

albedo map gσ(m)ρ∗ ρ(m)(x, y) is used to compute p(m+ 1),

q(m+ 1), and z(m+ 1) for the subsequent step (m+ 1). Asthe iteration proceeds, the kernel width σ(m)

ρ is decreased toallow an increasing spatial resolution of the albedo map.

3. EXPERIMENTAL EVALUATION

In our laboratory setting, depth data is obtained with a rangescanning device relying on the projection of structured light(Vialux zSnapper Vario), while pixel-synchronous images areacquired with the attached AVT Pike F-421 monochrome in-dustrial camera with a native resolution of 2048 × 2048 pix-els. Illumination is provided by a green (λ ≈ 525 nm)LED with calibrated position, illumination direction, andstrength. For the examination of metallic surfaces, high dy-namic range images are acquired which were normalized tothe gray value range 0–1. The lateral image resolution corre-sponds to 42.6 µm per pixel.

We found suitable values of γ to be of the order 10−6

for the small objects and 10−2 for the lunar surface sections.Values of δ were of the order 10−1 for the Lambertian objectsurface for which very accurate depth data were available, butwere decreased for the metallic surface (∼10−4) and the lunarsurface (∼10−5). The kernel width σDEM is of the order 10pixels such that it cancels out high-frequency structures of therefined 3D reconstruction not present in the depth data. Thekernel width σ(m)

ρ was chosen to be high enough to suppresstopography related artefacts in the albedo. Initially (m = 0),the value of σ(m)

ρ was set to ∼102 pixels, while it was gradu-ally reduced to about 10 pixels during the subsequent iterationsteps.

For the lunar surface, we used laser altimetry data ob-tained with the Lunar Orbiter Laser Altimeter (LOLA) in-strument on board the Lunar Reconnaissance Orbiter (LRO)spacecraft in combination with the Chandrayaan-1 MoonMineralogy Mapper (M3) 85-channel multispectral radi-ance imagery (http://pds.jpl.nasa.gov). For each channel,a division by the corresponding solar irradiance yields thewavelength-specific physical reflectance. The lateral resolu-tion of the images corresponds to 140 m per pixel. The M3

data set represents an ideal starting point for our reconstruc-tion algorithm since all data are provided in the same coordi-nate system. Specifically, we used the M3 image channel at1209.6 nm as it has a low noise level.

For the Lambertian plaster surface (cf. Fig. 1), details such

as small scratches invisible in the range scanner data are re-covered by our method while noise and spurious structuresinvisible in the image are suppressed. For the raw cast ironsurface of a flange examined in Fig. 3, we used Eq. (3) withK = 1, A1 = 1.5, and B1 = 2. “Holes” in the range scannerdata are filled based on the photometric image information,while small-scale noise and spurious structures that have nocounterpart in the image are suppressed. In both cases, thesurface albedo is largely uniform and is thus not shown here.

For the regions of the lunar surface shown in Figs. 2 and 4,our method is able to recover the non-uniform surface albedo.The amount of surface detail in our refined DEMs (especiallysmall craters and rilles) is much larger than in the laser altime-try DEMs, which appear rather blurred. No topographic mapsof sufficiently high resolution are available for the examinedregions as ground truth, but we found that the small craters inour refined DEMs have depths that correspond to one-fifth oftheir diameters, as expected from lunar crater statistics [17].In order to estimate the absolute accuracy, a synthetic im-age was created based on the constructed DEM together withthe non-uniform albedo map, the reflectance model, and theknown illumination geometry. This synthetic image is used asan input for our algorithm, which reconstructs the underlyingground truth surface with an elevation accuracy (RMSE) of10 m. The RMSE of the reconstructed albedo corresponds to0.005, where the mean albedo amounts to 0.44.

4. SUMMARY AND CONCLUSION

In this paper, we have presented an algorithm for the fu-sion of photometrically estimated surface normals with abso-lute depth data for non-Lambertian surfaces with non-uniformalbedos based on the minimization of a global error functionalinvolving image intensity, low-pass filtered depth data, andintegrability. Experiments regarding small real-world objectsand regions of the lunar surface have shown that the lateralresolution of the 3D reconstruction results comes close to thatof the images, such that a large amount of small-scale surfacedetail is recovered by our method while noise and spuriousstructures in the absolute depth data are suppressed. Futurework will address an evaluation in terms of independently ob-tained ground truth data.

5. REFERENCES

[1] B. K. P. Horn, “Shape from shading: A method for ob-taining the shape of a smooth opaque object from oneview,” TR 232, MIT, 1970.

[2] R. J. Woodham, “Photometric method for determiningsurface orientation from multiple images,” Optical En-gineering, vol. 19, no. 1, pp. 139–144, 1980.

[3] B. K. P. Horn, “Height and gradient from shading,” A.I.Memo 1105A, MIT, AI Lab, 1989.

[4] B. K. P. Horn and R. W. Sjoberg, “Calculating the re-flectance map,” AI Memo 498, MIT, AI Lab, 1978.

[5] T. Simchony, R. Chellappa, and M. Shao, “Direct an-alytical methods for solving poisson equations in com-puter vision problems,” IEEE Trans. PAMI, vol. 12, no.5, pp. 435–446, 1990.

[6] A. Agrawal, R. Raskar, and R. Chellappa, “What is therange of surface reconstructions from a gradient field?,”in ECCV, 2006, pp. 578–591.

[7] J. E. Cryer, P.-S. Tsai, and M. Shah, “Integration ofshape from shading and stereo,” Pattern Recognition,vol. 28, no. 7, pp. 1033–1043, 1995.

[8] D. Nehab, S. Rusinkiewicz, J. Davis, and R. Ramamoor-thi, “Efficiently combining positions and normals forprecise 3d geometry,” ACM Trans. Graphics (Proc. SIG-GRAPH), vol. 24, no. 3, pp. 536–543, 2005.

[9] P. d’Angelo and C. Wohler, “Image-based 3d surfacereconstruction by combination of photometric, geomet-ric, and real-aperture models,” ISPRS J. Phot. RemoteSensing, vol. 63, no. 3, pp. 297–321, 2007.

[10] C. Wohler and P. d’Angelo, “Stereo image analysis ofnon-lambertian surfaces,” IJCV, vol. 81, no. 2, pp. 529–540, 2009.

[11] N. Joshi and D. J. Kriegman, “Shape from varying illu-mination and viewpoint,” in ICCV, 2007.

[12] J. Lim, J. Ho, M.-H. Yang, and D. Kriegman, “Passivephotometric stereo from motion,” in ICCV, 2005, vol. 2,pp. 1635–1642.

[13] L. Zhang, B. Curless, A. Hertzmann, and S. M. Seitz,“Shape and motion under varying illumination: Uni-fying structure from motion, photometric stereo, andmulti-view stereo,” in ICCV, 2003, vol. 1, pp. 618–626.

[14] D. Simakov, D. Frolova, and R. Basri, “Dense shapereconstruction of a moving object under arbitrary, un-known lighting,” in ICCV, 2003, vol. 2, p. 1202.

[15] B. Hapke, Theory of reflectance and emittance spec-troscopy, Cambridge University Press, 1993.

[16] J. Warell, “Properties of the Hermean regolith: IV.Photometric parameters of Mercury and the Moon con-trasted with Hapke modelling,” Icarus, vol. 167, no. 2,pp. 271–286, 2004.

[17] R. J. Pike, “Control of crater morphology by gravity andtarget type: Mars, earth, moon,” in Proc. Lunar Planet.Sci. XI, 1980, pp. 2159–2189.

(a) (b) (c)

(d) (e) (f)

z [µm]

0

50

100

150

200

250

300

350

x [pixels]

smoothed depth data

depth dataoptimization result 201px / 8.55mm

1mm

Fig. 1. Results for an object with Lambertian surface: (a)Region of interest (intensity data). (b) Range scanner data,shaded with Lambertian reflectance. (c) High-frequency com-ponent of the range scanner data. (d) Cross-sectional profilethrough row indicated in (a). Vertical offsets were applied tothe curves for clarity. (e) Optimized 3D reconstruction shadedwith Lambertian reflectance. (f) High-frequency componentof the optimized 3D reconstruction.

(a) Reflec-tanceimage

(b) Albedomap

(c) Original DEM (LOLA)

(d) Refined DEM

Fig. 2. Lunar crater Alphonsus. The gray value range of thesingle-scattering albedo map (b) is 0–0.4. In the DEMs (c)and (d), the vertical axis is four times exaggerated, and thealbedo map is used as an overlay for the shaded perspectivicalviews. The low-albedo patches are dark volcanic ash deposits,while high-albedo regions correspond to parts of the craterrim consisting of bright highland material.

(a) (b) (c)

(e) (f)

(d) (g)

0

50

100

150

200

250

300

350

400

450

500z [µm]

20 60 100 140 180x [pixels]

optimizationresult

depthdata

smootheddepth data

201px / 8.55mm

1mm

Fig. 3. Results for the raw cast iron surface of a flange: (a)–(f) as in Fig. 1; the specular reflectance model according toEq. (3) has been used for the shading of (b) and (e). In (b),“holes” in the range scanner data appear as smooth patchessince they were filled by interpolation. (g) Rendered surfaceusing a new viewpoint. The conspicuous dent with its heightof about 120 µm is marked by an arrow.

(a) Reflectanceimage

(b) Albedo map

(c) Original DEM (LOLA)

(d) Refined DEM

Fig. 4. Mons Gruithuisen Gamma and Delta. The gray valuerange of the single-scattering albedo map (b) is 0–0.4. In theDEMs (c) and (d), the vertical axis is four times exaggerated,and the albedo map is used as an overlay for the shaded per-spectivical views.