Application of Image Enhancement Techniques for Shape Reconstruction Using Shape from

Shading loam Ullah and Syed Hassan Amin

Department o/Computer Science, FAST, National University o/Computer and Emerging Sciences

Peshawar Campus, Pakistan

inamullahlOl®gmail.com and Hassan.amin®nu.edu.pk

Abstract-Algorithms for reconstructing shape and geometry from 2D images are very sensitive to the light, camera, surface and texture parameters. It is very difficult to accurately recover these parameters under real life conditions. Variations in these parameters result in significant changes in the quality of the reconstructed 3D shapes. This paper proposes a novel method for dealing with the variations in light, camera and surface properties. This paper shows that image enhancement techniques can be used to improve performance of the shape reconstruction algorithms like shape-from-shading by altering these parameters in a controUed manner. As a consequence of image enhancement the reconstructed results are consistently improved regardless of variations in imaging conditions. The reasons for improved results of shape reconstruction algorithms are noise removal, removal of very high frequency content, and altered surface properties.

Keywords - image enhancement; shape reconstruction; shape from shading; image formation model

I. INTRODUCTION

Shape reconstruction from single 2D images is an active research area It involves extracting shape or depth information from single or multiple 2D images. It has several applications in different fields, like biometric, robotics, and medical. There are several algorithms in literature which employ different techniques to achieve the goal [2], [8]. One of them is shape from shading which uses shading information to estimate depth values [12], [5], [4], [6]. There are several techniques even in shape from shading to estimate the depth values, employing different approaches [15], [3]. Shape from shading and other shape reconstruction approaches suffer from problems in variations in lighting conditions, surface types or multi-material surfaces, ambiguity, and specular highlights [14]. Due to these problems, the shape reconstruction algorithms are difficult to apply for practical, real world applications [9], [10], [11].

In past, no significant work is done on pre-processing or enhancing input images for improving the results of shape reconstruction algorithms. It is due to the fact that image enhancement alters an image which in turn disturbs the under-lying imaging or camera model. Contrary to imaging model, the 3D reconstruction algorithms try to simulate the image-formation model in reverse to estimate the depth values. This apparently makes the

978-1-61284-941-6/11/$26.00 ©2011 IEEE

image enhancement for 3D reconstruction very complicated which might produce un-predictable results. But we showed in this paper that some simple image­enhancement techniques can be used to improve results of shape reconstruction algorithms, mitigating for unknown or erroneous lighting and surface reflectivity parameters. This at the other end enables one to even keep the light­source direction constant for a set of images with apparently different light-source directions.

II. SHAPE RECONSTRUCTION AND THE PROBLEM

Most of the shape reconstruction algorithms take several parameters related to the underlying imaging model. These parameters are the conditions at the time of capturing the image. Such parameters include light­source direction, nature of the light-source, surface types and reflectivity parameters, intrinsic camera parameters such as focal length, principal point, skew coefficient, and distortion [7].

All of the above described parameters are very difficult to recover in case of images captured in everyday life without any special lighting conditions and knowledge of the camera This makes it very hard for shape reconstruction algorithms to produce proper results for practical applications from such images. Besides this, the shape reconstruction algorithms are very sensitive to noise in the input image. The noise results in sharp peaks and disconnectivities in the reconstructed shape and obscures the overall structure [1].

III. PROPOSED SOLUTION: IMAGE ENHANCEMENT

In this paper, we propose a set of simple image enhancement techniques which improve the performance of the shape reconstruction algorithms, like shape from shading. The input images are enhanced before giving them to the specified shape reconstruction algorithm as input. The image enhancement techniques were selected from among the several techniques that were tried. These image enhancement techniques significantly improve the reconstructed shape quality.

A. Averaging Filters

Averaging filters are used for blurring. The blurring depends upon the size of the filter. When images are processed with the averaging filters, the smaller details

315

are washed out but the overall structure of the shape is enhanced. Moreover, it also reduces the noise in the image and improves the resultant reconstructed shape.

B. Gaussian Filters

Gaussian filter consists of two parameters, the filter size and the standard deviation of the filter. Gaussian filter is a low-pass filter, meaning that high-frequency contents like noise and finer details are blocked while the processed image is more blurred and unsharpened. Larger size and larger standard deviation result in more high-frequency contents being blocked. The affects of the Gaussian filter on shape reconstruction are comparable with those obtained using the averaging filters.

C. Median Filters

Median filters are used for noise and tiny details removal. They bring more improved results compared to the averaging and Gaussian filters. In most cases, the details washed out by averaging and Gaussian filters remain in the shape reconstructed from the images processed by the median filters. The median filters were found to be the best of all the enhancement techniques.

IV. EXPERIMENTS

We have used Lenna, pepper, and a face image for experiments in this paper. These images are often used as standards in image processing and shape reconstruction community. The linear approximation for shape from shading by Tsai and Shah [13] was used as experimental shape reconstruction algorithm. The reconstructed shapes consisted of sets of 3D points representing the depth values, each generated for every pixel. The Delaunay triangulation was then generated for those points for surface construction. This helped to produce somewhat smoother surface for the depth-representing points. For some shapes generated from smaller images, we have small number of representative 3D points due to which the resultant Delaunay triangulation has several disconnectivities. The figure 1 shows the original images with snapshots of the reconstructed shapes from two different angles. The shapes were reconstructed using the shape from shading algorithm by Tsai and Shah.

The light source directions were kept the same (0.875, 1.6125, 0.9) for all images and the surfaces were assumed to be pure Lambertian. These parameters were found to optimal after experimentation with several different parameters. The reconstructed shapes from the original images have significant noisy content which is mainly due to the erroneous light source direction and not fully correct assumption about the surface reflectivity. Moreover, there was some inherent salt and pepper noise in the original images, especially the Lenna and pepper images. All of these made the resultant reconstructed shape very unclear.

The figure 2, 3 and 4 show the result of applying shape from shading algorithm after the images were enhanced with averaging, Gaussian, and median filters. From these results, it can be seen that the quality of the shapes reconstructed by the shape from shading algorithm has

improved for the enhanced images as compared to those in figure 1. The main reasons for the improvement were the reduced noise, reduction in specular reflectivity of the surfaces, and reduction of the sharpness.

(a)

(d)

(g)

. -- -

(b)

(e)

(h)

(c)

(I)

(i)

Fig. I. Reconstructions obtained using Tsai Algorithm (a) Original images are shown in subimages a,d and g (b) Reconstructions are shown

in b,e and h (c) Rotated models are shown in c,f and i.

la) b)

lc d

(e) f

Fig. 2. Reconstructions obtained using Tsia algorithm after enhancing images using averaging filter (a) First column shows reconstructed

models in frontal pose (b) Second column shows reconstructed models in non-frontal pose.

V. DETAILS AND DISCUSSIONS

In this section, we will discuss what happens to the image in perspective of a shape from shading algorithm, when it is enhanced.

316

A. Modifying Suiface Properties

Smoothing filters reduce sharpening and remove high­frequency content. They also either remove smaller details and finer contents from a region or make the resulting gray level transitions smoother. If a region has specular spike, then the smoothness filters reduce the area of sharpness due the spike and make the surface so that the surface is more diffused. For Lambertian surfaces, we have brightness of same value in a region for all directions, and even if there is some shading or changing in gray-values of pixels, those are smoother and in a specific direction, the direction in which depth variations occurs. This helps the shape from shading algorithms to approximate the depth values for a region.

So, from this it can be deduced that for a fixed/specified region, the smoothness filters reduce specularity of a surface and makes it more Lambertian and diffused. After applying a smoothing filter, when an shape from shading algorithm is applied, more accurate results are obtained as the surfaces in the input image have become more Lambertian and diffused for which the shape from shading algorithm is designed.

B. Changes in Light Source Direction

Smoothing filters also correct light source direction for the image-surfaces. For each surface, the light source direction tends to come to the center of the projected region of the object. For a surface in an image which has been smoothed, the pixel values in it tend to become closer thus affecting the light source direction which results in improvement in the quality of the reconstructed shape.

a) b)

c) d

e)

Fig. 3. Reconstructions obtained using Tsia algorithm after enhancing images using gaussian filter (a) First column shows reconstructed

models in frontal pose (b) Second column shows reconstructed models in non-frontal pose.

'. -

(b)

(d)

(e) (t)

Fig. 4. Reconstructions obtained using Tsia algorithm after enhancing images using median filter (a) First column shows reconstructed models in frontal pose (b) Second column shows reconstructed models in non­

frontal pose.

C. Smoothness of High-Frequency Textured Regions

Face images have textured regions with very high frequency content like hair and shaved skin. In the image, these regions don't appear as pure Lambertian surfaces. When smoothing filters and related enhancement techniques are applied, the high-frequency content of the regions tend to reduce and at the end we have regions with somewhat smoothly varying pixel intensities. This makes the regions' surfaces near to Lambertian which in consequence provides good result in shape from shading algorithms.

D. Interactions with Smaller Artifacts and Finer Contents

Moles and scratches normally have different reflection properties and have very height content. They are often smaller in size when compared to other regions. More, if the filter window size is relatively bigger then these smaller artifacts are altered too much. The borders of these smaller regions are smoothed out and the abruptness is reduced. The proposed enhancement techniques adjusts the reflective parameters/properties and smoothes the variations as well. This helps the shape from shading algorithms to properly estimate the depth/height information when processing these regions with the surrounding regions.

E. Texture Boundaries

When texture boundaries vary smoothly through a larger distance/region, they affect the shape from shading algorithms which interpret the variations between textures as variations in depth. The proposed enhancement techniques which smoothes a region if the degree or amount of variation is less than a specified threshold. This helps in retaining the boundaries between textures in

317

face images as they are. This helps to reduce the problems in depth estimates due to texture-boundary variations.

VI. CONCLUSIONS AND FUTURE DIRECTIONS

Shape from shading techniques typically assume Lambertian surfaces, however real world surfaces seldom obey this property. This study proposes use of image enhancement techniques for shape reconstruction using shape from shading. The experimental results clearly show the qualitative improvements in reconstructed shape models as a result of applying proposed image enhancement techniques. The proposed image enhacement methods support the standard assumptions in shape from shading algorithms such as uniform light source direction and uniform surface refl ecti vity.

This paper highlights the simple logical assumptions of how the image enhancement techniques altered the parameters in the underlying imaging model and consequently how the results were affected. The techniques might bring considerable changes in parameters/constraints assumed in other shape from shading algorithms which employ different approaches and strategies for depth estimation. In contrast to image smoothing, other enhancement techniques like sharpening and contrast stretching do not improve quality of reconstruction because they increase specularity of surfaces.

318

REFERENCES

[1] A Hom Shape from shading: A method for obtaining the shape of a smooth opaque object from one view. height and gradient from shading. In PhD Thesis. MIT, 1970.

[2] AA Ahmed, AH. Farag. Shape from shading under various imaging conditions. In Computer Vision and Pattern Recognition. IEEE, 2007. S. H. Amin and D. Gillies. Analysis of 3d reconstruction In 14th International coTiference on Image Analysis and Processing, March 2007.

[3] AM. Bruckstein On shape from shading. Computer Vision, Graphics and Image Processing, pages 139-154, 1988.

[4] B. Hom Height and gradient from shading. International Journal o/Computer Vision, pages 37-75, August 1990.

[5] B. Horn Obtaining shape from shading information In The

Psychologyo/Computer Vision. McGraw-Hill, 1975. [6) E. Prados and O. Faugeras. Perspective shape from shading and

viscosity solutions. Proc. 0/ the Inti. CoTiference on Computer

Vision, 2:826-831, Oct 2003. [7) Faugeras 0. Prados E. Shape from shading. in handbook of

mathematical models in computer vision Springer, 12(2):375-388, 2006. P. Tsai and M. Shah Shape from shading using linear approximation Image and Vision Computing, pages 487-498, 1994.

[8) J. Oliensis. Shape from shading as a partially well-constrained problem CVGIP, 54(2):163-183, 1991.

[9) M. Falcone J. Durou and M. Sagona. A survey of numerical methods for shape from shading. In RopPOl1 de recherche, pages 84-89, 2004. K. Lee and C. Kuo. Shape from shading with a generalized reflectance map model. CVIU, 97(2):143-160, Aug 1997.

[10) P. Tsai J. Cryer and M. Shah. Integration shape from shading and stereo. Pattern Recognition, pages 1033-1043, 1995.

[11) Ruo Zhang, Ping-Sing Tsai, James Edwin Cryer, and Mubarak Shah Shape from shading: A survey. In IEEE Transactions on Pattern Analysis and Machine Intelligence. IEEE, March 1999.

[12) Ryan Murphy; Zo J. Wood. Using hybrid approaches to solve the challenges of shape from shading. In Proc. SPIE, 2008.