OMNIDIRECTIONAL VIEWS SELECTION FOR SCENE REPRESENTATION

Ivana Tosic and Pascal Frossard

Ecole Polytechnique Federale de Lausanne (EPFL)Signal Processing Institute

CH-1015 Lausanneivana.tosic, pascal.frossard}f@epfl.ch

ABSTRACT

This paper proposes a new method for the selection of sets of omni-directional views, which contribute together to the efficient represen-tation of a 3d scene. When the 3d surface is modelled as a functionon a unit sphere, the view selection problem is mostly governed bythe accuracy of the 3d surface reconstruction from non-uniformlysampled datasets. A novel method is proposed for the reconstruc-tion of signals on the sphere from scattered data, using a general-ization of the Spherical Fourier Transform. With that reconstructionstrategy, an algorithm is then proposed to select the best subset of nviews, from a predefined set of viewpoints, in order to minimize theoverall reconstruction error. Starting from initial viewpoints deter-mined by the frequency distribution of the 3d scene, the algorithmiteratively refines the selection of each of the viewpoints, in order tomaximize the quality of the representation. Experiments show thatthe algorithm converges towards a minimal distortion, and demon-strate that the selection of omnidirectional views is consistent withthe frequency characteristics of the 3d scene.

Index Terms- image based rendering, camera positioning, non-uniform sampling, omnidirectional vision, 3D scene representation

1. INTRODUCTION

The main objective of Image based rendering (IBR) is to describe a3-dimensional scene from a set of reference images, taken from dis-crete viewpoints in the space. It then allows to generate new views ofthe scene from arbitrary positions using the description provided byreference images. The increasing popularity of such approaches forthe coding of 3d scenes is certainly due to high complexity reduc-tion, compared to the classical 3d objects coding methods. Indeed,the 3d information is obtained from normal cameras, the coding stepessentially reduces to image coding, and IBR does not require spe-cial 3d rendering hardware at the user side.

IBR methods generally use the plenoptic function [2] to describethe light field that characterizes a 3d scene. Each image is a set ofsamples from the plenoptic function, and the performance of an IBRscheme therefore greatly depends on the choice of viewpoints forthe reference images, or equivalently the camera positions. Imageshave to be chosen such that all parts of the scene are visible, withsufficient sampling density for high quality reconstruction. More-over, the scene should be described with equal accuracy in all areas,

This work has been supported by the Swiss National Science Foundationunder grants 20001-107970/1 and PP002-68737.

in order to avoid over-sampling and under-sampling effects in thereconstruction of arbitrary views. In general, the number of imagesdirectly drives the size of the representation, and the bandwidth re-quirements for transmission.

Most of the previous work on camera positioning targeted the 3dscene modelling problem, where viewpoints were chosen in order tomaximize the visibility of 3d features [12], or similarly, to minimizeocclusions [7]. However, since these methods are primarily designedto ensure the complete coverage of the scene, they do not considerthe sampling accuracy of the scanned areas, which is certainly animportant factor for IBR tasks.

We propose here a framework where a 3d scene is captured bymultiple omnidirectional cameras, whose output images are appro-priately mapped into spherical images that represent the light in itsnatural radial form [5]. Processing the light field directly in thespherical domain allows to avoid discrepancies due to Euclidean ap-proximations in common IBR schemes. Moreover, omnidirectionalcameras offer generally a much wider field of view, which reducesthe number of necessary images for reconstruction of the scene. Weconsider that 3d objects can be represented as continuous functionsin the 3d space, in particular as function belonging to the space ofsquare integrable functions on a unit sphere. This approach is moti-vated by recent works on representation and compression of 3d ob-jects as functions on the sphere [1, 3, 6]. By modelling a 3d scenewith a continuous function we can achieve a consistent reconstruc-tion of arbitrary views without over-sampling nor under-sampling.The view selection problem can then be reduced to a non-uniformsampling problem on the sphere. From initial set of n viewpoints, aniterative algorithm is finally proposed to selectively alter the choiceof viewpoints, such that the overall distortion in the scene recon-struction is minimized.

The paper is organized as follows. In Section 2 we describethe framework for scene representation with omindirectional views.A new method for interpolation from non-uniform samples on thesphere is proposed in Section 3. Section 4 presents the view selectionalgorithm, and experimental results are given in Section 5. Section 6concludes the paper.

2. FRAMEWORK

We consider a framework where a 3d object is observed by multi-ple omnidirectional cameras, which represent the light field under aspherical fonn. The proposed approach for view selection is basedon sampling of the 3d object surface, which is modelled as a contin-

1-4244-0481-9/06/$20.00 C2006 IEEE 2213 ICIP 2006

the sphere as:

MSE(f, f) = Ilf- 12 (f _ff f)

41 j(f(Os)f(O))2sinOddo, (3)

where f is the signal on the sphere and f is its reconstruction.

3. RECONSTRUCTION FROM A SET OFOMNIDIRECTIONAL IMAGES

Fig. 1. A 2d slice of a synthetic scene captured with n cam-eras (only 3 are displayed for simplicity).

uous function on the sphere. It is described in the form r f (0, 0),where (r, 0, o) are the spherical coordinates of a point on the objectsurface, when the center of the coordinate system coincides with thecenter of the object. This parametrization is only valid for star-shapemodels, i.e., models where each point on the surface corresponds toonly one direction in the spherical coordinate system. However, aparametrization into a finite number of spheres is feasible for morecomplex models, with an appropriate tessellation of the surface.

Figure 1 shows a 2d slice of a synthetic scene, recorded by n om-nidirectional cameras. Suppose that camera i captures the distanceof the point X to the center of the sphere, i.e., the radial coordinateof this point in the coordinate system of camera i. We will denotethis system the camera i frame (Fi), while the coordinate system ofthe object will be called the world frame (Fo). Cartesian coordinatesof a point in different frames are related via the rigid body transfor-mation [13] in the following way:

Xi = Ri Xo + Ti, (1)

where Xi and X0 are the coordinates of the point X in Fi and Forespectively, and Ri and Ti are the rotation and translation matricesof a frame Fi with reference to the frame Fo. The world frame isuniquely defined as (r, 0, o) coordinates in a spherical coordinatesystem.

We then consider a set of viewpoints P that are distributed aroundthe object. In other words, we define a discrete sampling of the worldframe Fo. In order to obtain a discrete set of values for 0 and O co-ordinates, we have chosen to use the HEALPix [9] tessellation of thesphere, which results in sphere partitions of equal area and thereforegives the same importance to all possible viewpoints (placed in cen-ters of the partitions). In HEALPix, a base mesh consists of 12 partsof the sphere, and finer meshes are obtained by successive dyadicpartitioning of each part in the base mesh. The number of possibleviewpoints is given with Ndi, = 12N'ide, where Nside 2lr andIr is the resolution level. We finally choose n, arbitrary values forthe distance coordinate r, so the total number of viewpoints in oursystem is given by Np,, = 12 nf N 'ide

The view selection problem consists in finding in P, the subsetof n viewpoints, P, C P, such that the overall distortion in thereconstruction of the scene is minimized. The Mean Square Error(MSE) is used to assess the quality of the reconstruction, defined on

By capturing depth information from multiple arbitrarily positionedomnidirectional cameras, we obtain non-uniformly distributed sam-ples on the sphere. Signal reconstruction from non-uniform samplesfor Id and 2d signals on the plane, has been studied intensively in lit-erature, where both theoretical and practical issues have been consid-ered [10]. However, interpolation from scattered data on the spherehas been studied by only a few authors [4, 11], who have mostlydeveloped the theoretical frameworks, without considering the prac-tical implementation issues. We propose here a method for objectreconstruction from non-uniform samples obtained by multiple cam-eras, based on the Fast Spherical Fourier Transform - FST [8].FST decomposes a signal that belongs to the Hilbert space of square-integrable functions on the two-dimensional sphere L2 (S2, dw) intoa series of spherical harmonics Ylm (0, ):

f(),s)=,E3 f(l,m)Yim(0,so).1EN Iml

The previous relation can be rewritten in a matrix form as:

V -a =r,

where

V = {Yylm(Qi,Pi)}qXN27a = {a(l, m)}N2X 1r {ri}qXl,

(

factor then form an initial set of images, P,,o = {Vl,o, ..., Vr,,o}.Initial positioning obtained by using this frequency estimation ap-

(9) proach highly increases the probability that the view selection algo-rithm converges towards the global minimum of the error.

The view selection algorithm then iteratively refines each of the10) initial viewpoints, as described in Algorithm 1. It basically proceeds'I l) in three steps:vl l-

(12)

andq =E= qi.By solving this linear system we obtain the values for coeffi-

cients a (1, in), which are the approximates of Fourier coefficientsfor the signal f on the sphere:

a 1f (0), Wo) -a = V-1 r. (13)

Since V is not a square matrix, finding its inverse is not an easy task.The pseudo-inverse will give a minimal norm solution, resulting ina stable reconstruction, but it is computationally very expensive. In-stead of directly solving (9), we propose to multiply each side withV* = conj (VT), which results in:

V* V a= V* r,

or equivalently:

with

Ta =R,

(14)

(15)

T V V; R= V r. (16)

* Step 1: The initial viewpoints Pr,0o = {V1,o, ..., V,,o} arechosen as the positions with the highest importance factors.

* Step 2: The first viewpoint is moved to other possible posi-tions, while keeping all viewpoints fixed. In each case, the ob-ject is reconstructed using the FST-based method from depthmaps seen only from n cameras, for each of these possiblepositions of camera 1, and creates f0o; m = 1, ..., t. Thedistortion is calculated for each one of these approximations,with respect to fa11. The minimal value for MSE will givethe best position for camera 1 at this iteration, and V1,0 is as-signed the value of the best viewpoint for camera 1. The sameprocedure is repeated for all the viewpoints, by altering oneof the positions, while all the other stay unchanged.

* Step 3: After finding the best position for camera n, the algo-rithm goes back to the camera 1 and repeats the Step 2 witha new set of camera positions P,, 1, obtained from a previousiteration. The search is continued until it reaches the stablesolution (Pa,), i.e. when the change of position of any cameradoes not lead to reduction in MSE, i.e., Pn,i = P.,i- i .

The matrix T is now a square matrix of size N2, so that solvingthe system given in (15) instead of (9) is much less computationallydemanding. Another advantage of this transformation is that addingor removing samples does not change the size of the system, as itamounts to a simple addition (resp. subtraction), as given by:

(T + AT) a = R +IAR, (17)where AT and AR are obtained using (16) for set of added (resp.deleted) samples.

4. VIEW SELECTION ALGORITHM

Now that a method has been defined to reconstruct an object fromscattered data, we can address the problem of selecting the subset ofimages or viewpoints P, that minimizes the overall distortion afterreconstruction. The view selection algorithm first selects the ini-tial viewpoints based on the frequency distribution of the 3d scene.Depth estimation is performed from all images captured from eachviewpoint in P, in order to obtain a set of samples S U7NpI" Si,where Si is a set of depths seen from ith viewpoint. The scene fis then approximated into fa11 from S, using the FST-based method(see eq. 15). Substituting fa,1 in the spherical harmonic differentialequation, the distribution of frequency w = + m2of the 3d ob-ject can be estimated. An importance factorIFi is finally computedfor each view in P, as

IF , w(q1 wC (otI g )qi

where w((O, y?) is an estimate of the frequency w at a sample (0>, (pi)originating from a view i. The n viewpoints with highest importance

Algorithm 1 View SelectionInput: P, S, fallI Pn,O = {V1,0, , Vn,0}

O;j 0repeat

Si = Un where Sk,i is a set of surface samplesseen from view Vk,Pno,i = Pn,irepeat

j =j+1PVj,i = P \ (Pnj, \3ji)={,7i 3,7iPS7,i = Su{U k l;k#7j Sk,i} form 1,...,treconstruct fj7i from PSj7' using the FST method forTm= l, ...,ItLm = MSE(fj7,Ifall1)form = 1, ..., tVji> {= 7tV LM = minmT1.tLm}Pn i { Vl,i, ** Vn ,i}

until j ni i+1;tp ~P_ ,1nPn,i n,i_I

until Pn,i = Pn Xi-I

5. EXPERIMENTAL RESULTS

The performance of the proposed method is evaluated on an arbi-trarily shaped synthetic object, shown on Figure 2. The number ofpossible viewpoints used in simulations is set to Npos = 144, with

2215

8

6,

4,

2

ol n=6n=5

-2 n=4inital

4. positions-6

-8

5~~~~~~~~~-

05 5

Fig. 2. View selection algorithm: results for 4, 5 and 6 cam-eras, with initial viewpoints set.

Table 1. MSE for reconstruction from n best viewpoints, withn=4, 5 and 6.

fl MSEmin 14 8.8556e--13 165 1.816le-13 166 9.8843e-15 22

Nside =2 and n, 3 (ri 7,2r= 10, r3 =13). The view-points selected by the view selection algorithm are represented forthe cases of n =4, 5 and 6 cameras. The initial set of viewpointschosen from the frequency distribution is also represented for n =6(for n 4 and n 5 the initial set of viewpoints is a subset of theinitial set for n 6). The resulting selection of viewpoints is con-sistent with the object shape, namely camera positions are chosen sothat the object is sampled more densely in the areas with higher fre-quency content. Moreover, it is interesting to note that most of theviewpoints are placed on the sphere with the smallest radius, whichmeans that the influence of view resolution on positioning is higherthan the visibility of the object surface, when the number of view-points is large enough.

Table 1 presents the results of camera positioning for n =4, 5and 6 cameras respectively, in te,rs of MSE of the reconstruction. Italso represents the number of iterations of the algorithm, i, which arerequired to reach convergence. We see that the MSE decreases withthe number of viewpoints, as expected, and that the convergence isquite fast, even for increasing number of cameras in the system.

6. CONCLUSIONS

This paper introduces a view selection algorithm for 3d scene rep-resentation with a set of n omnidirectional cameras. 3d objects aremodelled as functions on the unit sphere, and a method for the recon-struction of 3d objects from non-uniformly spaced samples is pre-sented. The view selection algorithm first initializes the viewpointsbased on the frequency distribution of the object. It then iteratively

refines each of the camera positions, until the distortion of the recon-struction is minimized. Experimental results show that the proposedalgorithm selects viewpoints, in such a way that parts of the scenewith higher frequencies are sampled more densely; it is exactly theintuition one would start with in dealing with the positioning prob-lem. The proposed framework and view selection algorithm repre-sent a promising alternative in the representation and coding of 3dscenes, based on spherical image-based rendering approaches.

7. REFERENCES

[1] A. Khodakovsky, P. Schr6der and W. Sweldens. Progressivegeometry compression. Siggraph '00 Conference Proceedings,July 2000.

[2] E.H. Adelson and J.R. Bergen. Computational Models of Vi-sual Processing, pages 3 - 20. M. Landy and J.A. Movshon,eds., MIT Press, Cambridge, 2001.

[3] H. Hoppe and E. Praun. Shape compression using sphericalgeometry images. Symposium on Multiresolution in GeometricModeling. Cambridge, September 2003.

[4] S. Hubbert and T.M. Morton. lp-error estimates for radial basisfunction interpolation on the sphere. Journal ofApproximationTheory, (129):58-77, 2004.

[5] I. Tosic, I. Bogdanova, P. Frossard and P. Vandergheynst. Mul-tiresolution Motion Estimation for Omnidirectional Images. InProceedings ofEUSIPCO, September 2005.

[6] I. Tosic, P. Frossard and P. Vandergheynst. Progressive codingof 3d objects based on overcomplete decompositions. EPFL-ITS Technical Report TR-ITS-2005.026, 1015 Ecublens, Oc-tober 2005.

[7] J. Maver, R. Bajcsy. Occlusions as a guide for planning thenext view. IEEE Transactions on Pattern Analysis and Ma-chine Intelligence, 15(5):417-433, May 1993.

[8] J. R. Driscoll and D. Healy. Computing Fourier transforms andconvolutions on the 2-sphere. Adv. in Appl. Math, 15:202-250,1994.

[9] K.M. G6rski, E. Hivon, A.J. Banday, B.D. Wandelt, F.K.Hansen, M. Reinecke and M. Bartelmann. HEALPix:A Framework for High-Resolution Discretization and FastAnalysis of Data Distributed on the Sphere. The Astrophysi-cal Journal, 622(2), 2005.

[10] F. Marvasti. Nonuniform sampling: Theory and Practice,chapter 6, pages 284-290. Kluwer Academic/Plenum Publish-ers, 2000.

[11] F. Narcowich and J.D. Ward. Scattered data interpolation onspheres: error estimates and locally supported basis functions.SIAM Journal on Mathematical Analysis, 33(6):1393-1410,2002.

[12] P. K. Allen, M. K. Reed and I. Stamos. View Planning for SiteModeling. in Proc. DARPA Image Understanding Workshop,Monterey, pages 1181-1192, November 1998.

[13] Y. Ma, S. Soatto, J. Koseck'a and S.S. Sastry. An Invitationto 3-D Vision: From Images to Geometric Models, chapter 2,pages 19-34. Springer, 2004.

2216