factor graph methods for three-dimensional shape...

R. J. Drost and A. C. Singer Vol. 21, No. 10 /October 2004 /J. Opt. Soc. Am. A 1855

Factor graph methods for three-dimensional shapereconstruction as applied to LIDAR imaging

Robert J. Drost and Andrew C. Singer

Coordinated Science Laboratory, University of Illinois at Urbana-Champaign, 1308 West Main Street,Urbana, Illinois 61801

Received December 22, 2003; revised manuscript received May 19, 2004; accepted May 21, 2004

Two methods based on factor graphs for reconstructing the three-dimensional (3D) shape of an object from aseries of two-dimensional images are presented. First, a factor graph model is developed for image segmen-tation to obtain silhouettes from raw images; the shape-from-silhouette technique is then applied to yield the3D reconstruction of the object. The second method presented is a direct 3D reconstruction of the object usinga factor graph model for the voxels of the reconstruction. While both methods should be applicable to a va-riety of input data types, they will be developed and demonstrated for a particular application involving theLIDAR imaging of a submerged target. Results from simulations and from real LIDAR data are shown thatdetail the performance of the methods. © 2004 Optical Society of America

OCIS codes: 100.6890, 100.3010, 100.2000.

1. INTRODUCTIONThe relatively low cost and complexity of constructing anddeploying ocean mines have made their use widespread.As a result, the detection and the classification of suchthreats are of great interest, and obtaining a three-dimensional (3D) shape reconstruction of the target canaid in this process. In the past, such detection has beenperformed by towed sonars, and algorithms have been de-veloped to perform the subsequent classification of anydetected objects.1 A disadvantage of this method, how-ever, is that the sonar sensor must be submerged. Con-sequently, the scanning speed is limited to the speed atwhich such a sensor can be drawn through the water(typically by a helicopter), and the process can be danger-ous. Airborne light detection and ranging (LIDAR) sys-tems have recently been developed to overcome this limi-tation. The LIDAR system emits short laser pulses toilluminate a column of water and any object within thecolumn. The LIDAR instrument along with the sensorsfor capturing the returns from the pulses, is carried on-board an aircraft, such as a helicopter. Since the entiresystem is above the water, it can operate at a safe dis-tance from any threats and can scan a larger area than atowed sonar in the same amount of time.

Because the LIDAR returns from the surface of the wa-ter would severely degrade the quality of any capturedimages, a sensor gate is often employed, so that only thelight that is reflected off the target and the water beneaththe surface is imaged. When the timing gate is set to col-lect returns from the target, reflection data are obtained.Both intensity and range information can be gathered inthis mode. The general problem of target recognitionbased on LIDAR range data has been explored in previouswork,2 as have tomographic techniques for 3D reconstruc-tions from LIDAR intensity data.3 Here we focus on adifferent class of data. If the sensor timing gate is set sothat only the light that reflects off the water past the tar-get is imaged, it is as if the target were backlit. Hence

1084-7529/2004/101855-14$15.00 ©

the shadow of the target is imaged. Only intensity dataare available in this case. The nature of these imagessuggests the use of shape-from-silhouette (SFS) or relatedtechniques from computer imaging for the 3D reconstruc-tion of the target shape.

The SFS method for reconstructing the 3D shape of atarget uses a series of silhouettes and their known cam-era positions. The shape of an object is a binary set of 3Dvolumetric data in which each volume element (voxel) isan indicator function for the presence of the object in 3Dspace; a silhouette of an object is a binary two-dimensional (2D) image in which each pixel value indi-cates the presence (or the absence) of the object along theline of sight from the viewing position to the object.Given a silhouette and the location from which the silhou-ette was taken, the object is constrained to lie entirelywithin a properly oriented right cylinder with the crosssection of the shadow region of the silhouette. Note thatthe choice of a right cylinder is based on the parallel rayassumption. If this assumption is not valid, one wouldhave to consider a more complex geometry. Now, for eachimage, there exists a similar cylinder, and if the cylindersare all consistent, then the object must lie entirely withintheir intersection. The SFS process determines this in-tersection and for this reason is also known as silhouetteor volume intersection.

The SFS technique has been studied since at least theearly 1980s.4 Laurentini explored the accuracy of idealSFS reconstructions,5 developing the concept of the visualhull.6 The visual hull of an object is the largest objectthat will yield the same silhouettes as those from theoriginal object from any view. It is the visual hull of anobject that the SFS technique actually attempts to recon-struct. Research has also been conducted into algo-rithms for generating an octree, the 3D analogy of aquadtree, of an SFS reconstruction.7–9 New models havebeen developed specifically for efficient SFSreconstruction,10 while other variations on the SFS ap-

2004 Optical Society of America

1856 J. Opt. Soc. Am. A/Vol. 21, No. 10 /October 2004 R. J. Drost and A. C. Singer

Fig. 1. LIDAR data collection experiment: (a) experimental setup and (b) digital camera photograph of target.

proach itself have also been studied.11,12 For this paper,however, the basic SFS method along with the simplest3D data structure (voxel occupancy) is sufficient. Voxeloccupancy is a simple binary labeling of each voxel in thereconstruction to indicate whether or not the target occu-pies that space.

Throughout the body of literature on the SFS tech-nique, little emphasis has been placed on methods to ob-tain silhouettes from raw image data. Silhouette genera-tion can be considered an image segmentation problem,which has a rich body of literature outside the SFS frame-work. For example, basic probabilistic methods mightinvolve steps like Wiener filtering and thresholding.More ad hoc methods might include the use of active con-tours such as snakes13,14 or deformable templates.15 Thesegmentation problem has also been reformulated as anenergy minimization problem.16 The use of Markov ran-dom fields (MRFs) has been studied in this context aswell, for example in synthetic aperture radar images17

and in sonar images.18 Often the major drawback ofMRF-based methods is the required computational com-plexity. One attempt to reduce this complexity involveda tree-structured MRF model.16 Here we develop a factorgraph model and use the sum–product algorithm to ob-tain the segmented images.

An alternative to the formation of silhouettes and theuse of the SFS technique is a direct 3D reconstructionbased on the raw images. Since the formation of silhou-ettes can be error prone, one might expect that avoidingthis step could lead to better reconstructions. Unfortu-nately, it is not clear how to directly formulate such a re-construction. One method formulates it as an energyminimization problem that can be solved efficiently withgraphical techniques.19 In the second approach of thispaper, the voxels of the reconstruction are modeled in afactor graph on which the sum–product algorithm oper-ates to yield the reconstructed object.

In this paper, we consider the following image modelthat can be applied to LIDAR images. Let x@n1 , n2#,(n1 , n2) P I, be a silhouette of an object corresponding toa particular viewing position, where I P Z2 is an indexingset of the image pixels. We model x@n1 , n2# as a random

sequence with statistics that will be specified in Section 3.Let h@n1 , n2#, (n1 , n2) P Z2, be the impulse response ofa 2D linear shift-invariant filter, and let w@n1 , n2#,(n1 , n2) P I, be a 2D white-noise random sequence thatis independent of x@n1 , n2#. We model an observed im-age as

y@n1 , n2# 5 x@n1 , n2#** h@n1 , n2#

1 w@n1 , n2#, ~n1 , n2! P I, (1)

where ** denotes 2D convolution. While the algorithmswill be developed for this general model, we more pre-cisely define h@n1 , n2# as a 2D Gaussian-shaped pulseand w@n1 , n2# as a stationary white Gaussian random se-quence when modeling LIDAR images of a submerged ob-ject. The LIDAR returns are perhaps better modeled asnonstationary linear combinations of Poisson processes.However, since we are operating in a high-energy regime,the Poisson returns are well modeled as Gaussian. Also,by collecting returns over a sufficient period of time, wecan average out the nonstationary effects of the oceansurface and objects within the ocean. The additional av-eraging of images separated in time can further improvethis approximation. Finally, while modeling the linearcombination as a Gaussian-shaped blur is certainly a sim-plifying approximation to more complex data models,20–23

the resulting model is adequate for this paper.In addition to applying the algorithms presented in this

paper to simulated data, we also demonstrate the tech-niques on real LIDAR data collected during a recent ex-periment. This experiment, conducted at Scripps Pier,San Diego, California, consisted of using a pulsed LIDARsystem to illuminate an object that was suspended from aboom that hung over the edge of a pier and collecting theLIDAR returns with a gated intensified charge-coupleddevice camera. Figure 1 describes the experiment; Fig.1(a) is a diagram of the experimental setup, and Fig. 1(b)is a digital photograph of the object used in the experi-ment. The body of the object was a cylinder measuringapproximately 1.5 m in length and approximately 0.9 min diameter, and the arms of the object extended approxi-mately 0.6 m from the body. The object could be raised


above the water and lowered to depths as great as 20 ftbelow the surface. Furthermore, if the boom was swungoutward and the LIDAR imaging system moved along theedge of the pier, a variety of elevation angles from the ob-ject to the camera could be obtained. Finally, by rotatingthe object on its vertical axis, one could image it from anyhorizontal direction. Both reflection images and shadowimages were captured while the above parameters werevaried, yielding a diverse set of data.

The remainder of this paper is divided as follows. Abrief introduction to factor graphs and the sum–productalgorithm is provided in Section 2. Section 3 developsthe factor graph models for both the image segmentationproblem and the 3D volume reconstruction problem. InSection 4, segmentation and reconstruction results basedon both simulated images and real LIDAR images arepresented and discussed. Finally, some concluding re-marks are given in Section 5.

2. BACKGROUNDA factor graph is a relatively new concept that is closelyrelated to Bayesian networks and MRFs.24 Here wepresent a brief introduction to factor graphs, as they areapplied in this paper, based largely on the notation pre-sented in Ref. 24, which can be consulted for a more in-depth discussion.

Given a function of several variables, one is often inter-ested in finding the marginal functions of a subset of thevariables. When the original ‘‘global’’ function factorsinto several ‘‘local’’ functions, each of a subset of the vari-ables, factor graphs can be a useful tool in finding the de-sired marginals. Indeed, they have been used in such ap-plications as iterative decoding of error correction codes25

and phase unwrapping26 and will be the basis for thetechniques developed in this paper.

Let f(x1 , x2 ,...,xn) be a function such that

f~x1 , x2 ,..., xn! 5 )XPQ

gX~X !, (2)

where Q is a collection of subsets of $x1 , x2 ,..., xn%, and,for each element X, gX(X) is a function of the elements ofsubset X. (Throughout this paper, the notation of a func-tion with a set as an argument indicates a function of thevariables in the set.) A factor graph G 5 (V, E) is an un-directed bipartite graph that depicts the structure in Eq.(2), where V 5 $x1 , x2 ,..., xn% ø $ gX : X P Q% and E5 $$ gX , xi%: X P Q and xi P X%.

A node corresponding to one of the variables is knownas a variable node, whereas a node corresponding to oneof the local functions is known as a function node. As ameans of distinguishing between variable nodes and func-tion nodes in a graphical depiction, the former are oftendrawn as open circles and the latter are often drawn asdots. Note that it is common notation to ignore the dis-tinction between a node and its corresponding variable orfunction, as context should make the meaning clear.

The utility of representing a function in a factor graphlies in the efficiency of the sum–product algorithm, whichoperates on a factor graph to produce marginal functions.This algorithm is often formulated as a message-passing

scheme in which messages are passed along the edges of afactor graph. These messages are functions of the vari-able associated with the variable node from which or towhich the message is being passed. Algorithmically, thisrequires the storage of a set of parameters of the message.In the case of binary variables, the parameters can simplybe the values of the message corresponding to the twostates of the variable.

Let mn1→n2denote the message passed from node n1 to

node n2 . Then a message being sent along an edge froma variable node x to a function node g takes the form

mx→g~x ! 5 )nPN~x !\$ g%

mn→x~x !, (3)

and a message being sent along an edge from a functionnode g to a variable node x takes the form

mg→x~x ! 5 (;$x%

Fg~X ! )nPN~ g !\$x%

mn→g~n !G , (4)

where N(x) is the set of neighbors of x, X is the set of ar-guments of the function g, and (;$x% indicates the sum-mation over all configurations of variables other than x.24

The sum–product algorithm proceeds by passing mes-sages formed according to Eqs. (3) and (4), initialized bysending the unity message from each variable node.There is freedom in choosing the message-passing sched-ule, but perhaps the simplest, which is used in this paper,involves global updates. During each iteration, all mes-sages from every function node are sent, and then all mes-sages from every variable node are sent.

For cycle-free factor graphs, messages will cease tochange after a finite number of iterations, signaling thetermination of the algorithm. The marginal function foreach variable is then given by the product of all incomingmessages to the corresponding variable node.

On graphs with cycles, convergence is guaranteed foronly certain types of graphs, such as those with only asingle cycle.27 As a result, termination criteria are oftenemployed, such as when a chosen number of iterations isattained or when the sequence of updates leaving eachnode satisfies a chosen convergence metric. As in thecase of graphs without cycles, the desired output functionof each variable is taken as the product of the incomingmessages to the corresponding variable node. However,in this case, the resulting functions are only approxima-tions to the true marginal functions. Despite these addi-tional complexities when dealing with graphs with cycles,good results can be obtained in many instances.24

3. FACTOR GRAPH MODELSA. Image SegmentationIn this section, a factor graph model is developed for thesegmentation of images that can be modeled according toEq. (1). For any image x@n1 , n2#, (n1 , n2) P I, let xi, j5 x@i, j#, (i, j) P I, and let x be a vector containingeach pixel xi, j , (i, j) P I, in a fixed order. The factorgraph model will depict a factorization of the conditionalprobability distribution f(xuy) of the underlying silhou-ette pixels x given the observed image pixels y. We have


f~xuy! 5f~yux!f~x!

f~y!

51

Zf~yux!f~x!, (5)

where Z is a normalization constant depending only onthe observation y. We examine the further factorizationof the two terms f(x) and f(yux) separately.

The silhouettes that we consider are of connected ob-jects that are large relative to the size of a pixel. Conse-quently, it is reasonable to assume that a silhouette pixelis conditionally independent of all nonneighboring pixelsgiven its nine neighboring pixels. For example, giventhat all of the neighbors of a silhouette pixel are part ofthe shadow region, then clearly that pixel is highly likelyto also be part of the shadow region, independent of allother pixels in the image. Hence we assume that x formsan MRF. Provided that the joint probability distributionf(x) is strictly positive, the Hammersley–Clifford Theo-rem then states that f(x) factors according to

f~x! 51

Z8)i, j

c i, j~xi, j!, (6)

where xi, j 5 @xi, j xi, j11 xi11,j xi11,j11#T are vectorscontaining 2 3 2 groups of neighboring pixels and Z8 is anormalization constant.28 [When the limits of a summa-tion or a product are omitted, as in Eq. (6), it is assumedthat the corresponding operation is performed over thelargest set of values such that the terms of the operationare well defined.] So, in summary, we assume that a de-sirable distribution can be obtained as a product of onlylocal potential functions c i, j .

One can ascertain some desired properties of the poten-tial functions from the constraints of the problem. Con-sidering that the object being imaged may appear any-where in the image and at any orientation, it isreasonable to assume that the potential functions are allspatially and rotationally invariant, so that c i, j( • )5 c ( • ) for all i and j. Note that the potential is a func-tion of four neighboring binary-valued pixels. So for eachof the 16 possible pixel configurations, a value must be as-signed to the potential. Increasing one of these valuesrelative to the others models an increased probability thatthe corresponding configuration appears throughout thesilhouette. Since the object being imaged is assumed tobe connected and large relative to a pixel, it is clear thatcertain pixel configurations should appear often whileothers should almost never appear. In particular, theconfigurations in which all the pixels are inside theshadow or all the pixels are outside the shadow should byfar be the most likely. Meanwhile, configurations inwhich the pixels on one diagonal are of one value and thepixels on the other diagonal are of the opposite valueshould practically never appear in the silhouette, as thiswould violate the assumption concerning the size and theconnectedness of the object. Note, however, that even inthis case the value of the potential function should not beset to zero for numerical reasons related to the sum–product algorithm, to satisfy the strict positivity conditionof the Hammersley–Clifford Theorem, and to permit

highly unlikely, but possible, image configurations. Assuch, a small nonzero value should be used. The remain-ing configurations occur on the border of the shadow re-gion and should result in potential function values some-where between those of the first two cases considered.

Given the above guidelines for choosing the potentialfunctions, a reasonable model can be obtained. Furtherguidance can be found by examining the relative fre-quency of the various pixel configurations in simulatedsilhouettes similar to those expected from real data.Though there is not necessarily any direct relationshipbetween the true probability distribution of the pixel con-figurations and the potentials, it is reasonable to assumethat they will be qualitatively similar. In fact, distribu-tions can be constructed such that they are indeed simi-lar. Finally, the potential functions can be further re-fined empirically.

The potential function used throughout this paper isgiven by

c ~xi, j!

5 H 1000 if xi, j 5 xi11,j 5 xi, j11 5 xi11,j11

1 if ~xi, j 5 xi11,j11! Þ ~xi11,j 5 xi, j11!

10 otherwise. (7)

The values of 1000 and 10 were chosen because there wasa difference of approximately 2 orders of magnitude in thefrequency of the corresponding pixel configuration in asimulated silhouette. The third set of pixel configura-tions did not appear in the simulated silhouette, but forreasons discussed above, the relatively small but nonzerovalue of unity was chosen for the potential function.

We reiterate that the above potential functions weredeveloped based on a priori assumptions concerning thelocation, the size, and the connectedness of desired silhou-ettes. If any of these assumptions is violated, a degradedperformance should be expected if the potential functionsare not modified to emphasize the characteristics of theparticular silhouettes of interest.

Now we examine the factorization of f(yux). First, wenote that since each observation pixel is modeled as a lin-ear combination of the elements of x in additive whitenoise, it is clear that the observations are conditionally in-dependent of each other given x. Furthermore, depend-ing on the particular blur function that generated the ob-served image, each observed pixel might directly dependon only a few silhouette pixels. More precisely, let Xi, jdenote the smallest set of silhouette pixels such that theobservation yi, j is conditionally independent of all othersilhouette pixels given Xi, j . This set can easily be deter-mined by locating the nonzero values of the impulse re-sponse of the blurring filter h@n1 , n2#. Then

f~yux! 5 )i, j

f~ yi, jux!

5 )i, j

f~ yi, juXi, j!. (8)

So our image segmentation technique proceeds by ap-plying the sum–product algorithm to the factor graphthat depicts the factorization


f~xuy! 51

Z9)i, j

c ~xi, j!f~ yi, juXi, j!, (9)

yielding an approximation of the marginal distributionsf(xi, juy). Choosing the most likely value for each xi, j ,conditioned on the observations y, is then one method forobtaining an estimate of the desired segmented image.In actuality, the normalization constant Z9 is ignored inthe factor graph model, since it is not easily computed anddoes not have an effect on the final result.

Note that when we perform image segmentation usingMRFs, a factorization of the conditional probability distri-bution similar to that of Eq. (9) might arise.29 However,the MRF framework has led to algorithms other than theone proposed in this paper. For example, the segmenta-tion was performed in Ref. 29 with a dynamic program-ming approach. When various forms of belief propaga-tion have been used for segmentation, they have beenapplied to different formulations of the image segmenta-tion problem (e.g., Ref. 30).

Next, we illustrate the factor graph model with a fewexamples. First, suppose that no blurring was intro-duced into the observed image; i.e., h@n1 , n2# is a (possi-bly scaled) unit impulse function ad@n1 , n2#. Then Xi, j5 $xi, j%, resulting in the factor graph depicted in Fig.2(a), where the notation fyi, j

(xi, j) 5 f( yi, juxi, j) is used toemphasize that f is treated as a function of xi, j param-eterized by the corresponding observation yi, j .

For the LIDAR images, an observed pixel, given thecorresponding silhouette pixel, can be modeled as beingnormally distributed with a mean that depends on thevalue of the silhouette pixel. This suggests the followingprobability distribution:

fyi, j~xi, j!

5 51

A2ps 2expF2

1

2s 2~ yi, j 2 m0!2G if xi, j 5 0

1

A2ps 2expF2

1

2s 2~ yi, j 2 m1!2G if xi, j 5 1

,

(10)

where m0 and m1 are the conditional means of an observedpixel outside and inside the shadow region, respectively,and s 2 is the variance of the noise. Each of these param-eters can be estimated from the observed images.

As another example, suppose that before being cor-rupted by zero-mean additive white Gaussian noise ofconstant variance s 2, the silhouette image is first blurredby a small 2 3 2 filter with filter taps a5 @a1 a2 a3 a4#T, so that an observed pixel can bewritten as yi, j 5 aTxi, j 1 wi, j . In this case, Xi, j5 $xi, j , xi11,j , xi, j11 , xi11,j11%, suggesting the factorgraph of Fig. 2(b). Clearly, yi, j , given xi, j , is normallydistributed with a conditional mean of aTxi, j and a vari-ance of s 2. Hence we obtain

fyi, j~Xi, j! 5

1

A2ps 2expF2

1

2s 2~ yi, j 2 aTxi, j!

2G .

(11)

Often the blur function that degrades an image has aregion of support much larger than 2 3 2 pixels.Clearly, the general model that we have developed couldbe used in such a case, resulting in a factor graph that issimilar to that of the above 2 3 2 case but that assumesthat the observed pixel is a function of a larger set of sil-houette pixels. While such a factor graph can be formed,a problem rapidly arises in the computational complexityof solving for the marginal distributions.

Consider the implementation of a k 3 k blur functionin a factor graph for an n 3 n-pixel image. Computingall the messages leaving the observation functions in-volves forming n2k2 messages. Each of these messagesrequires twice summing over k2 2 1 binary variables,and for each configuration of these k2 2 1 variables,k2 2 1 real multiplies are needed. Hence the total num-ber of multiplies in this one step of one iteration of thesum–product algorithm is approximately n2k42k2

.Clearly, this rapidly becomes infeasible. While optimiza-tion could reduce the complexity somewhat, the problemwould still be intractable for even moderate values of nand k.

The final example to be considered attempts to accountfor a blur function that spans a larger area than the 23 2 case but with the same order of complexity. This isaccomplished by approximating the true blur function byone that has a few, possibly sparse, nonzero taps. For ex-ample, in all of our factor graph image segmentation re-sults in which the true blur function is a 2D Gaussian-shaped pulse, we use the approximate blur functiondepicted in Fig. 3. In the case of a radially symmetricGaussian pulse, we can set a2 5 a3 5 a4 5 a5 so thatonly two constants must be assigned. The developmentof the corresponding factor graph closely resembles thatof the 2 3 2 case and will not be described in detail.Note that, in general, any form for the approximate blurfunction can be used, but the complexity of applying thesum–product algorithm to the resulting factor graph in-creases exponentially with the number of nonzero taps.

B. Factor Graph ReconstructionHere we present a factor graph model that can be used toreconstruct the shape of an object directly from its rawimages. This method has two primary advantages overthe SFS technique. First, it bypasses the error-prone im-age segmentation process, and second, it directly pro-motes smoothness and connectedness in the 3D recon-struction. The factor graph model will be developed herefor the case of no blurring and will be a 3D analogy to theimage segmentation model for unblurred images. Forconciseness of presentation, we also assume that thenoise w@n1 , n2# is a stationary white Gaussian sequence,which, as discussed in Section 1, is consistent with ourmodel for the LIDAR images.

Let v contain the voxels of the reconstruction. We firstgenerate a vector of observations zi corresponding to eachvoxel vi from the observed images as follows. For eachimage, we first determine where the line of sight from thecenter of each voxel intersects the image. Then, to eachvoxel, we assign by nearest-neighbor interpolation a pixelvalue from the image. Combining the observation ofvoxel vi from each image yields the observed vector zi .


Fig. 2. Factor graph models for segmenting images with (a) no blurring and (b) a 2 3 2 blurring.

Note that this simple method of obtaining these observa-tions is best suited for cases when the image and recon-struction resolutions are similar. More complicatedmethods, such as taking a weighted average of all pixelsfrom which the corresponding line of sight passes througha particular voxel, can be used, but the nearest-neighborinterpolation appears to be adequate for this paper.

Similar to our approach in the image segmentationcase, we consider the factorization

f~vuz! 51

Zf~v!f~zuv!, (12)

where z contains the observations zi for all i. Next, weanalogously assume that f(v) factors as a product of localpotential functions. The 3D generalization would be toconsider local functions of the eight voxels in 2 3 2 3 2blocks. While this would still result in a tractable prob-lem, we choose instead to simplify the model by assumingthat the potentials are functions only of each pair of ad-jacent voxels. The resulting model can be justified in asimilar manner to that in the image segmentation model,i.e., by considering the conditional independence of avoxel and its non-adjacent voxels given its adjacent voxelsand then applying the Hammersley–Clifford Theorem.

By examining the ratio of combinations of neighboringvoxels in a simulated object and then adjusting the resultfor symmetry, one can reasonably approximate the poten-tial functions. Based on such a ratio, we chose to use for

Fig. 3. Approximate blur point-spread function.

the results in this paper the following as the potentialfunction for a voxel vi and one of its neighbors vj :

c ~vi , vj! 5 H 1 if vi Þ vj

k if vi 5 vj, (13)

where k ' 500. The same caveat concerning the appli-cability of the potential function used in the image seg-mentation factor graph also applies to the potential func-tion here. In particular, if the objects of interest are notexpected to be connected and large relative to the size of avoxel, then it might be necessary to adjust the potentialfunction to more accurately reflect the expected proper-ties.

Now, to factor the conditional distribution f(zuv), wefirst note that since the observations zi, j , i.e., the compo-nents of the vectors zi , are deterministic functions of v inwhite noise, they are conditionally independent of eachother given v. We further assume that each zi, j is con-ditionally independent of all other voxels given the corre-sponding voxel vi . Hence

f~zuv! 5 )i, j

f~zi, juv!

5 )i, j

f~zi, juvi!. (14)

Note that the conditional independence assumptionthat was applied in the second line of Eq. (14) is not al-ways valid. It is true that an observation, given the vox-els that lie along the corresponding line of sight, is condi-tionally independent of all other voxels. However, theobservation is clearly a function of all voxels along theline of sight. Furthermore, the quality of the resultingapproximation f(zi, juv) ' f(zi, juvi) depends on the dis-tance from voxel vi to the object. For voxels within theobject, the approximation is exact, since in this case zi, j isa noisy version of the shadow of the object, independent ofthe rest of the voxels. For voxels a great distance fromthe object, observations corresponding to most viewingpositions will be a noisy version of the nonshadow of the


object, assuming that viewing positions are chosen uni-formly from around the object. Hence the approximationin this case is accurate. As voxels approach the object,however, the approximation degrades.

This observation suggests two methods to help ensurethe accuracy of the observation model. First, the factorsf(zi, juvi) in Eq. (14) might be replaced with f(zi, juvi, j),where vi, j is a vector containing voxel vi and some set ofvoxels next to vi along the line of sight corresponding tothe observation zi, j . This added conditioning would re-sult in an exact distribution for voxels near the object,where the original approximation was poorest. The re-gion where this alteration would have an effect dependson the size of the vector vi, j . However, the complexity ofthe resulting algorithm would grow exponentially withthis size. Hence a trade-off between performance andcomplexity results. In this paper, we choose to apply adifferent method. In particular, since our assumption ofconditional independence is poorest near the object, wechoose the function f(zi, juvi) to most accurately reflect theconditional distribution of observations of voxels in thisregion. This results in a less accurate observation modelof voxels far from the object, but the portion of the factorgraph that models the distribution of f(v) should suffi-ciently reinforce the connectedness of the object to ensurethat proper decisions are made.

To determine f(zi, juvi), we first note that when vi ispart of the object (vi 5 1), then the observation is a noisyversion of the shadow of the object. Hence we have thatzi, j , given vi 5 1, is distributed normally with mean m1and variance s 2 that can be estimated from the shadowregion of the images. When vi is not part of the object(vi 5 0), the distribution is not as simple. In this case,there is some probability p that every voxel along the lineof sight of the observation will also not be part of the ob-ject, resulting in a noisy version of the nonshadow of theobject. On the other hand, there is a probability 1 2 pthat at least one other voxel along the line of sight will bepart of the object, resulting in a noisy observation of theshadow of the object. So the resulting distribution con-ditioned on vi 5 0 will be a mixture of Gaussians of theform

f~zi, juvi 5 0 ! 5 p1

A2ps 2expF2

1

2s 2~zi, j 2 m0!2G

1 ~1 2 p !1

A2ps 2

3 expF21

2s 2~zi, j 2 m1!2G , (15)

where m0 can be estimated from the observed images asthe mean of the nonshadow region.

As discussed above, we choose the value of p to most ac-curately model the conditional distribution f(zi, juvi) cor-responding to voxels near the surface of the object.Based on observations of a simulated object from a smallset of viewing positions, we estimated the value of p to beapproximately 0.25 for voxels adjacent to the object. Wealso analytically calculated an estimate of p based on a

sphere of the same approximate size as that of the objectsthat we expect to reconstruct. In this case, we found thatif a point is chosen uniformly from within a distance ofone voxel from the surface of the sphere and a viewing po-sition is chosen uniformly from around the object, thenthe probability that the point will be occluded by thesphere is approximately 0.2. All of the factor graph mod-els in this paper use the value of p 5 0.2.

We have now fully described a factorization of the dis-tribution f(vuz). Applying the sum–product algorithm tothe corresponding factor graph yields approximations tothe marginal distributions. Choosing the most likelyvalue for each voxel conditioned on the observations is themethod that we employ for obtaining the final reconstruc-tion.

While the factor graph model developed here was de-signed for unblurred images, there is no reason that itcannot be applied to blurred images as well. Certainlymore complex factor graph models could be used in suchcases, but it will be seen that the above model producesgood results despite its lack of a direct mechanism to com-pensate for such blurring.

4. RESULTSA. SimulationsHere we examine the performance of both factor graphmethods on simulated data. A set of eight512 3 512-pixel silhouettes was produced from the simu-lated target shown in Fig. 4. Then three sets of blurredimages were produced by filtering the silhouettes withone of the following functions. In the no blur (NB) case, aunit impulse function was used, whereas in the sparseblur (SB) case, a function of the form shown in Fig. 3 wasused with ai 5 0.2 for all i. Finally, in the Gaussian blur(GB) case, we used a 2D Gaussian pulse

h@n1 , n2# 51

ZexpF2

1

2s 2~n1

2 1 n22!G , (16)

where s 2 5 50 and Z is a normalization constant.The resulting images were then corrupted with addi-

tive white Gaussian noise to obtain data sets with signal-to-noise ratios (SNRs) that varied in 5-dB incrementsfrom 230 dB to 20 dB. We define the SNR as SNR5 10 log10( sp

2/sw2 ), where sp

2 is the variance of the pixelsin the filtered image before the addition of noise and sw

2 isthe variance of the noise.

Fig. 4. Simulated target.


1. MethodsFor each simulation case, an appropriate factor graphmodel was used to segment the images. In both the NBand SB cases, the model implemented the exact impulseresponse of the blurring filter. In the GB case, however,we approximated the blur function in the model as thefunction depicted in Fig. 3 with a1 5 0.6 and a2 5 a35 a4 5 a5 5 0.1. These factor graph methods will belabeled FG.

Three additional image segmentation methods wereapplied. The first was a heuristic method (H) that con-sisted of the following steps. We thresholded the images,ignoring any blurring, by using the Bayes decision rulefor equal priors. Since the noise variance is constantthroughout each image, the threshold level is simply12 (m0 1 m1), where m0 and m1 are the conditional meansof the observed pixels outside and inside the shadow re-gion, respectively. Because of the blurring, the edges ofthe shadow region were not well defined in the resultingsilhouettes. To compensate, we considered the densenessof shadow pixels in a 5 3 5-pixel region about each pixelin the silhouette. If this density was greater than 0.5,then the center pixel was labeled as part of the shadow;otherwise it was labeled as not part of the shadow. Fi-nally, since the target was expected to be large relative tothe size of a pixel, it was clear that small ‘‘islands’’ of pix-els of one type surrounded by pixels of the opposite typemust have been mislabeled. A search for such occur-rences was conducted, and the pixels corrected.

The second image segmentation method relied onWiener filtering the output image and is termed the WFmethod. Let x@n1 , n2# be the original silhouette, lety@n1 , n2# be the observed image, and let H(v1 , v2) bethe frequency response of the blur function. Then, forzero-mean images, the Wiener filter G(v1 , v2) givesx@n1 , n2# 5 y@n1 , n2#** g@n1 , n2# such thatE$(x@n1 , n2# 2 x@n1 , n2#)2% is minimized, whereg@n1 , n2# is the inverse discrete-time Fourier transformof G(v1 , v2) and E denotes expectation. (Nonzeromeans can be handled by subtracting the mean of the ob-served pixels from the observed image before processing itand then by adding the mean of the silhouette pixels tothe resulting estimate x.) The filter can be written as

G~v1 , v2! 5Sxy~v1 , v2!

Syy~v1 , v2!(17)

5@Syy~v1 , v2! 2 s 2#/H~v1 , v2!

Syy~v1 , v2!, (18)

where Sxy is the cross spectral density of the silhouetteand observed images, Syy is the power spectral density ofthe observed image, and s 2 is the variance of the noise.31

The actual power spectral density, however, was notknown, so it was estimated by smoothing the periodogramof the image. Applying this estimate Syy of Syy to Eq.(18) results in an estimate of the Wiener filter. Note thatat frequencies where H(v1 , v2) ' 0, we have Syy ' s 2.However, since we cannot guarantee that Syy ' s 2 whenH(v1 , v2) ' 0, this estimate of the filter might deviate

substantially from the true Wiener filter at these frequen-cies. To limit this distortion, we replaced the functionH(v1 , v2) in Eq. (18) with

H~v1 , v2! 5 H H~v1 , v2! if uH~v1 , v2!u . g

gH~v1 , v2!

uH~v1 , v2!uotherwise

,

(19)

where g was chosen to minimize (n1 ,n2(x@n1 , n2#

2 x@n1 , n2#)2.31 Of course, in a real setting, one wouldhave to choose g blindly, since the original silhouettewould not be available for this optimization. The result-ing images were thresholded to obtain the final result.

The final method involved finding the optimal 113 11 impulse response g@n1 , n2# that minimizes

(n1

(n2

~x@n1 , n2# 2 g@n1 , n2# ** y@n1 , n2# !2, (20)

assuming that the images are zero mean. (Nonzeromeans can be handled as in the WF method.) This opti-mization was performed by using the true silhouettex@n1 , n2# and hence cannot actually be implemented in areal setting. So the performance of this method shouldbe considered a bound on that of any similar technique.After filtering the observed image with g@n1 , n2#, wethresholded the result to obtain the final silhouette.Since this method involved finding the deterministicleast-squares filter, it is termed the LS method.

The silhouettes obtained from the above methods werecombined with the SFS technique to obtain reconstruc-tions to compare with the factor graph reconstructionmethod, which is labeled FG3D. For an additional com-parison, an energy minimization reconstruction (EMR)was also employed that uses the raw observation imagesto reconstruct the shape of the target.19 This approachformulates the reconstruction as an energy minimizationproblem that can be solved with graphical techniques.As in the FG3D reconstructions, we derive a set of obser-vations zi for each voxel vi from the image data. Graphcuts are then used to minimize an energy function of aparticular form over all voxel labelings. In this paper,the following empirically chosen energy function is used:

E~v! 5 (i

Dzi~vi!

1 70(i

($ j:vjPN~vi!%

1

2~1 2 d@vi 2 vj# !, (21)

where

Dzi~vi! 5 H roundH 40(

j@j~zi, j!#

2J if vi 5 0

300 if vi 5 1

,

(22)

j~x ! 5 H 0 if x , 0

x if 0 < x , 1.4

1.4 if x > 1.4, (23)


Fig. 5. Image segmentation error in NB case: (a) in entire image, (b) away from edges, and (c) near edges.

Fig. 6. Image segmentation error in SB case: (a) in entire image, (b) away from edges, and (c) near edges.

Fig. 7. Image segmentation error in GB case: (a) in entire image, (b) away from edges, and (c) near edges.

and round(x) is the nearest integer to x. Details of thisalgorithm can be found in Ref. 19.

2. Simulation ResultsFigures 5–7 detail the performance of the various imagesegmentation methods on the simulated data. Figure 5describes the results of the NB case, Fig. 6 describes theresults of the SB case, and Fig. 7 describes the results ofthe GB case. Each figure consists of three plots that dis-play the probability of mislabeling a pixel in a particularregion of the image. This error probability Pe is the ratioof mislabeled pixels to total number of pixels consideredand is given as a function of SNR.

The first plot in each figure gives Pe for all of the pixelsin the image. The second plot gives Pe for only those pix-els that are at least 6 pixels from the nearest edge of theshadow region of the true silhouette, while the third plot

gives Pe for those pixels that lie within 6 pixels of thenearest edge. We chose these regions because they di-vide where the blurring in the blurred images appearedvisually to have and to not have a significant effect on theobserved pixel values. The error probability is divided inthis manner because its behavior in these two regions dif-fers dramatically, and errors in each region can have dif-ferent effects on resulting SFS reconstructions. The re-gions away from the edge of the shadow region are mucheasier to properly label, and errors in this region can re-sult in degraded 3D reconstructions, since holes might bepunched through the reconstruction. Meanwhile, re-gions on the edge of the shadow are much harder to cor-rectly label but are not as vital to obtaining decent recon-structions. To yield high-quality reconstructions,however, even these pixels must be correctly labeled.

The figures clearly show that the image segmentation


factor graph models outperform the other methods by asmuch as 10 dB. The only instance where the factorgraph method fails is in the GB case at high SNR. Con-sidering that this does not occur in the SB case, we con-clude that it is largely the result of approximating theGaussian blur function as a simple five-tap filter. Tomitigate the effects of this approximation error, we alsosimulated the factor graph algorithm in the GB case athigh SNR with a factor graph model that assumes a con-stant SNR of 210 dB. In effect, we are modeling the fil-ter approximation error as an additional source of addi-tive white Gaussian noise. The resulting plot is alsoshown in Fig. 7 and is labeled FG2. A significant im-provement is evident that makes the factor graph methodcompetitive with the other algorithms.

Figure 8 shows the 3D shape reconstruction perfor-mance for each simulation case. Similar to the imagesegmentation plots, the error probability, given by the ra-tio of mislabeled voxels to total voxels, is shown as a func-tion of SNR. Comparing the factor graph SFS (FG-SFS)results with the other image segmentation SFS resultsshows that, again, the factor graph method outperformsthe others in all cases. Note that each of these methodsreaches an error floor at high SNR because the SFS tech-nique reconstructs the largest volume that is consistentwith the given silhouettes. It was verifed that even withideal silhouettes, better performance could not be at-tained.

Examining the direct reconstruction methods revealsthat the EMR method can give the best results in someinstances but does not usually outperform the FG-SFS orFG3D method. Meanwhile, the FG3D method most con-sistently outperforms all other methods. Only in a fewcases does the FG-SFS, EMR, or LS-SFS method give bet-ter results.

Note that some care must be taken in comparing the di-rect reconstruction results and the SFS results. If no as-sumptions are made about the target, then there are aninfinite number of objects that could have generated theideal silhouettes used in the simulation. At high SNR,the FG3D and EMR methods would have undoubtedlyperformed worse than the other methods for some of theseobjects. However, since the actual target was quiterounded, the smoothness that the FG3D and EMR meth-ods promoted resulted in better reconstructions. Hencethe choice between an SFS technique and a direct recon-struction technique might depend on what is assumedabout the target to be reconstructed.

B. Real DataIn this subsection, we demonstrate the performance of thefactor graph methods on real LIDAR data. Since a quan-titative measure of the performance was not available, weinstead show actual segmented images and reconstruc-tions and comment qualitatively on the results. Recallthat two types of LIDAR images are available: reflectionimages and shadow images. Because of the high qualityof the reflection images when the target was suspended inair, these images will be used in addition to the shadowimages of the submerged target.

1. Image Segmentation ResultsThe formation of silhouettes is first demonstrated on therepresentative in-air image in Fig. 9(a). The silhouettesthat resulted from the application of the WF method andthe factor graph method are shown in Figs. 9(b) and 9(c),respectively. Because the target was suspended in air,there was no detectable blurring in the image, suggestingthe use of the factor graph model depicted in Fig. 2(a).

As one might expect, the high quality of the raw imageresulted in high-quality silhouettes regardless of whichmethod was used. However, a few errors are apparent inthe WF segmentation near the edges of the shadow re-gion. One factor that may have contributed to these er-rors is that the images are not exactly of the assumedform. In particular, our image model dictates that thevalues of all of the pixels in the region that the target oc-cupies should be distributed normally with the samemean. Inspection of the image reveals that this is notthe case, as the mean intensity of the pixels appears tovary across the image and is probably related to the ori-entation of the corresponding surfaces. Hence the qual-ity of the silhouette based on the factor graph model dem-onstrates a degree of robustness of the technique to modelinaccuracies.

Next, silhouettes based on a representative time-averaged submerged shadow image are shown. Figure10(a) shows the result of taking the pixel-by-pixel medianof five time-adjacent shadow images. This averaging wasperformed to lessen the effect of ocean waves on the im-age, since these ocean waves result in a time-varying andspatially varying filter for which none of the techniquespreviously discussed has a direct mechanism to compen-sate. The resulting averaged image had an estimatedSNR of approximately 5 dB.

The scattering effects of the water are often modeled asa large Gaussian blurring of the image. Again, we ap-

Fig. 8. Reconstruction error in (a) NB case, (b) SB case, and (c) GB case.


proximate the Gaussian blur in the factor graph modelwith the function shown in Fig. 3, where a1 5 0.6 anda2 5 a3 5 a4 5 a5 5 0.1. Also, as in the GB simulationat high SNR, we attempt to reduce the effect of the filterapproximation error in the factor graph model by assum-ing a lower SNR than was actually estimated from theimages. Figures 10(b) and 10(c) show the silhouettesthat resulted from the use of the WF method and the fac-tor graph method, respectively, on the averaged image.

First, note that the edges of the silhouette derived fromthe factor graph method are much smoother and morewell defined than the edges in the silhouette from the WFmethod. Perhaps a more apparent observation is thatthe effects of the water degrade the quality of the silhou-ettes from both techniques. This is most evident in theinability of either technique to detect the deep internalcorners of the true silhouette. While it is true that theseerrors could be corrected during the SFS reconstruction ifanother silhouette indicates the presence of these deepcorners, this is unlikely. There is no reason to believethat any of the other silhouettes based on the shadow im-

Fig. 9. Silhouette formation from (a) in-air image: (b) with theWF method and (c) with the FG method.

Fig. 10. Silhouette formation from (a) shadow image: (b) withthe WF method and (c) with the FG method.

ages would not suffer from the same defect. There aretwo probable causes for this error. First, the models usedsimply might not adequately account for the blurring in-duced by the water. Second, since the target was con-stantly being rotated throughout the experiment to obtainimages from a diverse set of viewpoints, the target mighthave rotated a significant amount in the time span of fiveimages. In this case, averaging the five images would re-sult in an additional rotational blurring that is not ac-counted for.

2. ReconstructionsIn this subsection, several 3D reconstructions of the ex-perimental LIDAR target are presented. Two pointsshould be made concerning all of the reconstructions.First, to perform any reconstruction, one must have de-tailed information for each image about the location andthe orientation of the imaging system with respect to thetarget. While some measurements were taken duringthe imaging experiment, such as the elevation angle ofthe system, other measurements were not. To estimatethese unknown quantities from the images, we usedknowledge of the shape of the target. In a more generalsetting, where the shape of the target is not knowna priori, more attention to acquiring the necessary mea-surements or more complex estimation techniques wouldbe required. For example, as in some Navy LIDAR im-aging systems, the ATD-111 system of BAE Systems em-ploys a global positioning system and additional rate sen-sors and gyros to measure the location and the preciseorientation of the imaging system, as well as the preciseslant depth to the target, when each image is taken.Note that we used a priori knowledge of the target shapefor the sole purpose of estimating camera position andthat this should not affect the evaluation of the results,since all of the techniques discussed in this paper assumethat the camera position is known.

The second point is that the range of elevation anglesfrom which the experimental target could be imaged wasinherently limited by the experimental setup. When thetarget is submerged, the imaging system must be suffi-ciently elevated for the laser to adequately penetrate thesurface. The reconstructions shown here are all based onimages taken from elevation angles of at least 58°. Theuse of Snell’s law to compensate for the refraction of lightat the ocean surface gives effective elevation angles of atleast 66.5° when the target was submerged. Since data

Fig. 11. LIDAR reflection images used in reconstruction of target in air.


are not available from lower elevation angles, one wouldexpect that any reconstruction is severely elongated inthe vertical direction. This is indeed the case, as anyview of a reconstruction from a low elevation will reveal.

The first reconstructions to be discussed are based onthe eight in-air images, each from an elevation angle of62°, shown in Fig. 11. While the images were chosen sothat a variety of views from around the target were rep-resented, no effort was made to choose the ‘‘best’’ images.Figure 12 shows a WF-SFS reconstruction. While thegeneral shape of the target is captured, the reconstructionlacks the smoothness present in the true object. Next, anFG-SFS reconstruction is presented in Fig. 13. Just asthe high quality of the original image resulted in high-quality silhouettes, so did they result in a high-quality re-construction. Next, Fig. 14 gives an FG3D reconstruc-

Fig. 12. WF-SFS reconstruction from in-air data.

Fig. 13. FG-SFS reconstruction from in-air data.

Fig. 14. FG3D reconstruction from in-air data.

tion from the same set of data. Again, the reconstructionseems quite accurate. The smoothness that this methodpromotes is apparent both in the more cylindrical bodyand in the rounded corners. Hence we see a trade-off be-tween the accuracy of the FG-SFS and FG3D reconstruc-tions in these two areas.

Figure 15 shows eight images that resulted from aver-aging each of eight groups of five time-adjacent sub-merged shadow images. Four of these groups were takenfrom an elevation of 76.1°, and the remaining four weretaken from an elevation of 58°. The resulting averagedimages were used to form silhouettes with the WF and FGmethods as in the second LIDAR image segmentation ex-ample. The subsequent application of the SFS techniqueto the WF silhouettes yields the reconstruction shown inFig. 16. It is clear that the distortion of the ocean has ledto a significant degradation in the quality of the recon-struction. In addition to lacking the appropriate smooth-ness, the technique also failed to reconstruct the cornersof the object. The SFS reconstruction based on the FGsilhouettes, shown in Fig. 17, also displays these roundedcorners. However, the FG-SFS reconstruction more ac-curately reconstructs the smooth surfaces of the target.

It should be noted that applying the SFS methods tothe images without first averaging the groups results inworse reconstructions. The errors that are made in form-ing silhouettes from the original images corrupt the sub-sequent reconstruction to such an extent as to offset thebenefit of additional silhouettes.

Finally, the data in the previous reconstruction wereused in an FG3D reconstruction. The same 40 imageswere used, but unlike the SFS reconstructions, better re-sults were obtained by not averaging the groups of time-adjacent images. The reconstruction based on the unav-eraged images is shown in Fig. 18 and is clearly smootherand more rounded than the previous reconstructions.This certainly gives a more polished appearance, but un-fortunately it does little to help distinguish the deep cor-ners that were undetected in the previous reconstruc-tions. Hence the FG-SFS and FG3D reconstructionsappear comparable.

Despite the inaccuracies in the FG-SFS and FG3D re-constructions of the submerged target, the general 3Dshape of the target has been captured in each. The re-constructions present more information than the indi-vidual LIDAR images, and hence such reconstructionsmight be of use in identifying an unknown target.

Fig. 15. LIDAR shadow images used in reconstruction of submerged target.


5. CONCLUSIONSIn this paper, two factor graph methods were examinedthat can be used in reconstructing the 3D shape of an ob-ject from a series of 2D images. The first reconstructiontechnique was based on an image segmentation methodcombined with the SFS technique. The second techniquedeveloped a factor graph model for the direct 3D recon-struction of a target from its raw images. The perfor-mance of these techniques was compared with that ofother methods on simulated data and was demonstratedon actual LIDAR data.

The simulation results indicate the strength of the fac-tor graph methods in many cases. The high quality ofthe silhouettes and the reconstructions based on the in-air LIDAR data shows that the factor graph methods per-form well on unblurred images. The degraded quality ofthe reconstructions based on submerged data suggeststhe possibility of improvement. The major obstacle in re-alizing this improvement is the limitation in the accuracyof the factor graph model imposed by the computationalcomplexity of the sum–product algorithm. However, al-ternatives to the sum–product algorithm, such as thetree-based reparameterization algorithm,32 have been de-veloped. Hence it may be possible to develop a reducedcomplexity algorithm that will allow for the tractable useof more accurate factor graph models.

ACKNOWLEDGMENTSThis work was supported in part by the Department ofthe Navy, Office of Naval Research, under grant N00014-

Fig. 16. WF-SFS reconstruction from shadow data.

Fig. 17. FG-SFS reconstruction from shadow data.

Fig. 18. FG3D reconstruction from shadow data.

01-1-0117 and by the National Science Foundation undergrant CCR-0092598 (CAREER).

Robert Drost and Andrew Singer can be reached bye-mail at [email protected] and [email protected], respec-tively.

REFERENCES1. I. Quidu, J. Ph. Malkasse, G. Burel, and P. Vilbe, ‘‘Mine

classification based on raw sonar data: an approach com-bining Fourier descriptors, statistical models, and geneticalgorithms,’’ in Proceedings of Oceans 2000 MTS/IEEEConf. and Exhibition (IEEE Press, Piscataway, N.J., 2000),Vol. 1, pp. 285–290.

2. Q. Zheng, S. Z. Der, and H. I. Mahmoud, ‘‘Model-based tar-get recognition in pulsed ladar imagery,’’ IEEE Trans. Im-age Process. 10, 565–572 (2001).

3. P. J. Shargo, N. Cadalli, A. C. Singer, and D. C. Munson, Jr.,‘‘A tomographic framework for LIDAR imaging,’’ in Proceed-ings of the IEEE International Conference on Acoustics,Speech, and Signal Processing (IEEE Press, Piscataway,N.J., 2001), Vol. 3, pp. 1893–1896.

4. W. N. Martin and J. K. Aggarwal, ‘‘Volumetric descriptionsof objects from multiple views,’’ IEEE Trans. Pattern Anal.Mach. Intell. PAMI-5, 150–158 (1983).

5. A. Laurentini, ‘‘How far 3D shapes can be understood from2D silhouettes,’’ IEEE Trans. Pattern Anal. Mach. Intell.17, 188–195 (1995).

6. A. Laurentini, ‘‘The visual hull concept for silhouette-basedimage understanding,’’ IEEE Trans. Pattern Anal. Mach.Intell. 16, 150–162 (1994).

7. Lavakusha, A. K. Pujari, and P. G. Reddy, ‘‘Linear octreesby volume intersection,’’ Comput. Vision Graph. Image Pro-cess. 45, 371–379 (1989).

8. M. Potemsil, ‘‘Generating octree models of 3D objects fromtheir silhouettes in a sequence of images,’’ Comput. Vis.Graph. Image Process. 40, 1–29 (1987).

9. N. Ahuja and J. Veenstra, ‘‘Efficient octree generation fromsilhouettes,’’ in Proceedings of the IEEE Conference on Com-puter Vision and Pattern Recognition (IEEE Press, Piscat-away, N.J., 1986), pp. 537–542.

10. M. Jones and J. P. Oakley, ‘‘Efficient representation of ob-ject shape for silhouette intersection,’’ IEE Proc. Vision Im-age Signal Process. 142, 359–365 (1995).

11. J. C. Carr, W. R. Fright, A. H. Gee, R. W. Prager, and K. J.Dalton, ‘‘3D shape reconstruction using volume intersectiontechniques,’’ in Proceedings of the IEEE International Con-ference on Computer Vision (IEEE Press, Piscataway, N.J.,1998), pp. 1095–1100.

12. Lavakusha, A. K. Pujari, and P. G. Reddy, ‘‘Volume intersec-tion algorithm with changing directions of view,’’ in Pro-ceedings of the International Workshop on Industrial Appli-cations of Machine Intelligence and Vision (IEEE Press,Piscataway, N.J., 1989), pp. 309–314.

13. M. Kass, A. Witkin, and D. Terzopoulos, ‘‘Snakes: activecontour models,’’ Int. J. Comput. Vision 1, 321–331 (1988).

14. C. Xu and J. L. Prince, ‘‘Snakes, shapes, and gradient vec-tor flow,’’ IEEE Trans. Image Process. 7, 959–969 (1998).

15. A. K. Jain, Y. Zhong, and S. Lakshmanan, ‘‘Object matchingusing deformable templates,’’ IEEE Trans. Pattern Anal.Mach. Intell. 18, 267–278 (1996).

16. G. Poggi and A. R. P. Ragozini, ‘‘Image segmentation bytree-structured Markov random fields,’’ IEEE Signal Pro-cess. Lett. 6, 155–157 (1999).

17. R. A. Weisenseel, W. C. Karl, D. A. Casanon, and R. C.Brower, ‘‘MRF-based algorithms for segmentation of SARimages,’’ in Proceedings of the International Conference onImage Processing (IEEE Computer Society Press, LosAlamitos, Calif., 1998), Vol. 3, pp. 770–774.

18. M. Mignotte, C. Collet, P. Perez, and P. Bouthemy, ‘‘Unsu-pervised Markovian segmentation of sonar images,’’ in Pro-


ceedings of the IEEE International Conference on Acoustics,Speech, and Signal Processing (IEEE Press, Piscataway,N.J., 1997), Vol. 4, pp. 2781–2784.

19. D. Snow, P. Viola, and R. Zabih, ‘‘Exact voxel occupancywith graph cuts,’’ in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition (IEEE ComputerSociety Press, Los Alamitos, Calif., 2000), Vol. 1, pp. 345–352.

20. R. E. Walker and J. W. McLean, ‘‘LIDAR equations for tur-bid media with pulse stretching,’’ Appl. Opt. 38, 2384–2397(1999).

21. J. W. McLean, J. D. Freeman, and R. E. Walker, ‘‘Beamspread function with time dispersion,’’ Appl. Opt. 37, 4701–4711 (1998).

22. N. Cadalli, D. C. Munson, Jr., and A. C. Singer, ‘‘Bistatic re-ceiver model for airborne LIDAR returns incident on an im-aging array from underwater objects,’’ Appl. Opt. 41, 3638–3649 (2002).

23. R. E. Walker, Marine Light Field Statistics (Wiley, NewYork, 1994).

24. F. R. Kschischang, B. J. Frey, and H.-A. Loeliger, ‘‘Factorgraphs and the sum–product algorithm,’’ IEEE Trans. Inf.Theory 47, 498–519 (2001).

25. F. R. Kschischang and B. J. Frey, ‘‘Iterative decoding ofcompound codes by probability propagation in graphicalmodels,’’ IEEE J. Sel. Areas Commun. 16, 219–230 (1998).

26. B. J. Frey, R. Koetter, and N. Petrovic, ‘‘Codes on images

and iterative phase unwrapping,’’ in Proceedings of theIEEE Information Theory Workshop (IEEE Press, Piscat-away, N.J., 2001), pp. 9–11.

27. S. M. Aji, G. B. Horn, and R. J. McEliece, ‘‘Iterative decod-ing on graphs with a single cycle,’’ in Proceedings of theIEEE International Symposium on Information Theory(IEEE Press, Piscataway, N.J.), p. 276.

28. G. Winkler, Image Analysis, Random Fields and MarkovChain Monte Carlo Methods: A Mathematical Introduction(Springer, New York, 2003).

29. H. Elliot, H. Derin, R. Cristi, and D. Geman, ‘‘Application ofthe Gibbs distribution to image segmentation,’’ in Proceed-ings of the IEEE International Conference on Acoustics,Speech, and Signal Processing (IEEE Press, Piscataway,N.J., 1984), Vol. 9, pp. 678–681.

30. A. Minagawa, K. Uda, and N. Tagawa, ‘‘Region extractionbased on belief propagation for Gaussian model,’’ in Pro-ceedings of the International Conference on Pattern Recog-nition (IEEE Computer Society Press, Los Alamitos, Calif.,2002), Vol. 2, pp. 507–510.

31. J. S. Lim, Two-Dimensional Signal and Image Processing(Prentice Hall, Englewood Cliffs, N.J., 1990).

32. M. Wainwright, T. Jaakola, and A. Willsky, ‘‘Tree-basedreparameterization framework for approximate estimationof stochastic processes on graphs with cycles,’’ Tech. Rep.P-2510 (Laboratory for Information and Decision Systems,MIT, Cambridge, Mass., 2001).

factor graph methods for three-dimensional shape...

Documents