a linear elastic force optimization model for shape …...shapes. after proving the existence and...

J Math Imaging Vis manuscript No.(will be inserted by the editor)

A Linear Elastic Force Optimization Model for ShapeMatching

Konrad Simon · Sameer Sheorey · David Jacobs and Ronen Basri

Abstract We employ an elasticity based model toaccount for shape changes. In general, to solve theunderlying equations for the deformation, bound-ary conditions have to be incorporated, e.g., in theform of correspondences between contour points.However, exact boundary correspondences are usu-ally unknown. We propose a method that is ableto optimize pre-selected boundary conditions suchthat external forces causing the shape change areminimized in some sense. Thus we seek simple phys-ical explanations of shape change close to a pre-selected deformation. Our method decomposes thefull nonlinear optimization problem into a sequenceof convex optimizations. Artificial and natural ex-amples of shape change are given to demonstratethe plausibility of the algorithm.

Keywords Shape matching · Convex optimi-zation · Force optimization · Linear elasticity ·Finite elements

K. SimonDepartment of Computer Science and Applied Mathematics,The Weizmann Institute of Science, Rehovot 76100, IsraelE-mail: [email protected]

S. SheoreyUtopiaCompression Corporation, Los Angeles, CA 90064,USA

D. JacobsDepartment of Computer Science, University of Maryland,College Park, MD 20742, USAE-mail: [email protected]

R. BasriDepartment of Computer Science and Applied Mathematics,The Weizmann Institute of Science, Rehovot 76100, IsraelE-mail: [email protected]

1 Introduction

Given two shapes of an object, as seen for exam-ple in images of the same or similar scenes, shapematching denotes the process of finding point cor-respondences between them. Finding correspond-ences between shapes yields valuable informationabout their differences and is important in many ap-plications such as recognition, medical image reg-istration (where shapes represent substructures likeorgans) or motion correction during surgery. In theseand other imaging situations the observed shapesoften depict actual non-rigid physical bodies. Hence,shape change takes place due to and can be ex-plained by forces that cause an elastic deformation.Elastic deformable models can be used to describethe motion of animals, people, hands etc. and caneven describe changes in shape across instances ofthe same visual category (e.g., different faces).

1.1 Related Work

Shape matching and image registration are very ac-tive fields of research and many different methodshave been presented in recent decades, see [21,11]for surveys. A common scheme applied in elastic-ity or fluid based matching is to derive a local (dis-)similarity measure between the transformed andthe target image, e.g., based on intensity difference.Then forces are derived from this similarity mea-sure, which drive the registration under a certainsmoothness prior on the deformation field. Exam-ples include [2,3,16], which employ a linear elas-ticity prior and [31], which employs a hyperelasticsmoother. Boundary conditions are usually set tozero on the image boundary such that deformationonly takes place inside the image and thus possibly

2 Konrad Simon et al.

far away from the boundary. Also the derived imageforce introduces an (additional) nonlinearity whichmay complicate the energy functional to be mini-mized and hence requires careful numerical treat-ment.

Elastic models were used in a number of studiesto model deformation. Ilic and Fua [22] use a non-linear Timoshenko beam model to track large de-formations of beam like structures in image sequen-ces. There displacements and forces are computedin an alternate manner and are derived from imageforces. Actual physical forces that are responsiblefor the deformation are then derived from the ob-tained displacements.

A parameter-free approach to elastic registra-tion was proposed by [29]. In this work the au-thors employ an elasticity model with zero bound-ary conditions on the image boundary and constraindisplacements in the interior by prescribing corre-spondences between identifiable substructures suchas object boundaries. This approach implicitly in-corporates driving forces as described above.

The authors of [26] use a linear elasticity-basedapproach to reconstruct a three-dimensional surfacefrom a single image by minimizing the stretch en-ergy of a template surface. Their optimization isconstrained by a set of boundary point correspond-ences whose objective is to limit the search spaceto deformations subject to unknown external for-ces. As such unknown external loads usually pre-vent the deformed surface from achieving a stressfree equilibrium state they show that knowing a setof correspondences on the boundaries is necessaryto obtain good results.

In [1] the authors use a linear elastic frame-work combined with an extended Kalman filter toprovide a method for non-rigid structure from mo-tion. In this method boundary correspondences arenot needed since these are computed within theirframework.

The authors of [18] use a linear elastic modelto set up a Riemannian structure on a manifold ofshapes. After proving the existence and uniquenessof energy minimizers they compute geodesic pathsbetween shapes to compare shapes that differ bypossibly large deformations.

Another approach is to model the template im-age as a viscous fluid such that pixels are trans-ported by a flow deforming the template to the de-sired target image. As fluids do not carry memoryabout their previous state, these methods are veryflexible and therefore suitable for large deforma-tions. We refer the reader to [10] for one of the firstworks in a non-variational framework and to [5]

for a variational setting. Smoothness of the trans-formation is ensured by requiring the velocity tobe smooth enough [14]. Despite impressive regis-tration results this approach might cause physicallyunreasonable transformations [36]. For a compari-son measure between shapes or images Beg et al. [5]give a geodesic distance in the space of diffeomor-phisms (which is not invariant under rigid motionsof objects observed in the scene). All theseapproaches are in the spirit of Grenander’s defor-mable templates, see [19], modeling shape defor-mations as the action of a group of diffeomorphismsand have been well investigated.

One disadvantage of the above mentioned elas-ticity and fluid methods is their computational com-plexity. Another way to define a shape is by its out-line. This representation is more compact in termsof dimensionality and effectively amounts in match-ing (closed) curves. In [37,38] curves are modeledby their angle functions using their arc-length para-metrization, and shape change is modeled by a groupaction on the shape contours leading to an elas-tic energy. Correspondences and a distance mea-sure are obtained by solving a nonlinear variationalproblem derived from that energy. A similar ap-proach was chosen in [27], where curves are mod-eled as points on an (infinite dimensional) manifoldand elastic properties are incorporated by imposinga suitable Riemannian metric. Curves are matchedby computing geodesics on this manifold, and thegeodesic distance is used as a comparison measure.The representation of shapes as a closed curve doesnot a priori take into account the interior of theshape. A good survey and references to additionalliterature are given in [39].

Other works focus on comparing shapes bymeans of their intrinsic geometric properties [8],e.g., the Gromov-Hausdorff distance. Also see workon statistical dynamics shape models [12]. Thesemodels capture correlations of deforming shapes(silhouettes) over time and are based on the prin-ciple that observations of a certain shape at a giventime may depend on previous observations of thatshape.

1.2 Our Method

In the present work we propose a method for regis-tering shapes around a pre-selected (possibly usersupplied) deformation that is able to account forsmall deformations by employing a linear elasticitymodel. This model is invariant under translationsand under infinitesimal rotations. Once the shape

Elastic Force Optimization for Shape Matching 3

boundaries of the template and target objects areextracted our task is to find physically reasonableboundary correspondences. Exact boundary condi-tions are in general unknown [17] but one can of-ten give an initial guess. For example, the authorsof [29,17] employ active contour models (activesurface models respectively) to obtain boundary cor-respondences of objects observed in the images thatare to be compared.

To measure the complexity of shape change ex-ternal forces acting on the shape boundaries canbe used. In particular, if these forces are sparse (orsmall), one can consider the deformation to be small.Sparse forces, we believe, can provide a suitableand simple comparison measure for deformations.Given two shapes and a guess of boundary corre-spondences we minimize the forces on the bound-ary (in some norm) by simultaneously allowing theestimated correspondences to drift after paying apenalty in terms of a restoring force. The full opti-mization problem is highly nonlinear but we presenta convex relaxation which is subject to a linear PDE.Nevertheless, there are possibly many local minimaof the optimization problem. Hence, an initial guessof the boundary displacements localizes the searchfor a plausible (and simple) explanation of the ob-served deformation. The initial guess can be inter-preted as a pre-selected deformation. So the convexrelaxation of the full optimization that we sugguestcan be interpreted as a method to provide a simplerexplanation (in terms of forces) of the pre-selecteddeformation close to the initial guess.

The discretization of the PDE is done by meansof a displacement based finite element method(FEM), which is a suitable tool for connecting bound-ary displacements and external forces. This is doneby a splitting of the stiffness matrix. The decompo-sition of the stiffness matrix used for our purposewas also applied in [7] in the context of surgerysimulation. This decomposition has the advantagethat it incorporates also the elastic properties of theinterior of the shape although the force optimizationis effectively only performed at boundary nodes.This reduces the dimension of the full problem sig-nificantly. The difference is that in [7] forces areknown and the displacements are computed whereasour method assumes neither is known.

A similar idea to our force optimization prob-lem has been introduced in [33] in the context ofmotion tracking. Unknown forces are assumed todrive the motion of an object in a scene accord-ing to Newton’s second law. This is a dynamicalapproach and it is used to give a convex formula-tion of the tracking problem. Their approach is very

scene specific, i.e., the optimization problem has tobe adapted to the specific situation in order to in-corporate as much information as possible aboutknown forces and other problem constraints. Ourforce optimization problem, in contrast, is nonlin-ear, and its relaxation consists, in contrast, of a se-quence of convex problems and does not depend onthe specific deformation observed.

Our method works on silhouette shapes in R2

and ignores intensities. The idea can easily be gen-eralized to shapes in R3. In our approach we dothe actual optimization of the correspondences onthe boundary curve only but at the same time wekeep information of the interior of the shape. We donot employ more information than the elastic prop-erties of the shape. After the optimized boundaryconditions are computed one can solve the elas-ticity PDE, post-process the solution, and obtainvarious valuable (comparison) measures like exter-nal forces, strain, stress or stored energy of the de-formed shape. Knowing such quantities can help toassess how severe a given deformation is and henceenables us to compare different deformations.

As a motivating example imagine a physicianwho wants to compare different CT-scans of a pa-tients liver. Taking a scan of the healthy liver we cancompare it to other scans and compare the differentdeformations to decide whether or not a pathophys-iological deformation is progressing by looking, forexample, at the external forces that are responsiblefor this deformation.

However, note that assessing an observed defor-mation is an (in the sense of Hadamard) ill-posedinverse problem and observing a small deformationdoes not necessarily mean that forces are sparse (orsmall). The situation is even worse since the de-formation can be nonlinear. In a forthcoming paperwe will investigate a generalized nonlinear elastic-ity setting in order to be able to treat rotations andlarger deformations not covered by linear elasticity.

In the following we will give a short overviewof the relevant elasticity model and the FEM weemployed. A detailed description of the decompo-sition of the optimization procedure is given in Sec-tion 2.3 and experimental examples are shown inthe following section. We end the paper with a dis-cussion of the method and of our results.


2 Description of Methods

2.1 The Physical Model of a Linear Elastic Body

This section intends to give a concise overview ofthe basic principles of elasticity theory that are rele-vant for our model. More material on elasticity canbe found, e.g., in [4], [6] and [32]. We will followthe notation introduced in [9].

Elasticity theory regards the state of a body sub-mitted to forces. An elastic body is a body that re-acts to applied forces with a deformation, but re-turns to its original shape after removing the for-ces. Furthermore, an elastic body does not memo-rize previous deformations.

Strain, Stress and Equilibrium. Our starting pointis the assumption that there is a known referenceconfiguration B ⊂ R3 of the deformable body thatwill serve as a template body in the template image.This set B shall be bounded, and it describes thesubset in R3 which is occupied by the body whenno forces are applied, i.e., the body is free of stress.The shape of the deformed body will be describedby a vector field Φ : B→R3, i.e., Φ(x) is a point ofthe deformed body corresponding to a point x in thereference configuration. We use the decompositionΦ = I+ u, where I denotes the identity transfor-mation and u the displacement field. Now, Φ is adeformation if subsets of the body are mapped tosets of positive volume, i.e., the determinant of thedeformation gradient det(∇Φ)> 0.

A Taylor approximation of Φ shows that thequantity that is responsible for a local change oflength is the (right) Cauchy-Green strain tensor C =

(∇Φ)T ∇Φ . Its half deviation from the identity E =

1/2(C− I) is the Green-Saint-Venant strain tensoror simply strain. Intuitively, strain is a measure thatlocally represents a change of distance of nearbypoints relative to each other. In terms of the de-formation field u Cauchy-Green strain reads E =12

(∇u+(∇u)T +(∇u)T ∇u

). Neglecting the quad-

ratic nonlinearity leads to the linear theory of elas-ticity (small deformations) and E simply becomesthe symmetric part of the derivative of the displace-ment field. We denote it as ε = 1

2

(∇u+(∇u)T

).

The next important quantity is the stress tensoror simply stress. A given deformation induces, asdescribed above, a certain strain. Stress is a mea-sure that quantifies how difficult it is to achieve acertain strain, i.e., stress is a function of strain (seenext section). Intuitively it is clear that rigid trans-forms do not change stresses in a body. One canprove that if B is connected, a deformation Φ is

a rigid transform (rotation and translation) if theCauchy-Green tensor satisfies C(x) = I for all x ∈B. More generally, one can prove that two defor-mations corresponding to the same strain tensor Ccan be obtained by a composition with a rigid trans-form, see [9]. As a consequence we can identifytwo deformations if they have the same strain ten-sor. Hence, the strain E can be viewed as a measureof the deviation of a deformation Φ from a rigidtransformation. However, for the measure ε this isnot true. As ε is a linear approximation to E it isinvariant under translations and infinitesimal rota-tions but not under general rotations. This causessome difficulties in comparing shapes through for-ces applied to the template body. But if the shapesthat are to be compared differ from each other withno or minimal global rotation one can still expecta reasonable comparison by strain or stress or anyquantities that are derived from them like externalforces. The latter will be explained in the following.

Forces acting on a body can either be body for-ces or surface forces. Body forces are proportionalto mass and are related to an outside source (e.g.,gravity or magnetic forces). These forces are vol-ume forces and can be described by a force den-sity F : B→ R3 (measured per unit volume). Thesurface forces will be particularly interesting in ourformulation and are described in the following. Letus take a point x ∈ B and an arbitrary cross-sectionwithin the body, described by its (unit) normal n ∈R3. A surface force t(x,n)∈R3, called traction andmeasured per unit area, acts on the cross-section inx and is described by the Cauchy stress tensor σ ∈R3×3. The rows of this stress tensor represent thethree tractions (normal and shear stresses) withinthree canonical coordinate planes, usually the threedifferent orthogonally intersecting cartesian coor-dinate planes. One can then write the surface forceat x ∈ B as t(x,n) = σ(x)n, see Figure 1 for an il-lustration and [32] for more details. In particular, ifx ∈ ∂B and if n is its corresponding outward unitnormal the surface force t(x,n) is the traction onthe body’s boundary in x.

The equations of static equilibrium (Cauchy prin-ciple) can be formulated as a consequence of New-ton’s second law: In any closed subvolume V thebody forces on its boundary and inside V have tobalance, meaning∫

∂Vt(x,n)dS+

∫V

F(x)dV = 0 . (1)

The divergence theorem then yields∫V(divσ +F) dV = 0 , (2)


Fig. 1: Left: an illustration of a surface force t(x,n) at some point x in an arbitrary cross section with normal n of a body B. Right: illustration of the threecomponents of the stress tensor with respect to the canonical coordinate planes. The normal stress is orthogonal to the considered plane shear stresses liewithin the plane. The figure was created using Incscape.

and since this shall hold for any subvolume the in-tegrand must vanish. Hence, the equations of equi-librium can be written as

divσ +F = 0 . (3)

By the same principle for the momenta one showsthat σ is symmetric. The divergence of the tensoris to be taken row-wise. Note that the equationsof equilibrium, although we did not point that outclearly, must be formulated in the (unknown) de-formed configuration Φ(B). Transformingthese back to the reference configuration would in-troduce a nonlinearity, namely an additional fac-tor that depends on ∇Φ . However, in the setting ofsmall deformations this factor is close to the iden-tity and will be neglected, so that it does not makea difference to formulate the equations in terms ofthe reference configuration. We will deal with thisnonlinearity in a separate paper that addresses thecase of larger deformations.

Constitutive Equations for Linear Elastic Ma-terials. For elastic materials stress will depend onstrain and material properties. Below we briefly ex-plain the relation between strain and stress.

Intuitively it is clear that forces that are of thesame magnitude will cause different deformationswhen acting on different physical materials, i.e.,stress depends on the physical properties of the ma-terial. It measures how difficult a deformation is toachieve since it describes the internal force distri-bution in the body, i.e., it has to balance the ex-ternal force according to Newton’s second law (1).On the contrary, any deformation field in a body in-duces a certain strain ε = ε(u). The cause of thedeformation is an external force, gravity for exam-ple. Again, stress measures how much force at each

point inside the body is necessary to balance the ex-ternal cause of the deformation and this will dependon the material properties.

The relation between stress and strain is expressedby so-called constitutive laws. This is a priori justany function that relates strain and stress. The as-sumption of frame invariance states that a physicalquantity observed is independent of the observer,which is a common physical principle. Using thisone can restrict the class of suitable functions. Fur-ther restriction can be made by assuming that thematerial is homogenous, i.e., the material proper-ties are the same at any point inside the body. An-other restriction can be deduced from the assump-tion of isotropy. This means that the reaction ofa body to an applied external force at any pointwill essentially be the same in any direction. Steel,for example, is an isotropic material in contrast towood. Now, one can mathematically prove that anyfunction relating stress and strain that satisfies theseassumptions has a first order approximation that onlyinvolves two physical constants. The first order ap-proximation assumes small strains (and this is whatwe do in linear elasticity). This simple constitutivelaw is called Hooke’s law for linear isotropic mate-rials and can be written as

σ = λ tr(ε)I+2µε (4)

with only two independent positive constants. Heretr(·) denotes the trace and ε is the linearized straintensor. The interested reader is referred to [9]. Thematerial constants involved are the well-known Laméconstants λ and µ . In the literature sometimes sev-eral other constants are used to formulate this lawand they can be transformed into one another. Of-ten used are, for example, Young’s modulus E and


Poisson’s ratio ν which relate to the Lamé con-stants by

E =µ(3λ +2µ)

λ +µ, ν =

λ

λ +µ. (5)

Plane Strain. While our formulation can readilybe applied to the three-dimensional case, this paperdeals with deformations of two-dimensional shapes,in which case we do not have to take into accountthe full three-dimensional model. Denote x=(x1,x2,x3)

T

the three spatial coordinates. If we assume that thebody forces and the tractions on the body’s bound-ary are independent of x3 and do not have a com-ponent pointing out of the (x1,x2)-plane, we canassume u = (u1,u2,0)

T , u1 = u1(x1,x2) and u2 =

u2(x1,x2). This leads to a reduced 3-by-3 strain ten-sor in which the third row and column are zero.Nevertheless, by Hooke’s generalized law the ma-terials response leads to a stress tensor in which thethird row and column also vanish apart from σ33 =

λ (ε11 + ε22). Here one can see that even for two-dimensional forces and displacements that stress isin general still a three-dimensional quantity, but tak-ing into account that all these quantities only de-pend on x1 and x2 the equilibrium equations re-duce to a two-dimensional system. Combining thestress-strain relation (4) and the equilibrium equa-tion (3), this system, the well-known Navier-Laméequation, reads

µ∆u+(λ +µ)∇divu+F = 0 . (6)

2.2 The Pk-FEM

As an elliptic partial differential equation (PDE) themodel for small deformations can be handled com-fortably by finite element methods (FEMs). Thisshall be explained briefly. The interested reader isreferred to literature on FEMs and PDEs, [6,9,15,24].

In order to ensure unique solvability of the prob-lem, one has to impose boundary conditions so thatthe PDE can be written as

µ∆u+(λ +µ)∇divu+F = 0 in B,

u = u0 on ΓD,

σ(u)n = g on ΓN

(7)

where ΓD and ΓN is a decomposition of the body’sboundary ∂B into a part ΓD on which Dirichlet bound-ary conditions u0 are given and a part ΓN with Neu-mann boundary conditions, i.e., a force distributiong. The outward normal on ΓN is denoted by n. For

the sake of simplicity we use zero boundary condi-tions on the Dirichlet boundary, that is u0 = 0. Thegeneralization to non-zero conditions is just techni-cal. Also note that (7)1 is equivalent to

2µ divε(u)+λ∇ tr(ε(u))+F = 0 . (8)

The FEM relies on the variational formulationof the problem. We multiply (7)1 with a test func-tion v in the Sobolev space H1

ΓD

H1ΓD

:={

v ∈ H1(B)3 | v(x) = 0 on ΓD}

(9)

and integrate by parts. Here H1(B) denotes the stan-dard Sobolev space of square integrable functionson B which have one (weak) derivative in L2(B).The superscript “3” in (9) means that each compo-nent of v is in H1(B). Utilizing (8) this yields∫

Bσ(u) : ε(v)dV =

∫ΓN

g · v dS−∫

BF · v dV (10)

where A : B = tr(AT B) denotes the Frobenius scalarproduct. If equation (10) is satisfied for all v ∈ H1

ΓDthen u is called a weak solution of (7). Note thatin order to satisfy the weak form of the originalequation we require the displacement field to haveonly one weak derivative instead of two classicalderivatives and any classical solution will satisfythe weak form. The left-hand side of (10) is linearin u and v and so it is referred to as the bilinear formof the equation. The unique solvability of (10) inthe function space H1

ΓDis ensured by means of the

Lax-Milgram lemma that requires the bilinear formto have certain properties (continuity and coerciv-ity). The latter one is ensured by Korn’s inequali-ties. The well-posedness principles in the continu-ous case are inherited to our finite element model.The interested reader is referred to [6]. In engineer-ing the weak form is also known as the principle ofvirtual work.

As Galerkin techniques FEMs approximate thesolution u of the full PDE in a (suitable) finite di-mensional subspace Vh ⊂ H1

ΓD. Denote a(u,v) the

associated bilinear form of the left-hand side of (10)and b(v)−c(v) its right-hand side. Now, given a ba-sis (ϕi)

ni=1 of Vh one can make the ansatz

uh =N

∑i=1

uiϕi (11)

with real coefficients ui. Plugging this into (10) andtesting only with v ∈ Vh (it is of course sufficientto test only with the base functions) the Galerkinequation reads

N

∑i=1

uia(ϕi,ϕ j) = b(ϕ j)− c(ϕ j) ∀ j = 1, . . . ,N .


(12)

This is a linear equation for the coefficients of uhwith stiffness matrix A=(a(ϕi,ϕ j))i j and load vec-tor b(ϕ j)− c(ϕ j).

Since the bilinear form a(·, ·) is symmetric andcoercive the stiffness matrix is positive definite andsymmetric and hence the linear equation has a uniquesolution. In case of pure Neumann boundary condi-tions, i.e., a force distribution is prescribed on theentire boundary of the body, the stiffness matrixwill be singular and its image will have the codi-mension of the spatial dimension of the elasticityproblem plus one, two in our case. This is due tothe invariance of the stress tensor (and hence ofthe elastic force) under translations and infinitesi-mal rotations. Furthermore in this case a compati-bility condition between the prescribed tractions onthe boundary and the given body forces has to besatisfied. To see this one has to integrate (7)1 andapply the divergence theorem to the left-hand sidewhich shows that the mean value of the boundaryforce distribution has to be the same as the mean ofthe body force taken over the entire body.

We now have to choose Vh in a suitable manner.We do this using triangular finite elements. For thispurpose we generate a triangulation Th of the rele-vant shape, preferably a Delaunay triangulation inorder to avoid skinny triangles. Here h just denotesa grid parameter indicating its coarseness.

Consider a Delaunay triangulation Th of ourbody B with M nodes (pi)

Mi=1 on the boundary of

B and L interior nodes (pi)Ni=M+1, M +L = N. We

choose

Vh = {v ∈ H1 | v is continuous

and v ∈ Pk(K)∀K ∈Th}(13)

where either k = 1 or k = 2 and

Pk(K) = {v : K→ R | v is a polynomial

of degree k} .(14)

Note that for a triangle K ∈ Th each P1-functionis uniquely determined by its values at the trian-gle’s corners and each P2-function is uniquely de-termined by its values at the corners and the mid-points of the edges. Hence, if we define a basis forVh by

ϕi : B→ Rϕi(p j) = δi j ∀i, j = 1, . . . ,N

(15)

we get a set of N (scalar valued) base functions,where the base function corresponding to node iis supported only in adjacent cells (triangles). The

values inside each cell are obtained by linear inter-polation in case of P1-elements and quadratic inter-polation in case of P2-elements. This property leadsto a stiffness matrix which is sparse. In our case ofplain strain we have to choose two linear indepen-dent base functions at each node i, i.e., one for eachcomponent of the displacement. We use the vector-ized base functions (ϕi,0)T and (0,ϕi)

T . To incor-porate the Dirichlet boundary conditions we simplyset the values at the relevant nodes to the prescribedvalue of the solution at these points, which will thencontribute to the right-hand side of the equation.

2.3 Boundary Correspondence

We are now equipped with the described deforma-tion model, the Navier-Lamé equation, and a suit-able tool for computing its solution, the FEM. Asmentioned, we will have to supplement this modelwith boundary conditions. Suppose we knew thecorrespondences between the boundaries of the tworelevant shapes, B and B′, i.e., we are given pureDirichlet boundary conditions on the boundaries ofthe shapes Γ and Γ ′ respectively. We could com-pute the solution of the finite element equation andin post-processing compute various elasticity mea-sures which will tell us how difficult it is to achievethe observed shape change. However, exact corre-spondences are usually not known. In the follow-ing we will not take into account volume forces, al-though these can readily be incorporated. Of courseboundary displacements cause forces on the bound-ary and vice versa.

Our working assumption is that even if one ob-serves a very complicated deformation (think of adeforming car in a crash test or leaves of a tree ex-posed to blowing wind from one direction) its phys-ical cause will often be a (boundary) force of re-markable simplicity. Hence, we would like to opti-mize the boundary correspondences such that theircause is simple, i.e., that boundary forces act ac-cording to a certain prior. This can be formulatedas a nonlinear optimization problem subject to theelasticity PDE:

minimizeu

∫Γ

‖σ(u)n‖p dS

subject to µ∆u+(λ +µ)∇divu = 0

Φ(Γ ) = Γ′ .

(16)

Recall that Φ = I+ u. The notation Φ(Γ ) = Γ ′

simply means that we want the boundary Γ to bemapped onto the boundary Γ ′. The norm used is


(a) Template shape (b) Target shape 1 (c) Initial boundarydisplacements (for targetshape 1)

(d) Target shape 2 (rotated) (e) Initial boundarydisplacements (for targetshape 2)

(f) Optimal boundarydisplacements afterL1,2-optimization (target1)

(g) Optimal boundarydisplacements afterL2,2-optimization (target1)

(h) Optimal boundarydisplacements afterL1,2-optimization (target2)

(i) Optimal boundarydisplacements afterL2,2-optimization (target2)

(j) Optimal boundary for-ces after L1,2-optimization(target 1)

(k) Optimal boundary for-ces after L2,2-optimization(target 1)

(l) Optimal boundary for-ces after L1,2-optimization(target 2)

(m) Optimal boundary for-ces after L2,2-optimization(target 2)

Fig. 2: Shearing force experiment. Comparison of two deformations of a shape into two target shapes differing only by a rotation of π/12 around theshape center. Green dots in (c) – (i) represent the target shape and red dots in (j) – (m) the template shape. Force vectors have been scaled and not alldisplacement vectors are plotted for better visibility.

a mixed norm, i.e., we take the Euclidean magni-tude of the force σ(u)n and raise it to the power p.We will specify p later. With regard to applying thefinite element method the linear constraint can begiven in the weak form:

minimizeu

∫Γ

‖σ(u)n‖p dS

subject to∫

Bσ(u) : ε(v)dV

=∫

Γ

g · v dS ∀v ∈ H1(B)

Φ(Γ ) = Γ′ .

(17)

This optimization problem of finding the right bound-ary correspondences is difficult and a priori a highlynonlinear problem but in the case of small defor-mations one can often give a (more or less good)guess of correspondences and solve a simpler prob-lem instead of the original problem. If we were ableto relate boundary displacements to forces on theboundary directly we could optimize these forcesaccording to our prior. Simultaneously we can al-low the boundary correspondences to drift away from

the initial guess after paying some penalty and ob-tain more likely boundary displacements. This shouldresult in a better shape match. This can be formu-lated as another nonlinear optimization problem:

minimizeu

∫Γ

‖σ(u)n‖p +k2‖u−u0‖2 dS

subject to∫

Bσ(u) : ε(v)dV

=∫

Γ

σ(u)n · v dS ∀v ∈ H1(B) .

(18)

This is still a nonlinear optimization problem sub-ject to a linear PDE but without the nonlinear con-straint Φ(Γ ) = Γ ′. This constraint is localized bythe second term in the objective function of (18)around a given initial guess u0. It should be men-tioned that this localization changes the full prob-lem of finding a simple explanation of an observedshape change to finding a simple explanation of thedeformation that is close to the pre-selected initialguess. The second term in the objective function


of (18) can be interpreted as some kind of a restor-ing spring force that increases the more the bound-ary displacements u deviate from the initial guessu0, while k can be interpreted as a spring constant.This restoring spring force does not directly act onthe physical model itself in contrast to σ(u)n. Also,note that for p > 1 the optimization problem (18) isconvex. In order to take into account curvature in-formation of the target boundary one can replacethe Euclidean distance of in the objective with theMahalanobis distance as we shall describe in thesequel. Note that there is only one linear constraintleft, and that it contains only boundary forces mak-ing it a pure Neumann problem where the force dis-tribution σ(u)n depends on the yet unknown dis-placements on Γ .

Next, we translate this optimization problem intoa discretized setting. The finite element frameworkwill allow us to connect displacements and bound-ary forces conveniently. Assume a pure Neumannboundary value problem with unknown boundaryforce distribution. Let

A =

(ABB ABIAIB AII

)(19)

be the stiffness matrix of the linear system that weassemble using the FEM (as explained in section 2.2,equation (12)) in block decomposition. The blockABB corresponds to the boundary part of A, i.e.,the entries of ABB are the numbers a(ϕi,ϕ j) for allboundary nodes i, j. The load vector undergoes asimilar decomposition, such that the system can bewritten as(

ABB ABIAIB AII

)(uBuI

)=

(fB0

), (20)

where uB and fB denote the (column-) vectors ofdisplacements at boundary nodes and uI denotesthe displacements at inner nodes, see also equa-tion (22) below. Note that fI = 0 on the right-handside of (20) since we do not have volume forces.We can solve for the boundary displacements uB bytaking the Schur complement of A w.r.t. ABB whichgives

SuB = fB where S = ABB−ABIA−1II AIB . (21)

This splitting was also done in [7] in the contextof surgery simulation. Having K grid points on theboundary the vectors uB and fB are of the length2K (in case of plane strain). We arrange the dis-placements ui ∈ R2 and the forces fi ∈ R2 at each

boundary node in uB ∈ R2K and fB ∈ R2K in thefollowing manner:

uB =

u(1)1

u(2)1...

u(1)K

u(2)K

and fB =

f (1)1

f (2)1...

f (1)K

f (2)K

, (22)

where u( j)i and f ( j)

i denote the j-th component ofthe displacement (force resp.) at boundary node i.The matrix S can then be split into blocks Si of size2× 2K, such that fi = SiuB ∈ R2. Given the initialguess u0

B for the boundary displacements, the opti-mization problem can then be formulated as

minimizeub

K

∑i=1‖Siub‖p +

k2

∥∥uB−u0B∥∥2

2 (23)

meaning that we allow deviation from the estimatedcorrespondence but at the same time we want theforces to be simpler in some norm. We will comeback to this point later. Note that the decomposi-tion of the stiffness matrix takes into account thebehaviour of the interior of the shape, and the PDEconstraint is a priori satisfied by finding its uniquesolution to the corresponding Dirichlet boundaryconditions. The parameter k plays the role of a springconstant, increasing or decreasing the penalty dueto deviations from the initial guess. Also note thatour objective contains numerical integrals of theforce distribution weighted by the elements’ (local)base functions. Hence, optimization over it is ef-fectively an optimization over a smoothed versionof the external force distribution.

Focusing on the second term of the objective,the reader can see that it allows the approximatecorrespondences to drift. Unfortunately, the drift cango away from the target shape’s boundary which isclearly not desirable. Also, the objective does notinclude any kind of shape information (it only in-cludes the prior on the forces for the unknown de-formation). This means that our relaxed formula-tion is still an inappropriate approximation of thereal problem because only drifts on or close to theboundary manifold are desired. One way to dealwith this is by incorporation of distorted metrics,generated by positive definite forms, around eachof the target points of the initial guess u0

B on thedeformed shape. At each such point q on the tar-get shape we take the orthogonal coordinate sys-tem given by the unit normal n and unit tangent t.


(a) Template Shape(b) Initial boundary dis-placements 1

(c) Initial boundary dis-placements 2

(d) Target Shape(e) Optimal boundarydisplacements afterL1,2-optimization 1

(f) Optimal boundarydisplacements afterL2,2-optimization 1

(g) Optimal boundarydisplacements afterL1,2-optimization 2

(h) Optimal boundarydisplacements afterL2,2-optimization 2

(i) Optimal boundary for-ces after L1,2-optimization1

(j) Optimal boundary for-ces after L2,2-optimization1

(k) Optimal boundary for-ces after L1,2-optimization2

(l) Optimal boundary for-ces after L2,2-optimization2

Fig. 3: Beam bending experiment. Comparison of results under different initial boundary correspondences. Green dots in (b) – (h) represent the targetshape and red dots in (i) – (l) the template shape. Force vectors have been scaled and not all displacement vectors are plotted for better visibility.

(a) Template shape 1 (b) Template shape 2 (c) Target shape

(d) Initial boundary dis-placements (for templateshape 1)

(e) Optimal bound-ary displacementsafter L1,2-optimization(template 1)

(f) Optimal boundary for-ces after L1,2-optimization(template 1)

(g) Optimal bound-ary displacementsafter L2,2-optimization(template 1)

(h) Optimal boundary for-ces after L2,2-optimization(template 1)

(i) Initial boundary dis-placements (for templateshape 2)

(j) Optimal boundarydisplacements afterL1,2-optimization(template 2)

(k) Optimal boundary for-ces after L1,2-optimization(template 2)

(l) Optimal boundarydisplacements afterL2,2-optimization(template 2)

(m) Optimal boundary for-ces after L2,2-optimization(template 2)

Fig. 4: Sphere deformation experiment. The template in (a) has a radius of 90 pixels and the sphere in (b) 70. Green dots in this figure represent the targetshape and red dots represent the template shape. Force vectors have been scaled and not all displacement vectors are plotted for better visibility.


These vectors weighted by two positive real num-bers (rn,rt) form the principal axes of an ellipse de-scribed by the unit level set of the (affine) quadraticform generated by

Q =

(nT

tT

)(rn 00 rt

)(n t). (24)

The values rn and rt are chosen such that their prod-uct is one and such that the number of points in atangential direction that are within the unit circleof the metric described by ‖x‖2

Q := 〈x,Qx〉 is maxi-mized. This is a simple way to introduce curvatureinformation into the metric and can be seen as anapproximation of a metric in the tangent space ofthe shape boundary (within the ambient space) atthe relevant point. The objective function then takesthe form

f (uB) =K

∑i=1‖SiuB‖p +

k2‖uB−uB0‖2

Q . (25)

Note that

‖uB−uB0‖2Q =

K

∑i=1

⟨ui

B−ui,0B ,Qi(ui

B−ui,0B )⟩

(26)

where Q= diag(Q1, . . . ,QK). This unfortunately stillallows drifts away from the target boundary, but canbe overcome now by first optimizing (25) and thenprojecting the optimized correspondences back tothe target boundary. Note that the projection makesthis problem nonlinear but doing this iteratively amountsto a sequence of convex optimization problems forwhich we have efficient solvers. Also note that wereduced the full problem (16) to a problem on theboundary which reduces the computational dimen-sion.

So far we have not yet specified what norm wewant to use to minimize the forces (i.e., we didnot specify p in (23)). As mentioned above we areseeking simple explanations, not only for shape changebut also for the sake of comparing different defor-mations to each other. A set of forces that is sparsein some sense will be suitable for this comparison.Now, sparsity can mean several things in this con-text. First, one could seek a set of forces that onlyacts on a small subset of boundary points and is iso-tropic, i.e., it is sparse and does not favor any direc-tion. With regard to the example of the tree exposedto wind from some direction one would rather pre-fer the force derivative to be sparse so as to favor

a minimal number of directions. Here, we show re-sults with the first prior. Taking the objective as

fsparse(uB) =K

∑i=1‖SiuB‖+

k2‖uB−uB0‖2

Q (27)

will satisfy these needs. This equation is identicalto (25) with p = 1. We will refer to the optimi-zation in this norm as the L1,2 or mixed norm opti-mization. In our experiments we compare this ob-jective to

fsmall(uB) =K

∑i=1‖SiuB‖2 +

k2‖uB−uB0‖2

Q , (28)

which favors uniform and small forces. This is equa-tion (25) with p = 2

The optimization of (28) is a least squares prob-lem and has a closed form solution, but the prob-lem involving (27) has more structure. Introducingthe slack variables fi,e, where i = 1 . . .K, yields theproblem

minimizefi,e,ub

K

∑i=1

fi +k2

e

subject to ‖SiuB‖ 6 fi , i = 1 . . .K ,

‖uB−uB0‖2Q 6 e .

(29)

This is a second order cone program (SOCP). Specif-ically the second constraint can be transformed intoan SOCP constraint by rewriting it as

‖uB−uB0‖2Q 6

14((e+1)2− (e−1)2) (30)

which is equivalent to∥∥∥∥(2R(uB−uB0)

e−1

)∥∥∥∥26 e , (31)

where R=(diag(Q1, . . . ,QK))1/2. Putting everything

in one equation the optimization problem then reads

minimizefi,e,ub

K

∑i=1

fi +k2

e

subject to ‖SiuB‖ 6 fi , i = 1 . . .K ,∥∥∥∥(2R(uB−uB0)

e−1

)∥∥∥∥26 e .

(32)

Several fast solvers are available for this typeof problem. We used SeDuMi [35] together withthe YALMIP [28] interface for MATLAB.


3 Implementation and Results

To demonstrate our method the implementation, in-cluding our own finite element code, was done inMATLAB on a common PC. First we extracted theboundaries of the relevant shapes and constructed aDelaunay mesh of the template shape with a moder-ate number of cells (usually between 500 and 650).Initial boundary conditions were either given by handfor a small number of boundary points of the mesh(the remaining correspondences were interpolatedon the boundary of the target shape) or we alignedthe centers of mass of the two shapes and took thenearest neighbours of the boundary nodes of themesh and the target’s boundary. Generally, the lat-ter one gives a very poor estimate of correspond-ences but as we shall demonstrate in the first ex-periments with simple shapes the optimization pro-cedure is quite robust against a (moderate) changeof boundary conditions or a bad initial guess. Ofcourse there are many more ways to obtain initialcorrespondences, e.g., active contour models likesnakes, which have been used, for example, in [29,17].

Next we assembled the stiffness matrix with zeroNeumann boundary conditions and computed its Schurcomplement with respect to the block correspond-ing to the inner nodes of the mesh as described inequation (21). We implemented two finite elementmethods: The P1-method, i.e., the local base func-tions in each element are linear, and the P2-methodwith quadratic base functions. The P2-method wasmodified to be isoparametric, i.e., the boundary edgesof boundary cells were modified to locally give aquadratic approximation of the shape boundary (byprojecting the midpoint of the edge onto the bound-ary) instead of a linear one as in the P1-method.This modification took minor effort and can accountfor an improved approximation of the shape in re-gions with high curvature.

After the optimized boundary displacements werefound we computed the solution of the correspond-ing Dirichlet problem and several elasticity mea-sures (e.g., stored energy, von Mises stress, etc.).Such measures, based on stress and strain, can serveas measures for comparing different deformationsof a given template. If the material properties areknown, i.e., the Lamé constants are known, one canof course incorporate these into the model. In casethey are unknown one can make a reasonable choice.Our method showed robustness of the quality of theresults under different choices of the Lamé-constants.We will refrain from showing this here. All exper-iments that we present in this paper have been car-

ried out with Lamé-constants λ = 0 and µ = 1, al-lowing material to grow without being shrunk later-ally. This choice makes the model depend only onthe optimization parameter k since the remainingconstant µ drops out in the elasticity equation (6)(it can be taken into the right-hand side as a scalingfactor). This is in the spirit of [29].

Algorithm 1: Sketch of the algorithm

1 Generate a Delaunay Mesh on the templateshape;

2 Assemble the stiffness matrix (λ = 0,µ = 1);

3 Decompose the stiffness matrix according toEquation (21);

4 Give an initial guess of boundarycorrespondences for mesh boundary points;

5 Save the stored energy E of the deformedbody by using the initial guess inEquation (33);

6 for i = 1 to maxIter do7 For each boundary correspondence

compute its Mahalanobis metric〈·,Qi(·)〉 according to (24);

8 Minimize the relevant cost function, i.e.,either (27) or (28) to obtain newboundary correspondences;

9 Save the stored energy E of the deformedbody for the new correspondencesaccording to (33);

10 Project the optimized correspondencesonto the target shape boundary to getnew correspondences;

11 Save the stored energy E of the deformedbody by using the projectedcorrespondences in (33);

12 if E is stationary then13 Return boundary correspondences

and forces;14 end15 end16 Find the FEM solution and compute

elasticity measures (strain, stress, etc.);

A sketch of the algorithm is given in Algorithm 1.In the algorithm we used the internal energy of thedeformed shape, given by

E = ((uB,uI)T A(uB,uI))

1/2 = (uTBSuB)

1/2 , (33)

as a simple indicator for convergence. Of course itis possible to use different measures such as basedon forces as well. It also turned out to be good to pe-


nalize deviations from the initial guess less duringthe first iterations and then increase the penalty, i.e.,we choose a low k in the beginning and increase itbased on the convergence measure.

In all experiments shown here we will comparetwo optimization procedures, differing by the prioron the unknown forces, i.e., we will solve the abovementioned mixed norm problem formulated as aSOCP given by (32) and compare this to the so-lution of the minimization problem for the objec-tive (28).

Shear and beam bending. This experiment wasdone in order to demonstrate the algorithm’s plau-sibility in terms of physical intuition with regard toforces. These can be hard to describe intuitively inmore complicated situations. We also demonstrateits robustness against different poor initial corre-spondences. In Figure 2 we compared the plane de-formation of a square to a sheared square and thesame sheared square rotated by π/12. Intuitively,one would expect (approximately) the same forcesbut since the linear elastic model used here is notinvariant under rotations we obtain, as expected,slightly different sets of forces. In both cases initialboundary displacements have been given by align-ing the shapes’ centers of mass and taking the near-est neighbours between each boundary point of themesh and the boundary of the target shape. This isfar from being physically reasonable but both op-timizations, the L1,2 and the L2,2, reached similarminima in terms of displacements, while the forcestend to be more sparse in the former case.

Figure 3 shows another physically motivated ex-periment, in which we estimate the deformation ofa bent beam. We assigned two different initial bound-ary correspondences to show the robustness of themethod presented to different choices of initial cor-respondences. The first set of boundary correspond-ences was given as in Figure 2 and the second setwas assigned by hand, i.e., we chose two corre-spondences by hand and interpolated the remainingones on the target boundary. As the reader can ob-serve the optimized quantities obtained with bothinitial correspondences are nearly the same. Notethat in the case of the L2,2-optimization the result-ing forces look slightly different. Such robustnessagainst changes of initial correspondences cannotbe expected when more complicated shapes or shapechanges are involved or if the template is rotation-ally symmetric.

Rotationally Symmetric Template Shapes. As wementioned above, in our linear model the recovered

forces are not invariant under general rotations. Nev-ertheless, linear strain and hence forces are invari-ant under infinitesimal rotations, i.e., deformationfields of the form u(x,y) = α(y,−x)T . Hence, itis interesting to test cases of (almost) rotationallysymmetric templates. Indeed, as Figure 4 shows,the forces and displacements in both cases, mixednorm and L2,2-optimization, tend to introduce a ro-tation during the optimization. The rotation devel-ops faster if the radius is reduced (compare the tem-plate shapes in Figure 4a and 4b). The reason isthat the aforementioned invariance allows growthof spheres, since the infinitesimal rotation fields arenot isometries, with zero cost. This will produce akind of trade-off between zero cost sphere growth,interpreted as infinitesimal rotation, and the desireddeformation. As the reader can observe in Figure 5infinitesimal rotations did not affect the forces. Wealso tested translations of the target shape and thesealso had no effect on the forces. This is due to thethree-dimensional kernel of the stiffness matrix thatcontains translations and infinitesimal rotations thathave one degree of freedom in two dimensions. It ispossible to avoid this additional rotation by choos-ing for example a minimal Euclidean norm prior onthe deformation field. As we shall show in a forth-coming paper, a nonlinear elasticity model will nothave this drawback since it penalizes non-isometricdeformations. The force optimization in this case,however, will be more challenging.

Starfish and Hand Shapes. For these two experi-ments we chose a starfish matching experiment (Fig-ure 6) and an example in which we matched handshapes of two different persons (Figure 7). Theseexamples are less robust to poor initial conditions.Nonetheless, even here taking the nearest neigh-bour of each boundary point resulted in a good match-ing. One could argue that any cyclic permutation ofthe starfish arms would be reasonable. But as weare using a linear model and as we prefer smallforces this is a reasonable choice. The results ofwarping the first shape (and texture) to the targetare shown in both figures.

Jumping Person and More Shapes. We took avideo sequence of a jumping person with preseg-mented silhouettes of that person in each frame,see Figure 8. The first frame served as a referenceframe and we compared it to two other non-consecutiveframes. Here, we only show the sparse L1,2-optimization.The algorithm gave reasonable correspondences un-til a certain frame after which the deformation be-


(a) Template sphere (b) Target sphere (c) Initial boundary displacements

(d) Optimized boundary displacements afterL1,2-optimization

(e) Optimized boundary displacements afterL2,2-optimization

(f) Maximum of the L2-norms of boundaryforces in each iteration

Fig. 5: Growing sphere experiment. Matching of two spheres has low cost due to invariance of the model under infinitesimal rotations. The template andtarget sphere have radii of 135 and 150 pixels. Green dots represent the target shape. Not all displacement vectors are plotted for better visibility. In (f)the maximum of the L2-norm of the boundary forces is shown in every iteration for both optimizations. The forces are close to zero once the templatesphere is matched to the target by an infinitesimal rotation.

(a) Template shape (b) Initial boundary displacements (c) Optimal boundary displace-ments after L1,2-optimization

(d) Optimal boundary displace-ments after L2,2-optimization

(e) Target shape (f) Warped shape (L1,2-deformation)

(g) Optimal boundary forces afterL1,2-optimization

(h) Optimal boundary forces afterL2,2-optimization

Fig. 6: Starfish experiment. Green dots in (b) – (d) represent the target shape and red dots in (g) – (h) the template shape. Force vectors have been scaledand not all displacement vectors are plotted for better visibility.

came too large, see Figure 9. The video was takenfrom [20].

The last experiment, see Figure 10, was done onshapes taken from a database that was used in [34]


in the context of shock graph matching. It showstwo experiments in which we compared two defor-mations of two different shapes, heart shapes and aglass. Also here the algorithm gave reasonable re-sults.

4 Discussion

In this work we propose a physically motivated methodfor obtaining shape correspondences and compari-son measures using a linear elastic model. Our ap-proach is based on the assumption that many de-formations that are observed in images can be de-scribed as deformations of actual physical bodiesand that the external force responsible for a changeof shape is often of a very simple nature. To solvethe underlying Navier-Lamé equation, boundary cor-respondences need to be given. These are usuallyunknown. We find the optimal correspondences bychoosing them such that the external forces are min-imal in some norm. The optimization problem ishighly nonlinear but we show how to relax it by em-ploying a displacement based FEM. This enablesus, after an initial guess of boundary correspon-dence is given, to approximate the full problem bya sequence of convex steps. The initial guess local-izes the search space so that we ultimately find aplausible explanation of the observed shape changeclose to the pre-selected deformation. Each step isoptimized by simultaneously minimizing the for-ces and allowing the estimated correspondences todrift after paying a penalty in terms of a restoringforce. We compare the optimization of two differ-ent norms, an L2-norm preferring small, uniformforces, and an L1,2-norm that seeks sparse and iso-tropic forces. In several experiments we show thatboth optimization procedures perform well and thatthe results are reasonable and intuitive. Both meth-ods can be used to asses the amount of deforma-tion applied to a given shape, but we believe that asparse solution for the forces often better accountsfor real-life scenarios. The deformations found byour model are natural due to the physical model,provided the deformation is not too large.

Our model is limited to small deformations andsmall rotations due to the use of a linear elasticitymodel. A nonlinear model using nonlinear strainand perhaps also nonlinear material laws may re-solve this issue. Also it would make the compari-son forces invariant to rotations. Such a model iscurrently in preparation. Another difficulty was ob-served near corners and high curvature sections ofthe silhouette in which case the projection was some-

times unable to bring a displacement back to a rea-sonable point of the target contour. Furthermore,the algorithm is often sensitive to initial conditions.Indeed, since the method solves a nonlinear prob-lem the algorithm might converge to a local mini-mum, depending on the initial conditions. Finally,the method does not handle topological changes,i.e., material is not allowed to be divided or to de-velop holes.

Acknowledgements This research was supported in partby the U.S.-Israel Binational Science Foundation, Grant No.2010331, by the Israel Science Foundation, Grant No. 764/10,by the Israel Ministry of Science, by the National ScienceFoundation, Grant No. 0915977, and by the Citigroup Foun-dation.

References

1. A. Agudo, B. Calvo, J.M.M. Montiel Finite Elementbased sequential Bayesian Non-Rigid Structure from Mo-tion, Computer Vision and Pattern Recognition, pp. 1522–1529 (2013)

2. Y. Amit A Nonlinear Variational Problem for ImageMatching, SIAM Journal on Scientific Computing Vol. 15,no. 1, 207–224 (1994)

3. Y. Amit A Non-Linear Variational Problem for ImageMatching, SIAM J. Sci. Comput. Vol. 15, no. 1, pp. 1–30(1994)

4. S.S. Antman Nonlinear Problems of Elasticity, Springer-Verlag, Applied Mathematical Sciences Vol. 107 (1995)

5. M.F. Beg, M.I. Miller, A. Trouvé, L. Younes Comput-ing Large Deformation Metric Mappings via GeodesicFlows of Diffeomorphisms, International Journal of Com-puter Vision Vol. 61, no. 2, 139–157 (2005)

6. D. Braess Finite elements. Theory, fast solvers, and ap-plications in solid mechanics, Cambridge University Press(2001)

7. M. Bro-Nielsen Finite Elements Modeling in SurgerySimulation, Proceedings of the IEEE Vol. 86, no. 3, March1998

8. A. M. Bronstein, M. M. Bronstein, A. M. Bruckstein andR. Kimmel Analysis of two-dimensional non-rigid shapes,International Journal of Computer Vision Vol. 78, no. 1,67–88 (2008)

9. P.G. Ciarlet Mathematical Elasticity. Volume I: Three-Dimensional Theory, Series “Studies in Mathematics andits Applications”, North-Holland, Amsterdam (1988)

10. G.E. Christensen, R.D. Rabbitt, M.I. Miller DeformableTemplates Using Large Deformation Kinematics, IEEETransactions on Image Processing Vol. 5, no. 10, 1435–1447 (1996)

11. U. Clarenz Wang, M.Droske, S. Henn, M. Rumpf,K.Witsch Computational Methods for Nonlinear ImageRegistration, Mathematical Models for Registration andApplications to Medical Imaging, Mathematics in Indus-try, Volume 10 (2006)

12. D. Cremers Dynamical statistical shape priors for levelset based tracking, Transactions on Pattern Analysis andMachine Intelligence Vol. 28 (2008)

13. C. Davatzikos, J.L. Prince Brain Image RegistrationBased on Curve Mapping, IEEE Workshop BiomedicalImage Analysis, pp. 245–254 (1994)


(a) Template shape (b) Initial boundary displacements (c) Optimal boundary displace-ments after L1,2-optimization

(d) Optimal boundary displace-ments after L2,2-optimization

(e) Target shape (f) Warped shape (L1,2-deformation)

(g) Optimal boundary forces afterL1,2-optimization

(h) Optimal boundary forces afterL2,2-optimization

Fig. 7: Hand comparison experiment. Green dots in (b) – (d) represent the target shape and red dots in (g) – (h) the template shape. Force vectors havebeen scaled and not all displacement vectors are plotted for better visibility.

14. P. Dupuis, U. Grenander, M.I. Miller Variational Prob-lems on Flows of Diffeomorphisms for Image Matching,Quarterly of Applied Mathematics, LVI:587–600 (1998)

15. L.C. Evans Partial Differential Equations, GraduateStudies in Mathematics Vol. 19, AMS (2010)

16. M. Ferrant, S.K. Warfield, C.R.G. Guttmann, R.V.Mulkern, F.A. Jolesz, R. Kikinis 3D Image Matching Us-ing a Finite Element Based Elastic Deformation Model,Proc. of MICCAI 1999, LNCS 1679 , pp. 202–209 (1999)

17. M. Ferrant, S.K. Warfield, A. Nabavi F.A. Jolesz, R.Kikinis Registration of 3D Intraoperative MR Images ofthe Brain Using a Finite Element Biomechanical Model,Proc. of MICCAI 2000, LNCS 1935, pp. 19–28 (2000)

18. M. Fuchs, B. Jüttler, O. Scherzer, H. Yang Shape Met-rics Based on Elastic Deformations, Journal of Mathemat-ical Imaging and Vision Vol. 35, 86–102 (2009)

19. U. Grenander, M. Miller Pattern Theory: From Repre-sentation to Inference, Oxford University Press, (2007)

20. M. Blank, L. Gorelick, E. Shechtman, M. Irani and R.Basri Actions as Space-Time Shape, International Confer-ence on Computer Vision, pp. 1395–1402 (2005)

21. M. Holden A review of geometric transformations fornonrigid body registration, IEEE Transactions on MedicalImaging Vol. 27, no. 1, pp. 111–128 (2008)

22. S. Ilic, P. Fua Non-Linear Beam Model for TrackingLarge Deformations, International Conference on Com-puter Vision, pp. 1–8 (2007)

23. M. Kass, A. Witkin, D. Terzopoulos Snakes: ActiveContour Models, International Journal of Computer Vi-sion Vol. 1, no. 4, pp. 321–331 (1988)

24. P. Le Tallec Numerical Methods for Nonlinear Three-Dimensional Elasticity, Handbook of Numerical AnalysisVol. III, pp. 465–622 (1994)

25. T. Lin, I. Dinov, A. Toga, L. Vese Nonlinear ElasticRegistration and Sobolev Gradients, Biomedical ImageRegistration, LNCS 6204, pp. 269–280 (2010)

26. A. Malti and R. Hartley and A. Bartoli and Jae-HakKim Non-Linear Beam Model for Tracking Large Defor-mations, Computer Vision and Pattern Recognition, pp.1522–1529 (2013)

27. W. Mio, A. Srivastava, S. Joshi On Shape of Plane Elas-tic Curves, International Journal of Computer Vision Vol.73, no. 3, pp. 307–324 (2007)

28. J. Löfberg YALMIP: A Toolbox for Mod-eling and Optimization in MATLAB, Pro-ceedings of the CACSD Conference (2004),http://users.isy.liu.se/johanl/yalmip

29. W. Peckar, C. Schnörr, K. Rohr, H.S. Stiehl Parameter-Free Elastic Deformation Approach for 2D and 3D Regis-tration Using Prescribed Displacements, J. Math. Imagingand Vision Vol. 10, no. 2, pp. 143–162 (1999)

30. A. Quarteroni, R. Sacco, F. Saleri Numerical Mathemat-ics, Second Edition, Texts in Applied Mathematics no. 37,Springer Verlag (2007)

31. R.D. Rabbit, J.A. Weiss, G.E. Christensen, M.I. Millerand the Applied Mechanics Group (Lawrence LivermoreNational Laboratories, Livermore CA) Mapping of Hyper-elastic Deformable Templates Using the Finite ElementMethod, Proc SPIE 2573, pp. 252–265 (1995)

32. M.H. Sadd, Elasticity. Theory, Applications, and Nu-merics, Elsevier Butterworth-Heinemann (2005)

33. M. Salzmann, R. Uratsun Physically-based MotionModels for 3D Tracking: A Convex Formulation, Inter-national Conference on Computer Vision, pp. 2064–2071(2011)

34. T.B. Sebastian, P.N. Klein and B.B. Kimia Recognitionof Shapes by Editing Shock Graphs, International Confer-ence on Computer Vision, pp. 755–762 (2001)

35. J.F. Sturm Using SeDuMi 1.02, a MATLAB toolbox foroptimization over symmetric cones, Version 1.05 avail-able from http://fewcal.kub.nl/sturm, OptimizationMethods and Software, Vol. 11–12, pp. 625–653 (1999)


(a) First video frame with template shape (b) Video frame #3 with target shape 1 (c) Video frame #8 with target shape 2

(d) Initial boundary displacementsfor target shape 1

(e) Initial boundary displacementsfor target shape 2

(f) Optimal boundary displace-ments after L1,2-optimization fortarget shape 1

(g) Optimal boundary displace-ments after L1,2-optimization fortarget shape 2

(h) Computed deformation fieldusing the optimized boundary dis-placements for target shape 2

(i) Optimal boundary forces afterL1,2-optimization for target shape1

(j) Optimal boundary forces afterL1,2-optimization for target shape2

(k) Template shape warped to tar-get shape 2

Fig. 8: Jumping person experiment. We took the first frame of a video sequence and two more (non-consecutive) frames of a thirty frames sequence. Allframes contained the presegmented deforming shape of the person. We compare the deformations of the first frame to frame #3 (containing target shape1) and to frame #8 (containing target shape 2). Green dots in the figures represent the target shapes and red dots the template shape. Only the sparseL1,2-optimization is shown. Force vectors have been scaled and not all displacement vectors are plotted for better visibility.

36. Y. Wang, L.H. Staib Elastic Model Based Non-RigidRegistration Incorporating Statistical Shape Information,In Proc. of MICCAI 1998, pp. 1162–1173 (1998)

37. L. Younes Computable Elastic Distances BetweenShapes, SIAM J. Appl. Math. Vol. 58, no. 2, pp. 565–586

(1998)

38. L. Younes Optimal Matching Between Shapes via Elas-tic Deformations, Image and Vision Computing Vol. 17,pp. 381–389 (1999)


Fig. 9: Jumping person experiment. This figure shows the (zoomed) deformed meshes of the first video frame (Fig 8a) after L1,2-optimization for the(consecutive) frames #8 – #13 in which the deformation starts becoming too large. The red outline represents the target shape of the relevant frame. Readthe figures row-wise from top left to bottom right.

39. L. Younes Spaces and manifolds of shapes in computervision: An overview, Image and Vision Computing Vol. 30,pp. 389–397 (2012)

Konrad Simon wasborn in Magdeburg,Germany, in 1983 andreceived his Diplomain Mathematics at theUniversity of Leipzig,Germany, in 2009. From2009 until 2010 he wasa research assistant atthe Otto-von-GuerickeUniversity of Magde-burg. Currently he isa Ph.D. Candidate atthe Weizmann Insti-tute of Science, Israel,

at the Department ofComputer Science and

Applied Mathematics. His research interests are in the areaof computer vision, in particular shape deformation andmatching, partial differential equations as well as finiteelements and their applications to computer vision andgraphics.

Sameer Sheorey re-ceived his B.Tech. atthe Indian Institute ofTechnology, Bombay,in 2003. He obtainedhis M.Sc. and his Ph.D.In computer scienceat the University ofMaryland, College Park,under the supervisionof David Jacobs. Aftergraduating he was aResearch AssistantProfessor at ToyotaTechnological Institute,

Chicago. His researchinterests comprise

object recognition and detection, image segmentation,medical image processing as well as harmonic analysis andpartial differential equations with applications to imageprocessing and computer vision. Currently he is working forUtopiaCompression in Los Angeles, CA.

Dr. David W. Jacobs isa professor in the De-partment of ComputerScience at the Univer-sity of Maryland witha joint appointment inthe University’s Institutefor Advanced ComputerStudies (UMIACS). Hereceived the B.A. de-gree from Yale Univer-sity in 1982. From 1982to 1985 he worked forControl Data Corpora-tion on the development

of data base manage-ment systems, and at-

tended graduate school in computer science at New York


(a) Template heart (b) Target heart 1 (c) Initial boundarydisplacements (for targetheart 1)

(d) Target heart 2 (e) Initial boundarydisplacements (for targetheart 2)

(f) Optimal boundarydisplacements afterL1,2-optimization (targetheart 1)

(g) Optimal boundary for-ces after L1,2-optimization(target heart 1)

(h) Optimal boundarydisplacements afterL1,2-optimization (targetheart 2)

(i) Optimal boundary for-ces after L1,2-optimization(target heart 2)

(j) Template glass (k) Target glass 1 (l) Initial boundarydisplacements (for targetglass 1)

(m) Target glass 2 (n) Initial boundarydisplacements (for targetglass 2)

(o) Optimal boundarydisplacements afterL1,2-optimization (targetglass 1)

(p) Optimal boundary for-ces after L1,2-optimization(target glass 1)

(q) Optimal boundarydisplacements afterL1,2-optimization (targetglass 2)

(r) Optimal boundary for-ces after L1,2-optimization(target glass 2)

Fig. 10: Additional experiments. For each experiment we compare of two deformations into two different target shapes. Green dots in the figures representthe target shape and red dots the template shapes. Only the sparse L1,2-optimization is shown. Force vectors have been scaled and not all displacementvectors are plotted for better visibility.

University. From 1985 to 1992 he attended M.I.T., where hereceived M.S. and Ph.D. degrees in computer science. From1992 to 2002 he was a Research Scientist and then a SeniorResearch Scientist at the NEC Research Institute. In 1998he spent a sabbatical at the Royal Institute of Technology(KTH) in Stockholm, and in 2008 spent a sabbatical at theEcole normale supérieure de Cachan. In 2002, he joined theCS department at the University of Maryland.

Dr. Jacobs’ research has focused on human and com-puter vision, especially in the areas of object recognitionand perceptual organization. He has also published articlesin the areas of motion understanding, memory and learning,computer graphics, human computer interaction, and com-putational geometry. He has served as an Associate Editor ofIEEE Transactions on Pattern Analysis and Machine Intelli-gence, and has assisted in the organization of many work-shops and conferences, including serving as Program co-Chair for CVPR 2010. He and his co-authors received hon-

orable mention for the best paper award at CVPR 2000. Healso co-authored a paper that received the best student pa-per award at UIST 2003. In collaboration with researchersat Columbia University and the Smithsonian Institution hecreated Leafsnap, an app that uses computer vision for plantspecies identification. Leafsnap has been downloaded overa million times, and has been used in biodiversity studiesand in many classrooms. Dr. Jacobs and his collaboratorshave been awarded the 2011 Edward O. Wilson BiodiversityTechnology Pioneer Award for the development of Leafs-nap.


Ronen Basri receivedthe BSc degree from TelAviv University in 1985and the PhD degreefrom the WeizmannInstitute of Science in1991. From 1990 to1992 he was a post-doctoral fellow at theMassachusetts Instituteof Technology in theDepartment of Brainand Cognitive Scienceand the Artificial Intelli-gence Laboratory under

the McDonnell-Pew andRothchild programs.

Since then, he has been affiliated with the WeizmannInstitute of Science in the Department of Computer Scienceand Applied Mathematics, where he currently holds theposition of professor and is the incumbent of the Elaineand Bram Goldsmith Chair of Applied Mathematics. In2007, he served as chair for the Department of Com-puter Science and Applied Mathematics. He has furtherheld visiting positions at the NEC Research Institute inPrinceton, Toyota Technological Institute at Chicago, theUniversity of Chicago, and the Janelia Farm Campus ofthe Howard Hughes Medical Institute. His research hasfocused on computer vision, especially in the areas of objectrecognition, shape reconstruction, lighting analysis, andimage segmentation. His work deals with the developmentof algorithms, analysis, and implications to human vision.

a linear elastic force optimization model for shape …...shapes. after proving the existence and...

Documents