Final Class: Range Data registrationCISC4/689

Credits: Tel-Aviv University

The ProblemAlign two partially- overlapping meshes given initial guess for relative transform

Data TypesPoint setsLine segment sets (polylines)Implicit curves : f(x,y,z) = 0Parametric curves : (x(u),y(u),z(u))Triangle sets (meshes)Implicit surfaces : s(x,y,z) = 0Parametric surfaces (x(u,v),y(u,v),z(u,v)))

MotivationShape inspectionMotion estimationAppearance analysisTexture MappingLabeling (atlas registration)

MotivationRange images registration

Range Scanners

Aligning 3D Data

Iterative Closest Point AlgorithmAlso called ICP algorithm proposed in 1992.Many variants have come into existence after the original algorithm proposed by Besl and Mackay.

Corresponding Point Set AlignmentLet M be a model point set. Let S be a scene point set.

We assume :NM = NS.Each point Si correspond to Mi .

Corresponding Point Set AlignmentThe objective function :

The alignment is :

Aligning 3D DataIf correct correspondences are known, can find correct relative rotation/translation

Aligning 3D DataHow to find correspondences: User input? Feature detection? Signatures?Alternative: assume closest points correspond

Aligning 3D DataHow to find correspondences: User input? Feature detection? Signatures?Alternative: assume closest points correspond

Aligning 3D DataConverges if starting position close enough

Closest PointGiven 2 points r1 and r2 , the Euclidean distance is:

Given a point r1 and set of points A , the Euclidean distance is:

Finding MatchesThe scene shape S is aligned to be in the best alignment with the model shape M.The distance of each point s of the scene from the model is :

Finding Matches Finding each match is performed in O(NM) worst case.Given correspondence, Y we can calculate alignment

S is updated to be :

The AlgorithmInit the error to Calculate correspondenceCalculate alignmentApply alignmentUpdate errorIf error > thresholdY = CP(M,S),e(rot,trans,d)S`= rot(S)+transd` = d

Convergence TheoremCorrespondence error :

Alignment error:

ICP Variants

Variants on the following stages of ICP have been proposed:Selecting sample points (from one or both meshes)Matching to points in the other meshWeighting the correspondencesRejecting certain (outlier) point pairsAssigning an error metric to the current transformMinimizing the error metric w.r.t. transformation

ICP VariantsSelecting sample points (from one or both meshes).Matching to points in the other mesh.Weighting the correspondences.Rejecting certain (outlier) point pairs.Assigning an error metric to the current transform.Minimizing the error metric w.r.t. transformation.

ICP VariantsSelecting sample points (from one or both meshes).Matching to points in the other mesh using invariants.Weighting the correspondences.Rejecting certain (outlier) point pairs.Assigning an error metric to the current transform.Minimizing the error metric w.r.t. transformation.

ICP VariantsSelecting sample points (from one or both meshes).Matching to points in the other mesh.Weighting the correspondences.Rejecting certain (outlier) point pairs.Assigning an error metric to the current transform.Minimizing the error metric w.r.t. transformation.

ICP VariantsSelecting sample points (from one or both meshes).Matching to points in the other mesh.Weighting the correspondences.Rejecting certain (outlier) point pairs.Assigning an error metric to the current transform.Minimizing the error metric w.r.t. transformation.

Rejecting PairsInconsistent Pairs

ICP VariantsSelecting sample points (from one or both meshes).Matching to points in the other mesh.Weighting the correspondences.Rejecting certain (outlier) point pairs.Assigning an error metric to the current transform.Minimizing the error metric w.r.t. transformation.

Error metric and minimizationSum of squared distances between corresponding points .There exist closed form solutions for rigid body transformation :SVDQuaternionsOrthonoraml matricesDual quaternions.

3D Surface-to-surface Motion AnalysisDirect Shape-based method:J. S. Duncan, et al. 1991J. Feldmar, et al. 1996Y. Wang, et al. 2000D. Meier, et al. 2002

Nonrigid Shape-based method: Nonrigid shape relationship between the before-motion and after-motion surfaces is described by the undergoing nonrigid motion. C. Kambhamettu, et al. CVPR 1992 C. Kambhamettu, et al. CVGIP:IU 1994 C. Kambhamettu, et al. IVC 2003 P. Laskov, et al. PAMI 2003

3D Surface-to-surface Motion AnalysisPrevious Nonrigid Shape-based MethodsA local coordinate system is constructed at each point of interestDefined motion has no explicit physical meaningEach point of interest is looking for its corresponding point independentlyMotion consistency can not be guaranteed

New Approach of Nonrigid Shape-based Method Nonrigid motion is modeled with a single spline-based motion field (GRBF) over the whole 3D surface. Nonrigid shape relationship is still described in the local coordinate system constructed at each point of interest

BackgroundAt each point in the local coordinate system

BackgroundAssume orthogonal parameterization (F=0)

From World to Local CoordinatePrincipal local coordinate system

Motion transformation:

Problem StatementMotion estimation: recover the GRBF motion What we want to know:

What we already know:

What we should do:

Paper bending

ExperimentsCorrespondences between frame 1 and Frame 6 is first estimated. Intermediate faces are reconstructed using linear interpolation, based on obtained correspondences between frame 1 and 6

Tongue Motion AnalysisSagittal+CoronalSagittal+Axial

Tongue Motion AnalysisA tagged MRI image. Tags are used for validation only.

Tongue Motion AnalysisCorrespondence errors for 11 tongue deformations

Evaluation of Structure and Nonrigid Motion(evaluation of both structure and motion)TorsoBullfightMore..

Face Motion ApplicationFacial Animation Parameters (FAPs)Facial Definition Parameters (FDPs)

Drive a face:movieBuild and drive an Avatar:DemoFace Anatomy Motion: Movie1 (US)Movie2 (MRI)

REVIEWwww.cis.udel.edu/~chandra/courses.htm

Exam will only take 1.30 min., though you are given 2 hours.

Thank you!Please complete EvaluationsCome to my office and..Take your mid-term2Show project progressIf you have to run to a class: show me quick progress, meet again.5/20 is deadline for the project unless there is a great reason for extension for few more days.I prefer to give you an extra day or two rather than evaluating half finished product.Its easy to come show me the demo for project evaluation. However, project html reports will be gladly accepted.

Our general framework can also be used for 3D surface-to-surface motion analysis. In 3D, the shape-based motion analysis method can be further classified as direct shape-based method and nonrigid shape-based method.

The DSM directly compare shapes, such as curvatures and normals, of the before-motion and after-motion surfaces

Different with DSM, the NSM focuses on the nonrigid relationships between Since the nonrigid shape relationship is described in a local coordiante system, previous NSM works in a local coordinate systme and each point is looking for its corresponding point independently.

With our general motion analysis framework, the .In the local coordinate system, r is a point on the before motion surface , r is its corresponding point on the after-motion surface and the displacement between them is the small s.E,F ,G are the coefficients of the first fundamental form. D is the discriminant

The surface unit normal can be defined like this

We can also define the modulus of dilation and the motion divergence.

By omitting the higher orders of motion, modulus of dilation equals motion divergence

Modulus of dilation: unit change to the surface areaDivergence: rate at which "density" exits a given region of space

By assuming orthogonal parameterization which indicates F equals zero, the before motion normal and the after normal has this relationship. This is the nonrigid shape relationship used in our method.

previous nonrigid shape based methods use this nonrigid shape relationship, they have one additional small deformation assumptionThe GRBF motion is defined in the world coordinate system but the nonrigid shape relationship is defined in a local coordinate system. We have to transform the GRBF motion from the world to the local coordinate system.

Usually the local coordinate system is the principal local coordiante system which is defined by two principal direction r1 , r2 and the surface normal n at each point of interest

r1 r2 and n define a rotation Matrix which can transform the GRBF motion from the world to the local coordinate

Now we want to recover the GRBF motion vector, we already know the nonrigid shape relationship so we can define a least square Error and recovery the motion vector by minimizing this error plus some additional constraints such as the distanceBetween corresponding points is smallOur method is also tested with real motion. Real motion is represented by motion of some feature points and reconstructed from stereo images. It is then mapped to a before-motion surface to create the after-motion surface.

Results for five small-to-large paper bending deformations are shown shown here. In all stiutations our method outperforms small deformation method

Real face motion represented by motion of some feature points are also tested.

Correspondence errors for five s-2-l smile deformations, and five s-2-l open-mouth deformations are shown in these two figures. Similar results as the paper bending experiments are observedwe also tested our method on cyberware face data. Correspondence between frame 1 and frame 6 is recovered by our method.Intermediate faces are obtained with linear interpolation of frame 1 and frame 6 according to the recovered correspondence.

These interpolated faces display a smooth transition from the neutral face to the smile face. This smooth transition indicates that the undergoing deformation has been successfully captured by our method.

To test our method on 3D tongue surface, we have to reconstruct the tongue surface from images. We have mri tonue images in sagittal, coronal and axial planes. Left is a sagitall image and a coronal image, right is the same sagital image with an axialImage.

Tongue contours extracted from multiple sagittal, coronal and axial are used to reconstruct the 3D tongue surface. 12 tongue surfaces during a speech eeoo are reconstructedPoint correspondence between any two consecutive tongue surfaces is recovered by our method and compared with ground truth. The ground truth is obtained from the tagged MRI image of the same speech. Tags are created image features that move with the tongue. Correspndence errors for the eleven tongue deformations during speech eeoo are shown in this figure. Our method is compared with wangs method and still has better results.