3D Facial Deformable using Cubic Spline and Thin Plate Spline

C. Sinthanayothin, W. Bholsithi

Image Technology Laboratory, National Electronics and Computer Technology Center (NECTEC) National Science and Technology Development Agency (NSTDA)

112 Thailand Science Park, Phahon yothin Rd, Klong 1, Klong Luang, Pathumthani, 12120, Thailand

Abstract- Orthognathic surgery is a routine procedure for the

correction of facial deformity based on measured values of skeletal anomaly and asymmetry. Until recently, the measurements have been made from two-dimensional (2D) radiographs and any prediction of the patient’s post-operative appearance could only be carried out using two-dimensional software. However, recent advances in virtual reality and three-dimensional (3D) medical imaging technology now make three-dimensional predictions possible. In this paper we present an approach to facial surgery treatment simulation based on the following steps: Initially the patient has an optical surface scan of the facial features. Next was to apply a 3D cubic spline technique to user- or template-specified control points before fitting a mesh on the soft tissue area. The clinician can now move the mesh to the direction that the user requires. Finally a 3D Thin Plate Spine is applied to simulate the 3D facial deformable.

I. INTRODUCTION

Orthognathic surgery is a routine procedure for the correction of facial deformity [1]. Surgical intervention is required when conventional orthodontic treatment does not suffice. Software packages for simulating and predicting the outcome of orthodontic and orthognathic surgical treatment generally operate in 2D, such as Dolphin, Dentofacial Planner, OPAL, Quick Ceph, TIOPS, CephSmile, etc. Various computer techniques have been employed to superimpose a photograph with facial features of a patient over a standardized lateral skull radiograph. Various portions of soft tissue and underlying bone are then moved around to produce the optimal treatment result. The major constrains of current software packages are the postoperative appearance predictions, which have been limited by the lateral sections of the patient’s facial profile. To address these limitations, a true 3D modality for imaging and planning is required. With VISIA® Complexion Analysis, an optical 3D surface scanning system from Surface Imaging Solutions Ltd. [2], we could capture the 3D geometry of the soft tissue, complete with the surface texture of the face itself. In this paper we present a 3D computer graphics approach to 3D facial transformation to simulate an othognathic surgery treatment in advance by applying a 3D cubic spline technique to user- or template-specified control points to fill the missing coordinates while a mesh of control points has been put into the facial soft tissue area to allow facial changes in specified areas by the mesh movement and deformation. Finally a 3D Thin Plate Spine (3D TPS) has been applied to simulate the 3D soft tissue deformation.

II. METHOD

A. Preparing the 3D Facial Data The 3D facial data have been captured by optical surface

imaging which provided a 3D facial representation in VRML format by Optical Surface Imaging as shown in Figure 1. The 3D facial model combined with the triangular meshes can be seen in the expansion of the red rectangular region. To display the models in 3D, we have developed our own software written in C and OpenGL. The positions of 3D facial have been normalized and shifted to the center to facilitate image display.

Figure 1: Soft tissue facial data in VRML

B. Creating the user mesh grid overlay on soft tissue B1. Converting Window Coordinates to 3D Model Coordinates.

Normally, computer programs could recognize the screen

coordinates by mean of manual positing by user a 2D window on the screen, and then applying gluUnproject function from openGL library to convert screen coordinates to 3D model coordinates [3]. Apply the above-mentioned method to locate the mouse position

To calculate the x, y and z coordinate of the corresponding mouse position by applying this method, several variables have been passed into the function, including:

1. Viewport Origin and Extent 2. The Modelview Matrix 3. The Projection Matrix 4. The Windows Screen Coordinate 5. Variables to store the output OpenGL Coordinates The current viewport variable can be obtained by using

glGetIntegerv(GL_VIEWPORT, viewport), then the viewport variable will hold X and Y coordinates of GL_VIEWPORT along with the viewport width and height. Once obtaining viewport data, the Modelview information (that is the matrix

978-1-4244-3388-9/09/$25.00 ©2009 IEEE 668

determining how the vertices of OpenGL primitives are transformed to eye coordinates) can be retrieved by applying function glGetDoublev(GL_MODELVIEW_MATRIX, modelview). After that, the projection matrix transforming vertices in eye coordinates to clip coordinates by using glGetDoublev(GL_PROJECTION_MATRIX, projection), and the windows screen coordinates at the current mouse position along with the depth can be calculated by function glReadPixels(winX, winY, 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &winZ). Finally the windows screen coordinates can be converted to the object coordinates by using gluUnProject( winX, winY, winZ, modelview, projection, viewport, &posX, &posY, &posZ) as shown in Figure 2.

Figure 2: Screen Coordinates vs. Object Coordinates.

B2. Fitting the Control points with 3D Cubic Spine. Initially, user located the soft tissue landmarks by entering a

set of N x N control points (knots). The cubic spline example [4] displays sections of polynomial curves connected at these knots. The polynomials of a given spline all have the same degree in X and Y and fit the control points as a mesh. The spline [5] is obtained by calculating the second derivatives of the interpolating function at the tabulated points xi, with xi = (i, y[i]), [6]. One dimensional cubic spline fitting can be explained as (1) – (6). However, some modifications have been applied on the interpolation of one dimension to three dimensions. With this method, a series of unique cubic polynomials is fitted between each of the data points with the condition that the curve obtained be continuous and appears smooth. This spline consists of weights attached to a flat surface at the points to be connected. A flexible line is then bent across each of these weights, resulting in a smooth curve. The first step of this approach is applying the three dimensional cubic spline where the one dimensional cubic spline line can be expressed as G(x) in (1) and gi is the third degree polynomial defined in (2) [7, 8].

(1)

i = 1, 2, …, n-1.(2) Since the piecewise function G(x) will interpolate all of the

data points, we can conclude that

i = 1, 2, …, n-1 (3) The coefficients ai, bi, ci, and di in (2) are determined from

the first and second derivatives of the n-1 equations of gi(x). The routines must ensure that and are equal at the interior node points for adjacent segments. Substituting a variable Si for the and yi for si(xi) will reduce a number of equations and simplify the tasks.

Given the spacing between successive data points are

(4)

Finally, the variables ai, bi, ci and di in (2), or termed the weights of each n-1 equations, will be

(5)

When substituting these weights in (2), the first and second

derivatives, and including the condition of natural spline where 2nd derivative is equal to zero at the endpoints (S1 = Sn = 0), the system can be determined in matrix equation as follows

(6)

Therefore, (6) is applied to calculate the interpolation values

in all dimensions. For the application in this paper, data points can be any numbers with 10 control points on each segment such that the smooth surface from defined landmarks can be drawn. The result of fitting the mesh with cubic-spline interpolation on the 2D knots can be seen as shown in Figure 3.

Figure 3: Fitting the control points with 2D Cubic Spline.

hi xi xi 1

4 1 0 0 0 01 4 1 0 0 00 1 4 0 0 0

0 0 0 4 1 00 0 0 1 4 10 0 0 0 1 4

S2

S3

S4

S n 3

S n 2

S n 1

6h2

y1 2y2 y3

y2 2y3 y4

y3 2y4 y5

y n 4 2yn 3 yn 2

yn 3 2yn 2 yn 1

yn 2 2yn 1 yn

669

12 1( 1) 1

21 2( 1) 2

( 1)1 ( 1)2 ( 1)

1 2 ( 1)

0 ( ) ... ( ) ( )( ) 0 ... ( ) ( )... ... ... ... ...

( ) ( ) ... 0 ( )( ) ( ) ... ( ) 0

n n

n n

n n n n

n n n n

U r U r U rU r U r U r

KU r U r U r

U r U r U r

−

−

− − −

−

⎡ ⎤⎢ ⎥⎢ ⎥⎢ ⎥=⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦

Then this mesh is projected onto the 3D object model or soft tissue using the window-to-3D model coordinate transformation technique as describe previously. The 3D mesh projection can be seen as in Figure 4.

Figure 4: Fitting the 3D mesh onto the soft tissue.

C. Move Mesh to simulate Soft Tissue Distortion. Soft tissue movement is simulated by letting the middle area

of the mesh move the most while the edges of the mesh stay still based on a Gaussian distribution as given in (7)(8). The points move in the direction of the normal vector (the vector that perpendicular to the surface) as can be seen in Figure 5.

Figure 5: Surface and the normal vector.

(7)

Where v is the coordinate of the mesh (x, y, z) and v’ is the new coordinate of the mesh after moving (x’, y’, z’). Gaussian function G and Normal Vector N can be calculated as follows:

(8)

Where R is the distance from the coordinate v (x, y, z) to the

middle of the mesh while both A and σ are constants. The Normal Vector can be calculated from S x T where S = (a1i + a2j + a3k) and T = (b1i + b2j + b3k). The normal vector N can be calculated as (9). [8] The result after moving the mesh can be seen in Figure 6.

N = (a2b3 – a3b2)i + (a3b1 – a1b3)j + (a1b2 – a2b1)k (9)

Figure 6: Move mesh for soft tissue deformation.

D. Facial distortion using Thin Plate Spline Once the original mesh and the destination mesh have been

created, the soft tissue is deformed to follows the two sets of meshes using the Thin Plate Spline technique. Normally, thin plate splines are used to interpolate two sets of landmark points. The spline maps in the same way as the spline surface into linear (affine transformation) and non-linear (transformation with weighting factors) parts. In our implementation, the thin plate spline based on the paper by Bookstein [9] has been modified from 2 to 3 dimensions. The spline is applied to the two sets of soft tissue data (before and after deformation) by solving (10) to calculate the weighting factors and the affine factors.

(10) Where PT is the transpose of matrix P, and O represents 4x4,

4x3 matrixes of zeros respectively. K is a matrix of U(r) as follows:

(11) Function U(r) can be calculated as

(12)

P is the matrix of coordinates of the original soft tissue.

(13) V is the matrix of coordinates of the soft tissue after

deformation.

(14) W is the matrix of weighting factors for the non-rigid

registration to be calculated.

(15) A is the matrix of affine factors for rigid transformation

(Rotation, Translation and Scaling) to be solved.

(16)

νν +×= NG'

⎟⎟⎠

⎞⎜⎜⎝

⎛ −=2

2exp

σRAG

670

),,( zyxf xi

Use weighting and affine factors to define a function of everywhere in the plane as in (17)-(19).

(17)

(18)

(19)

Thin plate spline (TPS) technique has been tested on the simple box mesh attached with texture. After moving the corners to the different positions, the results can be seen as in Figure 7.

Figure 7: Simple mesh box distortion using TPS technique.

III. RESULT

The experiment results show the soft tissue deformation as shown in figure 8. Figure 8(A) shows the normal soft tissue appearance in different directions. Figure 8(B) shows computational simulation of pre-operative orthodontics/orthognathic treatment which soft tissue is moving backward. And figure 8(C) shows the soft tissue appearance after moving the tissue forward respectively. .

Figure 8: Soft Tissue Deformation by 3D TPS Technique.

IV. DISCUSSIONS & CONCLUSION

The experimental results show that the techniques discussed above and used in developing our program are good tools for orthognathic surgery simulation applications. However, further works needed to be done including comparing the computer simulation to the clinical results to measure the accuracies and discrepancies of the soft tissue deformation. Further refinement of the program will include faster algorithm with higher accuracy in moving the soft tissue with respect to the bones as well as other relating algorithms to ensure that the simulation results will be consistent with the actual surgery.

ACKNOWLEDGMENT

I wish to acknowledge the support and encouragement of Dr.Anthony Raynold at Image Diagnostic Technology Ltd (UK) during the performance of this work. Thanks to Mr. Nicolas Miedzianowski-Sinclair of Surface Imaging International Limited for providing the 3D surface scanning data. Also thanks to all IDT’s members for their friendship and encouragement during my visiting research work at IDT. Finally thanks to NECTEC, NSTDA for the grant supporting.

REFERENCES [1] T.M. Graber, and R.L. Vanarsdall, Orthodontics: Current Principles and

Techniques. Year Book Medical Pub; 2000 [2] Surface Imaging International Ltd., VISIA®, available online at

http://www.surfaceimaging.co.uk/index.php?option=com_content&view=category&layout=blog&id=5&Itemid=6

[3] L. Benstead, Using gluUnProject, available online at http://nehe.gamedev.net/data/articles/article.asp?article=13

[4] F. Buß, Spline Interpolation in Java, available online at http://www.frank-buss.de/spline.html

[5] C. O’Neill, Cubic spline interpolation, available online at http://www.caselab.okstate.edu/ocharle/projects/cubicspline.pdf

[6] W. H. Press et. al., Numerical Recipes in C: The Art of Scientific Computing, Cambridge University Press, 1992, available online at http://www.library.cornell.edu/nr/bookcpdf/c3-3.pdf

[7] S. McKinley and M. Levine, Cubic spline interpolation, Student Projects in Linear Algebra, College of the Redwood, available online at http://online.redwoods.cc.ca.us/instruct/darnold/laproj/Fall98/SkyMeg/Proj.pdf

[8] R. L. Ward, Analytic Geometry Formulas: Three Dimensions, available online at http://mathforum.org/dr.math/faq/formulas/faq.ag3.html

[9] F.L. Bookstein, Principal Warps: Thin Plate Splines and the Decomposition of Deformations, IEEE Trans. on Patt. Anal. And Mach. Intel., Vol. 11, No.6, 567-585, June 1989, available online at http://www-cse.ucsd.edu/classes/sp03/cse252/bookstein.pdf

∑++++==

n

iixxxxxi rUwzayaxaazyxf

14321 )(),,(

∑++++==

n

iiyyyyyi rUwzayaxaazyxf

14321 )(),,(

∑++++==

n

iizzzzzi rUwzayaxaazyxf

14321 )(),,(

671