f Special Section: Modelling and simulation in biomedicine

Anatomy- and physics-based facial animation for craniofacial surgery

simulations

\

E. Gladilin S. Zachow P. Deuflhard H.-C. Hege

Konrad-Zuse-Zentrum for Informationstechnik Berlin (ZIB), Berlin, Germany

Abstract--A mode~ring approach for the rea/istic simu/ation of facia/ expressions of emotion in craniofacial surgery planning is presented. The method is different from conventional, non-physical techniques for character animation in computer graphics. A consistent physiological mechanism for facial expressions was assumed, which was the effect of contracting muscles on soft tissues. For the numerical solution of the linear elastic boundary values, the finite element method on tetrahedral grids was used. The approach was validated on a geometrical model of a human head derived from tomographic data. Using this model, individual facial expressions of emotion were estimated by the superpositioning of precomputed single muscle actions.

Keywords--Soft tissue modelling, Craniofacial surgery simulation, Muscle-driven facial animation, FEM

Med. Biol. Eng. Comput., 2004, 42, 167-170

J

1 Motivation

HUMAN FACE modelling is a complex and challenging field of study that has links to many other fields, such as computer graphics, animation, medicine and engineering. One of the interesting applications of human face modelling is computer- assisted craniofacial surgery (CAS). Patients with facial deformities or paralysis are severely limited in their ability to communicate with other people, in such cases, the re-establish- ment of aesthetic appearance and normal facial expressions is the primary concern of corrective surgery.

For a realistic estimation of the patient's post-operative appearance, a detailed geometric model of the individual anatomy and an adequate physical model of soft tissue and muscles are ideally needed, in fact, modern medical imaging techniques, such as computer tomography (CT) and magnetic resonance imaging (MRI), are widely used for diagnostic and visualisation purposes and enable the derivation of useful 3D models of individual anatomy. On the other hand, state-of-the- art numerical techniques, such as the finite element method (FEM), provide a modelling platform for the realistic simulation of soft-tissue biomechanics.

in this work, we present an approach to the estimation of individual facial expressions of emotion that takes advantage of both the correct geometric modelling of an individual anatomy and the consistent physical modelling of soft tissue and muscles on the basis of underlying biomechanical laws.

Correspondence should be addressed to Dr E. Gladilin; emaih gladilin@zib.de

Paper received 30 Apr i l 2003 and in final form 17 November 2003

MBEC online number: 20043859

© IFMBE: 2004

Medical & Biological Engineering & Computing 2004, Vol. 42

2 Previous work

Most work in the field of facial animation is based on non- physical techniques, such as 3D shape interpolation, ad hoc surface shape parameterisation or abstract muscle models. Shape interpolation and facial parameterisation are early, simple techniques to control synthetic faces (PARKE and WATERS, 1996). Extended landmark-based methods, e.g. the motion- capturing technique*, are still very popular in computer graphics, as they are efficient and flexible when animating cartoon-based characters. However, these approaches do not consider individual anatomy and tissue biomechanics and thus are not adequate for the prediction of individual facial expressions.

Muscle-based models mimic, at a simple level, the action of the primary facial muscle groups (MAGNENAT-THALMANN e t al., 1988). These models are independent of particular facial geometry and they map directly onto muscle-based facial action coding systems (FACSs) (EKMAN and FRIESEN, 1975). Physically based models attempt to model the face on the basis of the underlying physical properties of soft tissue and muscles (TERZOPOULOS and WATERS, 1990). Most of these models are based on less realistic mass-spring approximations of mechan- ical continuum.

in KOCH et al. (1998), a framework for the estimation of facial expressions based on a linear elastic FE model of soft tissue and an heuristic parameterisation of mimic muscles was proposed. The approach presented in this work follows the same path of consistent, physically based modelling of the human face. To improve the realism of the muscle-driven facial animation, we use highly resolved geometric models of the human head

*Famous 3D, Facial Animation famous3D.com

Solutions, URL: ht tp: / /www.

167

(> 10 6 elements) and a more sophisticated parameterisation of facial musculature based on the virtual fibres approach (GLADILIN et al., 2001).

3 Material and methods

In this Section, we briefly describe our approach to the generation of geometric models of human anatomy and the FE-based modelling of soft tissue.

3.1 Geometric modelling

The starting point for the whole simulation pipeline is the generation of a geometric model of individual anatomy from tomographic data. The whole process of 3D model generation, including input, preprocessing and segmentation o ftomographic images, is carried out on the basis of the multipurpose visualisa- tion and modelling system AMIRA (STALLING et al.). A final model consists of addaptively refined, triangulated surfaces corresponding to the botmdaries between the essential tissue regions (bones, muscles, skin), which are filled up with an tmstrucmred tetrahedral grid.

(PESKIN and MCQUEEN, 1989). For the parameterisation of facial muscles, 'mainline'-based approximations of acting forces are usually applied (LEE et al., 1995).

In accordance with the sliding filament theory, the contractive ability of muscles is effected through the shortening of muscle fibres, which develops a tension along their longitudinal axis (FUNG, 1993). Any consistent biomechanical model of muscle contraction should pay attention to the fact that muscle forces act along the fibre tangents. The major difficulty of the macroscopic modelling of muscles, which is based on the reconstruction of individual anatomy from tomographic datasets, is that micro- scopic structures such as muscle fibres are not sufficiently resolved in the normal tomographic images. In the final geometric model, muscles are merely represented by their shapes. To overcome the problem of missing information, the following heuristic construction is applied.

3.3.1 Muscle forces. Muscles are represented in the general model of deformable soft tissue by the force densityfacting in the direction of fibre tangents

f ( x ) = )~(x)'c(x)x E ~ . . . . ]e C (4)

3.2 General soft-tissue model

In our approach, soft tissue is approximated as a St Venant- Kirchhoff material (CIARLET, 1988) that is basically charac- terised by the linear stress-strain relationship

E ( v ~) o-(e) = ~ \1 _---Z-~v tr(e)l + (1)

where o- denotes the Cauchy stress tensor, e is the strain tensor, E is the Young's modulus, which describes the material stiffness, and v is the Poisson ratio, which describes the material com- pressibility. Typical values for the Yotmg's modulus are EE[2200] kPa (FUNG, 1993). The Poisson ratio for soft tissues lies in the range [0.3, 0.49], but, depending on particular tissue type, age, sex and other factors, it can vary. In this work, we assume v = 0.45. The strain tensor in (1) is generally a non- linear ftmction of the displacement u

~(u) = l (vur + Vu + VurVu) (2)

in the case of small deformations, i.e. max IVul ~ 1, the quadratic term in (2) can be neglected, and the strain tensor can be linearised: e(u) = 1/2 (V u r + V u).

The deformation of a body occupying the domain ~2 is obtained as a solution of the botmdary value problem (BVP), which is given by the equation of static equilibrium between external loadsfand inner forces (stresses)

divcr = - f (3)

and the botmdary conditions. The boundary conditions for the static soft-tissue prediction are given implicitly in the form of prescribed displacements of removed bones. The deformation of facial tissue trader the impact of contracting muscles is obtained by solving (3) with an inhomogeneous right-hand side. To solve the given BVP, the finite element method (FEM) on the tetrahedral grids is used (GLADILIN et al., 2002).

3.3 Muscle modelling

Several different approaches for the modelling of muscles and facial mimics have been proposed in the past. Some of these works outline the general approach for muscle modelling based on the physiological findings and mechanics of fibrous materials

168

where 2 is the magnitude of the force density acting inside the muscle. Both r(x) and 2(x) are generally ftmctions of the co- ordinates x.

3.3.2 Lines o f action. Macroscopically seen, the direction of the muscle fibres enveloped by the muscle capsule somewhat resembles the isolines of vector fields observed in computa- tional fluid or continuum mechanics. Accordingly, the spatial arrangement of muscle fibres can be obtained as a solution of the botmdary value problem with suitably defined boundary conditions, in our previous work (GLADILIN et al., 2001), we used a technique based on the interpolation of fibre tangents as a kind of 'flow field' running inside ~2 ..... l~ from the insertion to the origin of a muscle. The vector field of fibre tangents z is obtained as a solution of the linear elastic BVP, i.e. a kind of test deformation of the subdomain ~2 ..... le corresponding to the muscle. In the present work, we apply a B-spline technique to interpolate the fibre tangents connecting the insertion and the originating areas of muscle, which yields the lines of acting muscle forces inside ~2 ..... l~.

3.3.3 Force magnitude, in the first approximation, the mag- nitude of the muscle forces acting along the lines of action can be assumed to be co-ordinate-independent in the whole domain occupied by a muscle: 2(x) c [0, 2 .. . . ] x c ~2 ..... le. The value 2 .. . . corresponding to the maximum allowed contraction for the particular muscle has to be determined empirically by adaptation to experimental results.

3.3.4 Insertion area. Muscle fibres do not abruptly end in the insertion area. In fact, they branch out to the surrounding soft tissue or other muscles. The spatial arrangement of muscle fibres in the insertion area decisively determines the impact area of muscle, which is essential for the correct modelling of the resulting muscle forces, in our approach, we model the insertion area as a cone-shaped prolongation of muscle sub- domain ~2 ..... le characterised by the effective radius of action R and the angle e. The magnitude of the muscle forces acting in the insertion area differs from those inside the muscle and has also to be determined empirically.

Medical & Biological Engineering & Computing 2004, Vol. 42

4 Experimental results

For the simulation of facial expressions of emotion, a detailed model of the human head, including all groups of mimic muscles, was generated. For this purpose, MRI volunteer data were used; see Fig. 1. A detailed description of 3D model generation can be found in ZACHOW et al. (2002). The final model consisted of surfaces corresponding to the boundaries between the skin, bones and different muscles.

As any facial expression of emotion in the linear approx- imation can be obtained as a superposition of single muscle actions, we first computed all the elementary sequences

needed for the modelling of more complex facial expressions by solving the BVP for each facial muscle. Consequently, canonical facial expressions of emotion were estimated by applying the FACS. Fig. 2a demonstrates the simulation of a facial expression of happiness, estimated by the superposition of single muscle actions of zygomaticus major, zygomaticus minor, risorius and orbicularis otis. Fig. 2b illustrates a facial expression of disgust, simulated as a superposition of single muscle actions of depressor angularis oris left, depressor labii left, mentalis left, levator labii right and orbicularis oris left and right.

a b

Detailed geometric model of human head generated on basis of MRI volunteer data (J?om ZACHOW et al. (2002)). (a) Inner view: surface model of mimic musculature. (b) Outer view: skin surface

Fig. 1

a b Fig. 2 (a) Simulation of Jacial expression of happiness by superpositioning of single muscle actions of zygomaticus majo~ zygomaticus mino~

risorius and orbicularis oris. (b) Simulation of Jacial expression of disgust by superpositioning of single muscle actions of depressor angularis oris left, depressor labii left, mentalis left, levator labii right and orbicularis oris left and right

Medical & Biological Engineering & Computing 2004, Vol. 42 169

5 Conclusions

in this work, an anatomy- and physics-based approach for the realistic estimation of individual facial expressions of emotion is presented. We assume a consistent, physiological mechanism of facial expressions that is the impact of contracting muscles on surrotmding facial tissue. As the microscopic structures, such as muscle fibres, cannot be obtained from tomographic data, an heuristic construction, considering the natural relationship between the muscle shape and its biomechanical functionality, is applied. To compute the tissue deformation under the impact of muscle forces, the finite element method on tetrahedral grids is used. Based on this approach, single muscle actions, as well as the complex FACS-coded facial expressions, are simulated. The results of the feasibility study presented appear more realistic in comparison with conventional, non-physical techniques used in computer graphics for character animation and have also been positively evaluated by our collaborating surgeons.

References

CIARLET, P. G. (1988): 'Mathematical elasticity, vol. I: Three- dimensional elasticity', taken from the series: Studies in Mathe- matics & its Applications (North-Holland, Amsterdam, 1988)

EKMAN, P., and FRIESEN, W. (1975): 'Unmasking the face. A guide to recognizing emotions from facial clues', (Prentice-Hall, Englewood Cliffs, New Jersey, 1975)

FUNG, Y. C. (1993): 'Biomechanics--mechanical properties of living tissues' (Springer, Berlin, 1993)

GLADILIN, E., ZACHOW, S., DEUFLHARD, E, and HEGE, H.-C. (2001): 'Virtual fibers: a robust approach for muscle simulation'. Proc. MEI)ICON'01 Conf. Pula, Croatia, pp. 961-964

GLADILIN, E., ZACHOW, S., DEUFLHARD, E, and HEGE, H. C. (2002): 'Adaptive nonlinear elastic FEM for realistic prediction of soft tissue in craniofacial surgery simulations'. Proc. SPIE Medical Imaging Conference, San Diego, USA

KOCH, R. M., GROSS, M. H., and BOSSHARD, A. A. (1998): 'Emotion editing using finite elements', Proc. Eurographics '98 Conf, 17, pp. 295-302

LEE, Y., TERZOPOULOS, D., and WATERS, K. (1995): Realistic model- ing for facial animation, Proc. SIGGRAPH'95 Conf.

MAGNENAT-THALMANN, N., PRIMEAU, N. E., and THALMANN, D. (1988): 'Abstract muscle actions procedures for human face animation', Visual Comput., 3, pp. 290-297

PARKE, E I., and WATERS, K. (1996): 'Computer facial animation' (A. K. Peters, Wellesley, 1996)

PESFdN, C., and MCQUEEN, I). (1989): 'A three-dimensional compu- tational method for blood flow in the heart. Immersed elastic fibers in a viscous incompressible fluid', J. Comput. Phys., 81, pp. 372-405

STALLING, D., Z{SCKLER, M., and HEGE, H.-C. 'Amira: An advanced 3i) visualization and modeling system', URL: http://axnira.zib.de

TERZOPOULOS, D., and WATERS, K. (1990): 'Physically based facial modeling, analysis and animation', Proc. Visual. Cornput. Anirn., 1, pp. 73-80

ZACHOW, S., GLADILIN, E., HEGE, H. C., and DEUFLHARD, P. (2002): 'Towards patient specific, anatomy based simulation of facial mimics for surgical nerve rehabilitation'. Proc. CARS'02 Conf., Paris, France

Authors" biographies

EVGENY GLADILIN is a research scientist at the Zuse-Institute-Berlin (ZIB), Department of Numerical Analysis and Modelling. He began his undergraduate study of biophysics and medicine at the Russian State Medical University (RSMU) in Moscow in 1989. He resumed his education at the University of Hamburg, receiving a degree (i)ipl. Phys.) in 1999. He wrote his diploma thesis titled "Theoretical and experimental investigation of linear elastic boundary element method for registration of medical images" within the scope of the IMAGINE project (IMage- and Atlas-Guided Interventions in NEurosurgery) carried out in collaboration with Philips Research Hamburg. In 1999, he joined the CAS project at ZIB, where he started his research work in the field of numerical modelling of deformable soft tissue for the computer-assisted surgery planning (CASP). In 2003, he defended his Phi) thesis "Biomechanical modelling of soft tissue and facial expressions for craniofacial surgery planning" at the Free University Berlin.

STEFAN ZACHOW is a research fellow at the Zuse-Institute Berlin (ZIB), I)epaxtment of Scientific Visualization. He received a degree in computer engineering (i)ipl. Ing.) in 1991 from the University of Applied Sciences (TFH) in Berlin, and a degree in computer science (i)ipl. Inform.) in 1999 from the Technical University (TU) in Berlin. Since 1999, Stefan has worked on his Phi) thesis within the field of cranio-maxillofacial surgery planning, (www.zib.de/Visual/projects/ cas). He is a member of IEEE CS/EMBS, Curac, and ACM.

PETER DEUFLHARD founded the Zuse Institute Berlin (ZIB) in 1986 as a research institute in scientific computing. Since then he has been its president and, at the same time, full professor at the Freie Universitfit Berlin, with a chair in scientific computing. He studied physics at the Technical University of Munich, where he obtained his diploma degree in pure physics in 1968. He then turned to mathematics and received his PhD in mathematics in 1972 at the University of Cologne. In 1977, he completed his postdoctoral thesis in mathematics at the University of Munich. From 1978 to 1986 he held the chair of numerical analysis at the University of Heidelberg. He is co-editor of a number of scientific journals. His research interests axe modelling, simulation, inverse problems, and optimization in all kinds of differ- ential equations (ordinary, partial, countable). For further information see his homepage http://www.zib.de/denflhard.

HANS-CHRISTIAN HEGE is head of the Scientific Visualization depaxtment at Zuse Institute Berlin (ZIB) and managing director of the spin-offcompany "Indeed- Visual Concepts". He studied physics at the Freie Universitfit Berlin and worked as a research assistant in quantum field theory and computational physics at the Institut ffir Theorie der Elementaxteilchen from 1984-1989. He is a co-founder of the companies "Mental Images" and "Indeed - Visual Concepts". From 1986 to 1989 he worked as part-time researcher and software developer at Mental Images. He has been with ZIB (initially as scientific consultant) since 1989. In 1991 he established the Scientific Visualization department where research in visualization and data analysis techniques is performed. In his group the 3i) visualization and volume modelling software Amira has been developed. He has edited books on mathematical visualization and acts as member of the editorial boards of the book series "Mathematics + Visualization" and the video/dvd series "videoMath" (Springer Verlag). He has co- organized international summer schools, workshops, the conference series "Visualization and Mathematics", and the film festival Video- Math. He served as program committee member for IEEE Visualiza- tion, Eurographics/IEEE Symposium on Visualization VisSym, and WSCG. Since 2003 he has been Honorary Professor at the German Film School (University for Digital Media Production).

170 Medical & Biological Engineering & Computing 2004, Vol. 42