realistic modeling of animatable faces in mpeg-4 marco fratarcangeli and marco schaerf university of...

Realistic Modeling of Animatable Faces in MPEG-4

Marco Fratarcangeli and Marco SchaerfUniversity of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

2

Outline

Definition of the Problem Introducing MPEG-4 FBA Facial Structure Overview Numerical Simulation Results and Conclusions

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

3

Definition of the Problem

MPEG-4 FBA standard mainly focuses on networking capabilities for sending/ receiving animations.

It allows employing model-based coding of human-like faces in a systematic manner.

It does not provide any solution on how to achieve the movements of the face/body.

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

4

Definition of the Problem

We use some already developed techniques to build a face model conforming to the anatomical structure of the human head.

Our contribution is in devising a fast and automatic building process using MPEG-4.

The resulting virtual talking head is used to perform general facial animation coming from an encoded MPEG-4 FAB stream.

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

5

Definition of the Problem

MPEG-4 FBA Encoded Stream

MPEG-4 Player

Animatable Face Model

Modeling Process

3D Mesh + Info MPEG-4 FBA

Output

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

6

Outline

Definition of the Problem Introducing MPEG-4 FBA Facial Structure Overview Numerical Simulation Results and Conclusions

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

7

Introducing MPEG-4 FBA

MPEG-4Standard dealing with efficient

transmission of multimedia on the net

FBAEfficient coding of shape and

animation of human-like faces and bodies

FDPControl points used to define

the shape of a proprietary face model

FAPSet of animation parameters

suitable to define facial movements

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

8

MPEG-4 Facial Definition Pts

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

9

MPEG-4 Facial Animation Pts

68 low-level FAPs used to express basic actions that the face can perform

20 high-level FAPs used to express more complex movements like expressions and visemes

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

10

Outline

Definition of the Problem Introducing MPEG-4 FBA Facial Structure Overview Numerical Simulation Results and Conclusions

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

11

Facial Structure OverviewFrom a VRML model

to a mass-spring structure

Skin structure{Masses [or nodes] from VRML vertices

Springs from VRML topologyActing on the skin model through muscle actions, movement is achieved

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

12

Skin is composed by 3 layers

Skin – Prism Element

subdivided in little triangular prisms

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

13

Skin – Prism Element

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

14

From FDPs

Muscles – Mapping

muscles are mapped automatically

Green points are where the muscle is attached to the skull

Red points are where the muscle is inserted in the skin

Two kinds of muscle: linear and sphincter

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

15

Muscle Mapping - Examples

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

16

Outline

Definition of the Problem Introducing MPEG-4 FBA Facial Structure Overview Numerical Simulation Results and Conclusions

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

17

Semi-Implicit Integrations

A ordinary differential equation system governs the movement of the spring-mass network.

Explicit methods are simple but unstable if applied to stiff systems [e.g. Euler steps].

Implicit methods are precise and stable but computationally expensive [e.g. Backward Euler].

Semi-Implicit Integration = implicit integration approximated with some explicit steps.

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

18

Skin nodes can be displaced by muscular force and by the elastic wave coming from the muscle movement.

Adaptive Displacement

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

19

Outline

Definition of the Problem Introducing MPEG-4 FBA Facial Structure Overview Numerical Simulation Results and Conclusions

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

20

Achieving Facial Animation

MPEG-4 FBAdata stream

MPEG-4 player(PC, PDA, mobile phones)

Animatable Face Model

anAtomic

Audio

FDPs

Static Model

Output

One action set for each MPEG-4

FAP

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

21

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

22

Conclusions - Pros

This method generates realistic facial animation encoded in a MPEG-4 FAP stream of general face meshes using only a standard PC.

It is automatic and fast. It can be used to animate talking

heads in MPEG-4 compliant applications.

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

23

Conclusions - Cons

Animation becomes inaccurate if the polygonal resolution of the face mesh is not enough fine (too few springs!) and/or the mesh topology is not regular (e.g. not simmetric).

Solution: Motion Cloning

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

24

Main References Y. Lee, D. Terzopoulos and K. Waters. ”Constructing

Physics-Based Facial Models of Individuals”

K. Waters. ”A Muscle Model for Animating Three-

Dimensional Facial Expressions”.

D. Baraff and A. Witkin. ”Large Steps in Cloth

Simulation”.

I.S. Pandzic and R. Forchheimer, editors. ”MPEG-4 Facial

Animation – The Standard, Implementation and

Applications”.

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

25

Acknowledgements

Image Coding Group, Department of Electrical Engineering, Linköping University, Sweden

Visage Technologies AB [www.visagetechnologies.com]

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

26

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

27

Skull

iiinet nnF )(

skullF 0

0 iinet nFwhe

notherwise

inetF is the net force on node i

inis the nodal normal of node i

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

28

Skull - Example

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

29

Skull – Lower Jaw

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

30

Muscle Mapping - Examples

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

31

Linear Muscles

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

32

Linear Muscles

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

33

Sphincter Muscles

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

34

Semi-Implicit Integration

txFtxF,KtxFdt

tdxD

dt

txdM musskullela

2

2

Starting from the differential system governing the net:

Equations for one mass mi:

tvdt

tdxi

i

tDvKtxFtxFdt

tdvm iielaimus

ii ,

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

35

Semi-Implicit IntegrationImplicit method (Lagrangian), it is obtained:

i

ni

niela

nimusn

ini m

DvKxFxFtvv

1111 ,

11 ni

ni

ni tvxx Non linear functions

very expensive extract xin+1 for each time step

xin+1 is approximated using an Euler step (explicit method):

ni

ni

ni tvxx 1

Trade-off between stability of the implicit and speed of the explicit

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

36

Adaptive Displacement

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

37