vision-based control of 3d facial animation jin-xiang chai jing xiao jessica hodgins carnegie mellon...
Post on 21-Dec-2015
213 views
TRANSCRIPT
Vision-based Control of 3D Facial Animation
Jin-xiang Chai
Jing Xiao
Jessica Hodgins
Carnegie Mellon University
Our Goal
Interactive avatar control• Designing a rich set of realistic facial actions for a
virtual character
• Providing intuitive and interactive control over these actions
+ High quality- Expensive
- Intrusive
- Noisy
- Low resolution
+ Inexpensive
+ Non-intrusive
Control Interface Quality
Control Interface vs. Quality
Vision-based animation
Online motion capture
Our Idea
Vision-based interface
Motion capture database
Interactive avatar control
+
Motion capture• Making faces [Guenter et al. 98]
• Expression Cloning [Noh and Neumann 01]
Vision-based tracking for direct animation• Physical markers [Williams 90]
• Edges [Terzopoulos and Waters 93, Lanitis et al. 97]
• Dense optical flow with 3D models [Essa et al. 96, Pighin et al. 99, DeCarlo et al. 00]
• Data-driven feature tracking [Gokturk et al. 01]
Vision-based animation with blendshape• Hand-drawn expression [Buck et al. 00]
• 3D model avatar model [FaceStation]
Related Work
VideoAnalysis
Avatar animation
PreprocessedMotion Capture
Data
ExpressionControl andAnimation
ExpressionRetargeting
Act out expressions
System Overview
Video Analysis
VideoAnalysis
Vision-based tracking
• 3D Head Poses [Xiao et al. 2002]
• 2D facial features
Video Analysis
Expression Control Parameters
Extracting 15 expression control parameters from 2D tracking points
Distance between two feature points
Distance between a point and a line
Orientation and center of the mouth
Expression control signalt
Avatar animation
PreprocessedMotion Capture
Data
ExpressionControl andAnimation
ExpressionRetargeting
Act out expressions
VideoAnalysis
System Overview
Motion Capture Data Preprocessing
3D PosesExpression separation
Expression control
parameter extraction
70000 frames (10 minutes) including:
• 6 basic facial expressions
• typical everyday facial expressions
• speech data
Avatar animation
ExpressionControl andAnimation
ExpressionRetargeting
Act out expression
VideoAnalysis Preprocessed
Motion CaptureData
System Overview
Expression Control
2D tracking data
Vision-based interface
Motion capture database
19*2 dofs
Expression control parameters
Expression control parameters
15 dofs 15 dofs
76*3 dofs3D motion data
Challenges
• Visual expression control signals are very noisy
• One to many mapping from expression control parameter space to 3D motion space
Temporal coherence
Control parameter space 3D motion space
15 dofs 76*3 dofs
Data-driven Dynamic Filtering
Nearest Neighbor Search
Noisy control signal
Online PCA
K=120 closest examples
PreprocessedMotion Capture
Data
W = 0.33s
7 largest Eigen-curves (99.5 % energy)
Filtered control signal
Filter by eigen-curves
Expression Mapping
Nearest Neighbor Search
From expression control parameter space to 3D motion data space
d1
d2
dK
...
w(d2)
w(dK)
w(d1)
...
Filtered control signalSynthesized motion
Avatar animation
Act out expression
VideoAnalysis Preprocessed
Motion CaptureData
ExpressionControl andAnimation
System Overview
ExpressionRetargeting
Expression Retarget
Synthesized expression Avatar expression
Expression Retargetxs xt
• Learn the surface mapping function using Radial Basis Functions such that xt=f(xs)
• Transfer the motion vector by local Jacobian matrix Jf(xs) by xt=Jf(xs) xs
xs xt
?
Run time computational cost depends on the number of vertices
Precompute Deformation Basis
…
T0 T1T2 T3 T4 T5
…
S0 S1 S2 S3 S4 S5
PreprocessedMotion Capture
Data
PCA
Precompute deformation basis
25 source motion bases –99.5% energy
25 precomputed avatar motion bases
Target Motion Synthesis
…
…
Synthesized expression
Avatar expression
0,…. NS0
T0
S1
T1T2
S2
iSi
iTi
Run time computational cost is O(N)
N is the number of bases
S3
T3 TN
SN
Avatar animation
Act out expression
VideoAnalysis Preprocessed
Motion CaptureData
ExpressionControl andAnimation
System Overview
Expression Retargeting
Results
Conclusions
Developed a performance-based facial animation system for interactive expression control
• Tracking real-time facial movements in video • Preprocessing the motion capture database • Transforming low-quality 2D visual control signal to high quality 3D facial expression• An efficient online expression retarget
• Formal user study on the quality of the synthesized motion
• Controlling and animating 3D photorealistic facial expression
• Size of database
Future Work