![Page 1: Facial Motion Retargeting...Facial Motion Retargeting Bindita Chaudhuri Paul G. Allen School of Computer Science & Engineering, UW (Linda Shapiro, Alex Colburn, Barbara Mones, Gary](https://reader034.vdocument.in/reader034/viewer/2022052000/6012189cc29d2c5a9c6dd2c4/html5/thumbnails/1.jpg)
Facial Motion Retargeting
Bindita Chaudhuri
Paul G. Allen School of Computer Science & Engineering, UW
(Linda Shapiro, Alex Colburn, Barbara Mones, Gary Faigin)
Visual Intelligence Group, Microsoft AI&R, Redmond
![Page 2: Facial Motion Retargeting...Facial Motion Retargeting Bindita Chaudhuri Paul G. Allen School of Computer Science & Engineering, UW (Linda Shapiro, Alex Colburn, Barbara Mones, Gary](https://reader034.vdocument.in/reader034/viewer/2022052000/6012189cc29d2c5a9c6dd2c4/html5/thumbnails/2.jpg)
Introduction
• Goal: Predict and transfer facial motion from 2D images to 3D models
expression
and pose
![Page 3: Facial Motion Retargeting...Facial Motion Retargeting Bindita Chaudhuri Paul G. Allen School of Computer Science & Engineering, UW (Linda Shapiro, Alex Colburn, Barbara Mones, Gary](https://reader034.vdocument.in/reader034/viewer/2022052000/6012189cc29d2c5a9c6dd2c4/html5/thumbnails/3.jpg)
Introduction
• Goal: Predict and transfer facial motion from 2D images to 3D models
• Recent applications include:
per-frame
expression
and pose
bounding box
detection
and tracking
![Page 4: Facial Motion Retargeting...Facial Motion Retargeting Bindita Chaudhuri Paul G. Allen School of Computer Science & Engineering, UW (Linda Shapiro, Alex Colburn, Barbara Mones, Gary](https://reader034.vdocument.in/reader034/viewer/2022052000/6012189cc29d2c5a9c6dd2c4/html5/thumbnails/4.jpg)
Methodology
• Blendshape based approach (better generalizability to multiple characters):
• Example based approach (better generalizability to out-of-space expressions):
Network3D Morphable
Model (3DMM)
expression
+ pose
Network
![Page 5: Facial Motion Retargeting...Facial Motion Retargeting Bindita Chaudhuri Paul G. Allen School of Computer Science & Engineering, UW (Linda Shapiro, Alex Colburn, Barbara Mones, Gary](https://reader034.vdocument.in/reader034/viewer/2022052000/6012189cc29d2c5a9c6dd2c4/html5/thumbnails/5.jpg)
Blendshape based Approach
T = V x Bexp x Bid
T = V x Bexp
M = V
identity coefficients wid
expression coefficients wexp
pose (R, t) and projection (f) parameters
user-specific blendshapes
3DMM tensor
3D vertex coordinates
2D landmarks
![Page 6: Facial Motion Retargeting...Facial Motion Retargeting Bindita Chaudhuri Paul G. Allen School of Computer Science & Engineering, UW (Linda Shapiro, Alex Colburn, Barbara Mones, Gary](https://reader034.vdocument.in/reader034/viewer/2022052000/6012189cc29d2c5a9c6dd2c4/html5/thumbnails/6.jpg)
Single Face Network
pose from global features
identity and expression from
local + global features
B. Chaudhuri, N. Vesdapunt, B. Wang, Joint Face Detection and Facial Motion Retargeting
for Multiple Faces , CVPR 2019
![Page 7: Facial Motion Retargeting...Facial Motion Retargeting Bindita Chaudhuri Paul G. Allen School of Computer Science & Engineering, UW (Linda Shapiro, Alex Colburn, Barbara Mones, Gary](https://reader034.vdocument.in/reader034/viewer/2022052000/6012189cc29d2c5a9c6dd2c4/html5/thumbnails/7.jpg)
Multi Face Network
face detection
• two sequential networks; memory inefficient
• runtime increases linearly with number of faces
B. Chaudhuri, N. Vesdapunt, B. Wang, Joint Face Detection and Facial Motion Retargeting
for Multiple Faces , CVPR 2019
![Page 8: Facial Motion Retargeting...Facial Motion Retargeting Bindita Chaudhuri Paul G. Allen School of Computer Science & Engineering, UW (Linda Shapiro, Alex Colburn, Barbara Mones, Gary](https://reader034.vdocument.in/reader034/viewer/2022052000/6012189cc29d2c5a9c6dd2c4/html5/thumbnails/8.jpg)
Multi Face Network
• YOLO loss function; bounding box and 3D face prediction help each other
B. Chaudhuri, N. Vesdapunt, B. Wang, Joint Face Detection and Facial Motion Retargeting
for Multiple Faces , CVPR 2019
![Page 9: Facial Motion Retargeting...Facial Motion Retargeting Bindita Chaudhuri Paul G. Allen School of Computer Science & Engineering, UW (Linda Shapiro, Alex Colburn, Barbara Mones, Gary](https://reader034.vdocument.in/reader034/viewer/2022052000/6012189cc29d2c5a9c6dd2c4/html5/thumbnails/9.jpg)
Network Performance for Test Images
![Page 10: Facial Motion Retargeting...Facial Motion Retargeting Bindita Chaudhuri Paul G. Allen School of Computer Science & Engineering, UW (Linda Shapiro, Alex Colburn, Barbara Mones, Gary](https://reader034.vdocument.in/reader034/viewer/2022052000/6012189cc29d2c5a9c6dd2c4/html5/thumbnails/10.jpg)
Results for Single Face Based Application
Hardware: Google Pixel 2
![Page 11: Facial Motion Retargeting...Facial Motion Retargeting Bindita Chaudhuri Paul G. Allen School of Computer Science & Engineering, UW (Linda Shapiro, Alex Colburn, Barbara Mones, Gary](https://reader034.vdocument.in/reader034/viewer/2022052000/6012189cc29d2c5a9c6dd2c4/html5/thumbnails/11.jpg)
Live Performance Capture for Multiple Faces
![Page 12: Facial Motion Retargeting...Facial Motion Retargeting Bindita Chaudhuri Paul G. Allen School of Computer Science & Engineering, UW (Linda Shapiro, Alex Colburn, Barbara Mones, Gary](https://reader034.vdocument.in/reader034/viewer/2022052000/6012189cc29d2c5a9c6dd2c4/html5/thumbnails/12.jpg)
Example based Approach (semi-supervised)
HCNN
SCNN
Binary
similarity score
E E+ E-
( )Online triplet generation
based on distance metric
fused-CNN
D. Aneja, B. Chaudhuri, A. Colburn, G. Faigin, L. Shapiro, B. Mones, Learning to Generate
3D Stylized Character Expressions from Humans, WACV 2018
Expression feature vectors Geometry feature vectors
![Page 13: Facial Motion Retargeting...Facial Motion Retargeting Bindita Chaudhuri Paul G. Allen School of Computer Science & Engineering, UW (Linda Shapiro, Alex Colburn, Barbara Mones, Gary](https://reader034.vdocument.in/reader034/viewer/2022052000/6012189cc29d2c5a9c6dd2c4/html5/thumbnails/13.jpg)
Example based Approach (semi-supervised)
HCNN
SCNN
Binary
similarity score
3D-CNN
E E+ E-
( )Online triplet generation
based on distance metric
fused-CNNmatching pairs
256 𝖷 256
100 𝖷 1
D. Aneja, B. Chaudhuri, A. Colburn, G. Faigin, L. Shapiro, B. Mones, Learning to Generate
3D Stylized Character Expressions from Humans, WACV 2018
![Page 14: Facial Motion Retargeting...Facial Motion Retargeting Bindita Chaudhuri Paul G. Allen School of Computer Science & Engineering, UW (Linda Shapiro, Alex Colburn, Barbara Mones, Gary](https://reader034.vdocument.in/reader034/viewer/2022052000/6012189cc29d2c5a9c6dd2c4/html5/thumbnails/14.jpg)
Results for Videos
Frame-by-frame transfer; jitter removed by temporal smoothening using Savitzky-Golay filter
![Page 15: Facial Motion Retargeting...Facial Motion Retargeting Bindita Chaudhuri Paul G. Allen School of Computer Science & Engineering, UW (Linda Shapiro, Alex Colburn, Barbara Mones, Gary](https://reader034.vdocument.in/reader034/viewer/2022052000/6012189cc29d2c5a9c6dd2c4/html5/thumbnails/15.jpg)
Example based Approach (unsupervised)
Aim:
• Use single network that directly regresses 3D vertices of character
• Generalize to a broader range of expressions
![Page 16: Facial Motion Retargeting...Facial Motion Retargeting Bindita Chaudhuri Paul G. Allen School of Computer Science & Engineering, UW (Linda Shapiro, Alex Colburn, Barbara Mones, Gary](https://reader034.vdocument.in/reader034/viewer/2022052000/6012189cc29d2c5a9c6dd2c4/html5/thumbnails/16.jpg)
3D
discriminator
landmark loss
Example based Approach (unsupervised)
• Compute facial landmarks:
• Convert 3D model to 2D position map:
• Train CycleGAN:
generator A→B
generator B→A
2D discriminator
![Page 17: Facial Motion Retargeting...Facial Motion Retargeting Bindita Chaudhuri Paul G. Allen School of Computer Science & Engineering, UW (Linda Shapiro, Alex Colburn, Barbara Mones, Gary](https://reader034.vdocument.in/reader034/viewer/2022052000/6012189cc29d2c5a9c6dd2c4/html5/thumbnails/17.jpg)
Results
Input Blendshape basedSemi-supervised
Example based
Unsupervised
Example based
![Page 18: Facial Motion Retargeting...Facial Motion Retargeting Bindita Chaudhuri Paul G. Allen School of Computer Science & Engineering, UW (Linda Shapiro, Alex Colburn, Barbara Mones, Gary](https://reader034.vdocument.in/reader034/viewer/2022052000/6012189cc29d2c5a9c6dd2c4/html5/thumbnails/18.jpg)
Results
Input Blendshape basedSemi-supervised
Example based
Unsupervised
Example based
![Page 19: Facial Motion Retargeting...Facial Motion Retargeting Bindita Chaudhuri Paul G. Allen School of Computer Science & Engineering, UW (Linda Shapiro, Alex Colburn, Barbara Mones, Gary](https://reader034.vdocument.in/reader034/viewer/2022052000/6012189cc29d2c5a9c6dd2c4/html5/thumbnails/19.jpg)