capturing facial details by space-time shape-from-shading
Post on 14-Feb-2016
25 Views
Preview:
DESCRIPTION
TRANSCRIPT
1
Capturing Facial Details by Space-time Shape-from-shading
Yung-Sheng Lo*, I-Chen Lin*, Wen-Xing Zhang*, Wen-Chih Tai†, Shian-Jun Chiou†
CAIG Lab, Dept. of CS, National Chiao Tung University*
Chunghwa Picture Tubes, LTD.†
2
Outline Introduction Acquisition of facial motion Space-time shape-from-shading Experiment and results Conclusion
3
Introduction Performance-driven method is one of the most
straightforward method for facial animation.
Expression details, e.g. wrinkles, dimples, are key factors but difficult for motion capture.
Original captured images Deformation without details
4
Introduction (cont.) Physics-based simulation and blend shape methods try
to mimic the details.
But, the synthesized details are not the exact expressions.
Muscle-based [E. Sifakis et al. 2005] Blend shape [Z. Deng et al. 2006]
5
Introduction (cont.) Our goal is to enhance existing motion capture tech.
and capture facial details.
With the captured images and directional lighting, our optimization-based shape-from-shading (SFS) can estimate details from shading in video.
With facial detailsCaptured video
6
The proposed method Combines the benefits of motion
capture and shape-from-shading. Motion capture and stereo reconstruction
accurate on feature points and general geometry.
Unreliable corresponding matching at textureless regions
Shape-from-shading don’t need detailed point correspondence for
textureless regions. Estimate relative undulation. Sensitive to noise.
Motion capture + Space-time shape-from-shading.
7
The proposed method
8
Approximate geometry by Mocap Tracking by block matching and stereo reconstruction.
Deforming a generic face model by radial-basis functions (RBF).
9
Facial details by SFS Estimating time-varying details by iterative
approximating shape V and reflectance R.
Input image
T PNum
t
Num
ptptp ISynRVO
1 1
2),(
Space-time constraints Only SFS is not enough. For more reliable detailed motions, we proposed
using space-time constraints.
10
Highly sensitive to noise After applying our spatial constraints
Spatial constraints Mostly continuous surface
High spatial coherence
11
j
tjj
tpCStp zw
zkCS 2)1(
∆ For J Є Neighbor(p)∆ Neighbor(p) denotes the 8-neighbor pixel set ∆ Wj is an adaptive weight ∆ Kcs is the weight for spatial constraints.
Reduce the noise
noiseZtp
Ztj
Ztp
Ztj
noise
ZtpZtj
Still flicker According to biomechanics properties:
A human facial surface should gradually transit between expressions.
Temporal constraints
12T0T1
flicker
T2
]3,3[ i ,)1( 2)( wherez
wzkCT
ipittpCTtp
T2T0
A video image sequence
Space-time shape-from-shading
Finally, our objective function becomes
13
])[(1 1
2tptp
Num
t
Num
ptptp CTCSISynO
T p
spatial constraints
+ temporal constraints
+shading constraints
=Space-time shape-from-shading
Performance issue if applied our optimization to the whole face.
DOF is too large Assigned some small windows.
We preferred areas with more wrinkles and creases.
14D.O.F=N*M (pixels) *i(frames)
N
M
Video image sequence
Fi
… F2F1
F3
Experiment Illumination-controlled (single light source) Two video streams. (HDV, 1280*720 ,30 fps)
We pasted 25 to 30 markers on human face.
15
{C1}
{C2}
Facial detailed results and Comparison
16
Result of synthesis
Generic model: 6078 vertices 6315 polygons
17deformation (RBF)subdivision per-pixel normal mapping
Result of animation
18
Conclusion & Future work We propose capturing detailed motion by
conventional Mocap and advanced shape-from-shading. Doesn’t need additional devices, paint pigments, or
restrict the wrinkle shape.
With spatial and temporal constraints, our optimal shape-from-shading is more reliable. Reflectance parameters are also estimated.
19
Conclusion & Future work In addition to Phong model, we will extend the
concept to other reflectance models. E.g Cook-Torrance BRDF model, BSSRDF, etc)
Currently, SFS is only applied to designated segments. An more efficient SFS for the whole face will make our animation more realistic.
20
Thank for your attention!
21
forehead details
between eyebrows
smile
top related