dense 3d modelling and monocular reconstruction of...
TRANSCRIPT
![Page 1: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/1.jpg)
Dense 3D Modelling and MonocularReconstruction of Deformable Objects
Anastasios (Tassos) RoussosLecturer in Computer Science, University of Exeter
Research Associate, Imperial College London
Overview of Recent & Ongoing ResearchMarch 2017
1
![Page 2: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/2.jpg)
Presentation Outline
1 Introduction
2 Model-free Dense 3D Reconstruction from Videos
3 Model-based Dense 3D Reconstruction from Videos
4 Craniofacial Surgery Applications
5 Conclusions
2
![Page 3: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/3.jpg)
3D Computer Vision
Inference of 3D information from 2D images
Wide variety of real-world applications
Match Moving Kinect
Photo Tourism, Photosynth Dense 3D models of buildings
3
![Page 4: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/4.jpg)
3D Computer Vision
Inference of 3D information from 2D images
Wide variety of real-world applications
Faceshift software Avatar motion capture
MPI & 3dMD 4D scanner The Digital Emily Project
3
![Page 5: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/5.jpg)
3D Computer Vision
(Thies et al., Face2Face: Real-time Face Capture and Reenactment of RGB Videos, CVPR’16).
4
![Page 6: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/6.jpg)
3D Computer Vision: Limitations
Assumption of rigidity:
input 2D images: different viewpoints of exactly the same3D scene
Multi-camera systems:
equivalent to single camera capturing a rigid sceneexpensive acquisition setups
Active sensors:
limitations in the acquisition conditions
Specific class of objects:
unrealistic shape model priors
5
![Page 7: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/7.jpg)
3D Computer Vision: Overcoming the Limitations
Core questions:How can we make detailed 3D reconstruction work in any
real-world scene?
How can we minimise the acquisition requirements?
Vision: robust and fast systems that:work under almost any conditionuse practical, low cost acquisition devices
6
![Page 8: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/8.jpg)
3D Computer Vision: Overcoming the Limitations
Core questions:How can we make detailed 3D reconstruction work in any
real-world scene?
How can we minimise the acquisition requirements?
Vision: robust and fast systems that:work under almost any conditionuse practical, low cost acquisition devices
Applications:
6
![Page 9: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/9.jpg)
Dense 3D Reconstruction from Monocular Sequences
Input: monocular sequence of non-rigid scene
���
Goal: estimation of 3D location of every pixel at every frame
Approaches:
model-free: no prior knowledge about the scene object(s)
model-based: object-specific prior shape models7
![Page 10: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/10.jpg)
Dense 3D Reconstruction from Monocular Sequences
Input: monocular sequence of non-rigid scene
���
Goal: estimation of 3D location of every pixel at every frame
Approaches:
model-free: no prior knowledge about the scene object(s)
model-based: object-specific prior shape models7
![Page 11: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/11.jpg)
Dense 3D Reconstruction from Monocular Sequences
Input: monocular sequence of non-rigid scene
���
Goal: estimation of 3D location of every pixel at every frame
Approaches:
model-free: no prior knowledge about the scene object(s)
model-based: object-specific prior shape models7
![Page 12: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/12.jpg)
Dense 3D Reconstruction from Monocular Sequences
Input: monocular sequence of non-rigid scene
���
Goal: estimation of 3D location of every pixel at every frame
Approaches:
model-free: no prior knowledge about the scene object(s)
model-based: object-specific prior shape models7
![Page 13: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/13.jpg)
Dense 3D Reconstruction from Monocular Sequences
Input: monocular sequence of non-rigid scene
���
Goal: estimation of 3D location of every pixel at every frame
Approaches:
model-free: no prior knowledge about the scene object(s)
model-based: object-specific prior shape models7
![Page 14: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/14.jpg)
Dense 3D Reconstruction from Monocular Sequences
Input: monocular sequence of non-rigid scene
���
Goal: estimation of 3D location of every pixel at every frame
Approaches:
model-free: no prior knowledge about the scene object(s)
model-based: object-specific prior shape models7
![Page 15: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/15.jpg)
Presentation Outline
1 Introduction
2 Model-free Dense 3D Reconstruction from Videos
3 Model-based Dense 3D Reconstruction from Videos
4 Craniofacial Surgery Applications
5 Conclusions
8
![Page 16: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/16.jpg)
Model-free: Non-rigid Structure from Motion (NRSfM)
Leap from sparse to dense NRSfM
Sparse
(Dai,Li,He, CVPR’12)
Dense
(Garg,Roussos,Agapito, CVPR’13)
9
![Page 17: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/17.jpg)
Our Pipeline
���
(Roussos, Russell, Garg, Agapito, IEEE ISMAR 2012)(Garg, Roussos, Agapito, International Journal of Computer Vision 2013)(Garg, Roussos, Agapito, IEEE CVPR 2013)
10
![Page 18: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/18.jpg)
Our Pipeline
���
���
Step 1: Dense
Video Registration ���
(Roussos, Russell, Garg, Agapito, IEEE ISMAR 2012)(Garg, Roussos, Agapito, International Journal of Computer Vision 2013)(Garg, Roussos, Agapito, IEEE CVPR 2013)
10
![Page 19: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/19.jpg)
Our Pipeline
Step 2: Dense
Shape Inference���
���
���
Step 1: Dense
Video Registration ���
(Roussos, Russell, Garg, Agapito, IEEE ISMAR 2012)(Garg, Roussos, Agapito, International Journal of Computer Vision 2013)(Garg, Roussos, Agapito, IEEE CVPR 2013)
10
![Page 20: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/20.jpg)
Our Pipeline
Step 2: Dense
Shape Inference���
���
���
Step 1: Dense
Video Registration ���
Priors �
(Roussos, Russell, Garg, Agapito, IEEE ISMAR 2012)(Garg, Roussos, Agapito, International Journal of Computer Vision 2013)(Garg, Roussos, Agapito, IEEE CVPR 2013)
10
![Page 21: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/21.jpg)
Our Pipeline
Step 2: Dense
Shape Inference���
���
���
Step 1: Dense
Video Registration ���
Low rank.
Spatial smoothness.�
(Roussos, Russell, Garg, Agapito, IEEE ISMAR 2012)(Garg, Roussos, Agapito, International Journal of Computer Vision 2013)(Garg, Roussos, Agapito, IEEE CVPR 2013)
10
![Page 22: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/22.jpg)
Multi-frame Subspace Flow (MFSF)
Robust Subspace Constraints for Video Registration
The code is now publicly available at:https://bitbucket.org/troussos/mfsf
11
![Page 23: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/23.jpg)
Model-Free Dense 3D Reconstruction from Videos
: input dense 2D tracks, computed with (Garg,Roussos,Agapito, IJCV’13)
, : the unknown rotations and shapes per frame
=
(Garg, Roussos, Agapito, IEEE CVPR 2013)
12
![Page 24: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/24.jpg)
Model-Free Dense 3D Reconstruction from Videos
(Garg, Roussos, Agapito, IEEE CVPR 2013)
12
![Page 25: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/25.jpg)
Model-Free Dense 3D Reconstruction from Videos
(Garg, Roussos, Agapito, IEEE CVPR 2013)
12
![Page 26: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/26.jpg)
Model-Free Dense 3D Reconstruction from Videos
(Garg, Roussos, Agapito, IEEE CVPR 2013)
12
![Page 27: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/27.jpg)
Model-Free Dense 3D Reconstruction from Videos
(Garg, Roussos, Agapito, IEEE CVPR 2013)
12
![Page 28: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/28.jpg)
Energy Minimisation Approach to NRSfM
Formulation of a single unified energy to estimate:
Orthographic projection matrices
3D shapes for all the frames
E(❘ , ❙
)= λ Edata
(❘,❙
)+ Ereg
(❙
)+ τ Etrace
(❙
)
reprojection error over all frames
spatial smoothness prior on 3D shapes
low rank prior on 3D shapes
13
![Page 29: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/29.jpg)
Reprojection ErrorE(
,)
= λEdata
(
,)
+ Ereg
( )
+ τEtrace
( )
Edata ( , ) = ‖ − ‖2F
14
![Page 30: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/30.jpg)
Spatial Smoothness PriorE(
,)
= λEdata
(
,)
+ Ereg
( )
+ τEtrace
( )
Ereg
( )=
∑
i
TV (Si)
−−−−−−→
Without regularisation With regularisation15
![Page 31: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/31.jpg)
Low Rank PriorE(
❘,❙)
= λEdata
(
❘,❙)
+ Ereg
(
❙
)
+ τEtrace
(
❙
)
Etrace
(❙
)= ‖❙‖∗ =
∑
i
σi(❙)
lies in−−−−→ span
K ≪ F
Angst et al. ECCV’12, Dai et al. CVPR’12, Angst et al. ICCV’11, Dai et al. ECCV’10
16
![Page 32: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/32.jpg)
Minimisation of E(
,)
min,
λ ‖ − ‖2F︸ ︷︷ ︸Reprojection
error
+∑
i
TV (Si)︸ ︷︷ ︸Smoothness
prior
+ τ ‖ ‖∗︸ ︷︷ ︸Low rank
prior
17
![Page 33: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/33.jpg)
Minimisation of E(
,)
min,
λ ‖ − ‖2F︸ ︷︷ ︸Reprojection
error
+∑
i
TV (Si)︸ ︷︷ ︸Smoothness
prior
+ τ ‖ ‖∗︸ ︷︷ ︸Low rank
prior
Our Algorithm
Initialize and using rigid factorisation.
Minimize energy via alternation:
Step 1: Rotation estimation.Step 2: Shape estimation.
Efficient and highly parallelizable algorithm → GPU-friendly
17
![Page 34: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/34.jpg)
Minimisation of E(
,)
minλ ‖ − ‖2F︸ ︷︷ ︸Reprojection
error
Step 1: Rotation estimation
Robust estimation by using dense data.
Solved via Levenberg-Marquardt algorithm.
Rotations are parametrised as quaternions.
17
![Page 35: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/35.jpg)
Minimisation of E(
,)
minλ ‖ − ‖2F︸ ︷︷ ︸Reprojection
error
+∑
i
TV (Si)︸ ︷︷ ︸Smoothness
prior
+ τ ‖ ‖∗︸ ︷︷ ︸Low rank
prior
Step 2: Shape estimation
Convex sub-problem.
Optimisation via alternation between:
Per frame shape refinement: using primal dual algorithmEnforcing low rank: using soft impute algorithm.
17
![Page 36: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/36.jpg)
Results on real sequences
18
![Page 37: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/37.jpg)
Quantitative Evaluation
Average RMS 3D reconstruction errors.
Sequence TB MP Ours Ours(τ = 0)
Non-smooth rotations 4.50% 5.13% 2.60% 3.32%Smooth rotations 6.61% 5.81% 2.81% 3.89%
- TB: Akhter et al., Trajectory space: A dual representation for NRSfM, PAMI’11.
- MP: Paladini et al., Optimal metric projections for deformable and articulated SfM, IJCV’12.
- Synthetic data generated using (Vlasic et al., SIGGRAPH’05). 19
![Page 38: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/38.jpg)
Presentation Outline
1 Introduction
2 Model-free Dense 3D Reconstruction from Videos
3 Model-based Dense 3D Reconstruction from Videos
4 Craniofacial Surgery Applications
5 Conclusions
20
![Page 39: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/39.jpg)
But what about In-the-wild Videos?
Addressing the challenges of unconstrained, everyday-lifevideos
Focusing on human faces
21
![Page 40: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/40.jpg)
Our Pipeline
Step 2: Dense
Shape Inference���
���
���
Step 1: Dense
Video Registration ���
+ Face-specific priors
(Snape, Roussos, Panagakis, Zafeiriou, IEEE ICCV 2015)(Booth, Roussos, Zafeiriou, Ponniah, Dunaway, IEEE CVPR 2016)(Booth, Roussos, Ponniah, Dunaway, Zafeiriou, IJCV 2017, under minor revision)(Booth, Roussos, et al., T-PAMI 2017, submitted)
22
![Page 41: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/41.jpg)
Constructing Detailed 3D Face Models: Identity Variation
Synthetic faces generated by our LSFM model
High-resolution 3D statistical model
Automatically built from ∼10,000 3D scans
Largest-scale Morphable Model ever constructed
(Booth, Roussos, Zafeiriou, Ponniah, Dunaway, IEEE CVPR 2016)
23
![Page 42: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/42.jpg)
Constructing Detailed 3D Face Models: Identity Variation
2D
0° +90°-90°
3D3D
3D GLOBAL PCA MODEL
(INITIAL ESTIMATION)
NICP dense correspondence
Automatic pruning
Auto landmarking LSM-BESPOKE PCALSM-GLOBAL PCA
Fully automatic pipeline
State-of-the-art image localisation on synthetic views
Natively 3D approach to dense mesh correspondence
Building global model but also models tailored byage/gender/ethnicity
(Booth, Roussos, Zafeiriou, Ponniah, Dunaway, IEEE CVPR 2016)
24
![Page 43: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/43.jpg)
Constructing Detailed 3D Face Models: Identity Variation
Update:
(Booth, Roussos, Ponniah, Dunaway, Zafeiriou, Large scale
3D Morphable Models, IJCV, under minor revision):
extended evaluationadded texture model
source code for construction pipeline is now available:https://github.com/menpo/lsfm
shape models will be available very soon:
25
![Page 44: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/44.jpg)
Evaluation of Model Fitting on 3D Scans
200 1000 8000
- BFM: Basel Face Model (Paysan et al. AVSS’09)- Brunton et al.: PCA model of (Brunton et al., CVIU’14)- 100-7000: Proposed LSM, built with varying size of training set (100-7000 faces) 26
![Page 45: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/45.jpg)
Adding Expression to LSFM models
Overall model of identity & expression by effectively combining:
identity variation from our LSFM models, with
expression variation from (Cao et al., IEEE T-VG 2014)
S(pid,pexp) = µ+Uid pid +Uexp pexp
Synthetised faces, with random identity and expression
27
![Page 46: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/46.jpg)
Adding Expression to LSFM models
28
![Page 47: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/47.jpg)
Adding Expression to LSFM models
LSFM-bespoke for (White ; over 50 years) with first 4 expression coefficients
29
![Page 48: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/48.jpg)
Adding Expression to LSFM models
LSFM-bespoke for (Black) with first 4 expression coefficients
30
![Page 49: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/49.jpg)
Face Flow: Face-Specific Video Registration
... ...
(Snape, Roussos, Panagakis, Zafeiriou, IEEE ICCV 2015)
31
![Page 50: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/50.jpg)
Face Flow: Face-Specific Video Registration
... ...
Evaluation on synthetic videos with challenging conditions:
(Snape, Roussos, Panagakis, Zafeiriou, IEEE ICCV 2015)
31
![Page 51: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/51.jpg)
3DMM Fitting “In-The-Wild” (ITW)
Fitting on single images, under unconstrained conditions
3D shape model of identity + expression
Texture models for in-the-wild images
32
![Page 52: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/52.jpg)
3DMM Fitting “In-The-Wild” (ITW)
Dense image features
Simplified fitting: no need to estimate lighting
Robust to illumination changes, occlusions, etc.
33
![Page 53: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/53.jpg)
3DMM Fitting “In-The-Wild” (ITW)
Robust PCA with missing values:
33
![Page 54: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/54.jpg)
3DMM Fitting “In-The-Wild” (ITW)
Fitting on images:
Fast algorithm, AAM-style
Source code will be available
34
![Page 55: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/55.jpg)
3DMM Fitting “In-The-Wild” (ITW)
Results on 300W:
35
![Page 56: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/56.jpg)
3DMM Fitting “In-The-Wild” (ITW)
Results on 300W:
35
![Page 57: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/57.jpg)
3DMM Fitting “In-The-Wild” (ITW)
New benchmark:
Quantitative comparisons:
36
![Page 58: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/58.jpg)
3DMM Fitting on “In-The-Wild” Videos
Robust facial landmark tracking
Valuable for:
initialisationconstraints on the dense solution
37
![Page 59: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/59.jpg)
3DMM Fitting on “In-The-Wild” Videos
Initialisation via fitting on the sparse tracks:
formulate cost function that combines:
reprojection error
temporal smoothness over expression
quadratic priors on identity & expression coefficients
minimise wrt camera, identity and expression coefficients
simultaneous estimation over all frames
automatic fine-tuning of balancing weights of the cost function
38
![Page 60: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/60.jpg)
3DMM Fitting on “In-The-Wild” Videos
Initialisation via fitting on the sparse tracks:
estimation of camera parameters via rigid Structure fromMotion
large-scale quadratic optimisation for identity & expressioncoefficients
39
![Page 61: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/61.jpg)
3DMM Fitting on “In-The-Wild” Videos
Results on 300VW database:
40
![Page 62: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/62.jpg)
3DMM Fitting on “In-The-Wild” Videos
Using LSFM-bespoke models:
41
![Page 63: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/63.jpg)
Presentation Outline
1 Introduction
2 Model-free Dense 3D Reconstruction from Videos
3 Model-based Dense 3D Reconstruction from Videos
4 Craniofacial Surgery Applications
5 Conclusions
42
![Page 64: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/64.jpg)
Craniofacial Applications
Synthetic faces generated by our LSM model
Useful for: craniofacial surgery planning and assessment
before surgery after surgery
43
![Page 65: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/65.jpg)
Comparing Facial Morphology Representations
Representations of facial morphology:
dense modelling sparse anthropometry
Ideally:
different shapes ⇒ different parameterssimilar shapes ⇒ similar parameters
44
![Page 66: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/66.jpg)
Facial Manifold Visualisation
Including syndromic faces
46 scans of patients, including manually annotated landmarks
45
![Page 67: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/67.jpg)
Presentation Outline
1 Introduction
2 Model-free Dense 3D Reconstruction from Videos
3 Model-based Dense 3D Reconstruction from Videos
4 Craniofacial Surgery Applications
5 Conclusions
46
![Page 68: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/68.jpg)
Conclusions
Pioneering methodologies for dense 3D reconstruction fromnon-rigid videos
Non-rigid videos contain extremely rich information
most existing methods exploit only part of it
47
![Page 69: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/69.jpg)
Conclusions
Pioneering methodologies for dense 3D reconstruction fromnon-rigid videos
Non-rigid videos contain extremely rich information
most existing methods exploit only part of it
Using monocular input only, our methods yieldstate-of-the-art results on estimating:
multiframe optical flow
dense dynamic 3D shape
joint dense multibody segmentation, tracking and 3Dreconstruction
47
![Page 70: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/70.jpg)
Conclusions
Pioneering methodologies for dense 3D reconstruction fromnon-rigid videos
Non-rigid videos contain extremely rich information
most existing methods exploit only part of it
Key components:
dense variational methods
robust penalisers and low-rank matrix priors
efficient optimisation approaches
highly-detailed and realistic shape priors
47
![Page 71: Dense 3D Modelling and Monocular Reconstruction of ...users.ics.forth.gr/~troussos/ARoussos_ResearchOverview_March17.… · Robust facial landmark tracking Valuable for: initialisation](https://reader036.vdocument.in/reader036/viewer/2022071003/5fbfe16bbfb2600cc4600b0f/html5/thumbnails/71.jpg)
Conclusions
Dense 3D face modelling with unprecedented quality
large-scale datasets are extremely valuable
fully-automated construction pipeline
far more diverse than existing models
48