production and elivery of interactive 3d c ontentreact.ch.bbc.co.uk/documents/cvmp2014.pdf ·...
TRANSCRIPT
PRODUCTION AND DELIVERY OF
INTERACTIVE 3D CONTENT
Graham Thomas (BBC)
With contributions from all partners(Fraunhofer HHI, INRIA, University of Surrey,
Artefacto, Oxford Metrics Group)
The research leading to these results has received funding from the European Union’s Seventh Framework Programme (FP7/2007-2013) under grant agreement n°288369
© RE@CT Consortium 2 http://react-project.eu
RE@CT partners
© RE@CT Consortium 3 http://react-project.eu
Project goal
To develop a production pipeline for actor performance to
support low-cost photorealistic interactive game-like
applications
Traditionalmotion capture
TV production RE@CT
photorealistic � � �
natural capture
environment
� � �
free-viewpoint � � �
Editable
animation
� � �
© RE@CT Consortium 4 http://react-project.euI
RE@CT processing pipeline
© RE@CT Consortium 5 http://react-project.eu
Project structure
© RE@CT Consortium 6 http://react-project.eu
Partners and roles
BBC3D video capture &
annotation
VICONHybrid capture
HHIFacial animation
SurreyActor animation
INRIAPerformance analysis
ArtefactoInteractive platform & Demonstrator
BBCInteractive player &
Test production
© RE@CT Consortium 7 http://react-project.eu
Body capture system
� 12-15 HD / UHD cameras
� Chromakey (retroreflective)
� Uncompressed recording
� Real-time feedback for keying and actors leaving volume
© RE@CT Consortium 8 http://react-project.eu
Face capture system (1)Approach: build detailed 3D model that can be used to enhance detail in whole-body capture
� Capture multiple stereo pairs with DSLR cameras & flash lighting
� Build 3D model by merging models from each pair
� Repeat for different facial expressions
© RE@CT Consortium 9 http://react-project.eu
Face capture system (2)
� For capturing full details of an animated face
� Head-mounted rig with two pairs of stereo cameras
� Draw small blobs on face
� Blobs tracked to provide 3D shape on every frame
� Launched at SIGGRAPH 2013 as ‘Cara’
© RE@CT Consortium 10 http://react-project.eu
Performance Analysis
� Create time-consistent geometry1
� Super-resolution technique for surface textures by combining multiple views 2
1 B. Allain et al. On Mean Pose and Variability of 3D Deformable Models. ECCV 20142 V. Tsiminaki et al. High Resolution 3D Shape Texture from Multiple Videos. CVPR 2014
© RE@CT Consortium 11 http://react-project.eu
Actor animation
� Head tracked in full-body model and hi-res model merged
� Texture and geometry blended to allow gradual transitions between animations, e.g. walk -> run1
1 D. Casas et al. 4D Video Textures for Interactive Character Appearance. Eurographics 2014
© RE@CT Consortium 12 http://react-project.eu
Representation & rendering
� Multi-layer texture map to support view-dependent texturing with texture maps for most important views rather than all views1
1M. Volino et al. Optimal Representation of Multi-View Video. BMVC 2014
© RE@CT Consortium 13 http://react-project.eu
Authoring tools
� Scriptable authoring tool for creating sequence of blended animations
� Tool for creating interactive data overlays for educational applications
© RE@CT Consortium 14 http://react-project.eu
Rendering
� WebGL-based renderer: textures from multiple cameras packed efficiently into a small number of frames; can be updated at frame rate to convey changing appearance
� Renderer for mobile platform: single texture optimised across all viewpoints / frames for efficient loading
© RE@CT Consortium 15 http://react-project.eu
Thanks for your attention!
react-project.eu