production and elivery of interactive 3d c ontentreact.ch.bbc.co.uk/documents/cvmp2014.pdf ·...

PRODUCTION AND DELIVERY OF

INTERACTIVE 3D CONTENT

Graham Thomas (BBC)

With contributions from all partners(Fraunhofer HHI, INRIA, University of Surrey,

Artefacto, Oxford Metrics Group)

The research leading to these results has received funding from the European Union’s Seventh Framework Programme (FP7/2007-2013) under grant agreement n°288369

© RE@CT Consortium 2 http://react-project.eu

RE@CT partners

© RE@CT Consortium 3 http://react-project.eu

Project goal

To develop a production pipeline for actor performance to

support low-cost photorealistic interactive game-like

applications

Traditionalmotion capture

TV production RE@CT

photorealistic � � �

natural capture

environment

� � �

free-viewpoint � � �

Editable

animation

� � �

© RE@CT Consortium 4 http://react-project.euI

RE@CT processing pipeline

© RE@CT Consortium 5 http://react-project.eu

Project structure

© RE@CT Consortium 6 http://react-project.eu

Partners and roles

BBC3D video capture &

annotation

VICONHybrid capture

HHIFacial animation

SurreyActor animation

INRIAPerformance analysis

ArtefactoInteractive platform & Demonstrator

BBCInteractive player &

Test production

© RE@CT Consortium 7 http://react-project.eu

Body capture system

� 12-15 HD / UHD cameras

� Chromakey (retroreflective)

� Uncompressed recording

� Real-time feedback for keying and actors leaving volume

© RE@CT Consortium 8 http://react-project.eu

Face capture system (1)Approach: build detailed 3D model that can be used to enhance detail in whole-body capture

� Capture multiple stereo pairs with DSLR cameras & flash lighting

� Build 3D model by merging models from each pair

� Repeat for different facial expressions

© RE@CT Consortium 9 http://react-project.eu

Face capture system (2)

� For capturing full details of an animated face

� Head-mounted rig with two pairs of stereo cameras

� Draw small blobs on face

� Blobs tracked to provide 3D shape on every frame

� Launched at SIGGRAPH 2013 as ‘Cara’

© RE@CT Consortium 10 http://react-project.eu

Performance Analysis

� Create time-consistent geometry1

� Super-resolution technique for surface textures by combining multiple views 2

1 B. Allain et al. On Mean Pose and Variability of 3D Deformable Models. ECCV 20142 V. Tsiminaki et al. High Resolution 3D Shape Texture from Multiple Videos. CVPR 2014

© RE@CT Consortium 11 http://react-project.eu

Actor animation

� Head tracked in full-body model and hi-res model merged

� Texture and geometry blended to allow gradual transitions between animations, e.g. walk -> run1

1 D. Casas et al. 4D Video Textures for Interactive Character Appearance. Eurographics 2014

© RE@CT Consortium 12 http://react-project.eu

Representation & rendering

� Multi-layer texture map to support view-dependent texturing with texture maps for most important views rather than all views1

1M. Volino et al. Optimal Representation of Multi-View Video. BMVC 2014

© RE@CT Consortium 13 http://react-project.eu

Authoring tools

� Scriptable authoring tool for creating sequence of blended animations

� Tool for creating interactive data overlays for educational applications

© RE@CT Consortium 14 http://react-project.eu

Rendering

� WebGL-based renderer: textures from multiple cameras packed efficiently into a small number of frames; can be updated at frame rate to convey changing appearance

� Renderer for mobile platform: single texture optimised across all viewpoints / frames for efficient loading

© RE@CT Consortium 15 http://react-project.eu

Thanks for your attention!

graham.thomas@bbc.co.uk

react-project.eu