ss5305 – advanced motion capture 1. objectives facial capture performance capture hardware trends...
TRANSCRIPT
1
SS5305 – Advanced Motion Capture
2
Objectives
• Facial Capture• Performance Capture• Hardware Trends• Software Trends• Conclusions
Facial Capture
Frederick I. Parke, University of UtahComputer Generated Animation of Faces1972
1988 B. Robertson, Mike the Talking Head, Computer Graphics World 11 (7):57
Facial Capture
Method 1 – PhonemesWhen a particular type of sound is spoken: phonemesSpecific shapes of the whole face are captured. (top down)Phonemes – the sounds that make up a word, not letters
balloon b – ah – l – oo – n
Method 2 – Feature TrackingParts of the face are tracked separately.Each part contributes to overall motion. (bottom up)Motion is the sum of many features.Works for speech and other facial expressions
Trend – Markerless Facial Capture
http://www.youtube.com/watch?v=UYgLFt5wfP4&feature=player_embedded
Emily, Image Metrics, 2010
Surface is tracked based on image distortion rather than markers.
Problem:
Motion capture records the body over volumes up to:10 x 10 sq. meters (30 sq. ft)
Facial capture records subtle details over space of:30 x 30 cm (1 sq. ft)
How to capture both the large-scale motion of the body
and subtle motion of the face during a single performance?
Avater (2010),James Cameron
Solution:
Block off the face using individual, head-mounted cameras, which record only the face. Use motion cameras and passive markers for the body.Allows for both large volumes and small details.
Trend –Performance captureis a collection of techniques that combine torecord the totalmotion of an actor.
Markers include:Body capture Green lines, white dotsFacial capture Head-mounted device, /w camera boomsHair capture Blue and red ropes
Hardware Trends
Trend – Markerless capture:Origins in 3D laser scanning
3D Lego Digitizer http://www.rchoetzlein.com/project/digitizer/
Trend – Markerless capture: Structured Light
Faster: Do all lines at once
Projector with structured light mapped onto the object. Use two cameras to determine object structure.
Structured light can be linear, binary coded, gray coded, or color coded. The encoding allows you to uniquely identify points.
Light may be infrared (Kinetic).
Q: High frequency gives details about height of point. But how do we tell if the point is on left or right side of obj?
A: Low frequency gives overall characteristics of pixels.
No markers.
Structured light creates a point cloud.
Skeleton is fit inside point cloud from root joints to extremities.Torso defines primary orientation,and also constraints placement of next joint layer in hierarchy.
Volume construction
Pointcloud
Fittorse
Fitextremities
Trend – Markerless capture: Direct-to-3D models
http://www.youtube.com/watch?v=dTisU4dibSc&playnext=1&list=PLD31C3C36D294EEDB
Christian Theobalt, Stanford University
http://www.stanford.edu/group/biomotion/Markerless.html
Performance Capture from Sparse Multi-view Video, SIGGRAPH 2010
Trend – Monocular capture
Fabio Remondino, Andreas RoditakisInstitute for Geodesy and Photogrammetry - ETH Zurich, Switzerland3D Reconstruction of Human Skeleton from Single Images or Monocular Video Sequences2003, 25th Pattern Recognition Symposium
One camera, without depth, is under-constrained.
However, the human body has fixed limb lengths and ratios.
Use the body ratios as an additional constraint.
Trend – Low Cost Systems
Cheap hardware: Microsoft Kinect, Web cameras.
Open source software: OpenKinect open kinect drivers libfreenect open kinect drivers
OpenNI skeleton fittingFaceAPI facial tracking
Main challenges: 1) Integration into existing frameworks,2) Usually requires programming experience3) Can be difficult to modify for research
Software Trends
Motion Graph:
A database of motion capture clips, connected to one another to represent transitionsbetween actions.
Motion graphs can be represented by a finite state machine,a set of states with edges representing state transitions.
Stand Run
Jump
Motion Graphs
Planning and Directing Motion Capture For GamesMelianthe Kines, Gamasutra. January 19, 2000http://www.gamasutra.com/view/feature/3420/planning_and_directing_motion_.php
Trend – Motion Graphs in Gaming
What are the advantages and disadvantages ofmotion graphs for gaming?
Advantages1. Fast. Motion is simply played back from pre-recorded data.2. Interactive. Motion can be changed immediately by transitioning to a different state.3. Modular. Different motions can easily be swapped in.4. Extensible. More states can be added to the graph.
Disadvantages1. Jump transitions between capture clips2. Motion may not match scene exactly. e.g. jump over chasm3. Cannot grasp objects accurately. No inverse kinematics.4. Cannot move in any direction5. Interruptions from outside forces not easily handled
Trend – Motion Blending in Gaming
Michael Gleicher, Hyun Joon Shin, Lucas Kovar, Andrew JepsenSnap-Together Motion: Assembling Run-Time AnimationsInteractive 3D Graphics 2003
“In order to create streams of high-quality motion, current applications [games]assemble static clips of motion created with traditional animation techniques such as motion capture or keyframing. The assembly process requires making transitions between motions. Thesetransitions may be difficult to create, such as a transition between a running clip and one where the character is lying down, or trivial, if the end of one clip is identical to the beginning of the next. In practice,simple techniques such as linear blends are capable of creating transitions in cases where the motions are similar.”
Common solutions in Gaming:
1. Jump transitions Linear blending between motion clips
2. Motion may not match Blend with scene constraints scene exactly (extend jump over river)
3. Cannot grasp objects Add inverse kinematics toarms in game characters.
4. Cannot move in any Add steering. direction Simple: re-orient, then play walk cycle
Advanced: add IK to legs
5. Interruptions from Use a rag-doll physics switch. outside forces When object hits..
Turn on physics, apply force.
Trend – Motion Graphs
How would you instruct a character to follow an arbitrary pathusing a set of pre-recorded captured motion?
Lucas Kovar, Michael Gleicher, Frederic Pighin. Motion Graphs, SIGGRAPH 2002.
Trend – Motion Graphs
Alla Safonova Jessica K. Hodgins,Carnegie-Melon University.Construction and optimal search of interpolated motion graphsSIGGRAPH 2007
How do we make energy optimal motion based on several, arbitrary constraints?
Uses motion capture data, butin arbitrary, non-acted scenarios.
Overview
Motioncapture
Pointclouds
Skeletonfitting
Facialcapture
Performancecapture
Markerdata Re-targeting
3D modelcapture
Motion graphs (e.g Gaming)
Blending
Optimization
Sequencing
Animation
Post processing (cleaning)
Skinning
Secondary motion
Physicalcapture /Haptics
Jointdata
Monoccularvideo
INPUT OUTPUT
Conclusion• Facial Recognition is critical in real world
applications; cyber security, safety, etc• Performance trends lead to Virtual Reality• Markerless is good, it might not be
reference standard. • Software shall be more and more
embedded than programmer oriented, more towards to end user familiar.
CASE STUDIES
Gary Sanderson: the biomechanics of a Sprinter
Gary is an 18 year old sprinterthe only difference is that he has Cerebral Palsy and wants to run at the next Olympics
The Problem: Gary was fitted with an ankle foot orthosis (or splint) to help support the ankle.But Gary’s foot was regularly collapsing as the foot was loaded during running causing great strain around the foot and ankle.
Copyright of Professor Jim Richards, University of Central Lancashire
Motion analysis showed the degree to which the ankle was collapsing
Copyright of Professor Jim Richards, University of Central Lancashire
Re-think
• Based on the information a redesign of the ankle foot orthosis (or splint) was conducted.
• The focus of this change was to provide greater stability about ankle joint.
• This in turn should help performance !?!
Copyright of Professor Jim Richards, University of Central Lancashire
The new orthosis shows no collapsing although the foot is still internally rotated
Before After
Copyright of Professor Jim Richards, University of Central Lancashire
Do we get an improvement of performance about the ankle?
• The ankle is more stable
• This should allow a better platform from which to push off
• This should lead to a significant increase in the power production
Copyright of Professor Jim Richards, University of Central Lancashire
Do we get an improvement of performance about the ankle?
Shortly after the fitting of the new orthosis Gary recorded his fastest ever time for the 100 m, 13.8 seconds, 1.5 seconds off his previous best time!
Copyright of Professor Jim Richards, University of Central Lancashire
Entertainment Applications
• Films• Television• Computer and video
games
Animation
Facial Caption
Movie/Television• Seamless and believable
visual effects• Films
– “Titanic“– "Gladiator“– "The Mummy Returns", – "Star Wars Episode 1 - the
Phantom Menace”• Crowd Scenes• Stunt Work• Photorealistic foreground
characters