Reach Out and Touch Space(Motion Learning)

Luis Goncalves, Enrico Di Bernardo, Pietro Perona

California Institute of Technology

Motion is an important cue

A system for body tracking(Goncalves-Di Bernardo-Perona, ICCV ‘95)

• Single camera

• Real-time estimate of body pose

• 3D Model-based

• No markers required

• Able to track at frame rate (30 fr/sec)

• 8% max error (along the line of sight)

• No loose clothing• Calibration on the user • Loses track for fast movements

BUT

A system for body tracking(Goncalves-Di Bernardo-Perona, ICCV ‘95)

Arm silhouette generation 3D Model

BackgroundSubtraction

Camera Errorvector

Estimated arm position and velocity

RecursiveEstimator

Calibrationparameters

Arm modelparameters

DynamicalModel

Current model: Random Walk

Dynamical model equations:

),0(, Νwwv

v

Why Models for Human Motion ?

• Locomotion (biomechanics, robotics)

• Brain motor control (neuroscience)

• Human/Machine perception of biological

motion (neuroscience, psychophysics, computer vision)

• Realistic animation (computer graphics)

Invariant properties of ballistic point to point movements

• The path is approximately straight

• The tangential velocity profile has a smooth bell shape

• These properties are invariant wrt subject, execution time, load carried

(Hollerbach, Viviani, Flash-Hogan, Bizzi)

Far from the neuro-muscular limits and after practice:

No predictive power for general motion

Human figure animation in CG

• Keyframing, manual editing (Perlin, animation software)

• Physics-based (Hodgins et al.)

• Constraint optimization (Witkin and Kass, Badler et al.)

• Human motion capture (Bruderlin, Rose et al.)

The proposed method

• Acquire sample human motions

• Label each individual motion with a high level description (e.g. goal of motion)

• Learn a function that maps labels to motions

The 2D Motion Capture System

• 14 dots on the main body joints• single camera• UV lighting• real-time detection with sub-pixel accuracy

Title:/home2/dibe/matlab/MLD/reports/MLDibe.psCreator:XV Version 3.10 Rev: 12/16/94 - by John BradleyPreview:This EPS picture was not savedwith a preview included in it.Comment:This EPS picture will print to aPostScript printer, but not toother types of printers.

Example: Reaching motions

Start from a fixed initial pose.

Reach to various locations in space.

Pictorial representationLabel space:

Trajectory space:

LNLlTN*28Mm

The Functional Space

NLdN

ddj L

ll 10

11)( l1. Polynomial Basis Functions

2. Radial Basis Functions

T

jjj

ej

)()(21 1

)(μlΣμl

l

Learning the function

ki

ik

ik

wjk mfw

jk ,

2

}{))((arg min l

ΦMW

N

jjjkkk wfm

0)()(ˆ ll

Experiments: Reaching motions

• NL = 2

• NT = 120

Picking up apples in30 different locations

90 sample motions

Experiments: Drawing motionsDrawing strokes on a blackboard• NL = 8

• NT = 60110 sample strokes

How to evaluate performance?

• RMS error

• Perceptual evaluation by subjects

• other methods ...

Results: %RMS error for Reaching

Results: Visual DiscriminabilityReaching using 3rd order polynomial functional space

Results: %RMS error for Drawing

Results: Visual DiscriminabilityDrawing using 1st order polynomial functional space

Combining Reaching and Drawing

Blend out trajectory discontinuities at merge points of different motions.

Conclusions

• The method generates realistic synthetic reaching and drawing motions • The method can generate motions from a high level description

• The technique can be used for animation

Future work• Obtaining and running experiments with 3D captured data

• Develop a perceptually motivated metric

• Experiment with other high level labels such as speed, emotional state or gender

• Use models for tracking/prediction

Minimum jerk trajectories in reaching movements (Flash-Hogan ‘85)

Explains the experimental evidence for both straight and via point reaching motions

The trajectory minimizes:

Tt

t

dttC0

2)(x

The equilibrium point trajectory (Bizzi ‘91)

The muscular system has a spring-like behavior

Brain signals activate entire muscle groups

The activation signals depend only on the ideal (minimum jerk) trajectory

Motion is a powerful cue

From the trajectories of 12 dots attached to the main joints in the body, a subject could• distinguish human motion from other objects motion• identify gender• identify action and mood• perceive 3D structure from 2D trajectories(Johansson ‘73, Mather ‘94, Dittrich et al. ‘96)

Labeling the data