new motion editing - seoul national...

Motion Editing (Geometric and Constraint-Based Methods)

Jehee Lee Seoul National University

Animation from Motion Capture

• Motion capture provides a large collection of realistic motion data – What can we do with this data set ?

• Applications – Animation authoring: Film, music video, TV ad., … – Interactive characters: Video games, VR, … – Live performance: Computer puppetry

Motion DatabasePreprocess

On-line Controller

Motion Editing Toolbox

Motion Sensor

Data

Custom Animation

Controllable Characters

Mapping Live Performance

High-Level User Interfaces

The Art of Animation

Computer Puppetry

Animation Authoring

• Animators select motion segments from the database

• Motion segments will be modified, blended, and stitched – Animators need a variety of motion editing tools

Motion DatabasePreprocess Motion Editing

ToolboxMotion Sensor

Data

Custom Animation

The Art of Animation

Motion Retargetting

Interactive Characters

• Users interact (or control) with computer-generated characters through high-level user interfaces

• System selects appropriate motions from the database – User interfaces and control algorithms are required

Motion DatabasePreprocess On-line

Controller

Motion Sensor

DataControllable Characters

High-Level User Interfaces

Sketch Interface

Computer Puppetry

• Motion sensor data is mapped to an animated character in realtime – No database is required

Motion Representation

• Configuration of an articulated figure – Linear components: – Angular components:

3)( Rp ∈t3)( Sq ∈ti

!!!!!!

"

#

$$$$$$

%

&

=

)(

)()()(

)( 1

0

t

ttt

t

nq

qqp

m!

The position of the root segment

The orientations of body segments w.r.t. their parents (joint angles)

The orientation of the root segment

Simple Operations

• Translation

• Rotation

!!!!!!

"

#

$$$$$$

%

& +

=

!!!!!!

"

#

$$$$$$

%

&

⊕

!!!!!!

"

#

$$$$$$

%

&

=⊕=

)(

)()(

)(

)(

)()()(

)()(' 2

1

2

1

t

tt

t

t

ttt

tt

nn

t

q

qq

vp

0

00v

q

qqp

dmm!!!

!!!!!!

"

#

$$$$$$

%

&

=

!!!!!!

"

#

$$$$$$

%

&

⊕

!!!!!!

"

#

$$$$$$

%

&

=⊕=

)(

)()exp()(

)(

)(

)()()(

)()(' 2

1

2

1

t

ttt

t

ttt

tt

nn

r

q

qvq

p

0

0v0

q

qqp

dmm!!!

Simple Operations

• Exercise joints

• Motion displacements – independent translation and rotation

!!!!!!!!

"

#

$$$$$$$$

%

&

=

!!!!!!!!

"

#

$$$$$$$$

%

&

⊕

!!!!!!!!

"

#

$$$$$$$$

%

&

=⊕=

)(

)exp()(

)()(

)(

)(

)()(

)()('

11

t

t

tt

t

t

tt

tt

n

i

n

i

j

q

vq

qp

0

v

00

q

q

qp

dmm

!

!

!

!

!

!

!!!!!

"

#

$$$$$

%

& +

=

!!!!!

"

#

$$$$$

%

&

⊕

!!!!!

"

#

$$$$$

%

&

=⊕=

))(exp()(

))(exp()()()(

)(

)()(

)(

)()(

)()()(' 11

0

1

0

1

tt

tttt

t

tt

t

tt

ttt

nnnn vq

vqvp

v

vv

q

qp

dmm!!!

Motion Displacement

• Rigid Transformation at the root segment

!!!!!

"

#

$$$$$

%

& +

=

!!!!!

"

#

$$$$$

%

&

⊕

!!!!!

"

#

$$$$$

%

&

=⊕=

−

)exp(

)exp(

)(

)()(

)(

)()(

)()()(' 11

1101

1

0

1

nnnn t

tt

t

tt

ttt

vq

vqpqvq

v

vv

q

qp

dmm!!!

Motion Displacement

• Rigid Transformation at the root segment

!!!!!

"

#

$$$$$

%

& −

=

!!!!!

"

#

$$$$$

%

&

−

!!!!!

"

#

$$$$$

%

&

=−=

−

−

−

)'log(

)'log())()('(

)(

)()(

)('

)(')('

)()(')(

1

111

111

11

nnnn

tt

t

tt

t

tt

ttt

qq

qqqppq

q

qp

q

qp

mmd!!!

Simple Operations

• Valid operations

• Invalid operations)! careful Be()()()(

)()()()()(

)()()(

321

21

12

21

ttttttttttt

ddddddmmmdm

=+

=

=−

=⊕

α

?)(?)()( 21

=

=⊕

ttt

mmmα

Simple Operations

• Time warping

• Properties

))(()(' tst mm =

( ) ( ))()()()()()()()()()(

2121 tttttttttt

ddmddmmddm

+⊕≠⊕⊕

⊕≠⊕

Transition Graph

Stand

Right foot forward

Start

Stop

TurnRTurnL

Loop

Transition Graph

Connecting Motion Segments

• Alignment and warpingFirst Motion

Second MotionRotate and translate the second motion

to align two motions

Warp the motions at the boundary

so that they can be connected smoothly

Alignment

• The end of one motion A should be aligned to the beginning of the next motion B – The root location of the end of A: – The root location of the beginning of B:

– Apply to motion B, that is

),( An

An qp

),( 00BB qp

!!!!!!

"

#

$$$$$$

%

& +

=

!!!!!!

"

#

$$$$$$

%

&

⊕

!!!!!!

"

#

$$$$$$

%

&

=⊕

−

)(

)()exp()()()()(

)(

)()()(

)( 2

1

111

2

1

t

tt

ttt

t

ttt

t

nn

B

q

qvqpuqq

0

0vu

q

qqp

dm!!!

!!"

#$$%

&−!!"

#$$%

&=!!

"

#$$%

&B

B

An

An

0

0

qp

qp

vu

Alignment Using In-Plane Transformation

• Rigid transformation restricted within a plane – Rotation about the vertical axis, followed by – Translation along two horizontal axes

• How can we find in-plane rotation from arbitrary rotation ? – Euler angles

• Discard rotation about x- and z-axes

zyx RRRR =

Optimal In-Plane Transformation

• Closest point from a geodesic curve to a point – The geodesic curve represents a set of orientation that can be

reached by rotating about the vertical axis

Anp

B0p

BB yyG 00 )sinˆ,(cos),ˆ( pp ⋅= θθ

Optimal In-Plane Transformation

Point Cloud Matching

Warping

• Deform a motion smoothly so that it is seamlessly connected its previous motion

• A scalar transition function s(t)

First Motion Second Motiond BA

n 0mmd −=

{ }dm ⋅⊕ )()( tstB

)(tS

t

1

Motion Database

• Reusability and flexibility – Motion capture is not cheap ! – Motion capture data doesn’t always reflect our

intention • There is no live performer who can move like an ant or

Godzilla

Antz (1998) Godzilla (1998)

Motion Database

• Reusability and flexibility – Motion capture is not cheap ! – Motion capture data doesn’t always reflect our

intention • We are not always able to construct an appropriate physical

environment in a mocap studio

Motion Database

• Reusability and flexibility – Motion capture is not cheap ! – Motion capture data doesn’t always reflect our

intention • There is no live performer who can move like an ant or

Godzilla • We are not always able to construct an appropriate physical

environment in a mocap studio – We sometimes want to reuse existing data rather than

capturing new motion whenever needed • Video games, interactive characters

Adaptation/Retargetting/Editing

• Reusing motion is challenging, because the motion was acquired – For a specific character – Within a specific environment – In a specific style and mood

Constraint-Based Approaches

• Constraints specify – Features to be retained – New features to be accomplished

• Find a new motion – Satisfy given constraints – While preserving the characteristics of the original

motion as much as possible

• The core of constraint-based approaches can be formulated as an optimization problem

Constraints

• Preserve important features of the original motion – Contact with environments (foothold positions) – Balance (dynamic constraints) – Subtle details (frequency bands)

• Accomplish new features for the target motion – Interactive manipulation – New characters/environment – New style/mood (How do you define?)

Constraint-Based Approaches

Inter- and Intra-Frame Coherency

Within each frame • Enforce constraints • By inverse kinematics

Inter-frame coherence • Avoid discontinuity • By using smooth splines

Motion and Constraints

Original Motion

How would we warp the motion to meet the constraints ?

Motion and Constraints

Original Motion

Bad warping

Displacement Mapping

• Motion warping [Popovic & Witkin 95] • Motion displacement mapping [Bruderlin & Williams 95]

Displacement Map

Original Motion

Target Motion

Displacement Mapping

!!!!!

"

#

$$$$$

%

& +

=

!!!!!

"

#

$$$$$

%

&

⊕

!!!!!

"

#

$$$$$

%

&

=

⊕=

)exp(

)exp(

)()('

11

0

1

0

1

nnnn

tt

vq

vqvp

v

vv

q

qp

dmm

!!!

Original Motion

Displacement map

Non-Uniform Spline Interpolation

• How would you create a smooth curve (displacement map) that interpolates given joint displacements ? – Spline interpolation – You already know how to do it !

• Constraints are distributed non-uniformly – Variational constraints (constraints that stand over a time period) are

discretized as a sequence of constraints

• Small details are added for constrained intervals – Details (high frequency) of the original motion could be destroyed

• IK solution is not always coherent – Small change of start poses and constraints doesn’t necessarily

yeild small change of output poses

Least-Squares Curve Fitting

• We would like to discard outliers – Approximation rather than interpolation

• We don’t want to add unnecessary high-frequency – Uniform splines

• IK solving and curve fitting should be applied alternatively for several times

• The linear system could be partially over-specified and partially under-specified

Non-Linear Optimization

• Formulation [Gleicher 98] – Curve approximation and IK can be combined into a

single large non-linear optimization problem

– Objective function: minimize distortion

– Parameters • All joint angles at all time instances (It is a huge problem !) • Joint angles are often represented by B-splines

– in order to reduce the number of parameters – in order to maintain inter-frame coherency

∫∫ =−= dttdtttG )()()( 0 dmm

Adaptive Refinement

• Shape freedom vs. Smoothness

• Hard to determine the density of knots • Multi-level (hierarchical) B-splines

– Adaptive refinement of knot sequences – Sum of uniform B-splines with different knot spacings – Coarse-to-fine hierarchy

Hierarchical Displacement Mapping

)))((( hh dddmm 210 ⊕⊕⊕⊕= !!

Algorithm

Examples

• Adaptation to New Characters

Examples

• Character Morphing

Adapting Motion for Humanoids

• Different structures and DOFs

[Pollard 2002]

Adapting Motion for Humanoids

• Different structures and DOFs – Adaptation algorithm is similar to IK with

multiple goals – Gimbal lock avoidance

Adapting Motion for Humanoids

• Different structures and DOFs – Adaptation algorithm is similar to IK with

multiple goals – Gimbal lock avoidance

• Joint limit – Inequality constraints – Similar to collision avoidance

Inequality Constraints

Upper bound

Lower bound

Inequality Constraints

Upper bound

Lower bound

Inequality Constraints

Upper bound

Lower bound

Hierarchical displacement mapping can be used

Inequality Constraints

Upper bound

Lower bound

Much better in terms of motion quality

Adapting Motion for Humanoids

• Different structures and DOFs • Joint limit • Dynamic limit

– Velocity, torque, balance, …

Editing dynamic human motions via momentum and force

Kwang Won Sok, Katsu Yamane, Jehee Lee, Jessica Hodgins (SCA 2010)

더 빠르게, 더 높이

더 빠르게, 바닥에 착지

달나라 점프

더 높게, 물리적으로 맞게

Motorized Marionette by Yamane (2003)

• 6 motors for hands and 2 motors for legs • Limited kinematic range • Limited dynamic capability • Avoid undesired swing

Summary

• Motion data processing – Reusability and flexibility – Efficient and correct – A Toolbox of motion editing tools

• Motion adaptation/retargeting/editing – Displacement mapping

• It is just spline interpolation/approximation • It can be formulated as a non-linear optimization problem

– Hierarchical displacement mapping • Adaptive refinement is desired