Department of Computer & Information Science

Technical Reports (CIS)

University of Pennsylvania Year 2001

Synthesis and Acquisition of Laban

Movement Analysis Qualitative

Parameters for Communicative Gestures

Liwei Zhao

Norman I. Badler

University of Pennsylvania

University of Pennsylvania, badler@seas.upenn.edu

This paper is posted at ScholarlyCommons.

http://repository.upenn.edu/cis reports/116

XD

XDXDXDXDXDXD

XDXDXDXD

XDXDXDXDXD

XDXDXDXD

XDXDXDXD

XDXD

Discourse modelsituation knowledge

encyclopaedia

Longtermmemory

Workingmemory

Conceptualizer

Others

Propositional Spatial/Dynamic

Formulator

Grammaticalencoder

Phonologicalencoder

Articulator

Auditorymonitor

Lemma

Wordforms

Spatial/dynamicfeature selector

Motor planner

Motor system

Kinesic monitor

Lexicon

overtspeech

preverbal message

phoneticplans

spatial/dynamic feature

motor program

lexical movement

XD

semantic filteringtemporal filteringunification

statistical weightingneural computingpolicy and planning

user input

speech recognition gesture recognition

gestural languageinterpretation

spoken languageinterpretation

speech generation gesture generation

user

multimodalinterface

psychologicalor cognitivemodels

embodiedconversational agent models

XDXDXDXDXDXDXDXD

wrist

elbow

waist

shoulder

clavicle

hip

neck

knee

spine

S

ShoulderJoint

WristJoint

WElbowJoint E

R

c

u

a

ve

n

ew

w

s

y

z

w

w

a

b

A

vdy vdz

Z

Y

o

y

s sw

wz

x

s

y

z

w

a

b

vdz

Z

Y

w

ver>0 &&

C

ShoulderJoint

S

O

WristJoint

WCenterof Mass

W"W

w c

vu

XZ

Y

origin

XZ

Y

t

s

XD

c

u

n

Pe( )

Pe

P

P~

1 2 3 4 5 6 7 8 9 10 11 120

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

motion samples

Spon

tane

ity

BoundFlow FreeFlow NeutralFlow

0 100 200 3000

0.1

0.2

0.3

0.4

0.5DirectStrong

curv

atur

e

0 100 200 3000

200

400

600

800DirectStrong

curv

atur

e0 200 400 600

0

0.1

0.2

0.3

0.4

0.5Neutral

curv

atur

e

0 200 400 6000

100

200

300

400Neutral

curv

atur

e

0 200 400 600 8000

0.1

0.2

0.3

0.4

0.5IndirectLight

frame number

curv

atur

e

0 200 400 600 8000

50

100

150IndirectLight

frame number

curv

atur

e

S

ShoulderJoint

WristJoint

WElbowJoint E

R

c

u

a

ve

n

ew

w

0 500 100020

40

60

80

100Example 1

swiv

el a

ngle

s

Indirect

DirectNeutral

0 500 10000

50

100Example 2

Indirect

DirectNeutral

0 100 200 30020

40

60

80Example 3

Indirect

DirectNeutral

0 100 200 3000

20

40

60Example 4

swiv

el a

ngle

Indirect

Direct

Neutral

0 100 200 30020

40

60

80

100Example 5

Indirect

Direct

Neutral

0 100 200 3000

20

40

60

80Example 6

Indirect

Direct

Neutral

0 100 200 3000

20

40

60Example 7

swiv

el a

ngle

IndirectDirectNeutral

0 200 4000

20

40

60

80Example 8

IndirectDirect Neutral

0 100 200 3000

20

40

60Example 9

Indirect

DirectNeutral

0 100 200 3000

50

100Example 10

frame number

swiv

el a

ngle

Indirect

Direct Neutral

0 100 200 30010

20

30

40

50Example 11

frame number

IndirectDirect

Neutral

0 100 200 3000

50

100Example 12

frame number

Indirect

Direct

Neutral

0 200 400 600 80020

40

60

80

100Example 1

swiv

el a

ngle

s

Indirect

Direct

Neutral

0 200 400 600 80050

0

50

100Indirect

indirect normalizedlow freq high freq

0 200 400 60020

0

20

40

60

80

100Neutral

frame number

swiv

el a

ngle

s

neutral normalizedlow freq high freq

0 50 100 15030

20

10

0

10

20

Indirect_hfreq

neutral_hfreq

frame number

Indirect vs Neutral

0 200 400 6000

20

40

60

80Example 1

wris

t ang

le IndirectDirectNeutral

0 200 400 600

20

40

60

80

Example 2

IndirectDirect

Neutral

0 100 200 3000

20

40

60

80Example 3

Indirect

Direct Neutral

0 100 200 3000

20

40

60

80Example 4

wris

t ang

le

Indirect

DirectNeutral

0 100 200 3000

50

100Example 5

Indirect

DirectNeutral

0 100 200 3000

50

100Example 6

Indirect

DirectNeutral

0 100 200 3000

50

100Example 7

wris

t ang

le

Indirect

Direct

Neutral

0 200 4000

50

100Example 8

Indirect

Direct

0 100 200 3000

20

40

60

80Example 9

Indirect

Direct

Neutral

0 100 200 3000

20

40

60

80Example 10

frame number

wris

t ang

le

Indirect

Direct Neutral

0 100 200 3000

50

100Example 11

frame number

IndirectDirect

Neutral

0 100 200 3000

20

40

60

80

Example 12

frame number

IndirectDirect

Neutral

1 2 3 4 5 6 7 8 9 1011120

2

4

6

8

10

12

motion samples

ster

num

hei

ght

SuddenTime StrongWeight

1 2 3 4 5 6 7 8 9 1011120

1

2

3

4

5

6

7

8

9

10

motion samples

ster

num

hei

ght

SustainedTimeLightWeight

W1,1W2,1WI,1

W1,2W2,2WI,2

W1,MW2,MWI,M

InputLayer

bias1

XI

X2

X1

hiddenLayer

OutputLayerSigmoid

functionSigmoid function

bias1

bias1

bias1

bias1

h2

h1

hn

0 50 1000

50

100

150

200

velocity

acce

lera

tion

direct neutral indirect

0 100 2000

100

200

300

400

500

600

swivel velocity

wris

t vel

ocity

0 5 100

2

4

6

8

10

swivel zerocrossings

wris

t zer

ocr

ossi

ngs

0 0.5 1 1.50

0.1

0.2

0.3

0.4

curvature

tors

ion

direct neutral indirect

0 50 100 1500

20

40

60

80

100

120

swivel angle sum

wris

t pen

dulu

m s

um (s

cale

d)

20 40 60 800

20

40

60

80

100

swivel angle

max

wris

t ang

le

0 50 1000

20

40

60

80

100

swivel minmax diff

wris

t max

ang

le

0 50 1000

0.5

1

1.5

velocity

curv

atur

e

0 50 1002

4

6

8

10

max wrist angle

swiv

el z

ero

cros

sing

s

direct neutral indirect

0 500 1000 15000

0.05

0.1

0.15

0.2

0.25

0.3

0.35

Number of weight updates (in hundreds)

Erro

r

Error versus weight updates (BPNN 0)

Training error Validation error

0 500 1000 15000

0.05

0.1

0.15

0.2

0.25

0.3

0.35

Number of weight updates (in hundreds)

Erro

r

Error versus weight updates (BPNN 1)

Training error Validation error

0 100 200 300 4000

0.05

0.1

0.15

0.2

0.25

0.3

0.35

Number of weight updates (in hundreds)

Erro

r

Error versus weight updates (BPNN 2)

Training error Validation error

0 500 1000 15000

0.05

0.1

0.15

0.2

0.25

0.3

0.35

Number of weight updates (in hundreds)

Erro

r

Error versus weight updates (BPNN 3)

Training error Validation error

3DEstimation

FeatureExtraction

ExpressiveMotionEngine(EMOTE)

ImageAnalysis

ImageAnalysis

Camera 1

Camera 2

...

keystructure

Style

Noisefiltering

EMOTEparameters

1O 2O

1P

1P

2P2P

Q

(x,y)1 1

(x ,y )1 1

(x ,y )2 2(x,y)2 2

image plane

xx

y y

O

1T 2T

f2f1

z z

1(X ,Y ,Z )1 1 1(X ,Y ,Z )2 2 22Q(X,Y,Z)1 1 11Q

(X,Y,Z)2 2 22Q

100 150 200 250 300 350 4000

100

200

300

400

x coordinates

y co

ordi

nate

s

2D Data (Neutral Space)

0200

400

0200

400600

200

0

200

x coordinates

3D Data (Neutral Space)

y coordinates

200 250 300 350 400100

150

200

250

300

350

x coordinates

y co

ordi

nate

s

2D Data (Direct Space)

0200

400

0200

400600

200

0

200

x coordinates

3D Data (Direct Space)

y coordinates

150 200 250 300 350 400100

150

200

250

300

350

x coordinates

y co

ordi

nate

s

2D Data (Indirect Space)

0200

4000

200400

600

200

0

200

x coordinates

3D Data (Indirect Space)

y coordinates

100 150 200 250 300 350 40050

100

150

200

250

300

350

400

x coordinates

y co

ordi

nate

s

2D Data (Neutral Weight)

2000

200400

0200

400600

200

100

0

100

200

x coordinates

3D Data (Neutral Weight)

y coordinates

200 250 300 350 400

100

150

200

250

300

350

x coordinates

y co

ordi

nate

s

2D Data (Strong Weight)

0200

400

0200

400600

300200100

0100200

x coordinates

3D Data (Strong Weight)

y coordinates