Spatio-Temporal Reasoning Over Play-Scripts for Artificially

Intelligent Characters Christine Talbot

Richard Burton in Hamlet, directed by Sir Gielgud

http://www.youtube.com/watch?v=XRU5yLgs0zw&feature=player_detailpage

Background

2

of 49

Virtual Character Positioning

3

Ba

ck-

gro

un

d

EMA: A process model of appraisal dynamics (Stacy C. Marsella,

Jonathan Gratch), In Journal of Cognitive Systems Research, volume

10, 2009.

Ada and Grace: Toward Realistic and Engaging Virtual Museum Guides (William

Swartout, David Traum, Ron Artstein, Dan Noren, Paul Debevec, Kerry

Bronnenkant, Josh Williams, Anton Leuski, Shrikanth Narayanan, Diane Piepol, H.

Chad Lane, Jacquelyn Morie, Priti Aggarwal, Matt Liewer, Jen-Yuan Chiang, Jillian

Gerten, Selina Chu, Kyle White), In Proceedings of the 10th International

Conference on Intelligent Virtual Agents (IVA 2010), 2010.

of 49

Mocap Files and Hand-Coding

4

Ba

ck-

gro

un

d

Discovery News – Avatar: Motion Capture Mirrors Emotions

http://news.discovery.com/videos/avatar-making-the-movie/

of 49

BML and BML Realizers

5

Ba

ck-

gro

un

d

SmartBody Path Planning

http://smartbody.ict.usc.edu

Hamlet played by robots

Unity using SmartBody

MindMakers Wiki

http://www.mindmakers.org/

projects/bml-1-0/wiki/Wiki

of 49<act><participant id="GRAVEDIGGER2" role="actor" /><bml><gesture lexeme="POINT" target="GRAVEDIGGER1"

/></bml></act>

<act><participant id="GRAVEDIGGER1" role="actor" /><bml><speech id="sp1" ref="“ type="application/ssml+xml">Give me

leave!</speech></bml></act>

<act><participant id="GRAVEDIGGER1" role="actor" /><bml><gesture lexeme="POINT" target="GRAVEDIGGER2"

/></bml></act>

<act><participant id="GRAVEDIGGER1" role="actor" /><bml><gesture lexeme="POINT" target="GRAVE" /></bml></act>

<act><participant id="GRAVEDIGGER1" role="actor" /><bml><speech id="sp1" ref="" type="application/ssml+xml"> Here lies

the water -- good? </speech></bml></act>

<act><participant id="GRAVEDIGGER1" role="actor" /><bml><gesture lexeme="POINT" target="GRAVE" /></bml></act>

<act><participant id="GRAVEDIGGER1" role="actor" /><bml><speech id="sp1" ref="" type="application/ssml+xml"> Here

stands the man -- good! </speech></bml></act>

<act><participant id="GRAVEDIGGER1" role="actor" /><bml><speech id="sp1" ref="" type="application/ssml+xml"> If the

man go to this water and drown himself, it is willynilly he goes, mark you that! But, </speech></bml></act>

<act><participant id="GRAVEDIGGER1" role="actor" /><bml><gesture lexeme="POINT" target="GRAVE" /></bml></act>

<act><participant id="GRAVEDIGGER1" role="actor" /><bml><speech id="sp1" ref="" type="application/ssml+xml">if the

water come to HIM and drown him, he drowns not him-self; Argal, he that is not guilty of his own death

shortens not his own life!</speech></bml></act>

<act><participant id="GRAVEDIGGER1" role="actor" /><bml><locomotion target="GRAVE" type="basic" manner="walk"

/></bml></act>

<act><participant id="GRAVEDIGGER2" role="actor" /><bml><speech id="sp1" ref="" type="application/ssml+xml">But is

this LAW ?</speech></bml></act>

Still a Lot of Work…

6

Ba

ck-

gro

un

d

4 hours & 12 minutes for a 10 minute scene!!

Point

Speak

Move

of 49

So How Do We Do It?

7

Ba

ck-

gro

un

d

A: Excuse me…

B: Can I help you?

A: Yes, where is the post office?

B: Go straight and turn left.

A: Where do I turn left?

B: Turn left at the bus stop -

you can’t miss it.

A: Thank you very much!

B: No problem.

Play-Scripts

8

of 49

GRAVEDIGGER1

Give me leave!

(GRAVEDIGGER2 sits on the side steps)

(Pointing down into the grave)

Here lies the water--good?

(Pointing to the table ledge)

Here stands the man-good!

(Illustrating each point literally with his hands)

If the man go to this water and drown himself, it is willy-nilly he goes,

mark you that! But,

(Pointing first to the grave, then to the ledge)

if the water come to HIM and drown him, he drowns not him-self;

(Greatly pleased with his own logic)

Argal, he that is not guilty of his own death shortens not his own life!

(He goes behind the barricade down into the grave and

prepares to dig)

GRAVEDIGGER2

(Trying to disprove him)

But is this LAW?

Play-Scripts

9

Pla

y-

Scrip

ts

Character Directions

Stage Directions

Stage Directions

Character Directions

Character Directions

of 49

The BaselineP

lay-

Scrip

ts

Hamlet

Act 5, Scene 3

(Graveyard Scene)

Richard Burton in Hamlet, directed by Sir Gielgud

http://www.youtube.com/watch?v=XRU5yLgs0zw&feature=pla

yer_detailpage

10

400 BML Commands

4 hours & 12 minutes

10 minute scene

of 49

11

Pla

y-

Scrip

ts

Sentence

Subject NP Actor/Noun

VP

VP Action/Verb

NP Target/Noun

Example Nouns:

GRAVEDIGGER1

GRAVEDIGGER2

HAMLET

HORATIO

Steps

Grave

Audience

Center stage

Stage left

Example Verbs:

Move to

Follow

Look at

Pick up

Put down

Speak

Point to

Example:

(Pointing down into the grave)

Actor = current speaker

Verb = point

Target = grave

Annotation Parsing

of 49

What did it look like?

12

Pla

y-

Scrip

ts

of 49

How Did We Do?

13

Pla

y-

Scrip

tsH

am

let

Gra

veD

igger1

Ground Truth Simple NLP Method

Character Traces Over Time for Entire Graveyard Scene

C. Talbot and G. M. Youngblood. Spatial Cues in Hamlet. In Proceedings of the 12th International Conference on

Intelligent Virtual Agents, IVA '12, pages 252-259, Berlin, Heidelberg, 2012. Springer-Verlag.

Spatial Rules

14

of 49

What’s Next?

Applying Spatial Rules

Conversational Spatial Rules

Grouping Spatial Rules

Theatre Rules

General Rules

15

Sp

atia

l R

ule

s

E. Sundstrom and I. Altman.

Interpersonal Relationships and Personal Space:

Research Review and Theoretical Model. 1976

Counter-Crossing

of 49

Architecture

16

Sp

atia

l R

ule

s

BML

of 49

Architecture

17

Sp

atia

l R

ule

s

of 49

Rules Engine Logic

18

Sp

atia

l R

ule

s

of 49

Position Results

19

Sp

atia

l R

ule

s

C. Talbot and G. M. Youngblood. Shakespearean Spatial Rules. In Proceedings of the 2013 International

Conference on Autonomous Agents and Multi-agent Systems, AAMAS '13, pages 587-594, Richland, SC, 2013.

of 49

Position Results

20

Sp

atia

l R

ule

s

C. Talbot and G. M. Youngblood. Shakespearean Spatial Rules. In Proceedings of the 2013 International

Conference on Autonomous Agents and Multi-agent Systems, AAMAS '13, pages 587-594, Richland, SC, 2013.

Implied Movements

21

of 49

Grave Digger 1 Initiative

22

Imp

lied

M

vm

t

of 49

Implied Motion

23

Imp

lied

M

vm

t

To be, or not to be—

that is the question:

Whether 'tis nobler in

the mind to suffer ….I should move

towards

the audience

for my monologue

of 49

24

Imp

lied

M

vm

tInformation Captured

For Each Line of Speech:

Movement by Speaker or Other

Character

Number of Lines Spoken Before

Number of Lines Spoken After

Annotation Before

Annotation After

Number of Lines since Last

Movement

Number of Repeated Words

Number of Upper Case Words

Punctuation Counts

Parts of Speech Counts

Type of Movements:

Fighting

Jumping

Gestures

Object Manipulations

Locomotion

Pointing

Posture

Gaze

of 49

25

Imp

lied

M

vm

tMachine Learning

RTextTools in R

Maximum Entropy

Random Forests

Boosting

SVM

Specific Movements

General Movements

Any Movement By Speaker

Any Movement at All

of 49

26

Imp

lied

M

vm

tLearning Combinations

Movement Classifications

Specific Movements

Movement High-Level Categories

Big Movements

Any Movements

N-Gram Sizes

Unigrams

Bigrams

Trigrams

4-grams

5-grams

Training Sizes

Even Split Training / Testing

Even Split of Positive Examples for

Training / Testing

Feature Sets

Text Only

POS Counts Only

POS Counts & Text

POS Counts & Contextual Features

All Features

of 49

27

Imp

lied

M

vm

tEvaluation Criteria

Overall Accuracy

Recall

Precision

F1 score

F0.5 score

Matthews Correlation Coefficient

ROC curves

of 49

28

Imp

lied

M

vm

tEvaluation Criteria

Overall Accuracy

Recall

Precision

F1 score

F0.5 score

Matthews Correlation Coefficient

ROC curves

of 49

29

Imp

lied

M

vm

tEvaluation Criteria

Overall Accuracy

Recall

Precision

F1 score

F0.5 score

Matthews Correlation Coefficient

ROC curves

of 49

Best Performing

30

Imp

lied

M

vm

t

Boosting

SVM

MaxEnt

RandForest

Any Mvmt, POS, Unigrams Any Mvmt, No Text, Unigrams

Gestures, All, 4-grams Any Mvmt, Text, Unigrams

C. Talbot and G. M. Youngblood. Lack of Spatial Indicators in Hamlet. In Florida Artificial Intelligence Research

Society Conference, FLAIRS '13, pages 154-159. Association for the Advancement of Artificial Intelligence, 2013.

of 49

Best Performing

31

Imp

lied

M

vm

t

Boosting

SVM

MaxEnt

RandForest

Random

Any Mvmt, POS, Unigrams Any Mvmt, No Text, Unigrams

Gestures, All, 4-grams Any Mvmt, Text, Unigrams

C. Talbot and G. M. Youngblood. Lack of Spatial Indicators in Hamlet. In Florida Artificial Intelligence Research

Society Conference, FLAIRS '13, pages 154-159. Association for the Advancement of Artificial Intelligence, 2013.

Incorporating Human

Characters

32

of 49

So Far…

33

Hum

ans

Sentence

Subject NP Actor/Noun

VP

VP Action/Verb

NP Target/Noun

Speech Movement

Grouping Spatial Rules

Conversational Spatial Rules

Theatre Rules

General Rules

MindMakers Wiki

http://www.mindmakers.org/

projects/bml-1-0/wiki/Wiki

SmartBody Path Planning

http://smartbody.ict.usc.edu

of 49

Adding a Human

Move correctly, on-time

Move correctly, wrong time

Move incorrectly, on-time

Move incorrectly, wrong time

Don’t move at all

34

Hum

ans

of 49

35

Hum

ans

Equilibrium of Forces

Aesthetically Balanced

Easy to See Nodes

Crossings-Free (some)

Fixed Nodes

Varying Relationships Based on Data

Can be Arranged in Pre-defined Shapes (some)

Force-Directed Graphs (FDGs)

T. M. J. Fruchterman, Edward, and E. M. Reingold. Graph Drawing by Force-

Directed Placement. Software: Practice and Experience, 21(11):1129{1164, 1991.

of 49

Force-Directed Graph Structure

Node Representations:

Characters

Human

Target/Marks/Pawns

Audience

Central Grouping Point

36

Hum

ans

A

H

T

A

H

T

of 49

Force-Directed Graph Structure

Node Representations:

Characters

Human

Target/Marks/Pawns

Audience

Central Grouping Point

Linkages

Characters – Humans/Characters

Characters – Targets/Marks/Pawns

Characters – Audience

Characters – Central Grouping Point

Humans – Central Grouping Point

Central Grouping Point - Audience

Humans - Audience

37

Hum

ans

A

H

T

of 49

Force-Directed Graph Functions

Adding Characters

Characters Leaving

Moving Characters

Human Moves

38

Hum

ans

B

H

T

T

A

of 49

A

Force-Directed Graph Functions

Adding Characters

Characters Leaving

Moving Characters

Human Moves

39

Hum

ans

B

H

T

T

of 49

Force-Directed Graph Functions

Adding Characters

Characters Leaving

Moving Characters

Human Moves

40

Hum

ans

B

H

T

T

A

of 49

Force-Directed Graph Functions

Adding Characters

Characters Leaving

Moving Characters

Human Moves

41

Hum

ans

B

H

T

of 49

42

Hum

ans

Forces and Time

δ= distance between nodes

L = length of stage depth

α = constant

C. Talbot and G. M. Youngblood. Positioning Characters Using Forces. In Proceedings of the Cognitive Agents for

Virtual Environments Workshop (CAVE 2013) collocated with AAMAS (W08). IFAMAAS (International Foundation

for Autonomous Agents and Multi-agent Systems), 2013.

of 49

Evaluation Approaches

Optimal arrangement based on current relationships

Time-based / sequential arrangement through entire scene

User evaluation of appropriate positioning

43

Hum

ans

of 49

Hum

ans

Arrangement Based Upon Relationships

44

of 49

Hum

ans

Arrangement Based Upon Relationships

45

of 49

Hum

ans

Arrangement Based Upon Relationships

46

of 49

47

Hum

ans

Evaluation CriteriaAppropriate arrangement based on current relationships

Even Vertex Distribution

Measure character distances

Small Number of Vertices

Count number of vertices

Fixed Vertices

Measure distance from targets/marks

Centering and Encircling of Groups

Comparison to semi-circular shape

of 49

48

Hum

ans

Results100s of Random Relationship Scenarios

Even Vertex Distribution

3.14 feet (SD=1.54) between characters

Small Number of Vertices

At most 40 vertices in graph, with 12 characters

Fixed Vertices

3.30 feet (SD=1.52) from target

Centering and Encircling of Groups

Characters formed nice semi-circles

C. Talbot and G. M. Youngblood. Application of Force-Directed Graphs on Character Positioning. In Proceedings

of the Spatial Computing Workshop (SCW 2013) collocated with AAMAS (W09), pages 53-58. IFAMAAS

(International Foundation for Autonomous Agents and Multi-agent Systems), 2013.

of 49

Incorporating Forces for Time-

Based Arrangements 49

Hum

ans

of 49

Evaluation Criteria

Occlusion

Clustering

50

Hum

ans

of 49

51

Hum

ans

ResultsC

ase #

Case

Descri

pti

on

Avg

Occlu

sio

n

Ave

rag

e

Clu

ste

rin

g

X Ave

rag

e

Clu

ste

rin

g

Y

0Baseline All AI 3.60% 19.50% 14.60%

1Baseline Human 90% 3.60% 19.10% 15.40%

2Baseline Human 50% 2.90% 20.00% 14.70%

3Baseline Human 10% 4.40% 30.90% 28.70%

4Forces All AI 2.40% 16.80% 14.60%

5Forces Human 90% 2.40% 16.80% 14.60%

6Forces Human 50% 1.60% 20.40% 13.80%

7Forces Human 10% 2.40% 20.80% 14.00%

C. Talbot and G. M. Youngblood. Scene Blocking Utilizing Forces. In Florida Artificial Intelligence Research

Society Conference, FLAIRS '14, pages 91-96. Association for the Advancement of Artificial Intelligence, 2014.

of 49

52

Hum

ans

ResultsC

ase #

Case

Descri

pti

on

Avg

Occlu

sio

n

Ave

rag

e

Clu

ste

rin

g

X Ave

rag

e

Clu

ste

rin

g

Y

0Baseline All AI 3.60% 19.50% 14.60%

1Baseline Human 90% 3.60% 19.10% 15.40%

2Baseline Human 50% 2.90% 20.00% 14.70%

3Baseline Human 10% 4.40% 30.90% 28.70%

4Forces All AI 2.40% 16.80% 14.60%

5Forces Human 90% 2.40% 16.80% 14.60%

6Forces Human 50% 1.60% 20.40% 13.80%

7Forces Human 10% 2.40% 20.80% 14.00%

C. Talbot and G. M. Youngblood. Scene Blocking Utilizing Forces. In Florida Artificial Intelligence Research

Society Conference, FLAIRS '14, pages 91-96. Association for the Advancement of Artificial Intelligence, 2014.

User Studies

53

of 49

Block World 3D Representation

54

Use

r S

tud

ies

of 49

Survey Questions1. Characters showed evidence of engaged listening

2. Characters appeared to perform suitable movements on cue

3. The pace of the performance was too fast

4. The pace of the performance was too slow

5. The use of the space on stage was appropriate

6. The blocking (positioning and timing of the characters) was appropriate

7. There was adequate variety in the staging positions of the characters

8. The characters’ movement onstage during the performance was believable in the context of the performance

9. The performance is free from distracting behavior that does not contribute to the scene

10. The arrangement of the performers appropriately conveys the mood of the scene

11. The character movements provide appropriate dramatic emphasis

12. There is adequate variety and balance in the use of the performance space

13. All visible behaviors appear to be motivated and coordinated within the scene

14. The characters were grouped to give proper emphasis to the right characters at the right time

15. The characters frequently covered or blocked each other from your point of view

16. The movements of the characters were consistent with the play

17. There was a great deal of random movement

18. The characters’ reactions to other characters were believable

19. Characters showed a lack of engagement when listening

20. The arrangement of the performers contradicts the mood of the scene

21. The more prominent characters in the scene were hidden or masked from your view

22. The characters were too close together

23. The characters were too far apart

24. The stage space was not utilized toits full potential

25. All characters were visible fromyour point of view throughout thescene

55

Use

r S

tud

ies

of 49

Results

56

Use

r S

tud

ies

Str

on

gly

Dis

ag

ree

Ne

utr

al

Str

on

gly

Ag

ree

Me

an

of 49

Results

57

Use

r S

tud

ies

Str

on

gly

Dis

ag

ree

Ne

utr

al

Str

on

gly

Ag

ree

Me

an

Planned Future Work

58

of 49

Planned Future Work

Additional User Studies (shortened)

Random

Baseline

NLP

NLP + Rules

NLP + Rules + FDGs

Human Interaction User Studies

Baseline

NLP + Rules + FDGs

Generalization

Identify play-types based on organization

Apply & evaluate techniques for up to 10 of these

59

Pla

nned

Summary

60

of 49

Proposed Play-Scripts

Applied NLP

Added Rules Engine

Evaluated Speech for Implied Movement

Incorporated Human-Controlled Characters

Added FDGs and Algorithms

Created Spatial Performance Evaluation

Initial User Study

Summary

61

Su

mm

ary

of 49

Christine Talbot

ctalbot1@uncc.edu

Questions?

C. Talbot. Creating an Artificially Intelligent Director (AID) for Theatre and Virtual Environments. In Proceedings of the

2013 International Conference on Autonomous Agents and Multi-agent Systems, AAMAS '13, pages 1457-1458,

Richland, SC, 2013. International Foundation for Autonomous Agents and Multi-agent Systems.

C. Talbot and G. M. Youngblood. Spatial Cues in Hamlet. In Proceedings of the 12th International Conference on

Intelligent Virtual Agents, IVA '12, pages 252-259, Berlin, Heidelberg, 2012. Springer-Verlag.

C. Talbot and G. M. Youngblood. Application of Force-Directed Graphs on Character Positioning. In Proceedings of the

Spatial Computing Workshop (SCW 2013) collocated with AAMAS (W09), pages 53-58. IFAMAAS (International

Foundation for Autonomous Agents and Multi-agent Systems), 2013.

C. Talbot and G. M. Youngblood. Lack of Spatial Indicators in Hamlet. In Florida Artificial Intelligence Research Society

Conference, FLAIRS '13, pages 154-159. Association for the Advancement of Artificial Intelligence, 2013.

C. Talbot and G. M. Youngblood. Positioning Characters Using Forces. In Proceedings of the Cognitive Agents for

Virtual Environments Workshop (CAVE 2013) collocated with AAMAS (W08). IFAMAAS (International Foundation for

Autonomous Agents and Multi-agent Systems), 2013.

C. Talbot and G. M. Youngblood. Shakespearean Spatial Rules. In Proceedings of the 2013 International Conference on

Autonomous Agents and Multi-agent Systems, AAMAS '13, pages 587-594, Richland, SC, 2013. International

Foundation for Autonomous Agents and Multi-agent Systems.

C. Talbot and G. M. Youngblood. Scene Blocking Utilizing Forces. In Florida Artificial Intelligence Research Society

Conference, FLAIRS '14, pages 91-96. Association for the Advancement of Artificial Intelligence, 2014.

62

Qu

estio

ns

Selected Bibliography Highlighting This Work