voices 2015 - spatial temporal reasoning over play-scripts for artificially intelligent characters
Post on 17-Jul-2015
76 Views
Preview:
TRANSCRIPT
Spatio-Temporal Reasoning Over Play-Scripts for Artificially
Intelligent Characters Christine Talbot
Richard Burton in Hamlet, directed by Sir Gielgud
http://www.youtube.com/watch?v=XRU5yLgs0zw&feature=player_detailpage
Background
2
of 49
Virtual Character Positioning
3
Ba
ck-
gro
un
d
EMA: A process model of appraisal dynamics (Stacy C. Marsella,
Jonathan Gratch), In Journal of Cognitive Systems Research, volume
10, 2009.
Ada and Grace: Toward Realistic and Engaging Virtual Museum Guides (William
Swartout, David Traum, Ron Artstein, Dan Noren, Paul Debevec, Kerry
Bronnenkant, Josh Williams, Anton Leuski, Shrikanth Narayanan, Diane Piepol, H.
Chad Lane, Jacquelyn Morie, Priti Aggarwal, Matt Liewer, Jen-Yuan Chiang, Jillian
Gerten, Selina Chu, Kyle White), In Proceedings of the 10th International
Conference on Intelligent Virtual Agents (IVA 2010), 2010.
of 49
Mocap Files and Hand-Coding
4
Ba
ck-
gro
un
d
Discovery News – Avatar: Motion Capture Mirrors Emotions
http://news.discovery.com/videos/avatar-making-the-movie/
of 49
BML and BML Realizers
5
Ba
ck-
gro
un
d
SmartBody Path Planning
http://smartbody.ict.usc.edu
Hamlet played by robots
Unity using SmartBody
MindMakers Wiki
http://www.mindmakers.org/
projects/bml-1-0/wiki/Wiki
of 49<act><participant id="GRAVEDIGGER2" role="actor" /><bml><gesture lexeme="POINT" target="GRAVEDIGGER1"
/></bml></act>
<act><participant id="GRAVEDIGGER1" role="actor" /><bml><speech id="sp1" ref="“ type="application/ssml+xml">Give me
leave!</speech></bml></act>
<act><participant id="GRAVEDIGGER1" role="actor" /><bml><gesture lexeme="POINT" target="GRAVEDIGGER2"
/></bml></act>
<act><participant id="GRAVEDIGGER1" role="actor" /><bml><gesture lexeme="POINT" target="GRAVE" /></bml></act>
<act><participant id="GRAVEDIGGER1" role="actor" /><bml><speech id="sp1" ref="" type="application/ssml+xml"> Here lies
the water -- good? </speech></bml></act>
<act><participant id="GRAVEDIGGER1" role="actor" /><bml><gesture lexeme="POINT" target="GRAVE" /></bml></act>
<act><participant id="GRAVEDIGGER1" role="actor" /><bml><speech id="sp1" ref="" type="application/ssml+xml"> Here
stands the man -- good! </speech></bml></act>
<act><participant id="GRAVEDIGGER1" role="actor" /><bml><speech id="sp1" ref="" type="application/ssml+xml"> If the
man go to this water and drown himself, it is willynilly he goes, mark you that! But, </speech></bml></act>
<act><participant id="GRAVEDIGGER1" role="actor" /><bml><gesture lexeme="POINT" target="GRAVE" /></bml></act>
<act><participant id="GRAVEDIGGER1" role="actor" /><bml><speech id="sp1" ref="" type="application/ssml+xml">if the
water come to HIM and drown him, he drowns not him-self; Argal, he that is not guilty of his own death
shortens not his own life!</speech></bml></act>
<act><participant id="GRAVEDIGGER1" role="actor" /><bml><locomotion target="GRAVE" type="basic" manner="walk"
/></bml></act>
<act><participant id="GRAVEDIGGER2" role="actor" /><bml><speech id="sp1" ref="" type="application/ssml+xml">But is
this LAW ?</speech></bml></act>
Still a Lot of Work…
6
Ba
ck-
gro
un
d
4 hours & 12 minutes for a 10 minute scene!!
Point
Speak
Move
of 49
So How Do We Do It?
7
Ba
ck-
gro
un
d
A: Excuse me…
B: Can I help you?
A: Yes, where is the post office?
B: Go straight and turn left.
A: Where do I turn left?
B: Turn left at the bus stop -
you can’t miss it.
A: Thank you very much!
B: No problem.
Play-Scripts
8
of 49
GRAVEDIGGER1
Give me leave!
(GRAVEDIGGER2 sits on the side steps)
(Pointing down into the grave)
Here lies the water--good?
(Pointing to the table ledge)
Here stands the man-good!
(Illustrating each point literally with his hands)
If the man go to this water and drown himself, it is willy-nilly he goes,
mark you that! But,
(Pointing first to the grave, then to the ledge)
if the water come to HIM and drown him, he drowns not him-self;
(Greatly pleased with his own logic)
Argal, he that is not guilty of his own death shortens not his own life!
(He goes behind the barricade down into the grave and
prepares to dig)
GRAVEDIGGER2
(Trying to disprove him)
But is this LAW?
Play-Scripts
9
Pla
y-
Scrip
ts
Character Directions
Stage Directions
Stage Directions
Character Directions
Character Directions
of 49
The BaselineP
lay-
Scrip
ts
Hamlet
Act 5, Scene 3
(Graveyard Scene)
Richard Burton in Hamlet, directed by Sir Gielgud
http://www.youtube.com/watch?v=XRU5yLgs0zw&feature=pla
yer_detailpage
10
400 BML Commands
4 hours & 12 minutes
10 minute scene
of 49
11
Pla
y-
Scrip
ts
Sentence
Subject NP Actor/Noun
VP
VP Action/Verb
NP Target/Noun
Example Nouns:
GRAVEDIGGER1
GRAVEDIGGER2
HAMLET
HORATIO
Steps
Grave
Audience
Center stage
Stage left
Example Verbs:
Move to
Follow
Look at
Pick up
Put down
Speak
Point to
Example:
(Pointing down into the grave)
Actor = current speaker
Verb = point
Target = grave
Annotation Parsing
of 49
What did it look like?
12
Pla
y-
Scrip
ts
of 49
How Did We Do?
13
Pla
y-
Scrip
tsH
am
let
Gra
veD
igger1
Ground Truth Simple NLP Method
Character Traces Over Time for Entire Graveyard Scene
C. Talbot and G. M. Youngblood. Spatial Cues in Hamlet. In Proceedings of the 12th International Conference on
Intelligent Virtual Agents, IVA '12, pages 252-259, Berlin, Heidelberg, 2012. Springer-Verlag.
Spatial Rules
14
of 49
What’s Next?
Applying Spatial Rules
Conversational Spatial Rules
Grouping Spatial Rules
Theatre Rules
General Rules
15
Sp
atia
l R
ule
s
E. Sundstrom and I. Altman.
Interpersonal Relationships and Personal Space:
Research Review and Theoretical Model. 1976
Counter-Crossing
of 49
Architecture
16
Sp
atia
l R
ule
s
BML
of 49
Architecture
17
Sp
atia
l R
ule
s
of 49
Rules Engine Logic
18
Sp
atia
l R
ule
s
of 49
Position Results
19
Sp
atia
l R
ule
s
C. Talbot and G. M. Youngblood. Shakespearean Spatial Rules. In Proceedings of the 2013 International
Conference on Autonomous Agents and Multi-agent Systems, AAMAS '13, pages 587-594, Richland, SC, 2013.
of 49
Position Results
20
Sp
atia
l R
ule
s
C. Talbot and G. M. Youngblood. Shakespearean Spatial Rules. In Proceedings of the 2013 International
Conference on Autonomous Agents and Multi-agent Systems, AAMAS '13, pages 587-594, Richland, SC, 2013.
Implied Movements
21
of 49
Grave Digger 1 Initiative
22
Imp
lied
M
vm
t
of 49
Implied Motion
23
Imp
lied
M
vm
t
To be, or not to be—
that is the question:
Whether 'tis nobler in
the mind to suffer ….I should move
towards
the audience
for my monologue
of 49
24
Imp
lied
M
vm
tInformation Captured
For Each Line of Speech:
Movement by Speaker or Other
Character
Number of Lines Spoken Before
Number of Lines Spoken After
Annotation Before
Annotation After
Number of Lines since Last
Movement
Number of Repeated Words
Number of Upper Case Words
Punctuation Counts
Parts of Speech Counts
Type of Movements:
Fighting
Jumping
Gestures
Object Manipulations
Locomotion
Pointing
Posture
Gaze
of 49
25
Imp
lied
M
vm
tMachine Learning
RTextTools in R
Maximum Entropy
Random Forests
Boosting
SVM
Specific Movements
General Movements
Any Movement By Speaker
Any Movement at All
of 49
26
Imp
lied
M
vm
tLearning Combinations
Movement Classifications
Specific Movements
Movement High-Level Categories
Big Movements
Any Movements
N-Gram Sizes
Unigrams
Bigrams
Trigrams
4-grams
5-grams
Training Sizes
Even Split Training / Testing
Even Split of Positive Examples for
Training / Testing
Feature Sets
Text Only
POS Counts Only
POS Counts & Text
POS Counts & Contextual Features
All Features
of 49
27
Imp
lied
M
vm
tEvaluation Criteria
Overall Accuracy
Recall
Precision
F1 score
F0.5 score
Matthews Correlation Coefficient
ROC curves
of 49
28
Imp
lied
M
vm
tEvaluation Criteria
Overall Accuracy
Recall
Precision
F1 score
F0.5 score
Matthews Correlation Coefficient
ROC curves
of 49
29
Imp
lied
M
vm
tEvaluation Criteria
Overall Accuracy
Recall
Precision
F1 score
F0.5 score
Matthews Correlation Coefficient
ROC curves
of 49
Best Performing
30
Imp
lied
M
vm
t
Boosting
SVM
MaxEnt
RandForest
Any Mvmt, POS, Unigrams Any Mvmt, No Text, Unigrams
Gestures, All, 4-grams Any Mvmt, Text, Unigrams
C. Talbot and G. M. Youngblood. Lack of Spatial Indicators in Hamlet. In Florida Artificial Intelligence Research
Society Conference, FLAIRS '13, pages 154-159. Association for the Advancement of Artificial Intelligence, 2013.
of 49
Best Performing
31
Imp
lied
M
vm
t
Boosting
SVM
MaxEnt
RandForest
Random
Any Mvmt, POS, Unigrams Any Mvmt, No Text, Unigrams
Gestures, All, 4-grams Any Mvmt, Text, Unigrams
C. Talbot and G. M. Youngblood. Lack of Spatial Indicators in Hamlet. In Florida Artificial Intelligence Research
Society Conference, FLAIRS '13, pages 154-159. Association for the Advancement of Artificial Intelligence, 2013.
Incorporating Human
Characters
32
of 49
So Far…
33
Hum
ans
Sentence
Subject NP Actor/Noun
VP
VP Action/Verb
NP Target/Noun
Speech Movement
Grouping Spatial Rules
Conversational Spatial Rules
Theatre Rules
General Rules
MindMakers Wiki
http://www.mindmakers.org/
projects/bml-1-0/wiki/Wiki
SmartBody Path Planning
http://smartbody.ict.usc.edu
of 49
Adding a Human
Move correctly, on-time
Move correctly, wrong time
Move incorrectly, on-time
Move incorrectly, wrong time
Don’t move at all
34
Hum
ans
of 49
35
Hum
ans
Equilibrium of Forces
Aesthetically Balanced
Easy to See Nodes
Crossings-Free (some)
Fixed Nodes
Varying Relationships Based on Data
Can be Arranged in Pre-defined Shapes (some)
Force-Directed Graphs (FDGs)
T. M. J. Fruchterman, Edward, and E. M. Reingold. Graph Drawing by Force-
Directed Placement. Software: Practice and Experience, 21(11):1129{1164, 1991.
of 49
Force-Directed Graph Structure
Node Representations:
Characters
Human
Target/Marks/Pawns
Audience
Central Grouping Point
36
Hum
ans
A
H
T
A
H
T
of 49
Force-Directed Graph Structure
Node Representations:
Characters
Human
Target/Marks/Pawns
Audience
Central Grouping Point
Linkages
Characters – Humans/Characters
Characters – Targets/Marks/Pawns
Characters – Audience
Characters – Central Grouping Point
Humans – Central Grouping Point
Central Grouping Point - Audience
Humans - Audience
37
Hum
ans
A
H
T
of 49
Force-Directed Graph Functions
Adding Characters
Characters Leaving
Moving Characters
Human Moves
38
Hum
ans
B
H
T
T
A
of 49
A
Force-Directed Graph Functions
Adding Characters
Characters Leaving
Moving Characters
Human Moves
39
Hum
ans
B
H
T
T
of 49
Force-Directed Graph Functions
Adding Characters
Characters Leaving
Moving Characters
Human Moves
40
Hum
ans
B
H
T
T
A
of 49
Force-Directed Graph Functions
Adding Characters
Characters Leaving
Moving Characters
Human Moves
41
Hum
ans
B
H
T
of 49
42
Hum
ans
Forces and Time
δ= distance between nodes
L = length of stage depth
α = constant
C. Talbot and G. M. Youngblood. Positioning Characters Using Forces. In Proceedings of the Cognitive Agents for
Virtual Environments Workshop (CAVE 2013) collocated with AAMAS (W08). IFAMAAS (International Foundation
for Autonomous Agents and Multi-agent Systems), 2013.
of 49
Evaluation Approaches
Optimal arrangement based on current relationships
Time-based / sequential arrangement through entire scene
User evaluation of appropriate positioning
43
Hum
ans
of 49
Hum
ans
Arrangement Based Upon Relationships
44
of 49
Hum
ans
Arrangement Based Upon Relationships
45
of 49
Hum
ans
Arrangement Based Upon Relationships
46
of 49
47
Hum
ans
Evaluation CriteriaAppropriate arrangement based on current relationships
Even Vertex Distribution
Measure character distances
Small Number of Vertices
Count number of vertices
Fixed Vertices
Measure distance from targets/marks
Centering and Encircling of Groups
Comparison to semi-circular shape
of 49
48
Hum
ans
Results100s of Random Relationship Scenarios
Even Vertex Distribution
3.14 feet (SD=1.54) between characters
Small Number of Vertices
At most 40 vertices in graph, with 12 characters
Fixed Vertices
3.30 feet (SD=1.52) from target
Centering and Encircling of Groups
Characters formed nice semi-circles
C. Talbot and G. M. Youngblood. Application of Force-Directed Graphs on Character Positioning. In Proceedings
of the Spatial Computing Workshop (SCW 2013) collocated with AAMAS (W09), pages 53-58. IFAMAAS
(International Foundation for Autonomous Agents and Multi-agent Systems), 2013.
of 49
Incorporating Forces for Time-
Based Arrangements 49
Hum
ans
of 49
Evaluation Criteria
Occlusion
Clustering
50
Hum
ans
of 49
51
Hum
ans
ResultsC
ase #
Case
Descri
pti
on
Avg
Occlu
sio
n
Ave
rag
e
Clu
ste
rin
g
X Ave
rag
e
Clu
ste
rin
g
Y
0Baseline All AI 3.60% 19.50% 14.60%
1Baseline Human 90% 3.60% 19.10% 15.40%
2Baseline Human 50% 2.90% 20.00% 14.70%
3Baseline Human 10% 4.40% 30.90% 28.70%
4Forces All AI 2.40% 16.80% 14.60%
5Forces Human 90% 2.40% 16.80% 14.60%
6Forces Human 50% 1.60% 20.40% 13.80%
7Forces Human 10% 2.40% 20.80% 14.00%
C. Talbot and G. M. Youngblood. Scene Blocking Utilizing Forces. In Florida Artificial Intelligence Research
Society Conference, FLAIRS '14, pages 91-96. Association for the Advancement of Artificial Intelligence, 2014.
of 49
52
Hum
ans
ResultsC
ase #
Case
Descri
pti
on
Avg
Occlu
sio
n
Ave
rag
e
Clu
ste
rin
g
X Ave
rag
e
Clu
ste
rin
g
Y
0Baseline All AI 3.60% 19.50% 14.60%
1Baseline Human 90% 3.60% 19.10% 15.40%
2Baseline Human 50% 2.90% 20.00% 14.70%
3Baseline Human 10% 4.40% 30.90% 28.70%
4Forces All AI 2.40% 16.80% 14.60%
5Forces Human 90% 2.40% 16.80% 14.60%
6Forces Human 50% 1.60% 20.40% 13.80%
7Forces Human 10% 2.40% 20.80% 14.00%
C. Talbot and G. M. Youngblood. Scene Blocking Utilizing Forces. In Florida Artificial Intelligence Research
Society Conference, FLAIRS '14, pages 91-96. Association for the Advancement of Artificial Intelligence, 2014.
User Studies
53
of 49
Block World 3D Representation
54
Use
r S
tud
ies
of 49
Survey Questions1. Characters showed evidence of engaged listening
2. Characters appeared to perform suitable movements on cue
3. The pace of the performance was too fast
4. The pace of the performance was too slow
5. The use of the space on stage was appropriate
6. The blocking (positioning and timing of the characters) was appropriate
7. There was adequate variety in the staging positions of the characters
8. The characters’ movement onstage during the performance was believable in the context of the performance
9. The performance is free from distracting behavior that does not contribute to the scene
10. The arrangement of the performers appropriately conveys the mood of the scene
11. The character movements provide appropriate dramatic emphasis
12. There is adequate variety and balance in the use of the performance space
13. All visible behaviors appear to be motivated and coordinated within the scene
14. The characters were grouped to give proper emphasis to the right characters at the right time
15. The characters frequently covered or blocked each other from your point of view
16. The movements of the characters were consistent with the play
17. There was a great deal of random movement
18. The characters’ reactions to other characters were believable
19. Characters showed a lack of engagement when listening
20. The arrangement of the performers contradicts the mood of the scene
21. The more prominent characters in the scene were hidden or masked from your view
22. The characters were too close together
23. The characters were too far apart
24. The stage space was not utilized toits full potential
25. All characters were visible fromyour point of view throughout thescene
55
Use
r S
tud
ies
of 49
Results
56
Use
r S
tud
ies
Str
on
gly
Dis
ag
ree
Ne
utr
al
Str
on
gly
Ag
ree
Me
an
of 49
Results
57
Use
r S
tud
ies
Str
on
gly
Dis
ag
ree
Ne
utr
al
Str
on
gly
Ag
ree
Me
an
Planned Future Work
58
of 49
Planned Future Work
Additional User Studies (shortened)
Random
Baseline
NLP
NLP + Rules
NLP + Rules + FDGs
Human Interaction User Studies
Baseline
NLP + Rules + FDGs
Generalization
Identify play-types based on organization
Apply & evaluate techniques for up to 10 of these
59
Pla
nned
Summary
60
of 49
Proposed Play-Scripts
Applied NLP
Added Rules Engine
Evaluated Speech for Implied Movement
Incorporated Human-Controlled Characters
Added FDGs and Algorithms
Created Spatial Performance Evaluation
Initial User Study
Summary
61
Su
mm
ary
of 49
Christine Talbot
ctalbot1@uncc.edu
Questions?
C. Talbot. Creating an Artificially Intelligent Director (AID) for Theatre and Virtual Environments. In Proceedings of the
2013 International Conference on Autonomous Agents and Multi-agent Systems, AAMAS '13, pages 1457-1458,
Richland, SC, 2013. International Foundation for Autonomous Agents and Multi-agent Systems.
C. Talbot and G. M. Youngblood. Spatial Cues in Hamlet. In Proceedings of the 12th International Conference on
Intelligent Virtual Agents, IVA '12, pages 252-259, Berlin, Heidelberg, 2012. Springer-Verlag.
C. Talbot and G. M. Youngblood. Application of Force-Directed Graphs on Character Positioning. In Proceedings of the
Spatial Computing Workshop (SCW 2013) collocated with AAMAS (W09), pages 53-58. IFAMAAS (International
Foundation for Autonomous Agents and Multi-agent Systems), 2013.
C. Talbot and G. M. Youngblood. Lack of Spatial Indicators in Hamlet. In Florida Artificial Intelligence Research Society
Conference, FLAIRS '13, pages 154-159. Association for the Advancement of Artificial Intelligence, 2013.
C. Talbot and G. M. Youngblood. Positioning Characters Using Forces. In Proceedings of the Cognitive Agents for
Virtual Environments Workshop (CAVE 2013) collocated with AAMAS (W08). IFAMAAS (International Foundation for
Autonomous Agents and Multi-agent Systems), 2013.
C. Talbot and G. M. Youngblood. Shakespearean Spatial Rules. In Proceedings of the 2013 International Conference on
Autonomous Agents and Multi-agent Systems, AAMAS '13, pages 587-594, Richland, SC, 2013. International
Foundation for Autonomous Agents and Multi-agent Systems.
C. Talbot and G. M. Youngblood. Scene Blocking Utilizing Forces. In Florida Artificial Intelligence Research Society
Conference, FLAIRS '14, pages 91-96. Association for the Advancement of Artificial Intelligence, 2014.
62
Qu
estio
ns
Selected Bibliography Highlighting This Work
top related