bringing characters to life for immersive storytelling - dioselin gonzalez
TRANSCRIPT
Bringing characters to life for immersive storytelling
Dioselin GonzalezLead VR Engineer, Unity Labs
VR With The Best - March 19, 2016
Relevant background
Since Oct/2016
VR authoring tools
labs.unity.com
Character animation&
Partially autonomous character system
Networked VR simulation for dismounted soldiers
CAVE™Omnidirectional treadmill
M.S. Computer Graphics Technology
Collaborative virtual reality
InterPlay: Loose Minds in a Box (2005) - anotherlanguage.org/interplay/003_lmib/
Scenario
©2014-2016 AlloyRabbit - http://alloyrabbit.deviantart.com/
Sense-Think-Act-Communicate
M. Siegel, "The sense-think-act paradigm revisited," Robotic Sensing, 2003. ROSE' 03. 1st International Workshop on, 2003, pp. 5 pp.-.doi: 10.1109/ROSE.2003.1218700
Facial recognition
Action Units and the Facial Action Coding System
Shichuan Du et al. Compound facial expressions of emotion. PNAS 2014;111:E1454-E1462. doi: 10.1073/pnas.1322355111
Shichuan Du et al. Compound facial expressions of emotion. PNAS 2014;111:E1454-E1462. doi: 10.1073/pnas.1322355111
MPEG-4 Face andBody Animation (MPEG-4 FBA) [ISO14496] International Standard
Facial Feature Points (FP)
https://visagetechnologies.com/uploads/2012/08/MPEG-4FBAOverview.pdf
Shichuan Du et al. Compound facial expressions of emotion. PNAS 2014;111:E1454-E1462. doi: 10.1073/pnas.1322355111
Facial Feature Points (FP)
Fully Automatic Facial Feature Point Detection Using Gabor Feature Based Boosted Classifiers
Danijela Vukadinovic and Maja PanticDelft University of Technology
2005 IEEE International Conference on Systems, Man and Cybernetics
http://ibug.doc.ic.ac.uk/media/uploads/documents/VukadinovicPantic-SMC05-FINAL.pdf
Open Source Computer Vision Library
http://docs.opencv.org/3.1.0/d7/d8b/tutorial_py_face_detection.html#gsc.tab=0
OpenCV face Detection using Haar Cascades
http://docs.opencv.org/3.1.0/d7/d8b/tutorial_py_face_detection.html#gsc.tab=0
https://facerig.com/
Other facial features
Using Cr-Y Components to Detect Tongue Protrusion Gesturess
Crawford, C.; Bailey, S.; Badea, C.; Gilbert, J.
CHI EA '15 Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems
http://dl.acm.org/citation.cfm?id=2732916
Speech recognition
CMUSphinx toolkitPocketsphinx — lightweight recognizer library written in C.
Sphinxbase — support library required by Pocketsphinx
Sphinx4 — adjustable, modifiable recognizer written in Java
Sphinxtrain — acoustic model training tools
http://cmusphinx.sourceforge.net/
OpenEars®
Offline iPhone Voice Recognition and Text-To-Speech
http://www.politepix.com/openears/
#import <OpenEars/OELanguageModelGenerator.h>#import <OpenEars/OEAcousticModel.h>
- (void)viewDidLoad{
OELanguageModelGenerator *lmGenerator = [[OELanguageModelGenerator alloc] init];NSArray *words = [NSArray arrayWithObjects:@"WORD", @"STATEMENT", @"OTHER WORD", @"A
PHRASE", nil];NSString *name = @"NameIWantForMyLanguageModelFiles";NSError *err = [lmGenerator generateLanguageModelFromArray:words withFilesNamed:name
forAcousticModelAtPath:[OEAcousticModel pathToModel:@"AcousticModelEnglish"]]; NSString *lmPath = nil;NSString *dicPath = nil;if(err == nil) {
lmPath = [lmGenerator pathToSuccessfullyGeneratedLanguageModelWithRequestedName:@"NameIWantForMyLanguageModelFiles"];
dicPath = [lmGenerator pathToSuccessfullyGeneratedDictionaryWithRequestedName:@"NameIWantForMyLanguageModelFiles"];
} else {NSLog(@"Error: %@",[err localizedDescription]);
}}
OpenEars language model
The characters’ “brain”
Finite state machines
http://gamedevelopment.tutsplus.com/tutorials/finite-state-machines-squad-pattern-using-steering-behaviors--gamedev-13638
boost::statechart
http://www.boost.org/doc/libs/release/libs/statechart/doc/tutorial.html
http://www.boost.org/doc/libs/release/libs/statechart/doc/tutorial.html
namespace sc = boost::statechart;
struct EvShutterHalf : sc::event< EvShutterHalf > {};struct EvShutterFull : sc::event< EvShutterFull > {};struct EvShutterRelease : sc::event< EvShutterRelease > {};struct EvConfig : sc::event< EvConfig > {};
struct NotShooting;struct Camera : sc::state_machine< Camera, NotShooting >{ /* ... */};
struct Idle;struct NotShooting : sc::simple_state<NotShooting, Camera, Idle >{ typedef sc::custom_reaction< EvShutterHalf > reactions; sc::result react( const EvShutterHalf & );};
struct Idle : sc::simple_state< Idle, NotShooting >{ typedef sc::custom_reaction< EvConfig > reactions;
// ... sc::result react( const EvConfig & );};
boost::msm
http://www.boost.org/doc/libs/release/libs/msm/doc/HTML/ch03s02.html
http://www.boost.org/doc/libs/release/libs/msm/doc/HTML/ch03s02.html
struct transition_table : mpl::vector<// Start Event Target Action Guard // +---------+------------+-----------+---------------------------+----------------------------+ a_row< Stopped , play , Playing , &player_::start_playback >,a_row< Stopped , open_close , Open , &player_::open_drawer >, _row< Stopped , stop , Stopped >,// +---------+------------+-----------+---------------------------+----------------------------+ a_row< Open , open_close , Empty , &player_::close_drawer >,// +---------+------------+-----------+---------------------------+----------------------------+ a_row< Empty , open_close , Open , &player_::open_drawer >, row< Empty , cd_detected, Stopped , &player_::store_cd_info , &player_::good_disk_format >, row< Empty , cd_detected, Playing , &player_::store_cd_info , &player_::auto_start >,// +---------+------------+-----------+---------------------------+----------------------------+ a_row< Playing , stop , Stopped , &player_::stop_playback >,a_row< Playing , pause , Paused , &player_::pause_playback >,a_row< Playing , open_close , Open , &player_::stop_and_open >,// +---------+------------+-----------+---------------------------+----------------------------+ a_row< Paused , end_pause , Playing , &player_::resume_playback >,a_row< Paused , stop , Stopped , &player_::stop_playback >,a_row< Paused , open_close , Open , &player_::stop_and_open >// +---------+------------+-----------+---------------------------+----------------------------+ > {};
Behavior trees
Image from http://charliepark.tumblr.com/post/97667486/a-sample-behavior-tree-from-spore-from-chris
Behavior trees
Behavior trees
HALO 2 (2004)
GDC 2005 - Handling Complexity in the Halo 2 AI
Image from http://en.wikipedia.org
Behavior trees
Façade: a one-act interactive drama (2005)
GDC 2005 - AI and Interactive Storytelling: How We Can Help Each Other
Image from http://www.macobserver.com/
Behavior trees
SPORE
GDC 2010 - Behavior Trees: Three Ways of Cultivating Strong AI
Image from http://en.wikipedia.org
BT_root
decider
decider
decider
decider
decider
behavior(s)
behavior(s)
behavior(s)
behavior(s)decider
behavior(s)decider
Group decider
Behavior decider
BT_root
decider
decider
decider
decider
decider
behavior(s)
behavior(s)
behavior(s)
behavior(s)decider
behavior(s)decider
id type data
... ... ...
Blackboard
BT_root
decider
decider
decider
decider
decider
behavior(s)
behavior(s)
behavior(s)
behavior(s)decider
behavior(s)decider
bool GroupDecider::decide(Blackboard &);
BT_root
decider
decider
decider
decider
decider
behavior(s)
behavior(s)
behavior(s)
behavior(s)decider
behavior(s)decider
enum Status {RUNNING, SUCCESS, FAILURE};
bool BehaviorDecider::activate(Blackboard &);bool BehaviorDecider::deactivate(Blackboard &);Status BehaviorDecider::tick(Blackboard &);
BT_root
decider
decider
decider
decider
decider
behavior(s)
behavior(s)
behavior(s)
behavior(s)decider
behavior(s)decider
Group decider types - Priority List
BT_root
Flee
Feed_self
behavior(s)
behavior(s)
behavior(s)
eat_food
Find_food
Idle behavior(s)
Group decider types - Priority List
BT_root
Flee
Feed_self
behavior(s)
behavior(s)
behavior(s)
eat_food
Find_food
Idle behavior(s)
? ?
Group decider types - Priority List
BT_root
Flee
Feed_self
behavior(s)
behavior(s)
behavior(s)
eat_food
Find_food
Idle behavior(s)
12:00 pm
? ?
Group decider types - Priority List
BT_root
Flee
Feed_self
behavior(s)
behavior(s)
behavior(s)
eat_food
Find_food
Idle behavior(s)
12:00 pm
? ?
Group decider types - Priority List
BT_root
Flee
Feed_self
behavior(s)
behavior(s)
behavior(s)
eat_food
Find_food
Idle behavior(s)
? ?
Group decider types - Priority List
BT_root
Flee
Feed_self
behavior(s)
behavior(s)
behavior(s)
eat_food
Find_food
Idle behavior(s)
? ?
Group decider types - Priority List
BT_root
Flee
Feed_self
behavior(s)
behavior(s)
behavior(s)
eat_food
Find_food
Idle behavior(s)
? ?
Group decider types - Sequence
BT_root
Mix_ingredients
Bake
behavior(s)
behavior(s)
behavior(s)
Put_food_in
Preheat_oven
Decorate behavior(s)
→ →
Group decider types - Parallel
BT_root
Sing
Dance
behavior(s)
behavior(s)
behavior(s)
Jump
Wave_arms
Idle behavior(s)
→→
?
→→
More group decider types (and decorators)
randomrepeat <x>
timesloop_seq
Behavior Designer @ Unity Asset Store
http://www.opsive.com/assets/BehaviorDesigner/
Concurrency: Resource allocators
http://aigamedev.com/open/article/popular-behavior-tree-design/
Aborting/Enabling subtrees
http://aigamedev.com/open/article/popular-behavior-tree-design/
Hybrid architectures
Horizontal layering
Sensor input
Action output
http://www.inf.ed.ac.uk/teaching/courses/abs/slides/abs05-reactivehybrid.pdf
Hybrid architectures
Horizontal layering Vertical layering
Sensor input
Action output
Sensor input
Action outputOne-pass control
http://www.inf.ed.ac.uk/teaching/courses/abs/slides/abs05-reactivehybrid.pdf
Hybrid architectures
Horizontal layering Vertical layering
Sensor input
Action output
Sensor input
Action output
Sensor input Action output
One-pass control Two-pass control
http://www.inf.ed.ac.uk/teaching/courses/abs/slides/abs05-reactivehybrid.pdf
InteRRaP
http://www.sharprobotica.com/2011/01/interrap-hybrid-architecture-for-robotic-multi-agent-systems/
Behavior realization language(i.e. messaging protocol)
(some) Issues
AI Sound
Animation
Physics
(some) Issues
AI Sound
Animation
Physics
Behavior expression
(some) Issues
AI Sound
Animation
Physics
Timing
(some) Issues
AI Sound
Animation
Physics
Synchronization
(some) Issues
AI Sound
Animation
Physics
Interruption
Behavior Markup Language (BML)
Intent planning Behavior planning Behavior realization
SAIBA framework (Situation, Agent, Intention, Behavior, Animation)
FML BML
Towards a Common Framework for Multimodal Generation: The Behavior Markup Language http://xenia.media.mit.edu/~kris/ftp/BML-IVA-06-KoppEtAl.pdf
Behavior Markup Language (BML)
http://www.mindmakers.org/projects/bml-1-0/wiki
BML: synchronization
<bml id="bml1" xmlns="http://www.bml-initiative.org/bml/bml-1.0" character="Alice">
<pointing id="behavior1" target="blueBox" mode="RIGHT_HAND" start="speech1:start"/>
<speech id="speech1"><text>Look there!</text></speech>
...
<constraint>
<synchronize>
<sync ref="speech1:sync4"/>
<sync ref="beat1:stroke:2"/>
<sync ref="nod1:stroke"/>
</synchronize>
</constraint>
http://www.mindmakers.org/projects/bml-1-0/wiki
BML: behavior elements
Towards a Common Framework for Multimodal Generation: The Behavior Markup Language http://xenia.media.mit.edu/~kris/ftp/BML-IVA-06-KoppEtAl.pdf
What is interactive storytelling anyway?
What is interactive storytelling anyway?
Image from http://ragreynolds.net/review-the-last-of-us-remastered/
More referencesGDC 2005 Handling Complexity in the Halo 2 AI - Gamasutra, GDC Vault.
AiGameDev.com Behavior Trees for Next-Gen Game AI - Part 1, Part 2, Part 3.
University of Denver Video Game AI: Behavior Trees & Blackboards
GDC AI Summit 2010 Behavior Trees: Three Ways of Cultivating Strong AI - GDC Vault, AiGameDev.com
Chatting Up Façade’s AI: 23 Ideas to Talk Your Game Into - AiGameDev.com
Popular Approaches to Behavior Tree Design - AiGameDev.com