![Page 1: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/1.jpg)
A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture
Arno Hartholt (ICT), Thomas Russ (ISI), David Traum (ICT), Eduard Hovy (ISI), Susan Robinson (ICT) 05/30/2008
![Page 2: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/2.jpg)
2
Overview
Virtual Humans Project
Challenge
Virtual Humans Architecture
Ontology
Conclusion
Future Work
![Page 3: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/3.jpg)
3
Virtual Humans Project
![Page 4: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/4.jpg)
4
Virtual Humans Project
Reasoning andEmotion
Reasoning andEmotion
BehaviorAnimation
BehaviorAnimation
Non-Verbal BehaviorSensing
Non-Verbal BehaviorSensing
Natural LanguageDialogue
Natural LanguageDialogue
Non-Verbal Behavior
Understanding
Non-Verbal Behavior
Understanding
Natural LanguageUnderstanding
Natural LanguageUnderstanding
KnowledgeRepresentation
KnowledgeRepresentation
SpeechProcessing
SpeechProcessing
VirtualHuman
Architecture
VirtualHuman
Architecture
Non-Verbal Behavior
Generation
Non-Verbal Behavior
Generation
Natural LanguageGeneration
Natural LanguageGeneration
ConversationalMinor
Characters
ConversationalMinor
Characters
Reasoning andEmotion
Reasoning andEmotion
BehaviorAnimation
BehaviorAnimation
Non-Verbal BehaviorSensing
Non-Verbal BehaviorSensing
Natural LanguageDialogue
Natural LanguageDialogue
Non-Verbal Behavior
Understanding
Non-Verbal Behavior
Understanding
Natural LanguageUnderstanding
Natural LanguageUnderstanding
KnowledgeRepresentation
KnowledgeRepresentation
SpeechProcessing
SpeechProcessing
VirtualHuman
Architecture
VirtualHuman
Architecture
Non-Verbal Behavior
Generation
Non-Verbal Behavior
Generation
Natural LanguageGeneration
Natural LanguageGeneration
ConversationalMinor
Characters
ConversationalMinor
Characters
![Page 5: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/5.jpg)
5
Challenge – Knowledge Representation
Module-specific representation– How to communicate between modules?– How to make sure translations are correct?
Uniform representation– Common understanding– Reuse– Impoverished or rich?
NLU
Dialogue NLG
Gesture
TaskModeling
Speech
![Page 6: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/6.jpg)
6
Our Process
Multi-phase project life cycle
First, choose individual representations, suitable for the state of the art in that area
Next, bring languages closer together
![Page 7: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/7.jpg)
7
Solution - Old and New Architecture
Old architecture
NL Generation
NL Understanding
Dialogue Management
Action Planning
Emotion Reasoning
Gesture Generation
Speech Recognition
Speech Synthesis
Ontology1Interface1
Ontology2Interface2
Ontology3Interface3
Ontology4Interface4
New architecture
NL Generation
NL Understanding
Dialogue Management
Action Planning
Emotion Reasoning
Gesture Generation
Speech Recognition
Speech Synthesis
Common Ontology
Central Interface
![Page 8: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/8.jpg)
8
Knowledge Representation
Static knowledge:– Offline authoring– Social / psychological behavior– Domain knowledge
Dynamic knowledge:– Runtime – Current state / events– Communication through message protocol
![Page 9: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/9.jpg)
9
Environment
Natural Language
Understanding
Natural Language
Understanding
SpeechRecognition
SpeechRecognition
Non-VerbalBehavior
Generator
Non-VerbalBehavior
Generator
SmartbodyProcedural
Animation Planner
SmartbodyProcedural
Animation Planner
SpeechGeneration
SpeechGeneration
Dialog andDiscourse
Management
Dialog andDiscourse
ManagementEmotionModel
EmotionModel Task Planner
Task Planner
Body
Mind
Real Environment
VisionRecognition
VisionRecognition
VisionUnderstanding
VisionUnderstanding
Visual Game Engine
DomainSpecific
Knowledge
DomainSpecific
Knowledge
DomainIndependentKnowledge
DomainIndependentKnowledge
WorldState
Protocol
WorldState
Protocol
KnowledgeManagement
Intelligent Cognitive Agent
Virtual Humans Architecture
Human Trainee
“I want to move the clinic”
Cognition
Virtual Human
Body and Affective StateManagement
Body and Affective StateManagement
Natural LanguageGeneration
Natural LanguageGeneration
Speech-act: statement
Event: move
Agent: captain
Theme: clinicBody Planning
Speech-act: statement
Polarity: negative
Object: market
Attribute: resource
Value: medical-supplies“We have no medical supplies”
![Page 10: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/10.jpg)
10
Focus
Task model and NLU
Domain dependent knowledge
Strong interaction
Big authoring effort
![Page 11: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/11.jpg)
11
Task Model – Tasks & States
![Page 12: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/12.jpg)
12
Task Model - Plans
![Page 13: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/13.jpg)
13
Task Model - Whole
![Page 14: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/14.jpg)
14
Task Model - Task
defTask doctor-moves {
:agent doctor
:theme clinic
:event move
:source here
:destination there
:instrument locals
:pre {have-transport clinic-here }
:add {clinic-there}
:del {clinic-here}
}
![Page 15: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/15.jpg)
15
Task Model - State
defState clinic-here location clinic here <p> \
:belief true \
:initialize here \
:probability 0.8 \
:concern {doctor 20}
:sim-object *none*
![Page 16: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/16.jpg)
16
Natural Language Understanding (NLU)
<S> i want to move the clinic </S>
<S>.meta.id mcsw
<S>.mood declarative
<S>.sem.speechact.type statement
<S>.sem.modal.desire want
<S>.sem.type event
<S>.sem.event move
<S>.sem.theme clinic
<S>.sem.source here
<S>.sem.destination there
![Page 17: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/17.jpg)
17
Natural Language Understanding (NLU)
Framebank: sets of semantic frame / utterance tuples
NLU trains on framebank
Either building frames or retrieving frames
![Page 18: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/18.jpg)
18
Frames Ontology
Core Concepts
Module-specific knowledge
Module-specific knowledge
uses uses
Exporters
Module 1
Module 2
![Page 19: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/19.jpg)
19
Frames Ontology – Core Concepts
event move
agent captain-kirk
theme clinic
source market
destination downtown
object clinic
attribute location
value downtown
Task:
State:
![Page 20: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/20.jpg)
20
Frames Ontology – Task Model
event moveagent captain-kirktheme clinicsource marketdestination downtownpre: clinic-location-marketdel: clinic-location-marketadd: clinic-location-downtown
object clinicattribute locationvalue downtownbelief falseconcern {doctor-perez 10}
Task:
State:
![Page 21: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/21.jpg)
21
Frames Ontology – Natural Language
mood declarative
sem.speechact.type statement
sem.modality.deontic must
sem.polarity positive
sem.type event
sem.event move
sem.agent captain-kirk
sem.theme clinic
sem.source market
sem.destination downtown
![Page 22: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/22.jpg)
22
OWL
Hierarchy
Assertions on all levels
Automatic classification
![Page 23: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/23.jpg)
23
OWL Ontology
Scenario- independent
Ontology
ScenarioFamily 3
ScenarioFamily 2
Scenario family 1
Scenario 3-1Scenario 2-2Scenario 2-1
Scenario 1-2Scenario 1-1
• General world knowledge
• (generic actions, objects…)
• Linguistic structures
• Generic dialogue items
• Scenario actor classes
• Scenario action templates
• (generic preconds, effects...)
• Propositions for scenario
• Specific actors, locations, etc.
• Sentences for scenario
• Actor positions, attitudes, etc.
• Simulation initializationConfiguration
FileConfiguration
FileConfiguration
file
![Page 24: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/24.jpg)
24
Ontology Action Templates
Templates define semantics of action:Ex: One effect of a move action is that the destination will contain the thing being moved (the ‘theme’)
Templates define semantics of action:Ex: One effect of a move action is that the destination will contain the thing being moved (the ‘theme’)
Hierarchy allows inheritance of semanticsHierarchy allows inheritance of semantics
![Page 25: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/25.jpg)
25
Creating a new move action
1. Select class1. Select class
3. Add details• agent• theme• source• destination
3. Add details• agent• theme• source• destination
2. Create instance2. Create instance
4. Inherit details Preconditions, effects, etc. are added automatically (and can be edited)
4. Inherit details Preconditions, effects, etc. are added automatically (and can be edited)
![Page 26: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/26.jpg)
26
Creating a new move utterance
1. Select class1. Select class
3. Add details• semantic object• speech act• modality• addressee
3. Add details• semantic object• speech act• modality• addressee
2. Create instance2. Create instance
4. Add utterance(s)4. Add utterance(s)
![Page 27: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/27.jpg)
27
Conclusion
Multiple phase life cycle allows quick prototyping
Ontology pros– Module synchronization– Formal specification– Reuse of knowledge– Reference to data, rather than copy– Common user interface
Ontology cons– Extra learning curve– Some changes are more difficult– User interface not ideal
![Page 28: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/28.jpg)
28
Future Work
Extend natural language capabilities (NLG, Lexicon)
Integrating other modules
Completing consistency and integrity testing routines
Exploring using existing resources
We want to enable non-Computer Science people to author new scenarios…so we’re investigating the optimal point in the trade-off between:
– Programming for Non-Experts: Reduced, simple, scripting languages BUT limited in functionality
– Powertools for Experts: High functionality BUT need considerable expertise
Simpler, but limited
More powerful, but complex
![Page 29: A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture Arno Hartholt (ICT), Thomas Russ (ISI),](https://reader036.vdocument.in/reader036/viewer/2022070401/56649f165503460f94c2cd2f/html5/thumbnails/29.jpg)
Questions
Thank you