1 integrating learning in interactive gaming simulators david w. aha 1 & matthew molineaux 2 1...
TRANSCRIPT
1Integrating Learning in Interactive Gaming Simulators
Integrating Learning in Interactive Gaming Simulators
David W. Aha1 & Matthew Molineaux2
1Intelligent Decision Aids GroupNavy Center for Applied Research in AI
Naval Research Laboratory; Washington, DC2ITT Industries; AES Division; Alexandria, VA
AAAI’04 Workshop on Challenges in Game AI25 July 2004
2Integrating Learning in Interactive Gaming Simulators
Outline
1. Motivation: Learning in cognitive systems2. Objectives:
– Support empirical studies w/ simulators– Encourage studies that address industry
& military M&S concerns3. Design: TIELT functionality & components4. Example: Knowledge base content5. Status: Implementation, collaborations6. Summary
1. Motivation: Learning in cognitive systems2. Objectives:
– Support empirical studies w/ simulators– Encourage studies that address industry
& military M&S concerns3. Design: TIELT functionality & components4. Example: Knowledge base content5. Status: Implementation, collaborations6. Summary
Thanks to our sponsor:
3Integrating Learning in Interactive Gaming Simulators
Rough Anatomy of a Cognitive Agent
External Environment External Environment
Communication(language,gesture,image)
Prediction,planning
Deliberative Processes
Reflective Processes
Reactive Processes
Perception Action
STM
Sensors Effectors
Other reasoning
LTM
Concepts
SentencesCog
nit
ive
Ag
en
t
Affect
Attention
Learning
Learning
(Brachman, 2003)
4Integrating Learning in Interactive Gaming Simulators
Problem
Status of Cognitive Learning
Few deployed cognitive systems integrate techniques that exhibit rapid & enduring learning behavior on complex tasks
– It’s costly to integrate & evaluate embedded learning techniques
Few deployed cognitive systems integrate techniques that exhibit rapid & enduring learning behavior on complex tasks
– It’s costly to integrate & evaluate embedded learning techniques
Complication
The ML research community has been focusing on:¬Rapid: Knowledge poor algorithms
¬Enduring: Learning over a short time period
¬Embedded: Stand-alone evaluations
The ML research community has been focusing on:¬Rapid: Knowledge poor algorithms
¬Enduring: Learning over a short time period
¬Embedded: Stand-alone evaluations
5Integrating Learning in Interactive Gaming Simulators
Wanted: A New Interface (thanks to W. Cohen, others)
Supervised Learning ML
SystemDatabase Interface
(standard format)
(e.g., UCI Repository)
ReasoningSystem
Supervised Learning ML
SystemDatabase Interface
(standard format)
(e.g., UCI Repository)
ReasoningSystem
Supervised Learning ML
Systemj
DatabaseiInterface
(standard format)
(e.g., UCI Repository of ML Databases)
ReasoningSystemk
Cognitive Learning Reasoning Modules
Wor
ld(S
imu
late
d/R
ea
l)
Sensors ML Module
Interface(standard API)
ML Module
ML Module(e.g., TIELT)Effectors
Cognitive Learning Reasoning Modules
Wor
ld(S
imu
late
d/R
ea
l)
Sensors ML Module
Interface(standard API)
ML Module
ML Module(e.g., TIELT)Effectors
Cognitive Learning Reasoning Systemk
Wor
ldi
(Sim
ula
ted/
Re
al)
Sensors ML Module
Interface(standard API)
ML Module
ML Modulej
(e.g., TIELT)Effectors
Testbed for Integrating and Evaluating Learning TechniquesTestbed for Integrating and Evaluating Learning Techniques
6Integrating Learning in Interactive Gaming Simulators
Objectives & Domain
Objective
Facilitate the evaluation of learning techniques in CogSys1. Develop & transition TIELT to help reduce integration costs (time, $)2. Support DARPA Challenge Problems on Cognitive Learning3. Demonstrate research utility prior to approaching industry/military
Facilitate the evaluation of learning techniques in CogSys1. Develop & transition TIELT to help reduce integration costs (time, $)2. Support DARPA Challenge Problems on Cognitive Learning3. Demonstrate research utility prior to approaching industry/military
Domain: Why interactive gaming simulators?
1. Available implementations (cheap to acquire & run)2. Challenging problems for CogSys/ML research 3. Significant interest (military, industry, academia, funding, public)
1. Available implementations (cheap to acquire & run)2. Challenging problems for CogSys/ML research 3. Significant interest (military, industry, academia, funding, public)
7Integrating Learning in Interactive Gaming Simulators
TIELT Specification
1. Simplifies integration & evaluation!• Learning-embedded reasoning systems & gaming simulators• Inputs: 5 descriptions (simulator I/O, game model, learning &
performance tasks, reasoning system I/O, & evaluation strategy)• Free
2. Learning foci: Many• Task (e.g., learn how to execute, or advise on, a task)• Player (e.g., learn/predict a human player’s strategies)• Game (e.g., learn/refine its objects, their relations, & behaviors)
3. Learning methods: Many types• Supervised/unsupervised, immediate/delayed feedback, analytic,
active/passive, online/offline, direct/indirect, automated/interactive• Learning results should be available for inspection
4. Gaming simulators: Those with challenging learning tasks5. Reuse: Provide access to libraries of simulators & reasoning systems
• Abstracts interface definitions from game & task models
1. Simplifies integration & evaluation!• Learning-embedded reasoning systems & gaming simulators• Inputs: 5 descriptions (simulator I/O, game model, learning &
performance tasks, reasoning system I/O, & evaluation strategy)• Free
2. Learning foci: Many• Task (e.g., learn how to execute, or advise on, a task)• Player (e.g., learn/predict a human player’s strategies)• Game (e.g., learn/refine its objects, their relations, & behaviors)
3. Learning methods: Many types• Supervised/unsupervised, immediate/delayed feedback, analytic,
active/passive, online/offline, direct/indirect, automated/interactive• Learning results should be available for inspection
4. Gaming simulators: Those with challenging learning tasks5. Reuse: Provide access to libraries of simulators & reasoning systems
• Abstracts interface definitions from game & task models
8Integrating Learning in Interactive Gaming Simulators
Distinguishing TIELT
System Focus $ Game Engine(s)
Prominent Feature
Reasoning Activity
DirectIA (MASA)
AI SDK FPS, RTS, etc.
Behavior authoring Sense-act, …
SimBionic (SHAI)
AI SDK FPS, etc. Behavior authoring Sense-act, …
FEAR AI SDK Quake 2, etc. Behavior authoring Sense-act, …
RoboCup Research Testbed
RoboCup Soccer game play Sense-act, coaching, etc.
GameBots Research Testbed
UT (FPS) UT game play Sense-act
ORTS Research Testbed
RTS games Hack-free MM RTS Sense-act, strategy
TIELT Research Testbed
Several genres
Experimentation for Learning Systems
Sense-act, advice processing, prediction, model updating, etc.
1. Provides an interface for message-passing interfaces2. Supports composable system-level interfaces
9Integrating Learning in Interactive Gaming Simulators
TIELT’sInternal
CommunicationModules
TIELT’s KBEditors
TIELT’s KBEditors
Selected/Developed Knowledge BasesSelected/Developed Knowledge Bases
GameModel
Description
Task Descriptions
GameInterface
Description
ReasoningInterface
Description
EvaluationMethodology Description
GamePlayer(s)
GameEngineLibrary
GameEngineLibrary
Stratagus
Full Spectrum Command
TIELT’s User Interface TIELT’s User Interface
PredictionInterface
EvaluationInterface
CoordinationInterface
AdviceInterface
TIELTUser
TIELTUser
SelectedGameEngine
SelectedGameEngine
ReasoningSystemLibrary
ReasoningSystemLibrary
ReasoningSystem
ReasoningSystem
Learning Module
. . .
Learning Module
ReasoningSystem
ReasoningSystem
Learning Module
. . .
Learning Module
ReasoningSystem
ReasoningSystem
Learning Module
. . .
Learning Module
SelectedReasoning
System
SelectedReasoning
System
Learned Knowledge
(inspectable)
TIELT: Integration Architecture
Knowledge Base Libraries
Knowledge Base Libraries
GIDGID
GID
RIDRID
RID
GMDGMD
GMD
TDsTDs
TDs
EMDEMD
EMD
...
10Integrating Learning in Interactive Gaming Simulators
TIELT’s Knowledge Bases
GameModel
Description
Task Descriptions
GameInterface
Description
ReasoningInterface
Description
EvaluationMethodology Description
Defines communication processes with the game engine
Defines communication processes with the learning system
Defines interpretation of the game• e.g., initial state, classes, operators, behaviors (rules)• Behaviors could be used to provide constraints on learning
Defines the selected learning and performance tasks• Selected from the game model description
Defines the empirical evaluation to conduct
11Integrating Learning in Interactive Gaming Simulators
TIELT’sInternal
CommunicationModules
TIELT’s KBEditors
TIELT’s KBEditors
Selected/Developed Knowledge BasesSelected/Developed Knowledge Bases
GameModel
Description
Task Descriptions
GameInterface
Description
ReasoningInterface
Description
EvaluationMethodology Description
GamePlayer(s)
GameEngineLibrary
GameEngineLibrary
Stratagus
Full Spectrum Command
TIELT’s User Interface TIELT’s User Interface
PredictionInterface
EvaluationInterface
CoordinationInterface
AdviceInterface
TIELTUser
TIELTUser
SelectedGame
Engine
SelectedGame
Engine
ReasoningSystemLibrary
ReasoningSystemLibrary
ReasoningSystem
ReasoningSystem
Learning Module
. . .
Learning Module
ReasoningSystem
ReasoningSystem
Learning Module
. . .
Learning Module
ReasoningSystem
ReasoningSystem
Learning Module
. . .
Learning Module
SelectedReasoning
System
SelectedReasoning
System
Learned Knowledge
(inspectable)
Example: Controlling a Game Character
Knowledge Base Libraries
Knowledge Base Libraries
GIDGID
GID
RIDRID
RID
GMDGMD
GMD
TDsTDs
TDs
EMDEMD
EMD
Raw StateProcessed
State
DecisionAction
12Integrating Learning in Interactive Gaming Simulators
TIELT’sInternal
CommunicationModules
TIELT’s KBEditors
TIELT’s KBEditors
Selected/Developed Knowledge BasesSelected/Developed Knowledge Bases
GameModel
Description
Task Descriptions
GameInterface
Description
ReasoningInterface
Description
EvaluationMethodology Description
GamePlayer(s)
GameEngineLibrary
GameEngineLibrary
Stratagus
Full Spectrum Command
TIELT’s User Interface TIELT’s User Interface
PredictionInterface
EvaluationInterface
CoordinationInterface
AdviceInterface
TIELTUser
TIELTUser
SelectedGame
Engine
SelectedGame
Engine
ReasoningSystemLibrary
ReasoningSystemLibrary
ReasoningSystem
ReasoningSystem
Learning Module
. . .
Learning Module
ReasoningSystem
ReasoningSystem
Learning Module
. . .
Learning Module
ReasoningSystem
ReasoningSystem
Learning Module
. . .
Learning Module
SelectedReasoning
System
SelectedReasoning
System
Learned Knowledge
(inspectable)
Example: Updating a Game Model
Knowledge Base Libraries
Knowledge Base Libraries
GIDGID
GID
RIDRID
RID
GMDGMD
GMD
TDsTDs
TDs
EMDEMD
EMD
Raw StateProcessed
State
Edit
Edit
13Integrating Learning in Interactive Gaming Simulators
SelectedGame
Engine
EvaluationEditor
Game InterfaceEditor
Percepts
User
ReasoningInterface Editor
Game ModelEditor
TaskEditor
GameModel
Description
TaskDescriptions
Pe
rf.
Ta
sk
EvaluationInterface
Evaluator
Action / Control
Translator(Mapper)
Learning OutputsActions
Translated Model (Subset)
Learning Task
GameInterface
Description
ReasoningInterface
Description
LearningTranslator(Mapper)
CurrentState
ModelUpdater
Database
EvaluationMethodologyDescription
StoredState
AdviceInterface Database
EngineState
Controller
SelectedReasoning
System
TIELT’s Internal Communication Modules
14Integrating Learning in Interactive Gaming Simulators
Sensing the Game State(City placement example)
TIELT
Game Interface
Editor
Sensors
User
Game ModelEditor
GameModel
Description
Up
da
tes
GameInterface
Description
ActionTranslator
Actions
GameEngine
GameEngine
CurrentState
1
2
4 3
4
In Game Engine, the game begins; a colony pod is created and placed.
1
The Game Engine sends a “See” sensor message identifying the pod’s location.
This message template provides updates (instructions) to the Current State, telling it that there is a pod at the location See describes.
4
2
The Model Updater receives the sensor message and finds the corresponding message template in the Game Interface Description.
3
ControllerModel
Updater
3
The Model Updater notifies the Controller that the See action event has occurred.
5
5
15Integrating Learning in Interactive Gaming Simulators
Getting Decisions from the Learning System(City placement example)
TIELT
SelectedReasoning
System
SelectedReasoning
System
LearningModule #1
LearningModule #n
. . .
User
Learning InterfaceEditor
AgentEditor
TaskDescriptions
LearningTranslator
Translated Model (Subset)
ReasoningInterface
Description
ActionTranslator
LearningOutputs
The Controller notifies the Learning Translator that it has received a See message.
The Learning Translator finds a city location task, which is triggered by the See message. It queries the controller for the learning mode, then creates a TestInput message to send to the reasoning system with information on the pod’s location and the map from the Current State.
The Reasoning System transmits output to the Action Translator.
The Learning Translator transmits the TestInput message to the Reasoning System.
1
2 23
4
Controller
CurrentState
1
4
2
3
16Integrating Learning in Interactive Gaming Simulators
TIELT
Game InterfaceEditor
User
ActionTranslator
Actions
GameEngine
GameEngine
1
2
4.a
The Action Translator receives a TestOutput message from the Reasoning System.
The Action Translator finds the TestOutput message template, determines it is associated with the city location task, and builds a MovePod operator (defined by the Current State) with the parameters of TestOutput.
The Game Engine receives Move and updates the game to move the pod toward its destination, or
The Action Translator determines that the Move Action from the Game Interface Description is triggered by the MovePod Operator and binds Move using information from MovePod.
ReasoningInterface Editor
3
GameInterface
Description
ReasoningInterface
Description
AdviceInterface
The Advice Interface receives Move and displays advice to a human player on what to do next, or makes a Prediction.
4.b, c1
4.a
2
3
Acting in the Game World(City placement example)
4.b
CurrentState
2
PredictionInterface
4.c
3
17Integrating Learning in Interactive Gaming Simulators
Initial Work
Status (July 2004)
• TIELT v1 + documentation due 9/04– Message protocols
• Current: Console I/O, TCP/IP• Future: Library calls, HLA interface, RMI (possibly)
– Message content: Configurable• Instantiated templates tell it how to communicate with other modules
– Initialization messages: Start, Stop, Load Scenario, Set Speed– Game Model representations (w/ Lehigh University)
• Simple programs• TMK process models• PDDL (language used in planning competitions)
• Stratagus/Wargus module (Lehigh University)• Initial publicity (BRIMS’04, here)• Workshops being planned: ICCBR’05 (plus competition), ICML’05, ...?
• TIELT v1 + documentation due 9/04– Message protocols
• Current: Console I/O, TCP/IP• Future: Library calls, HLA interface, RMI (possibly)
– Message content: Configurable• Instantiated templates tell it how to communicate with other modules
– Initialization messages: Start, Stop, Load Scenario, Set Speed– Game Model representations (w/ Lehigh University)
• Simple programs• TMK process models• PDDL (language used in planning competitions)
• Stratagus/Wargus module (Lehigh University)• Initial publicity (BRIMS’04, here)• Workshops being planned: ICCBR’05 (plus competition), ICML’05, ...?
Example
18Integrating Learning in Interactive Gaming Simulators
TIELT Collaboration Projects (2004-05)
Organization Game Interface and Model
Reasoning Interface
Tasks and Evaluation Methodology
Mad Doc Software Empire Earth 2 (RTS)
Troika Games Temple of Elemental Evil (RPG)
ISLE SimCity (~RTS) ICARUS ICARUS w/ FreeCiv, design
Lehigh U. Stratagus/Wargus (RTS), and HTN/TMK designs
Case-based planner (CBP)
Wargus/CBP
NWU FreeCiv (discrete strategy), and qualitative game representations
U. Michigan SOAR SOAR w/ 2 games(e.g., FSW, ToEE), design
U. Minnesota-Duluth RoboCup (team sports) Advice-taking components
Advice processing
USC/ICT Full Spectrum Command(RTS)
SOAR with FSC
UT Arlington Urban Terror (FPS) DCA (lite version)
UT Austin Neuroevolution e.g., Neuroevolution/EE2
19Integrating Learning in Interactive Gaming Simulators
TIELT’sInternal
CommunicationModules
TIELT’s KBEditors
TIELT’s KBEditors
Selected/Developed Knowledge BasesSelected/Developed Knowledge Bases
GameModel
Description
Task Descriptions
GameInterface
Description
ReasoningInterface
Description
EvaluationMethodology Description
TIELT’s User Interface TIELT’s User Interface
PredictionInterface
EvaluationInterface
CoordinationInterface
AdviceInterface
TIELT User
TIELTUser
TIELT Collaborations (2004-05)
Knowledge Base LibrariesKnowledge Base Libraries
Game LibraryGame Library
Mad Doc
EE2 ToEE
Troika
FreeCiv
NWU ISLE
Platform LibraryPlatform Library
Stratagus
Lehigh U.
UT Arl.
FSC/R
USC/ICT
UrbanTerror
U. Minn-D.
RoboCup
Reasoning System LibraryReasoning System Library
Learning Modules
Soar: U.MichICARUS: ISLE
DCA: UT Arlington
Neuroevolution: UT Austin
Others: Many
LU, USC Mich/ISLEU. Mich.Many Many
U.Minn-D. USC/ICTU.Mich.
20Integrating Learning in Interactive Gaming Simulators
Summary: Questions? Concerns?
TIELT: Mediates between a (gaming) simulator and a learning-embedded reasoning system
• Goals: – Simplify running learning expts with cognitive systems– Support DARPA challenge problems in learning
• Designed to work with many types of simulators & reasoning systems
TIELT: Mediates between a (gaming) simulator and a learning-embedded reasoning system
• Goals: – Simplify running learning expts with cognitive systems– Support DARPA challenge problems in learning
• Designed to work with many types of simulators & reasoning systems
Status: • v1 scheduled for completion in 9/04
– Please see Matt Molineaux’s demo• 11 additional organizations about to start 1-year collaborations
– Enhances probability that TIELT will achieve its goals
Status: • v1 scheduled for completion in 9/04
– Please see Matt Molineaux’s demo• 11 additional organizations about to start 1-year collaborations
– Enhances probability that TIELT will achieve its goals