htn in videogames

Hierarchical Planning in Video Games

by Andrea Tucci

Outline

OutlineIntroduction

Hierarchical Task Networks

Overview

Definition

Expressivity

Complexity

Applications

Video Games AI : A Hierarchical Task Networks approach

Planning in games

HTNs in games

Example of Study : Killzone

Conclusions

References

OutlineIntroductionHTNsVideo Games Ai : HTNsConclusionsReferences

Introduction

Introduction

Hierarchical Planning in a real-time environment application : video games

Overview of Hierarchical Task Networks

Video Games AI, focusing on planning

Killzone : a commercial game with HTNs

Show issues and advantages of Hierarchical Planning in dynamic worlds

IndexIntroductionHTNsVideo Games Ai : HTNsConclusionsReferences

Hierarchical Task Networks

Overview - Classical planning vs HTNs

Classical AI planning : how reach a goal? → actions

an action is represented by an operator < c, e>, c is a precondition, e is the effect of the action

states : configurations of the world generated by applying actions

planner aim is to find the (minimal) path from initial state to the goal state, the path is composed of actions

Hierarchical Task Network (HTN) planning : actions to reach a (composite) task

task : composite or primitive

composite tasks have to be decomposed in subtasks

primitive tasks can be executed directly by applying an operator

planner decomposes compound tasks of a given task network until only primitive tasks are found

OutlineIntroductionHTNs>>Overview>Definition>Expressivity>Complexity>ApplicationsVideo Games Ai : HTNsConclusionsReferences

DefinitionGiven a first order HTN language L , we have a vocabulary < V, C, P, F, T, N > where

V = { v1, v2, ... } is an infinite set of variables

C is a finite set of constant symbols

P is a finite set of predicate symbols

F is a finite set of primitive tasks

T is a finite set of compound tasks

N = {n1, n2 ... } is an infinite set of task labels

Goals replaced by tasks. A task can be:

goal task, tasks to achieve

primitive task, obtained by executing the corresponding action

compound task, require decomposition

e.g. “travelByPlane” may require subtasks like buyTicket, goToAirport, fly, goToDestination

OutlineIntroductionHTNs>Overview>>Definition>Expressivity>Complexity>ApplicationsVideo Games Ai : HTNsConclusionsReferences

DefinitionA Task Network is a collection of tasks and constraints : (( n1 : α1), ... , (nm : αm), Φ)

Each αi is a task labeled ni

Φ is a boolean formula; can contain :

variable constraints (e.g x = y, x,y ∈ V or x ∈ V, y ∈ C )

temporal ordering (e.g n ≺ n’, n,n’ ∈ N)

truth constraints (e.g (n,l) i.e.“n must be true immediately after n, n ∈ N and l is a literal)

Operator is of the form (f(v1, ..., vk),l1, ..., lk) where f ∈ F, v ∈ V and li is a literal

in this representation there are no preconditions in operators : they are expressed as goal tasks in task networks

A method is a pair (α, d) where α is not a primitive task and d is a task network

This means : in order to achieve α, we use d (i.e. all its subtasks)

Example : (on(v1,v2),d) where d is a task network.


DefinitionA planning problem is composed of:

an initial state

a task network, that encode the problem to be solved

a planning domain, that contains methods and operators


DefinitionGiven a planning problem with task network d, a rough Algorithm is :

1.Perform a compound task by decomposing it in subtask (use a method); replace the compound task with the subtasks provided by the used method

2.Are there other compound tasks? If yes goto (1.);

3.Find a totally-ordered ground instantiation of d that satisfies all the constraints.

Example (from blocks world domain) : put v1 on v2


Syntax used :do precedes a primitive task

perform precedes a compound taskachieve precedes a goal task

ExpressivityHTNs provides all the STRIPS concepts (states, actions, goals)

translation of a STRIPS encoded domain to HTNs domain done in polynomial time

Is the opposite true?

NO!

HTNs can express situations impossible to express using STRIPS operators

HTNs provide a constraint language to express many type of interactions

STRIPS does not have the same level of abstraction

There exist HTN planning domains that can not be represented by any finite number of STRIPS-style operators1

OutlineIntroductionHTNs>Overview>Definition>>Expressivity>Complexity>ApplicationsVideo Games Ai : HTNsConclusionsReferences

ComplexityA planning problem P can be

propositional : no variables are allowed

totally ordered : all tasks in any task network are totally ordered

regular : all the task networks contain max one non-primitive task and is the first or the last task

primitive : the task network we need to plan for contains only primitive tasks

Plan existence is

strictly semi-decidable2;

decidable if methods are restricted to be acyclic3

is EXPSPACE-hard and DOUBLE-EXPTIME if the planning probem(P) is restricted to be totally ordered; PSPACE-hard and EXP-TIME if it is propositional4

EXPSPACE-complete if P is regular5

PSPACE-complete if P is regular and propositional6

is NP-complete if P is primitive7

OutlineIntroductionHTNs>Overview>Definition>Expressivity>>Complexity>ApplicationsVideo Games Ai : HTNsConclusionsReferences

ComplexityOutlineIntroductionHTNs>Overview>Definition>Expressivity>>Complexity>ApplicationsVideo Games Ai : HTNsConclusionsReferences

ApplicationsScheduling

Video Games

Automated Composition of Web Services

Evaluating Terrorist Threats

Evacuation

Military Tasks

Others

OutlineIntroductionHTNs>Overview>Definition>Expressivity>Complexity>>ApplicationsVideo Games Ai : HTNsConclusionsReferences

Applications - SHOPSimple Hierarchical Ordered Planner 2 (SHOP 2) is a

hierarchical planner developed in 2001, written in LISP.

OutlineIntroductionHTNs>Overview>Definition>Expressivity>Complexity>>ApplicationsVideo Games Ai : HTNsConclusionsReferences

(defdomain basic-example ( (:operator (!pickup ?a) () () ((have ?a))) (:operator (!drop ?a) ((have ?a)) ((have ?a)) ())

(:method (swap ?x ?y) ((have ?x)) ((!drop ?x) (!pickup ?y)) ((have ?y)) ((!drop ?y) (!pickup ?x)))))

(defproblem problem1 basic-example ((have banjo)) ((swap banjo kiwi)))

Plans:(((!DROP BANJO) (!PICKUP KIWI)))

Video Games AI : A Hierarchical Task Networks

Approach

Planning in GamesPlanning in games arises in the last years

Planning approach makes video games agents (bots,NPCs) more dynamic w.r.t. old approaches

Planning approach can split action planning from action execution

Planning increases autonomy of NPCs : they have alternate solutions to fulfill a goal

Planning can handle new situations

First Encounter Assault Recon (F.E.A.R.) developed by Monolith was the first commercial game that used planning in its AI

in 2005 won “best AI award” from GameSpot

still considered as one of the best games for AI

GOAP : Goal Oriented Action Planning,decide what to do AND how to do it

Hierarchical planning : more natural in some situations (reasoning on tasks rather than actions)

OutlineIntroductionHTNsVideo Games Ai : HTNs>>Planning in Games>HTNs in Games>Example of Study>KillzoneConclusionsReferences

Planning in GamesOutlineIntroductionHTNsVideo Games Ai : HTNs>>Planning in Games>HTNs in Games>Example of Study>KillzoneConclusionsReferences

Hierarchical Task Networks in Games

Used in both individual and coordination decisions

In video games AI, actions can be dictated also by

OutlineIntroductionHTNsVideo Games Ai : HTNs>Planning in Games>>HTNs in Games>Example of Study>KillzoneConclusionsReferences- error prone +little CPU overhead

- human effort- case specific

Script

HTNs can be used as script generators

offline planning, then the output actions are converted to scripts

Other approach : real-time planning

Examples of StudyUnderstand how HTNs work in practice

Real-Time planning

How planners are integrated with the entire system

Design of HTN based agents

Differences with other implementations

OutlineIntroductionHTNsVideo Games Ai : HTNs>Planning in Games>HTNs in Games>>Example of Study>KillzoneConclusionsReferences

Killzone 2

Killzone 2 First Person Shooter, developed by Guerrilla Games published in 2009

Need for a new multiplayer bot system...why?

In its predecessor, Killzone, coordination between bots of the same team were missing

Goals :

Extensible bot architecture

Improve Killzone AI

Include cooperative behavior

Outperform old AI

Entertain and challenge players

OutlineIntroductionHTNsVideo Games Ai : HTNs>Planning in Games>HTNs in Games>Example of Study>>KillzoneConclusionsReferences

Killzone 2 Quick overview of AI Outline

IntroductionHTNsVideo Games Ai : HTNs>Planning in Games>HTNs in Games>Example of Study>>KillzoneConclusionsReferences

Strategy

Bot

Squad

•Goal-based•Defines what to achieve in the mission by using heuristics•Creates squads•Execution of mission tasks

•Coordinate bots (e.g. targets) •Action synchronization

•Execution of Squad task• Inform their squad

Killzone 2 vs KillzoneKillzone 2

Killzone


Killzone 2 vs Killzone (2)OutlineIntroductionHTNsVideo Games Ai : HTNs>Planning in Games>HTNs in Games>Example of Study>>KillzoneConclusionsReferences

Killzone Killzone 2

Approach

Goal selection

Goal satisfaction

Atom part

Front end

Goal-based HTNs

Heuristics High-level HTN choices

Behavior Plan

Action Primitive task

Body goals Body goals

Killzone 2 - AIGoal behavior system of Killzone, replaced by task planning mechanism

Improve flexibility

Dynamic : decomposition of a compound task may lead to different plans (i.e. different behavior)

HTN planner based on SHOP syntax

Specialized primitive tasks : is possible to add facts in the HTN database for a certain amount of time and delete them in any moment(!remember ?period ?fact) (!forget ?fact)

HTN planner is used for evaluate the relevance of goals and find how to reach themKnowledge domains provides how tasks are performed and also tasks priorityAgents use a combination of generalized and specialized knowledge rules


Killzone 2 - AI (2)What should AI do?

Decided by high-level HTNs

Compound task(s) as output

How agent should perform decisions?

Low-level HTNs decisions

Each agent has a set of daemons that provide info about the world

The planner requires those facts before it starts

HTNs used in squad

There is an invisible leader that uses planner for coordination purposes

Informations shared

Squad plans are series of orders for team members


Killzone 2 - Squad AIOutlineIntroductionHTNsVideo Games Ai : HTNs>Planning in Games>HTNs in Games>Example of Study>>KillzoneConclusionsReferences

Strategy Team member messages

Facts

Daemons

Planner Plan

Task execution

Order to members

Killzone 2 - AI (2)(:method (order_member_defend ?inp_member ?inp_id ?inp_level ?inp_marker ?inp_context_hint)

… (branch_advance () ( (!forget member_status ?inp_member **) (!remember - member_status ?inp_member ordered_to_defend ?inp_id) (!start_command_sequence ?inp_member ?inp_level 1) (do_announce_destination_waypoint_to_member ?inp_member) (!order ?inp_member clear_area_filter) (!order ?inp_member set_area_restrictions (call find_areas_to_wp ?inp_member (call get_entity_wp ?inp_marker))) (!order_custom ?inp_member move_to_defend ?inp_marker ?inp_context_hint) (!order_custom ?inp_member send_member_message_custom completed_defend ?inp_id) (order_set_defend_area_restriction ?inp_member (call get_entity_area ?inp_marker)) (!order_custom ?inp_member defend_marker ?inp_marker ?inp_context_hint) (!end_command_sequence ?inp_member) ) ))


Killzone 2 - AI (2)HTN used in individual bot’s decisions

combat behavior

mission specific tasks

other (heal, repair ... )

OutlineIntroductionHTNsVideo Games Ai : HTNs>Planning in Games>HTNs in Games>Example of Study>>KillzoneConclusionsReferences(:method (select_weapon_and_attack_as_turret ?inp_threat)

(branch_use_bullets // Only use bullets against humanoids and turrets. (and

(or (threat ?inp_threat humanoid) (threat ?inp_threat turret) ) (distance_to_threat ?inp_threat ?threat_distance) (call lt ?threat_distance @weapon_bullet_max_range) ) ((attack_as_turret_using_weapon_pref ?inp_threat wp_bullets))

)

( branch_use_rockets // Don't use rockets against humanoids and turrets. (and

(not (threat ?inp_threat humanoid)) (not (threat ?inp_threat turret)) (distance_to_threat ?inp_threat ?threat_distance) (call lt ?threat_distance @weapon_rocket_max_range) ) ((attack_as_turret_using_weapon_pref ?inp_threat wp_rockets)) ))

Killzone 2 - agent designOutlineIntroductionHTNsVideo Games Ai : HTNs>Planning in Games>HTNs in Games>Example of Study>>KillzoneConclusionsReferences

Killzone 2 - real time environment

Planning ahead is not so good, we are in a real-time environment

Long plans tend to invalidate

☛ Local plans are made

Planning done 5 times per second, 2 times per second for squad

Invalidation of a decided plan of actions : better plan or changing environment

Problem : re-planning so often is


HEAVYHow to deal with this?

Killzone 2 - real time environment (2)

While re-planning, if we come up with a current task, just update it.

Only decompose the part of the plan which we are interested at the moment, delaying the other.

Re-plan only if a certain percentage of preconditions is not valid anymore (i.e. the world changed too much)

Strategies for optimize re-planning

tasks updating

subtasks priority order changes

partial decomposition of tasks

delayed decomposition of tasks


Killzone 2 - conclusionsOld AI (bots from Domination game mode) vs new AI (Group Strategy) : 174 tests

battles in a multiplayer modality


Conclusions

ConclusionsOverview of HTNs

Use of HTNs in video games

offline planning (e.g. Script generation)

online planning

Real-Time (HTN) planning

much more interesting

deal with re-planning

OutlineIntroductionHTNsVideo Games Ai : HTNs>>ConclusionsReferences

Conclusions - HTN in real-time environment

Make planners for real-time environments

build connection to the problem to be solved ... but is like “hard-code” the planner

make full logic representation, gaining expressive power and more generality ... but loosing efficiency

How can we maintain a “generic” planner with efficiency?

Some optimizations:

Use critics to prevent backtracking in an unfeasible plan

critics are used to remove conflicts within a plan as soon as possible

Partial re-planning

why re-plan an entire plan? We can adapt it :(example) agent A needs an object O, so a plan is made. A can’t find O, but he can replace it with another object that has the same function, no need to re-plan for the whole task (go to the shop etc..)



Domain compilation

Introduced for JSHOP planner

unique instance of the planner, generated compiling both the domain and the problem

☛ Expert planner for that problem

In real-time environment we only can compile the domain

Time slicing

Allows to pause and restart the planner

Spread time of planning in more frame

☛ Characters are able to perceive events

Other optimizations based on the problem

Make precondition checking light : heavy computations (e.g. pathfinding) are done by sensors of the agent and stored in a working memory

☛ Sensors provide up-to-date data to the planner

Caching perceptions as working memory facts, containing important informations

Garbage collector : a sensor should delete a fact added that is no relevant anymore



So...are HTNs good or not? Can be used well in a real time environment?

In general, planning in more than one layer is better

at any given level of the HTN, the number of applicable operators is smaller than the total number (reduction of a search space)

invalid plan can be detected at a top level and never be considered for expansion

more suitable for dynamic world : support of re-planning is good


ReferencesPresentation based on

HTN Planning : Complexity and Expressivity (K. Erol, J. Hendler, D. Nau)

Ms thesis : A hierarchically-layered multiplayer bot system for a first-person shooter (T. Verweij)

Papers consulted

SquadSmart Hierarchical Planning and Coordinated Plan Execution for Squads of Characters (P. Gorniak, I. Davis)

Agent Architecture Considerations for Real-Time Planning in Games (J. Orkin)

Hierarchical Plan Representations for Encoding Strategic Game AI (H. Hoang, S. Lee-Urban, H. Muñoz-Avila

Planning with Hierarchical Task Networks in Video Games (J.Kelly, A. Botea, S. Koening)

Other material

Hierarchical Planning in Dynamic Worlds (N.Wallace, Ai Programming Wisdom vol. 2)

Killzone 2 Multiplayer Bots slides (Slides by A.Champandard, T.Verweij, R.Straatman)

Images of STRIPS-based and HTNs-based games, are taken from AiGameDev.com

OutlineIntroductionHTNsVideo Games Ai : HTNsConclusions>>References

ReferencesNotes (from HTN Planning : Complexity and Expressivity)

1. Theorem 7

2. Theorem 1

3. Theorem 2

4. Theorem 3

5. Theorem 4

6. Theorem 5

7. Theorem 6

OutlineIntroductionHTNsVideo Games Ai : HTNsConclusions>>References

ThanksThank you for your attention!

@andreatux

OutlineIntroductionHTNsVideo Games Ai : HTNsConclusionsReferences

htn in videogames

Technology