Agent animation: capabilities, issues, and trends

Paolo Petta

Austrian Research Institute for Artificial Intelligence, Vienna

Introduction Computer animation developments

Geometry Resolution, detail

Model-driven dynamics Ambient physics modeling,

Behavioural modeling Control

Interactivity, communication techniques, autonomy, learning

Population Multiple actors, distributed systems

Typical Applications Synthetic characters,

virtual Humans,visualisation/simulation

Design choices “Sparse” top-down models vs.

“complete” bottom-up models Application requirements

deep-and-narrow vs. broad-and-shallow

Research topics

Facedesign

BehaviouralanimationSpatialrelationships

User interfacefor

Emotion control

Actor behaviouremotion control Kinematics

Dynamics

Vision-basedanimation Path planning

WalkingmodelsObjectgrasping

shape transformationCollision detection Facial animation

Clothanimation

Collisionresponses

Finite-element deforma-tions

Musclemodels

Hair

Skin texture

ArtificialIntelligence Robotics

UserInterface

Animation

Physics

GeometricModelling

ImageSynthesis

IMPROV (MRL, NYU)

Artistic and commercial applications Animated staging Choreography Interactive multi-user environments ...

Surface model of mood&emotions Productivity tool

API for “laypersons”(educators, historians, social scientists)

IMPROV Microlevel:

Procedural animation Accurate modeling of single actions

and all permissible transitions Statistically controlled parameter

randomization for variability and consistency

IMPROV Microlevel:

Behavioural layering Scripts are classified in a hierarchy

according to level of behaviour User-defined connections between

layers define the effective heterarchy Action selection:

deterministic linear scripts or stochastic selection from alternatives

Exclusion of pursuit of conflicting goals at same level

Parallelism across the hierarchy

IMPROV Macrolevel:

Blackboard architecture

Stage

Manager

Characters (attributes + scripts)

Avatars

Story agent („director“)

IMPROV

Macrolevel: Behaviour layers spanning across

groups of agents forcoordinated action

Distributed environment modeling: “Inverse Causality” (=> MOO) information about interactions is

attached to objects characters are “contaminated” by

use (new/update of state variables: competence learning)

Edge of Intention (Oz, CMU)

Interactive drama Believable autonomous characters

Goal-directed Emotional

(folk theory of emotions, OCC) Simple appearance, emphasis on

behaviours(-> internal processing)

Interaction modes Moving/gesturing, “talking” (typing)

TOK architecture Microlevel

Hap Goal-oriented reactive action engine Static plan library

• Action behaviours• Emotion behaviours• Sensing behaviours

Sensing of low-level actions of other Woggles

Action blending

TOK architecture Microlevel

Em Model of emotional and social

aspects Explicit state variables for beliefs and

standards of performance Variables are influenced by

comparison of current goal states with events and perceived actions (thresholding)

TOK architecture Microlevel

Behavioural features Mapping of emotional state to overt

behaviour Manifestation of “personality” Tight integration of Hap and Em No need for arbitration

TOK architecturestandardsattitudesemotions

Em

goalsbehaviours

Hap

behaviour featuresand raw emotions

goal successes,failures & creation

sensory routines andintegrated sense model

senselanguagequeries

senselanguage

queries

The world

TOK architecture Macrolevel:

Fixed plan library encodes all possible communications/interactions

ALIVE (MIT Media Lab)

Entertainment Magic mirror metaphore Unincumbered immersive

environment

ALIVE Microlevel:

Hamsterdam Behaviour system for action selection

• Based on ethological model

• Sensory inputs via release mechanism

• Loose hierarchy of behaviour groups

• “Avalanche effect” for persistent selection

• Inhibited behaviours can issue secondary and meta commands

Motor skills layer for coordination of motions

Geometry layer for animation rendering

ALIVEExternal World

World

SensorySystem

ReleasingMechanism Behaviour

Levelof Interest Inhibition

InternalVariable

InternalVariable

Motor Commands

Goals/Motivations

ALIVE

ALIVE Levels of control:

Motivations via variables of single behaviours “You are hungry”

Directions via motor skills “Go to that tree”

Tasks via sensory, release, and behaviour systems “Wag your tail”

ALIVE Increased situatedness

Synthetic vision For navigation Generic interface

Plasticity: reinforcement learning (conditioning)

ALIVE Macrolevel:

Totally distributed control

Virtual Humans (Miralab/EPFL)

Goal Simulation of existing people Real-time animation of virtual

humans that are realistic and recognizable

Inclusion of synthetic sensing capabilities allows simulation of (seemingly) complex capabilities,e.g. real-time tennis

Virtual Humans Issues requiring compromising

Surface modeling Deformation Skeletal animation Locomotion Grasping Facial animation Shadows Clothes Skin Hair

Virtual Humans Methodology

Modeling: Prototype-based Head and hand sculpting Layered body definition:

Skeleton, Volume, Skin Animation:

Skeleton motioncaptured, play-back, computed

Body deformationfor realistic rendering of joints

Detailled hand and facial animation

Virtual Humans Synthetic sensing as a main

information channel between virtual environment and digital actor(since ca. 1990)

Synthetic audition, vision and tactile Differs fundamentally from robotic

sensing:direct access to semantic information

Virtual Humans Example: synthetic vision

Environment is perceived from a field-of-view that is rendered from the actor’s point of view

Access to pixel attributes:color, distance,index to semantic information Simple case: color coding of objects

=> perception of color = recognition of object

Object attributes areretrieved directly from the simulation

Virtual Humans Navigation:

Path planning & obstace avoidance Global navigation:

Based on prelearned model Determines the global navigation goal

Local navigation Purely indexical, based on sensing

=> No need for model of environment=> No need for current position

Three modules: synthetic vision, controller, performer

Virtual Humans Navigation controller:

Regularly invokes vision to retrieve updated state of environment

Creates temporary local goals if an obstacle “up front”

Local goals are determined by obstacle-specific Displacement local automata

Virtual Humans Interaction with the environment:

Smart Objects Each modeled object includes

detailled solutions for each possible interaction with the object

Objects are modeled according to situated decomposition

Virtual Humans Smart Objects include:

Description of moving parts, physical properties, semantic index(purpose and design intent)

Information for each possible interaction: position of interaction part, position and gesture information for the actor (capacity limits!)

Object behaviours with state variables (=> actor state info)

Triggered agent behaviours

Virtual Humans Example: virtual tennis

Actor model based on stack machine of state automata

Actor state can change according to currently active automaton and sensorial input

Virtual Humans

Architectureof behaviourcontrol

Virtual Humans

Tennisgameautomatasequence

JACK (UPenn)

Ergonomic environment analysis Workplace assessment Product evaluation Device interfaces Logistics

JACK Microlevel:

Biomechanically correct model Synthetic sensors for high-level

behaviours Three-level architecture realising

“truly situated” low-level behaviour

JACK Microlevel

PaT-Net

object-specific and genericsymbolic reasoning capabilites

controlsystems

stimulus           perceptual                             motor             response modules behaviours

(learned sense-control-act loop parameters)

JACK Macrolevel

Taskable virtual agent Global intentions and expectations of

all characters are statically captured (explicitly anticipated)

Parallel Transition networks

JACK Macrolevel: PaT Net

Topics for Discussion “Completeness” of modeling “True” agent characteristics

(Wooldridge&Jennings)

Autonomy Social abilities Reactivity Pro-activeness

Topics for Discussion The “TLA Debate” Situatedness/synthetic sensing Variability/adaptiveness/plasticity Believability

Modelling completeness “Sparse” models

Abstract, “top down” Based on explicit, reified design

elements Bridging/obviating of full detail by

careful selection of modeled elements

Broader coverage at differing resolution

Believability/impression over fidelity (Bound to) Lose in the long run?

Modelling completeness “Complete” models

Situated, “bottom up” Depend on balanced design

(including environment&coupling) Limited coverage/complexity Allow for flexible action-selection Fidelity over believability/impression Win in the long run?

Autonomy (McFarland/Boesser)

Automaton:state-dependent behaviour

Autonomous agent:self-controlling, motivated

Motivation:reversable internal processes that are responsible for changes in behaviour

Multiple goals/actions are the rule!=> concurrency, transitioning

Insights on own skills&conditions of applicability

Social abilities “Deep” agent modeling

Of the self: BDI and variants Of others (recursively) Of the society

Coordination Communication

Generation&understanding of facial expressions, postures, gestures, task execution, text/speech,…

(social) Emotions(including display rules)

Social abilities From Action Selection to Action

expression Sign management: context-

dependent behaviour sematics What should an agent do at any point

in order to best communicate its goals and activities?

Goal: increase comprehensibility of behaviour

Believability Quality vs. correctness Self-motivation

pursuit of multiple simultaneous goals => entails requirement of broad

capabilities Personality/Emotion Plasticity/change over time Situatedness

social skills affordances

And then... Methodologies for assembly of

architectures with understandable/predicatable (motivated, goal-directed,…) behaviour

Agent control systems Persistency, plasticity Agent animation as simulation