Agents and Avatars 2

Ruth Aylett

Overview-

Crowds Speech and Expressive behaviour

– Embodied conversational characters

Creating autonomy– Scripting– Architectures

Crowds and flocking

Interactions amongmembers of a group– Local neighborhood

Reynolds Boids– Originally birds or

fish– Complex behaviour

determined bysimple rules

Separation: BoidAvoidance

Alignment

Cohesion

Motor Control

Steering Force Integrate to determine acceleration

– Thrust – determines speed– Lateral Steering Force – determines direction

Boid Object Representation

Point Mass Vehicle– Mass– Position– Velocity– Orientation

• Constrained to align with velocity– Force and Speed Limits

(No moment of inertia)

Same maths as point objects

acceleration = steering_force / mass– Given f = mavelocity = velocity + acceleration– Over a unit time period– Generically v = v x atposition = position + velocity– Over a unit time period– Generically s = s + vt

Seeking and Fleeing Aim towards target

Desired_velocity = Kp (position – target)Steering = desired_velocity – velocity

Seeking andFleeing Applet(Reynolds)

Pursuing and Avoiding

Target is another moving object Predict target’s future position Scale prediction time, T, based on distance to object, Dc

T=Dc

Pursuing andavoiding applet(Reynolds)

More Behaviors

Evasion– Like flee, but predict pursuer’s movement

Arrival– Like seek, but stop at target– Applet (Reynolds)

Obstacle Avoidance– Repulsive force– Aim to boundary– Adjust velocity to be perpendicular to surface normal

Do People Flock?

Social psychologist’sreport that peopletend to travel assingles or in groupsof size 2 to 5.

“Controlling Steering Behavior for Small Groups ofPedestrians in Virtual Urban Environments”

Terry Hostetler, Phd dissertation, 2002

Characteristics of Small Groups

Proximity Coupled Behavior Common Purpose Relationship Between

Members

Moving Formations

Pairs: Side by side Triples: Triangular shape

Stationary Formations

Moving pair approachesstationary triple

Stationary quintupleformed

Two Parameters - because in 2D– Acceleration

• Increase/reduce walking speed• Combination of step length and step rate

– Turn• Adjust orientation• Heading direction for forward walking

Locomotion Model for Walking

Avoiding an Obstacle --Trajectory

Small look-ahead distance Large look-ahead distance

ped 1

ped 2

walkway axis walkway axis

ped 1

ped 2

Interaction Between Pairs -- 1

Interaction Between Pairs -- 2

Interaction Between Pairs -- 3

Adding goals

People are usually going somewhere– Easy case: same goal

• Walking to a sports stadium• Political demonstrations

– Harder case: multiple goals• Campus traffic• Most public spaces: streets, shops, stations

Talking Heads

Human faces– Thus high expectations:

• Blinking, random head movement• Facial muscles• Lip sync

Rarely intelligent– Entirely scripted in most cases– No integration of speech production with

intelligent architecture and NL generation

Moving the face - 1

Simple approach– Define overlay frames with different facial

expressions• For example, six mouth shapes

– Use signal from TTS engine to pick amouth shape

– This works OK with cartoon-like characters

Moving the face - 2

Visemes– Mouth and lip position

for each phoneme– Correspond to a

submesh in thegeometry

Using visemes

Find viseme for each phoneme– Synchronise graphical change with sound

Catalan weather forecaster

Complete talking character– Weather forecasts have predictable content– Still have to match content and behaviour

• <demo via browser>

Embodied conversational characters

Beyond talking heads– Key difference is interactivity– Use of dialogue theory– Focus on body language - expressive

behaviour

Relating graphical to languagebehaviour

Expressive facial behaviour

Drawing on psychology:– FACS (Facial Action Coding System)– Developed by Ekman & Friesen (1978)– Distinguishes among 44 Action Units– Each action unit (AU) describes a separate,

visible, and muscle-based facial change

Duchenne de Bologne

French anatomist of19thC– In 1860s used

electric currents onfacially paralysedsubject

– Showed whichmuscles go withwhich expressions

The Duchenne smile

Associated withexpression ofhappiness

AU12 (Lip Corner Puller)

Plus

AU6 (Cheek Raiser)

Mapping onto mpeg-4

Defines FAPs– Facial action

parameters– Set of feature points– Model calibration– Morphology

parameters– Expression

parameters

Coordinating with language Use of Affective Performance Markup Language (APML)<performative type="inform"><theme> As far as <emphasis x-

pitchaccent="LplusHstar"> vitamins </emphasis> are concerned<boundary type="LH"/></theme></performative><rheme><emphasis x-pitchaccent="Hstar"> research </emphasis>

has shown <boundary type="LH"/></rheme> <performativetype="inform"> <rheme>that eating<emphasis x-pitchaccent="Hstar">the recommended </emphasis>levels of vitamin

<rheme affect="sorry-for">can have <emphasis x pitchaccent="Hstar">beneficial </emphasis><emphasis x-pitchaccent="Hstar">effects</emphasis>for your <emphasis x-pitchaccent="Hstar">appearance</emphasis> <boundary type="LH"/> and <emphasis x-pitchaccent="Hstar"> health</emphasis><boundary type="LL"/>

</rheme></performative>

Incorporating gesture

Additional markup to control gestures insynch with other expressive behaviour

Generating expressivebehaviour

Setting the internal state to be expressed Low-level accounts

– Focus on behaviour (non-symbolic)– Primitive emotions– Brain limbic system: physiological integration– Canamero, Velasquez, Damasio

High-level accounts– Focus on cognitive appraisal– Complex taxonomy of emotions– Cortex: cognitive integration– Ortony, Clore and Collins; many applications– Frijda; Lazarus; Sherer

Modelling emotion Appraisals

– Assessments of events, actions, objects

Valence– Whether emotion is positive or negative

Arousal– Degree of physiological response

Implementing appraisals– Domain-specific rules– Probability of impact on agent’s goals

Emotion theoryOrtony, Clore and Collins (OCC)

Extremely widely used for agents– Ortony, A; Clore, G. & Collins, A. 1988 The cognitive

structure of emotions. Cambridge University Press–

Group TypesWell-being joy, distressFortunes-of-others happy-for, gloating, resentment, sorry-forProspect-based hope, satisfaction, relief, fear, fears-confirmed, disappointmentAttribution pride, admiration, shame, reproachAttraction love, hateWell-being ! Attribution admiration + joy " gratitude(compound emotions) Reproach + distress " anger

Pride +joy " gratificationShame +distress " remorse

OCC Model - more detail

Group Specification Name & typeWellBeing

Appraisal ofsituation as event

Joy - pleased about eventDistress - displeased aboutevent

FortunesofOthers

Appraisal ofsituation as eventaffecting another

Happy-for: pleased about anevent desirable for anotherResentment: displeased aboutan event desirable for another

Prospect-based

Appraisal ofsituation as aprospective event

Hope: pleased about aprospective desirable eventFear: displeased about aprospective undesirable event

Emotions defined in terms ofsituations, goals, and others

Autonomy

An independent sense-reflect-act cycle: implies– An agent-based action-selection mechanism– Self-animation in real-time: cannot be pre-rendered– Local sensing– Virtual robotics

Not a natural graphics view– Global manipulation of polygons– Hierarchical scene-graph

Scripting

Typical of NPCs in computer games What representational content is

needed?– Can just invoke named animations– What parameters?

• Depends on animation approach

Parametrised ActionRepresentation

Badler: symbolic term -> movement Supports natural language interaction

– Named actions: walk, jump, run– Adverb modification: quickly, tiredly,

enthusiastically Laban Movement Analysis

– From choreography– Effort and shape components to

characterise movements

Badler’s PARs

Scripting with Finite State Automata

TennisgameFSMsequence

Limitations of scripting

Becomes predictable– Either independent of local sensing– Or single FSM transition stimulus

Limited ability to use internal state– Interaction memory– Affective state

Lacks sequencing flexibility– Sequence is hardwired

Low-level architecture for a fish…

Functional Architecture

High-level architecture

Using game engines

Offer some support for NPCs– More facilities than a scenegraph– Attaching animations to graphical bodies– Imports from popular 3D packages like character studio

Disadvantages– Proprietary; oriented to scripting– Hard (in some cases impossible) to implement sensing

• Fixed paths quite often– Often focused on shooting

Gamebots

Loosely coupled architecture based onUnreal Tournament– CMU– Agent ‘mind’ outside of UT– Defined interface: message types and

content– Javabots gives java interface

The mechanism Gamebots and TCP sockets

– Every robot possesses one TCP socket– Exchange data through the TCP socket– Unreal Client(s) can connect to the server at anytime

UnrealServer

(response to the command;

send out sensor data)

Gamebots

Robot Controller

Unreal Client(s)

Command

Sensor data

Command

Sensor data

Robot ControllerCommand

Sensor data

Command

Sensor data

………… TCP Socket

Virtual robot

Communication protocol Gamebots protocol

– Format: data_type {segment1} {segment2} …• data_type: the type of the data in upper case

characters– Examples: INIT, STA, SEN, DRIVE etc.

• segment: a list of name-value pairs separated by space– Examples: {Location 100,200,300}, {Name Left Range

800.0} etc.

– Example• INIT {ClassName USARBot.ATRVJr} {Location 200,600,-450}

– Two types of data• Message (from the server)• Command (from the client)

Communication protocol - 2 Messages

– State message: the robot’s current state• STA {Time t} {Camera pitch,yaw,roll} {Zoom fov} {Attitude

pitch,yaw,roll} {Location x,y,z} {Velocity x,y,z} {LightToggle bool}{LightIntensity int} {Battery float}

– Sensor message: the sensor data• Sonar Sensor SEN {Type Range} {Name string Range number}

{Name string Range number} …• Laser Sensor SEN {Type RangeScanner} {Name string} {Location

x,y,z Rotation pitch,yaw,roll} {Range r1,r2,r3…}• Human Motion Detection SEN {Type HumanMotion} {Name string}

{Prob float}• Sound Sensor SEN {Type Sound} {Name string} {Loudness float}

{Duration float}– Geometry message: sensor’s geometry information

• GEO {Type string} {Name string Position x,y,z Direction pitch,yaw,roll} {Namestring Position x,y,z Direction pitch,yaw,roll} …

– Configuration message: sensor’s configuration information• CONF {Type string} {Name Value} {Name Value} …

Communication protocol - 3

Command– Spawn robot

• INIT {ClassName robot_class} {Name robot_name} {Locationx,y,z}

– robot_class: the class name of the robot. Example:USARBot.P2AT, USARBot.P2DX etc.

– robot_name: the robot’s name. Can be any string.– x,y,z: the start position of the robot. For different arenas, we

need different positions.• Example

– INIT {ClassName USARBot.P2AT} {Location 312,-600,-305}