agent based multimedia interaction

© W. Wahlster, DFKI

IUI99, International Conference on Intelligent User InterfacesLos Angeles, January 6th, 1999

Agent-based Multimedia Interaction for Virtual Web Pages

German Research Center for Artificial Intelligence, DFKI GmbH

Stuhlsatzenhausweg 366123 Saarbruecken, Germany

phone: (+49 681) 302-5252/4162fax: (+49 681) 302-5341e-mail: [email protected]

WWW:http://www.dfki.de/~wahlster

Wolfgang Wahlster


What are Virtual Web Pages?

What are Agent-Based User Interfaces?

Using Life-like Characters for the Personalization of User Interfaces

Plan-based Media Transformation and Coordination

The Combination of Retrie ved and Generated Media Objects

for the Generation of Virtual Web Pages

Commercial Applications in Advanced WebCommerce

SmartKom: a Transportable and Transmutable Multimodal Interface

Our Research Agenda for Agent-based User Interfaces

Conclusion

Outline


First Generation Second Generation Third Generation

Static Web Sites

Fossils cast in HTML

Interactive Web Sites

JavaScripts and Applets

Database Access andTemplate-based Generation

Dynamic Web Sites

Virtual Web Sites

Netbots,Information Extraction,Presentation Planners

Adaptive Web Sites

User Modeling,Machine Learning,

Online Layout

Three Generations of Web Sites


A Virtual Web Page

is generated on the fly as a combination of various media objects frommultiple web sites or as a transformation of a real web page.

looks like a real web page, but is not persistently stored.

integrates generated and retrieved material in a coordinated way.

can be tailored to a particular user profile and adapted to a particularinteraction context.

has an underlying representation of the presentation context so that an Interface Agent can comment, point to and explain its components.

Virtual Memory, Virtual Relation, Virtual Reality...

What is a Virtual Web Page?


Virtual Webpage Retrieved from 5 Different Servers


Virtual Webpage Augmented by Persona


PROACTIVE

ACTI

VER

EACTIVE

anticipate the user's needs adopt the user's goals provide unsolicited comments

appear as life-likecharacters

plan interactive behaviorautonomously

can initiate inter-action

INTERFACEAGENTS

respond immediatelyto interruptions

criticism and clarification questions

direct manipulation

What are Agent-based User Interfaces?


Consumer Provider

sells InformationGoodsServices

buys InformationGoodsServices

Web Sites

Knowledge about: Usage PatternsUser ModelsConsumer Profiles

Netbot

Intelligent Parallel Retrieval Information Extraction and

SummarizationPersonalized Presentation Matchmaking Teleshopping Assistance Telemarketing Assistance Translation Services Data Mining Services

Intelligent Web Services


Netbots

Indices,Directories,

Search Engines

WWW

Netbot := Intelligent Agent that uses Internet tools on a person’s behalf

Traveller’s Netbot: Tries to achieve traveller’s goals(finding and executing plans)checks availability finds best price uses personal preferences (e.g. frequent flyer programme, seating preferences lets the traveller know, when seats become available (active help)

MassServices

PersonalAssistants

e.g. Ahoy!,Jango,AiA

Netbots as Personal Assistants for WWW Users


Portrait Photo

Netbot: Personal Picture Finder

Parallel Meta-Search of Webpages for<Name>

<Name>

Parallel Searchin Picture ArchivesHome Pages

Extraction of Images

Filtering ofLogos, Graphics,...

WWW Knowledge Sources

Applications: Journalism, Contact Preparation, Tracing Criminals

http://finder.dfki.de:7000

A Netbot for Portrait Photos: The Personal Picture Finder


The Personal Picture Finder


Information

Information

Information

Customer 1

Provider

1

2

3

Push Technologies

+ no effort for customer

- minimal adaptation

Interactive Pull

+ good adaptation

- major effort for customerCustomer

Query

Information

Netbots with Parallel Pull

Customer NetbotQuery

Information

Provider 1

Provider 2

Provider N

+ good adaptation

+ minimal effort for customer

Customer 2

Customer N

Provider

Netbots versus Push Technologies


System is able to flexibly tailor presentations to the individual user and the current situation.

Enhancement of User Interfaces through Personalization

An animated characterserves as “Alter Ego”

of the presentation system.

Personalized Presenters at DFKI


Personalization = adaption of system behavior

according to a user model

Personalization =the “agent” appears as

animated character which presents, explains and

comments an offer, and to which the user can talk

to.

Personalized Package Presentation by an Animated Agent


Video Character Presents the Interior of a Boeing 757


Reactive Behavior of the Persona Agent


PersonaServer

Behaviors

Presentation GesturesReactive BehaviorsIdle-time actionsNavigation actions

Auditory Characteristics

Sound effects, auditory iconsVoice: male, female

Visual Appearances

Hand-drawnCartoonBitmaps

GeneratedBitmaps from 3D-Models

Video Bitmaps

PPP’s Persona Server implements a generic Presentation Agent that can be easily adapted to various applications


More than 200 cartoon frames were drawn by a professional artist.

A real persona was filmed with a video-camera and the pictures weredigitized with a frame grabber.

The frames of the visual appearance of persona can be cartoon-like images or video-frames which show real persons


The Persona Editor


take-position (t1 t2 ) point-to (t3 t4 )

move-to (t1 t2 ) r-stick-pointing (t3 t4 )

High-LevelPersona Actions

Context-SensitiveExpansion(including Navigation Actions)

Decomposition into Uninterruptable Basic Postures

r-turn (t1 t21 )

r-step (t21 t22 )

f-turn (t22 t2 )

r-hand-lift (t3 t31)

r-stick-expose (t31 t4 )

Bitmaps

... ... ... ...

Context-Sensitive Decomposition of Persona Actions


MultimediaGeneration

TextGenerator

GraphicsGenerator

GestureGenerator

AnimationGenerator

PresentationPlanner

(PREPLAN)

KR & Reasoning

(RAT)

ProductionActs

GeneratedMaterial

PresentationActs

Signals +Events

MultimediaServer

LayoutManager

PersonaServer

MusicGenerator

PPP System Architecture


Plan multimedia material as well as presentation acts and their temporal coordination

Presentation Acts

Persona Acts Display Acts

This is thetransformer

Task of the Presentation Planner


Persona explains a modem


Production ActPresentation Act

Introduce

Create-Graphics

S-Show

S-Wait

S-Position Elaborate-Parts

S-Create-Window

S-Depict

Label Label

S-Point S-Speak S-SpeakS-Point

Qualitative constraints: Create-Graphics meets S-Show, ...

Metric constraints: 1 <= Duration S-Wait <= 1, ...

Distinction between production and presentation acts

(i.e. Persona- or display acts)

Explicit representation of qualitative and quantitative constraints

Extensions of the Representation Formalism


(define-plan-operator:HEADER (A0 (INTRODUCE P A ?object ?window)):INFERIORS ((A1 (CREATE-GRAPHICS P A ?window ?object))

(A2 (S-SHOW P A ?window ?object))(A3 (S-POSITION P A ?window))(A4 (S-WAIT P A))(A5 (ELABORATE-PARTS P A ?object ?window)))

:QUALITATIVE((A1 (m) A2) (A3 (s) A2) (A3 (m) A5) (A5 (m) A4) (A4 (f) A2)):METRIC((10 <= DUR A2) (2 <= DUR A4 <= 2)):START A1:FINISH A2)

(cf. [André/Rist 97])

Presentation Strategies in PPP contain qualitative and metric constraints


PRELIMINARY SCHEDULE

PPP first builds up a preliminary schedule at design time


UPDATED SCHEDULE

The preliminary schedule is continously updated at presentation time


Temporal

Consistency

Checker and

Propagator

Plan

Scheduler

Plan Nodes with

Links to Local

Temporal

Constraint

Networks

Plan Operators

with Metric and Qualitative

Temporal Constraints

Temporal Reasoning in the Presentation Planner for Dynamic Multimedia Coordination


Persona Presents an Automatically DesignedBusiness Chart


Multi-DomainProblem Specs

NETBOT

RetrievedResults

Information Structures Relations, Lists KR Terms

Media Objects Texts, Sounds, Videos Pictures, Maps,

Animations

DistributedInformation

MultipleData Sources

The Combination of Retrieved and Generated Media Objects for Virtual Webpages


RetrievedResults

Select & Design

Select CannedMedia Objects

Design NewMedia Objects

Information Structures Relations, Lists KR Terms

Graphics,Animation

Text, Speech,Mimic

Icons, Clip Art

Frames, Sounds

Reuse & Transform

CoordinateMedia Objects

TransformMedia Objects

Temporal Synchroni-zation

SpatialLayout

Clip, Convert,Abstract

Zoom, Pan,TransitionEffects

Media Objects Texts, Sounds, Videos Pictures, Maps,

Animations

The Combination of Retrieved and Generated Media Objects for Virtual Webpages


Virtual Webpage with Animation Effects Based on a Single GIF Image


Transition Effects in a Series of Retrieved Pictures


Persona as a Personal Travel Consultant


NetbotPAN

Trip Data

Pictures andGraphics

Pieces of Text

Coordinates forPointing Gestures

Input for SpeechSynthesis

Icons forHyperlinks

HotelAgent

MapAgent

Address

WeatherAgent

Train & FlightScheduling

Agent

Major EventAgent

VirtualWeb

Presentation

Constraint-basedOnlineLayout

PresentationPlanner

PersonaServer

Componentsof virtual

Webpages

AiA

The Generation of Virtual Webpages with PAN and AiA


Persona as a Personal Travel Consultant


S-Include-Photo

S-Include-Text

S-Include-Link

Introduce

S-SpeakIllustrate

Design-Intro-Page Emphasize

This hotel has a nice swimming

pool.Location

Elaborate

S-Speak S-Point

S-Include-Map Label

Your hotel is located here.

Location Link Selected

Default Time Over/ Up

Dynamic Node Expansion for the Conditional Generation of Virtual Web Pages


Use of a Life-like Character for Electronic Commerce


Plug-Ins Applets

Add features (character players) to browser Download triggered by user Requires disk space on client Unrestricted access to client Less appropriate for WebCommerce,

Guides Agents integrated in 3D environments Appropriate for Entertainment

Examples:

Extempo's Jennifer James (Hayes-Roth et al. 98)

PFMagic's virtual petz

Java animation code sent over the net Automatic loading Requires no disk space on client Restricted access to client Appropriate for WebCommerce, Guides Agents integrated in 2D environments Less appropriate for Entertainment

Examples:

DFKI's Web Persona(Müller et al. 98)

ISI's Adele (Johnson et al 98)

New in AiA/PAN: Balanced combination of Applets and ServeletsEfficient distribution of client-side Java and server-side Java for driving theInterface Agent

Sending Interface Agents to Clients: Plug-Ins or Applets?


Queries

TransactionsShopbots

2Provider

pays usage fee 3 pays fee after successfulcommercial transaction

InternetShop 1

.

.

.

.

.InternetShop n

Provider1

pays fee forbanner avertisement

1

2

3

Banner fee

Usage fee

Transaction fee (credit card model)

Alternative Business Models for Shopbots


Advanced WebCommerce

Advanced WebCommerce

Virtual WebPages

One-to-OneMarketing

One-to-OneMarketing

Intuitive,Multilingual

Access

Dialogue withVirtual Sales

Agents

Shopbots forAutomated

ComparisonShopping

Text Analysis and Generation

User Modeling and Language GenerationCoordinated Text & Graphics Planning

Robust Dialogue Understanding

Advanced SpeechSynthesis

Information Extraction from HTML/XML Documents

MachineTranslation

MultimodalInterfaces

MultimediaPresentation

Planning

Intelligent Interface Technology is a Prerequisitefor Advanced WebCommerce


aa

Mensch-Technik-Interaktion

HOME/OFFICE

aaa


aa


SmartKom-Home/Office:A Versatile Agent-based Interface

SmartKom-Public: A Multimodal

CommunicationBooth

SmartKom-Mobile: A Handheld

CommunicationAssistant

Media Analysis

Kernel ofSmartKomInterface

Agent

Interaction Management

ApplicationManage-

ment

MediaDesign

SmartKom: A Transportable and Transmutable Interface Agent


MediaInterfaceEuropean Media LabUinv. Of

MunichUniv. ofStuttgart

Saarbrücken

Aachen

Dresden Berkeley

Stuttgart

MunichUniv. ofErlangen

Heidelberg

Main ContractorProject Management

TestbedSoftware Integration

DFKISaarbrücken

The SmartKom Consortium:

Project Budget: $ 34 MProject Duration: 4 years

DAIMLERCHRYSLERUlm

SmartKom: Intuitive Multimodal Interaction


Smartcard/ Credit Cardfor authentication and billing

Docking stationfor PDA/Notebook/Camcorderhigh speed and broadbandwidth Internet connectivity

High-resolution scanner

Loudspeaker

Room microphone

Face-tracking camera

Virtual touchscreenprotected against vandalism

Multipoint video conferencing

SmartKom-Public:A Multimodal Communication Booth


MOBILE

Camera

GPS

Microphone

Loudspeaker

Stylus-Activated Sketch Pad

WearableComputeServer

Docking Stationfor Car PC

Biosensorfor Authentication

& Emotional Feedback

GSM for Telephone,Fax, Internet Connectivity

SmartKom-Mobile: A Handheld Communication Assistant


SpeechMikeVirtual Touchscreen

Natural Gesture Recognition

SmartKom-Home/Office: A Versatile Agent-based Interface


User(s)

MediaAnalysis

Design

Media Fusion

OutputRendering

Representation and Inference

UserModel

DiscourseModel

DomainModel

TaskModel

MediaModels

InteractionManagement

MediaAnalysis

InputProcessing

Information

Applications

People Intention

RecognitionMediaDesign

Ap

pli

cati

on

In

terf

aceDiscourse

Modeling

UserModeling

PresentationDesign

Language

Graphics

Gesture

Biometrics

Language

Graphics

Gesture

AnimatedPresentation

Agent

The Architecture of the SmartKom Agent (cf. Maybury/Wahlster 1998)


Personalized Presentation Agents (limited user interaction)

WIP: 1989-1993

1

Personalized Interface Agents (full user interaction)

2PPP:1994-1996

Multiple Interface Agents (agent-agent and user interaction)3

Multiple Presentation Agents in one scene (eg. pros and cons)

Multiple Role-Taking (eg. Travel Assistance vs. Comparison Shopper)

Multiple Interface Agents (eg. human-computer, human-human interaction)

AiA: 1997-2000

Our Research Agenda for Agent-based Interfaces(Wahlster, André, Rist, Müller, Graf etc; www.dfki.de/imedia)


I recommend you this SLX limousine.

Multiple Agents Discussing Pros and Cons of a Mercedes Model


IntelligentWeb

Services

Intelligent Web

Services

UserModeling

Planning

Natural LanguageUnderstanding

KnowledgeRepresentation

ImageUnderstanding

MachineLearning

PlanRecognition

InformationRetrievalMultimodal

UserInterfaces

Research on Intelligent Web Services brings disparate subfields in the area of intelligent systems together


ECommerce projects of DFKI have shown that research on agent-based multimodal interfaces can be transferred to real real-world applications:

Dekra (largest European organization of used car dealers): FairCar as an ECommerce platform with NL access and a comparison shopping agent for used cars

DaimlerChrysler: IKP for online user modelling in a one-to-one marketing system for Mercedes cars

Otto/Shopping24/Eddie Bauer (largest European mail order company): virtual sales agents for one-to-one marketing of fashion and computer hardware

Lufthansa/Condor: direct marketing of charter flights

Conclusion


Two Research Challenges:

Making the interface agents sensitive to temporary limitations of the user´s time and working memory capacity (cf. our READY project, Jameson et al., p. 79-85 in IUI99 Proceedings)

Making the agents instructible, so that they can learn from the user in a dialog, how to extract information in a new domain (cf. our PAN project, Bauer/Dengler, p. 153-156 in IUI99 Proceedings)

The generation of virtual web pages by agent-based multimodal interfaces leads to innovative applications in:

Electronic Commerce, Electronic TV Guides (EPG)Telelearning environments, Call Centers and Help Desks

Conclusion

agent based multimedia interaction

Technology

multiple web sites

generations of web sites

real web page

persona agent

intelligent agent

individual user

virtual webpage

enhancement of user