![Page 1: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/1.jpg)
German Research Center for Artificial IntelligenceDFKI GmbH
Stuhlsatzenhausweg 366123 Saarbruecken, Germany
phone: (+49 681) 302-5252/4162fax: (+49 681) 302-5341e-mail: [email protected]
WWW:http://www.dfki.de/~wahlster
Wolfgang Wahlster
SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies
Cyber Assist Consortium Second International Symposium- Information Environment for Mobile and Ubiquitous Computing Era -
Tokyo, 25 March 2003
![Page 2: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/2.jpg)
© W. Wahlster
Multimodal UMTS Systems
Intelligent Interaction with Mobile Internet Services
Access to web content and web services anywhere
and anytime
Access to corporate networks
and virtual private networks from any device
Access to edutainment and infotainment servicesAccess to edutainment
and infotainment services
Access to all messages (voice, email, multimedia, MMS)
from any single device
Access to all messages (voice, email, multimedia, MMS)
from any single device
PersonalizationPersonalization
LocalizationLocalization
![Page 3: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/3.jpg)
© W. Wahlster
MMDialogue
Back-Bone
Home:Consumer Electronics
EPG
Public:Cinema,
Phone, Fax,
Mail, Biometrics
Mobile:Car andPedestrianNavigation
Application
Layer
SmartKom-Mobile
SmartKom-Public
SmartKom-Home/Office
SmartKom: A Highly Portable Multimodal Dialogue System
![Page 4: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/4.jpg)
© W. Wahlster
A Demonstration of SmartKom’s MultimodalInterface for the Federal President of Germany Dr. Rau
![Page 5: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/5.jpg)
© W. Wahlster
SmartKom`s SDDP Interaction MetaphorSDDP = Situated Delegation-oriented Dialogue ParadigmAnthropomorphic Interface = Dialogue Partner
User
specifies goal delegates task
cooperate on problems
asks questions presents results
Service 1 Service 1
Service 2Service 2
Service 3Service 3
Webservices
PersonalizedInteraction Agent
See: Wahlster et al. 2001 , Eurospeech
![Page 6: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/6.jpg)
© W. Wahlster
SmartKom‘s Use of Semantic Web Technology
Three Layers of Annotations
PersonalizedPresentation
M3LContent high
Structure XML medium
Layout HTML low
![Page 7: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/7.jpg)
© W. Wahlster
SmartKom: Intuitive Multimodal Interaction
MediaInterfaceEuropean Media LabUinv. Of
MunichUniv. ofStuttgart
Saarbrücken
Aachen
Dresden Berkeley
Stuttgart
MunichUniv. ofErlangen
Heidelberg
Main ContractorScientific Director
W. Wahlster
DFKISaarbrücken
The SmartKom Consortium:
Project Budget: € 25.5 million, funded by BMBF (Dr. Reuse) and industryProject Duration: 4 years (September 1999 – September 2003)
Ulm
![Page 8: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/8.jpg)
© W. Wahlster
Outline of the Talk
1. The Markup Language Layer Model of SmartKom
2. Modality Fusion in SmartKom
3. The Role of the Semantic Web Language M3L
4. Providing Coherence in Multimodal Dialogs by Ontology-based Overlay
5. Conclusions
![Page 9: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/9.jpg)
© W. Wahlster
Personalization
Mapping Web Content Onto a Variety of Structures and Layouts
From the “one-size fits-all“ approach of static webpages to the “perfect personal fit“ approach of adaptive webpages
StructureXML1 XML2 XMLn
ContentM3L
LayoutHTML11 HTML1m HTML21 HTML2o HTML31 HTML3p
![Page 10: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/10.jpg)
© W. Wahlster
The Markup Language Layer Model of SmartKom
M3LMultiModal Markup
Language
OILOntology Inference
Layer
XMLSeXtended Markup Language Schema
RDFSResource Description Framework Schema
XMLeXtended Markup
LanguageRDF
Resource Description Framework
HTMLHypertext Markup
Language
![Page 11: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/11.jpg)
© W. Wahlster
Spoken Dialogue
Graphical Userinterfaces
GesturalInteraction
MultimodalInteraction
SmartKom: Merging Various User Interface Paradigms
Facial Expressions
Biometrics
![Page 12: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/12.jpg)
© W. Wahlster
Multimodal Input and Output in SmartKomFusion and Fission of Multiple Modalities
Input by the User
Output by the Presentation agent
Speech
Gesture
FacialExpressions
+
+
+
+
+
+
![Page 13: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/13.jpg)
© W. Wahlster
Symbolic and Subsymbolic Fusion of Multiple Modes
SpeechRecognition
GestureRecognition
ProsodyRecognition
Facial ExpressionRecognition
LipReading
SubsymbolicFusion
- Neuronal Networks- Hidden Markov Models
SymbolicFusion
- Graph Unification - Bayesian Networks
Reference Resolution and Disambiguation
Modality-Free Semantic Representation
![Page 14: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/14.jpg)
© W. Wahlster
Personalized Interaction with WebTVs via SmartKom (DFKI with Sony, Philips, Siemens)
User: Switch on the TV.
Smartakus: Okay, the TV is on.
User: Which channels are presenting the latest news right now?
Smartakus: CNN and NTV are presenting news.
User: Please record this news channel on a videotape.
Smartakus: Okay, the VCR is now recording the selected program.
Example: Multimodal Access to Electronic Program Guides for TV
![Page 15: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/15.jpg)
© W. Wahlster
Using Facial Expression Recognition forAffective Personalization
(3’) Smartakus: Which of these features do you want to see?
Processing ironic or sarcastic comments
(1) Smartakus: Here you see the CNN program for tonight.
(2) User: That’s great.
(3) Smartakus: I’ll show you the program of another channel for tonight.
(2’) User: That’s great.
![Page 16: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/16.jpg)
© W. Wahlster
The SmartKom Demonstrator System
Camera for Gestural Input
Microphone
Multimodal Control of TV-Set
Multimodal Control of VCR/DVD Player
Camera forFacial Analysis
![Page 17: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/17.jpg)
© W. Wahlster
Unification of Scored Hypothesis Graphs for Modality Fusion in SmartKom
Word HypothesisGraph with
Acoustic Scores
Clause and Sentence
Boundarieswith Prosodic
Scores
Scored Hypotheses
about the User‘s Emotional State
Gesture HypothesisGraph with Scores
of PotentialReference Objects
Intention RecognizerSelection of Most Likely Interpretation
Modality FusionMutual Disambiguation
Reduction of UncertaintyIntention Hypotheses Graph
![Page 18: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/18.jpg)
© W. Wahlster
SmartKom‘s Computational Mechanisms for Modality Fusion and Fission
Modality Fusion Modality Fission
OntologicalInferences
Unification
OverlayOperations
Planning
ConstraintPropagation
M3L: Modality-Free Semantic Representation
![Page 19: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/19.jpg)
© W. Wahlster
The Role of the Semantic Web Language M3LM3L (Multimodal Markup Language) defines the data exchange formats used
for communication between all modules of SmartKom
M3L is partioned into 40 XML schema definitions covering SmartKom‘s
discourse domains
The XML schema event.xsd
captures the semantic
representation of concepts
and processes in SmartKom‘s
multimodal dialogs
![Page 20: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/20.jpg)
© W. Wahlster
OIL2XSD: Using XSLT Stylesheets to Convert an OIL Ontology to an XML Schema
![Page 21: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/21.jpg)
© W. Wahlster
Using Ontologies to Extract Information from the Web
MyOnto-Movie
:title
:description
:actors
MyOnto-Person
:name:birthday
:director
Film.de-Movie
:title
:description
Kinopolis.de-Movie
:name :critics
:o-title
:main actor
Mapping of Metadata
![Page 22: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/22.jpg)
© W. Wahlster
I would like to send an email to Koiti.
<domainObject>
<sendTelecommunicationProcess>
<sender>....................</sender>
<receiver>..............</receiver>
<document>..........</document>
<email>...........</email>
</sendTelecommunicationProcess>
</domainObject>
M3L as a Meaning Representation Language for the User‘s Input
![Page 23: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/23.jpg)
© W. Wahlster
Exploiting Ontological Knowledge to Understand and Answer the User‘s Queries
<beginTime> <time> <function> <at> 2002-05-10T10:25:46 </at> </function></beginTime>
<domainObject> <epg> <broadcastDefault> <avMedium> <actors>
<actor><name>Schwarzenegger/name></actor>
</actors> </avMedium> <channel><name>Pro7</name></channel> </broadcastDefault> </epg> </domainObject>
Which movies with Schwarzenegger are shown on the Pro7 channel?
![Page 24: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/24.jpg)
© W. Wahlster
SmartKom’s Multimodal Dialogue Back-Bone
Communication BlackboardsData FlowContext Dependencies
Analyzers
ExternalServices
ModalityFusion
DiscourseModeling
ActionPlanning
ModalityFission
Generators
• Speech
• Gestures
• Facial Expressions
• Speech
• Graphics
• Gestures
Dialogue Manager
![Page 25: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/25.jpg)
© W. Wahlster
Smartkom‘s Three-Tiered Discourse Model
DO1 DO2
VO1
DO10
DO3 DO9
Modality Layer
Discourse Layer
System: This [] is a list of films showing in Heidelberg.
heidelberglist
LO2 LO3. . .
Domain Layer DomainObject1
ticket first
DO11 DO12
reserve
LO4 LO5 LO6
DomainObject2
GO1
. . .
. . .
User: Please reserve a ticket for the first one.
DO = Discourse Object, LO = Linguistic ObjectGO = Gestural Object, VO = Visual Object
cf. M. Löckelt et. al. 2002, N. Pfleger 2002
![Page 26: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/26.jpg)
© W. Wahlster
theater: MovieTheater
movie: Movie
reservationNumber: PositiveInteger
SmartKom’s Domain Model based on M3L• Used for communication in the back-bone• Frame-based ontology; representation as Typed Feature Structures in M3L (XML)
name: Stringdirector: Person
cast: PersonListyearOfProduction: PositiveInteger…
address: Addressseats: SeatStructure…
CinemaReservation
• Application objects composed of subobjects• Slots: Feature paths meaningful for the dialogue (entities that can be talked
about / referenced to); e.g. movie:director:lastName in a CinemaReservation object
• Slots can recursively contain other slots
firstName: StringlastName: String…
![Page 27: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/27.jpg)
© W. Wahlster
Overlay Operations Using the Discourse Model
Augmentation and Validation
– compare with a number of previous discourse states:
•fill in consistent information•compute a score
– for each hypothesis - background pair:
– Overlay (covering, background)
Covering:Background:
IntentionHypothesis
Lattice
SelectedAugmentedHypothesis
Sequence
![Page 28: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/28.jpg)
© W. Wahlster
The Overlay Operation Versus the Unification Operation
• Nonmonotonic and noncommutative unification-like operation
• Inherit (non-conflicting) background information
• two sources of conflicts:
– conflicting atomic values
overwrite background (old) with covering (new)
– type clash
assimilate background to the type of covering; recursion
Unification
Overlay
cf. J. Alexandersson, T. Becker 2001
![Page 29: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/29.jpg)
© W. Wahlster
Example for Overlay
User: "What films are on TV tonight?"
System: [presents list of films]
User: "That‘s a boring program, I‘d rather
go to the movies."
How do we inherit “tonight” ?
![Page 30: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/30.jpg)
© W. Wahlster
Domain Model: A Type Hierarchy of TFS
String:...
Time:entEntertainm
title
beginTime
...Cinema:
ePerformanccinema
...Channel:
Broadcastchannel
A named entertainment at
some time
A named TV program at some
time on some channel
A named Movie at some time at some cinema
![Page 31: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/31.jpg)
© W. Wahlster
Unification Simulation
Films on TV tonight
...:
"":
Broadcast
anychanneltonight
TimebeginTimeFail – type clash
String:...
Time:entEntertainm
title
beginTime
...Cinema:
ePerformanccinema
...Channel:
Broadcastchannel
...:
""Time:
ePerformanc
anycinematonightbeginTime
![Page 32: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/32.jpg)
© W. Wahlster
...:
""Time:
ePerformanc
anycinematonightbeginTime
Overlay Simulation
String:...
Time:entEntertainm
title
beginTime
...Cinema:
ePerformanccinema
...Channel:
Broadcastchannel
Go to the moviesFilms on TV tonight
...:
"":
Broadcast
anychanneltonight
TimebeginTime
...:
Time:ePerformanc
anycinemabeginTime
..."":
ePerformanc
tonightTimebeginTime
Assimilation
Background
Covering
![Page 33: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/33.jpg)
© W. Wahlster
"Formal" Definition Overlay
• Let– co be covering– bg be background
• Step 1:– Assimilate(co,bg)
T
bg
co
• Step 2:– Overlay(co,assimilate(co,bg))
• If co and bg are frames: recursion• If co is empty: use bg• If bg is empty: use co• If conflict: use co
![Page 34: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/34.jpg)
© W. Wahlster
Domain Models with Multiple Inheritance
• Assimilate(co,bg) – Compute the set of minimal
upper bounds (MUB)– Specialize the MUBs– Unify the specialized MUBs
T
co bg
• Overlay remains untouched
MUB MUB
![Page 35: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/35.jpg)
© W. Wahlster
Overlay - Scoring
• Four fundamental scoring parameters:– Number of features from Covering (co)– Number of features from Background (bg)– Number of type clashes (tc)– Number of conflicting atomic values (cv)
)()(
)()(),,,(
cvtcbgco
cvtcbgcocvtcbgcoscore
Codomain [-1,1]
Higher score indicates better fit (1 overlay(c,b) unify(c,b))
![Page 36: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/36.jpg)
© W. Wahlster
anychannel
avMedium
Tto
Tfrombetweenfunction
eveningdaytime
tonightdaydeicticdayinstantanalysis
timebeginTime
Broadcast
:
:
00:59:2326102001:
00:00:1826102001:::
:
::::
::
Analysis of U4:
Example: Enrichment and Validation
U4: What’s on TV tonight?S5: [Displays a list of films] Here you see a list of films running tonight.U6: That seems not very interesting, show me the cinema program.
Discourse context
...
:
00:59:2326102001:
00:00:1826102001:::
:
::::
::
avMedium
Tto
Tfrombetweenfunction
eveningdaytime
tonightdaydeicticdayinstantanalysis
timebeginTime
Broadcast
Heidelberg :name
Town:target
carnnsportatiomeansOfTra
iteratedctedTranslMotionDire
:
![Page 37: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/37.jpg)
© W. Wahlster
Example: Enrichment and Validation
U4: What’s on TV tonight?S5: [Displays a list of films] Here you see a list of films running tonight.U6: That seems not very interesting, show me the cinema program.
Discourse context
...
:
00:59:2326102001:
00:00:1826102001:::
:
::::
::
avMedium
Tto
Tfrombetweenfunction
eveningdaytime
tonightdaydeicticdayinstantanalysis
timebeginTime
Broadcast
...
:
:
avMedium
beginTime
ePerformanc
Analysis of U6:
Heidelberg :name
Town:target
carnnsportatiomeansOfTra
iteratedctedTranslMotionDire
:
anycinema
avMedium
Tto
Tfrombetweenfunction
eveningdaytime
tonightdaydeicticdayinstantanalysis
timebeginTime
ePerformanc
:
:
00:59:2326102001:
00:00:1826102001:::
:
::::
::
Overlay ( U6, U4)
Result: (Score: 0.8666)
![Page 38: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/38.jpg)
© W. Wahlster
Example: Enrichment and Validation
U4: What’s on TV tonight?S5: [Displays a list of films] Here you see a list of films running tonight.U6: That seems not very interesting, show me the cinema program.
Discourse context
...
:
00:59:2326102001:
00:00:1826102001:::
:
::::
::
avMedium
Tto
Tfrombetweenfunction
eveningdaytime
tonightdaydeicticdayinstantanalysis
timebeginTime
Broadcast
...
:
:
avMedium
beginTime
ePerformanc
Analysis of U6:
Heidelberg :name
Town:target
carnnsportatiomeansOfTra
iteratedctedTranslMotionDire
:
Overlay ( U6, U2)
...
:
:
avMedium
beginTime
ePerformanc
Result: (Score: -1)
![Page 39: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/39.jpg)
© W. Wahlster
Animation of Scoring Parameters
anychannel
avMedium
Tto
Tfrombetweenfunction
eveningdaytime
tonightdaydeicticdayinstantanalysis
timebeginTime
:
:
00:59:2326102001:
00:00:1826102001:::
:
::::
::
Broadcast
Background
...
:
:
ePerformanc
avMedium
beginTime
Covering
Number of features from Covering (co)Number of features from Background (bg)Number of type clashes (tc)Number of conflicting atomic values (cv)Result:
121
2
8666.0)01()122(
)01()122(),,,(
cvtcbgcoscore
0
![Page 40: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/40.jpg)
© W. Wahlster
The High-Level Control Flow of SmartKom
![Page 41: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/41.jpg)
© W. Wahlster
M3L Specification of a Presentation Task<presentationTask> <subTask> <presentationGoal> <inform> ... </inform> <abstractPresentationContent> ... <result> <broadcast id="bc1"> <channel> <name>EuroSport</name> </channel> <beginTime> <time> <at>2000-12-05T14:00:00</at> </time> </beginTime> <endTime> <time> <at>2000-12-05T15:00:00</at> </time> </endTime> <avMedium> <title>Sport News</title> <avType>sport</avType> ... </abstractPresentationContent> <interactionMode>leanForward</interactionMode> <goalID>APGOAL3000</goalID> <source>generatorAction</source> <realizationType>GraphicsAndSpeech</realizationType>
![Page 42: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/42.jpg)
© W. Wahlster
SmartKom‘s Presentation PlannerThe Presentation Planner generates a Presentation Plan by applying a set of Presentation Strategies to the Presentation Goal.
GlobalPresent
Present AddSmartakus DoLayout EvaluatePersonaNode
Inform
TryToPresentTVOverview
ShowTVOverview
ShowTVOverview SetLayoutDataSetLayoutData
ShowTVOverview
SetLayoutData SetLayoutData
PersonaAction
SendScreenCommand
....
...
...
Generation of Layout
Smartakus Actions
GenerateText
......
... Speak
cf. J. Müller, P. Poller, V. Tschernomas 2002
![Page 43: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/43.jpg)
© W. Wahlster
• Seamless integration and mutual disambiguation of multimodal input and output on semantic and pragmatic levels
• Situated understanding of possibly imprecise, ambiguous, or incom-plete multimodal input
• Context-sensitive interpretation of dialog interaction on the basis of dynamic discourse and context models
• Adaptive generation of coordinated, cohesive and coherent multimodal presentations
• Semi- or fully automatic completion of user-delegated tasks through the integration of information services
• Intuitive personification of the system through a presentation agent
Salient Characteristics of SmartKom
![Page 44: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/44.jpg)
© W. Wahlster
Various types of unification, overlay, constraint processing,
planning and ontological inferences are the fundamental
processes involved in SmartKom‘s modality fusion and
fission components.
The key function of modality fusion is the reduction of the
overall uncertainty and the mutual disambiguation of the
various analysis results based on a three-tiered
representation of multimodal discourse.
We have shown that a multimodal dialogue sytsem must not
only understand and represent the user‘s input, but its own
multimodal output.
Conclusions
![Page 45: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/45.jpg)
© W. Wahlster
First International Conference on Perceptive &Multimodal User Interfaces (PMUI’03)
November 5-7th, 2003 Delta Pinnacle Hotel, Vancouver, B.C., CanadaConference Chair Sharon Oviatt, Oregon Health & Science Univ., USAProgram Chairs Wolfgang Wahlster, DFKI, GermanyMark Maybury, MITRE, USA
PMUI’03 is sponsored by ACM, and will be co-located in Vancouver with ACM’s UIST’03. This meeting follows three successful Perceptive User Interface Workshops (with PUI’01 held in Florida) and three International Multimodal Interface Conferences initiated in Asia (with ICMI’02 held in Pittsburgh).
![Page 46: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/46.jpg)
© W. Wahlster
March 2003ISBN 0-262-06232-18 x 9, 392 pp., 98 illus.$40.00/£26.95 (CLOTH)
Edited by Dieter Fensel, James A. Hendler, Henry Lieberman and Wolfgang WahlsterForeword by Tim Berners-Lee
![Page 47: SmartKom: Modality Fusion for a Mobile Companion based on Semantic Web Technologies](https://reader034.vdocument.in/reader034/viewer/2022051517/568145fd550346895db30bdc/html5/thumbnails/47.jpg)
http://smartkom.dfki.de/