1 PhD Public Defense, 25 June 2008
A Methodology for Developing Multimodal User Interfaces of
Information Systems
Adrian Stanciulescu
Université catholique de Louvain (UCL)Louvain School of Management (LSM)
Information Systems Unit (ISYS)Belgian Laboratory of Computer-Human Interaction (BCHI)
http://www.isys.ucl.ac.be/bchi
2 PhD Public Defense, 25 June 2008
Outline
• Context & Demonstration
• State of the art
• Thesis statement & Focus
• Methodology:
– Models & Language
– Method
– Tool support
• Validation
– External
– Internal
• Conclusion
• Future work
3 PhD Public Defense, 25 June 2008
Context
• 5 senses: sight, hearing, touch, smell, taste• Different user-system interaction types
graphical vocal tactile olfactory gustatory
• Prevailing interaction: graphical
• Shortcomings:– restrictive (graphical)– less flexible than human-to-human
interaction– difficult to use in mobile
environments with new emerging devices
4 PhD Public Defense, 25 June 2008
Multimodal systems
• New interaction paradigm:– Combines two or more individual interaction – Generic benefits: naturalness, flexibility, robustness, error
avoidance
• Specific benefits: – wide range of users (disabled persons)
– environment (mobile)
– platform (heterogenous devices)
5 PhD Public Defense, 25 June 2008
DemonstrationInput: graphical (A) Input: vocal (A) Input: multimodal (E)
6 PhD Public Defense, 25 June 2008
Outline
• Context
• State of the art
• Thesis statement & Focus
• Methodology:
– Models & Language
– Method
– Tool support
• Validation
– External
– Internal
• Conclusion
• Future work
7 PhD Public Defense, 25 June 2008
State of the art: Concerns
• Concerns – lack of: – structuring framework for MM UI development, support for MM I/O,
separation of modalities, combination of modalities, modality-independent models, extendibility for new modalities, method extendibility
Concerns of multimodal UIs
Chapter 2
Real world
Real world
State of the Art
(2) analyseChapter 1
Thesis statement:1. Models2. Method3. Tool support
Methodology
(1) define
Shortcomings
(3) identify
12 Requirements
(4) elicit
(5) guide (6) validate
8 PhD Public Defense, 25 June 2008
State of the art: Shortcomings
• Survey of 8 languages and 11 development tools
• Shortcomings: lack of– MM application deployment, fast interaction, error recovery, platform mobility, usable
MM UIs, robust systems, device effectiveness, MM experience
• Multimodal Teresa[Paterno, 2004]– Design space with unexplicit
options and limited alternatives– Mix of modality dependent and
independent elements– CARE: no support for R and C– Pre-computed and hard-coded
transformations
9 PhD Public Defense, 25 June 2008
State of art: Requirements
• Modeling:1. Support for MM I/O2. Separation of modalities3. Support for CARE properties in I/O4. Ability of modeling a UI independent of any modality5. Extendibility to new modalities6. Ontology homogeneity7. Human readability
• Method:8. Approach based on design space9. Method explicitness10.Method extendibility
• Tool:11.Machine processability of involved models12.Support for tool interoperability
10 PhD Public Defense, 25 June 2008
Outline
• Context
• State of the art
• Thesis statement & Focus
• Methodology:
– Models & Language
– Method
– Tool support
• Validation
– External
– Internal
• Conclusion
• Future work
11 PhD Public Defense, 25 June 2008
Thesis statement & Focus
• Working hypotheses:– Model-based approach: a set of models specifying different abstractions of the final UI – Semi-automated transformational approach: rules manually selected and automatically
applied
• Focus:– Information systems– Predefined and constant contexts of use– Graphical, vocal and MM interactions– MM UIs familiar to vast majority of users and available on most platforms– Target audience: HCI reasearch comunity, professionals in the field of MM interaction
Define a design space-based method that is supported by model-to-model colored transformations in order to obtain multimodal user interfaces
of information systems from a task and a domain models.
12 PhD Public Defense, 25 June 2008
Outline
• Context
• State of the art
• Thesis statement & Focus
• Methodology:
– Models & Language
– Method
– Tool support
• Validation
– External
– Internal
• Conclusion
• Future work
13 PhD Public Defense, 25 June 2008
Models & Language
• Cameleon Reference Framework:– Task Model – Domain Model– Abstract User Interface Model – Concrete User Interface Model– Mapping Model– Transformation Model
• UsiXML v1.8: – integrates our extensions– 3 dimensions of linguistics: semantics,
syntax, stylistics
Task & Concepts
Abstract User Interface
Concrete User Interface
Final User Interface
• extended CTT + extensions• modality independent vocabulary + extensions
• toolkit independent vocablary + extensions
• inter-model relationship mappings
• conceptualizes transformation rules + extensions
• UML class diagram
14 PhD Public Defense, 25 June 2008
SemanticsVocal Concrete expansion
• Vocal Containers:– vocalGroup
– vocalForm
– vocalMenu
– vocalConfirmation
• Relationships:– vocalTransition– vocalAdjacency– vocalContainment– synchronization
• EventTypes: – error/help/noInput/ noMatch
• Vocal Individual Components:– vocalOutput:
• vocalFeedback• vocalPrompt• vocalMenuItem• audio
– vocalInput– grammar/part/item– vocalNavigation/submit– connect– record– vocalVar/setVar/resetVar– if/else/elseif– break/exit
15 PhD Public Defense, 25 June 2008
SyntaxUsiXML language
• USer Interface eXtensible Markup Language
• Particular motivations:– Structured according to Cameleon Reference Framework– Ensures the independence of modality (Req.4. Ability to model modality-
independent UI )– Logically structures the transformation steps (Req. 9. Method
explicitness)– Flexibility for adding/deleting sub-steps (Req. 10. Method extendibility)– Transformational approach with transformations expressed in the same
formalism– Supported by tools processing its format (Req.11. Machine
processability of involved models)
16 PhD Public Defense, 25 June 2008
Language: Stylistics
vocalGroup
A dashed rectangle to suggest the containment purposes and a callout symbol to indicate the vocal character.
vocalForm
A dashed rectagle to suggest the containment intentions, a user and a system icon next to a callout symbol.
vocalMenu
As it is a container that enables to select among different options and by analogy with the menu provided by the graphical toolkits the representation is composed of a blue dashed oval to suggest the containment purpouses, an overlaying callout symbol to indicate the vocal aspects and yellow ovals to indicate the vocal options.
vocalPrompt
A system icon and a callout symbol containing a question sign.
vocalInput
A user icon next to a callout symbol and a system icon.
?
17 PhD Public Defense, 25 June 2008
Outline
• Context
• State of the art
• Thesis statement & Focus
• Methodology:
– Models & Language
– Method
– Tool support
• Validation
– External
– Internal
• Conclusion
• Future work
18 PhD Public Defense, 25 June 2008
Design space
Design option 1
Design option 2
Design option 3
Designoption 4
Design option 5
Designoption 6
Design option 7
Designoption n
Value 14
Value 13
Values 12
Value 11
Value 22
Value 21
Value 31 Value 32
Value 41
Value 42
Value 51
Value 52
Value n2
Value n3
Value n1
Value 63
Value 61
Value 62
Value 73 Value 72 Value 71
19 PhD Public Defense, 25 June 2008
Design space (cont’d)
• Advantages– Clarifies the development process: structured, less design effort, consistent
results
– Every piece of development is reflected in a design option
– Abstractions covered by a software that do not require any further interpretation effort
• Rationale– Intrinsic: descriptive, comparative, generative
– Extrinsic:
• implementation and tool-independent: useful for any designer
• explicit guidance: flexibility to set the CARE properties
• simplifies the design decision: reduces the complexity of the design space
20 PhD Public Defense, 25 June 2008
Design space (cont’d)
A structured combination of modality-independent design options having assigned a finite set of design option values that support the
stakeholder’s design decisions during the development life cycle of multimodal user interfaces
Design options for UIs
Sub-task presentation
Sub-task navigation
Navigation type
Control type
Sub-task guidance
Support for default value and unit
Answer cardinality
Answer order
Confirmation answer
Navigation and controltype
Sub-task triggering
Prompt
Input
Simpleoutput
Immediate feedback
Guidance
Definition, Rationale, Values
21 PhD Public Defense, 25 June 2008
Sub-task presentation
• Specifies the way in which the system is presenting the sub-tasks to the user.
Sub-task presentation
Combined
One at once Many at once All at once
separated
extended task list
reducedtasklist
tabbedlist
singleexpansion
list
multipleexpansion
list
separatedlist
groupedlist
bulletedlist
orderedlist
22 PhD Public Defense, 25 June 2008
Sub-task 1
Sub-task 2
Sub-task 3
Sub-task 1
Sub-task 2
Sub-task 3
Sub-task 1
Sub-task 2 Sub-task 3Sub-task 1
Sub-task 1
Sub-task 2
Sub-task 3Sub-task 1
Sub-task 2
Sub-task 3
Sub-task 1
Sub-task 3
Sub-task 2
Sub-task 1
Sub-task 2
Sub-task 3
Sub-task 1
Sub-task 2
Sub-task 3
1. Sub-task 1
2. Sub-task 2
3. Sub-task 3
Co
ncr
etiz
atio
in in
gra
ph
ical
ob
ject
sC
on
cret
izat
ion
in v
oca
l ob
ject
s
Sub-task 1
Sub-task 2
Sub-task 3
“Select an option”
?
“Which are my options?”
Sub-task 1
Sub-task 2
Sub-task 3
?“Sub-task 1”
?“Sub-task 2”
?“Sub-task 3”
Audio
Audio
Audio
Sub-task 1
Sub-task 2
Sub-task 3
?“Sub-task 1”
?“Sub-task 2”
?“Sub-task 3”
Audio (beep)
Audio (beep)
Audio (beep)
Sub-task 1
Sub-task 2
Sub-task 3
?“Sub-task 1”
?“Sub-task 2”
?“Sub-task 3”
Audio (“ 1 ”)
Audio (“ 2 ”)
Audio(“ 3 ”)
Sub-task 1
Sub-task 2
Sub-task 3
separatedextended
task list
reducedtasklist
tabbedlist
singleexpansion
list
multipleexpansion
list
separatedlist
groupedlist
bulletedlist
orderedlist
Sub-task 1
Sub-task 2
Sub-task 3
Sub-task 1
Sub-task 2
Sub-task 3Sub-task 1
Sub-task 2
Sub-task 3
23 PhD Public Defense, 25 June 2008
Design options for multimodal text field
• Prompting: multimodal (R)• Input: multimodal (E)• Immediate feedback: multimodal (R)• Guidance:
– Input: iconic (A)
– Immediate feedback: iconic (A)
Prompt: multimodal
Please specify your name
(vocal +graphical)1
Input: multimodal
Juan Gonzalez
2
(vocal +graphical)
Juan Gonzalez
Feedback:multimodal
Your answer was: Juan Gonzalez
3
(graphical + vocal)
Guidance for input: iconic
Guidance for feedback: iconic
24 PhD Public Defense, 25 June 2008
Design Rationale Approach
• Supports different alternative solutions for a design issue by providing a justification and enabling its evaluation
• Advantages:– detects the concistency and completeness issues in the early design phases– clarifies the reasons for a particular design solution and enables to argue it
• Example: Dream/Team ([Laca05 ])– Team notation– Dream tool
25 PhD Public Defense, 25 June 2008
• Existing shortcomings:– repeated common
parts– huge set of rules
• Contributions:– colored graph (rules) +
operations
• Benefits:– reduced number of
rules (250 -> 130)– high scalability
Colored ontology
26 PhD Public Defense, 25 June 2008
abstractContainment
isComposedOf
isReifiedBy
vocalContainments
vocalContainments
vocalContainment
isReifiedBy
abstractContainment
isComposedOfisReifiedBy
abstractContainment
isReifiedBy
graphicalContainment
isComposedOf
graphicalContainment
isReifiedBy
graphicalContainment
abstractContainment
isReifiedBy isComposedOf
isReifiedBy
NAC
LHS
RHS
Rule for GUI Rule for VUI
isReifiedBy
NAC
LHS
RHS
• merging
Operations over colored rules:
• splitting
Rule for MMUI
27 PhD Public Defense, 25 June 2008
Transformation rule catalog
• Complete and systematic arrangement of rules (130) into a structured catalog:– Transformation rules supporting design options
– Additional transformation rules
28 PhD Public Defense, 25 June 2008
Sub-task 1
Sub-task 2
Sub-task 3
Sub-task 1
Sub-task 2
Sub-task 3
Sub-task 1
Sub-task 2 Sub-task 3Sub-task 1
Sub-task 1
Sub-task 2
Sub-task 3Sub-task 1
Sub-task 2
Sub-task 3
Sub-task 1
Sub-task 3
Sub-task 2
Sub-task 1
Sub-task 2
Sub-task 3
Sub-task 1
Sub-task 2
Sub-task 3
1. Sub-task 1
2. Sub-task 2
3. Sub-task 3
Co
ncr
etiz
atio
in in
gra
ph
ical
ob
ject
sC
on
cret
izat
ion
in v
oca
l ob
ject
s
Sub-task 1
Sub-task 2
Sub-task 3
Sub-task 1
Sub-task 2
Sub-task 3
“Select an option”
?
“Which are my options?”
Sub-task 1
Sub-task 2
Sub-task 3
?“Sub-task 1”
?“Sub-task 2”
?“Sub-task 3”
Audio
Audio
Audio
Sub-task 1
Sub-task 2
Sub-task 3
?“Sub-task 1”
?“Sub-task 2”
?“Sub-task 3”
Audio (beep)
Audio (beep)
Audio (beep)
Sub-task 1
Sub-task 2
Sub-task 3
?“Sub-task 1”
?“Sub-task 2”
?“Sub-task 3”
Audio (“ 1 ”)
Audio (“ 2 ”)
Audio(“ 3 ”)
Sub-task 1
Sub-task 2
Sub-task 3
separatedextended
task list
reducedtasklist
tabbedlist
singleexpansion
list
multipleexpansion
list
separatedlist
groupedlist
bulletedlist
orderedlist
Sub-task 1
Sub-task 2
Sub-task 3Sub-task 1
Sub-task 2
Sub-task 3
R2
R2
R5,R6
R5,R6
R7,R8
R7,R8R9,R10
R11,R12
R11,R12
R9,R10 R13,R14
R15,R16
R15,R16
R17,R18
R17,R18
R19,R20
R19,R20
R21,R22
R21,R22
29 PhD Public Defense, 25 June 2008
Steps and sub-steps of the transformational approach
• Step 1: Task & Domain Models
• Step 2: From Task & Domain to AUI
• Step 3: From AUI
to CUI
• Step 4: From CUI
to FUI
sub-task presentationnavigation typecontrol type
30 PhD Public Defense, 25 June 2008
Outline
• Context
• State of the art
• Thesis statement & Focus
• Methodology:
– Models & Language
– Method
– Tool support
• Validation
– External
– Internal
• Conclusion
• Future work
31 PhD Public Defense, 25 June 2008
TransformiXML tool
2
Step 2
TransformiXML tool
2
Step 3
VoiceXML Generator tool
4
GrafiXML tool
3XHTML+Voice Generator tool
5
Tool support MultimodaliXML
Task & Domain Models
Abstract UI Model
Concrete UI Model
(graphical)
Concrete UI Model(vocal)
Concrete UI Model
(multimodal)
XHTML+Voicecode
IdealXML toolStep 1
Step 4
VoiceXML code
Multimodal BrowserVoiceXML Browser
Web browser
XHTML code
1
32 PhD Public Defense, 25 June 2008
Input: Graphical (A)Input: Vocal (A)Input: Multimodal
33 PhD Public Defense, 25 June 2008
Outline
• Context
• State of the art
• Thesis statement & Focus
• Methodology:
– Models & Language
– Method
– Tool support
• Validation
– External
– Internal
• Conclusion
• Future work
34 PhD Public Defense, 25 June 2008
External validation: case studies
• Feasability of the approach - 3 case studies ( ≠ levels of complexity and ≠ design options):– 2 web-form: virtual polling application (low complexity), car rental application
(medium complexity)– 1 non web-form: map browsing application
Abstract User Interface
Task and Domain
Concrete User Interface(multimodal)
Concrete User Interface(vocal)
Concrete User Interface(graphical)
Final User Interface(graphical)
Final User Interface(vocal)
Final User Interface(multimodal)
35 PhD Public Defense, 25 June 2008
Validation: Empirical assessment
• Assesment of 3 applications: – Session I (2 web-forms), Session II (1 non web-form)– Relative usability level of interaction modalities
• Input design option: G, V, MM (equivalence)
• Evaluation measures:– task completion time– error rate– interaction preference– task procentage completion– learning time– n° of mouse clicks
36 PhD Public Defense, 25 June 2008
Results: Task completion mean time
• Web-form: – Efficiency (G > MM > V)– Experienced faster then non-experienced
• experience has a significant influence for G (t-Test analysis p=0.0339)• higher dispersion for V and MM
37 PhD Public Defense, 25 June 2008
Results: Error rate
• Higher error rate for V and MM (synchronization and pronunciation errors )• Correlation (Pearson correlation analysis):
– task completion time (web form): V(Pearson= 0.75), true MM (Pearson= 0.73)– number of mouse clicks (web form): V(Pearson= 0.77), true MM (Pearson= 0.63)– task percentage completion (non-web form): V(Pearson= -0.72), MM (Pearson= -0.48)
38 PhD Public Defense, 25 June 2008
Results: Interaction modality preference
• Satisfaction: strong preference for MM all commands conveyed MM [Oviatt, 1997]– web-form: G ≈ MM > V– non web-form: G > V > MM– mix of monomodal (radio buttons, list box) and multimodal (check boxes with numerous items /
combo boxes with items to select at the end) [Oviatt, 1999]
39 PhD Public Defense, 25 June 2008
Outline
• Context
• State of the art
• Thesis statement & Focus
• Methodology:
– Models & Language
– Method
– Tool support
• Validation
– External
– Internal
• Conclusion
• Future work
40 PhD Public Defense, 25 June 2008
Internal validation: Requirements level of support
Req 1. Support for MM I/O
Req 2. Separation of modalities
Req 3. Support for CARE properties in I/O
Req 4. Ability of modeling a UI independent of any modality
Req 5. Extendibility to new modalities
Req 6. Ontology homogeneity
Req 7. Human readability
Req 8. Approach based on design space
Req 9. Method explicitness
Req 10. Method extendibility
Req 11. Machine processability of involved models
Req 12. Support for tool interoperability
Mo
del
ing
Met
ho
dT
oo
l
41 PhD Public Defense, 25 June 2008
Outline
• Context
• State of the art
• Thesis statement & Focus
• Methodology:
– Models & Language
– Method
– Tool support
• Validation– External– Internal
• Conclusion
• Future work
42 PhD Public Defense, 25 June 2008
ConclusionMethodology barriers
• Ontological– Lack of model expressiveness
• Method– Difficulty in defining an orthogonal design space
– Difficulty in finding an appropriate level of generalization when defining rules
– Difficulty in managing large sets of transformation rules:
• High reusability
• Low reusability
• Tool– Low interoperability: high number of software modules
43 PhD Public Defense, 25 June 2008
Conclusion (cont’d): Main Contribution
• Ontological contribution– Expanded task model to support the requirements of
MM UIs– Expanded vocal ontology– Multimodal instruction structure– Synchronization between the modalities– Formalization according to UsiXML syntax– Stylistics of vocal ontology
44 PhD Public Defense, 25 June 2008
Conclusion (cont’d): Main Contribution
• Method contribution– Design space
• 16 design options: low treshold, high ceiling, wide walls
– Expanded model-to-model transformation
• colored concepts → colored graph → colored rules
• operations over colored transformation rules
– Transformation rule catalog: identify rules supporting the design options
– 4 transformational development steps: identify the involved design options
– Introduction of a new methodogical sub-step: synchronization of CICs
45 PhD Public Defense, 25 June 2008
Conclusion (cont’d): Main Contribution
• Tool contribution– 5 interoperable software modules– Address specific steps of the development process– Proof of concept reasons
46 PhD Public Defense, 25 June 2008
Outline
• Context
• State of the art
• Thesis statement & Focus
• Methodology:
– Models & Language
– Method
– Tool support
• Validation
– External
– Internal
• Conclusion
• Future work
47 PhD Public Defense, 25 June 2008
Future work
• Extend the methodology to support development of context-aware systems: dynamic migration from one modality to another based on a set of rules
• Design space improvement: reduce dependent dimensions, introduce new dimensions and new values
• Knowledge base of inference rules: recommandation of suitable design options
• New interaction modalities: – ontological extension– method extension– tool support extension
48 PhD Public Defense, 25 June 2008
Future work (cont’d)
• Design space-based tool
49 PhD Public Defense, 25 June 2008
Thank you !
Questions ?