measuring human multimodal behavior and developing multimodal interfaces jean-claude martin...
TRANSCRIPT
Measuring Human Multimodal Behaviorand
Developing Multimodal Interfaces
Jean-Claude MARTIN
LIMSI-CNRS, FRANCE
LINC, University Paris 8, FRANCE
www.limsi.fr/Individu/martin
2
• Is it possible to identify a “minimal set” of “generic” types of cooperation between communication modalities that• are independent of the modalities• are observed in human behavior• should be used in computer I/O
behavior
3
CONTENT
1. Framework2. Measuring Behavior3. Futures applications
4
1. FRAMEWORKTYCOON (Martin 93)
5
1. FRAMEWORKDefinitions
6
2. MEASURING BEHAVIORSRI WOZ Toronto Map (Cheyer et al. 98)
• How to analyze the multimodal behavior ?
• How to specify a multimodal system ?
• Find a common grid ?• 10 annotated subjects
7
2. MEASURING BEHAVIORSRI WOZ Toronto Map (Cheyer et al. 98)
GRAPHICS Royal ontario museum selected
SPEECH Can you give me more information about the museum
GESTURE -
8
2. MEASURING BEHAVIORSRI WOZ Toronto Map
• [Kehker et al. 98]
Form Total
UnselectedSelected
ImmediateSelected Not
ImmediateUnselected
Selected Immediate
Selected Not Immediate
Full NP 20 32 5 10 4 0 71Definite NP 2 1 1 0 0 0 4"here" 0 0 0 5 3 0 8"there" 0 7 3 0 3 1 14"this" NP 0 0 0 2 10 0 12"that" NP 0 1 0 0 0 0 1"this" 0 4 0 8 5 0 17"they" 0 1 0 0 0 0 1"it" 0 6 0 0 2 0 8empty : 'Info" 0 22 2 13 1 0 38Total 22 74 11 38 28 1 174
No Gesture Simultaneous gesture
9
2. MEASURING BEHAVIORSRI WOZ Toronto Map (Cheyer et al. 98)
GRAPHICS Campbell House not selected but name displayed
SPEECH What is there in the Campbell House?
GESTURE Draws an arrow towards Campbell House
• Annotating and managing ambiguity• Annotate cooperation at a finer level• Nb modalities, nb of attributes
10
2. MEASURING BEHAVIORIST-ISLE Project
• 2000-2002• Natural Interaction and MultiModality • NIS, URome, IMS, DFKI, MPI, ELRA, LIMSI• Survey on Data Resources
• 22+5 on gestures + speech• Survey on Coding Schemes
• 10+3 on gestures + speech
http://isle.nis.sdu.dk/
11
2. MEASURING BEHAVIORXML DTD + Java
12
2. MEASURING BEHAVIORXML DTD + Java
For each object oFor each reference r
If the target of r is oThen sum += salienceOf o in r
Rate red-compl = sum / nbReferences
13
3. FUTURE DIRECTIONSApplication to teaching
• How do teachers modify their mm communication behavior depending on student level, time slots …
• Towards mm pedagogical agents• BQR Université Paris 8
14
3. FUTURE DIRECTIONSSpeech Therapist Corpus
• Multiple tasks• Goals
• Who types ?• Improvement
• Multimodal• Mediated• Collective gesture
15
3. FUTURE DIRECTIONSApplication to home automation
• Goals• Usability of mm for in-home application
(interactive TV) • Applying / Adapting the TYCOON methodology
• RNRT Project• 2 years• Partners :
• MediaLive (simulation room of in home environment)• Laboratoire Conception de Produits Nouveaux et Innovation
(Ecole Nationale Supérieure d’Arts et Métiers)
Measuring Human Multimodal Behaviorand
Developing Multimodal Interfaces
Jean-Claude MARTIN
LIMSI-CNRS, FRANCE
LINC, University Paris 8, FRANCE
www.limsi.fr/Individu/martin