fusion in multimodal interactive systems: an hmm-based algorithm for user-induced adaptation

2 December 2005

Fusion in Multimodal Interactive Systems: An HMM-Based Algorithm for User-Induced Adaptation

Bruno Dumas1, Beat Signer1 and Denis Lalanne2

1 WISE Research Lab, Vrije Universiteit Brussel, Belgium2 DIVA Research Group, University of Fribourg, Switzerland

Bruno Dumas – WISE resarch lab – Department of Computer Science - [email protected] 2June 26, 2012

Outline• Fusion of multimodal input

• Multimodal interaction: from design to implementation• Algorithms for fusion of multimodal input

• A Hidden Markov Model-based fusion algorithm• Instantiation of the algorithm

• Evaluation of the algorithm• Qualitative test• Quantitative evaluation• Performance assessment

• Conclusion


Fusion of Multimodal Input

Challenges in multimodal interaction engineering: continous and probabilistic inputs, real-time…


Challenging/Ambiguous Fusion Cases

• “Put-that-there” Complementarity case

• “Play next track” Different meanings based on order and context


CARE Properties at the Design Level

Describe how modalities can be combined Complementarity

All modalities are necessary

Assignment Absence of choice

Redundancy Same expressive power between modalities

Equivalence Any modality is sufficient


HephaisTK Multimodal Framework


Description of Multimodal Dialogues

• Syntactic-level description

• For more details: check SMUIML language


Fusion Algorithms

• Frame-based algorithms & Unification-based algorithms• Symbolic approaches• Both are a de facto standard

• Symbolic-statistical algorithms


Symbolic-Statistical Approaches

Symbolic data Interpretation of low-level data

Statistical processing Machine learning algorithms


Our HMM-Based Fusion Algorithm

Why hidden Markov models? Very good modelling of time-related events Takes advantage of input data coming from

probabilistic input modalities On-the-fly adaptation of the model

- e.g. based on user feedback


Put-That-There Example


Instantiation of the Algorithm


Evaluation of the Algorithm

Evaluation on three levels Qualitative “gut check” test Benchmarks on real-world examples Performance assessment


Evaluation: Qualitative Test

• Three expert users were asked to assess the behaviour of the algorithm frames-based and HMM-based fusion 5 minutes per condition + interview

• Coherent behaviour between the algorithms

• No noticeable difference in responsiveness


Evaluation: Benchmark


Benchmark Results (1)

• CARE properties: equivalence and redundancy tests



• Sequential and non-sequential complementarity tests



• “Play next track” example

Mea

ning

fra

mes

HM

M-b

ased


Evaluation: Performance

• Per fusion algorithm: • 5 runs of 40 input pieces• 5 x 20 expected fusion results

• Frame-based: 18.2 ms• Standard deviation: 12.7 ms

• HMM-based: 16.6 ms• Standard deviation: 11.6 ms


Conclusion

• A new symbolic-statistical multimodal fusion algorithm based on Hidden Markov Models

• Integrated into the HephaisTK framework

• Superior recognition rates compared to existing algorithms

• Handling of ambiguous cases

• Processing of real-time user feedback

http://wise.vub.ac.be/bruno-dumas

fusion in multimodal interactive systems: an hmm-based algorithm for user-induced adaptation

Technology