quality of sensory experience (quase)

Quality of Sensory Experience (QuaSE)

Chris8an Timmerer

Alpen-‐Adria-‐Universität Klagenfurt (AAU) w Faculty of Technical Sciences (TEWI) w Department of Informa8on Technology (ITEC) w Mul8media Communica8on (MMC) w Sensory Experience Lab (SELab)

blog.&mmerer.com w selab.itec.aau.at w dash.itec.aau.at chris&an.&[email protected]

QUALINET Final Workshop October 10, 2014

Acknowledgments. This work was supported in part by the European Commission in the context of the NoE INTERMEDIA (NoE 038419), the P2P-‐Next project (FP7-‐ICT-‐216217), the ALICANTE project (FP7-‐ICT-‐248652), the SocialSensor project (FP7-‐ICT-‐287975),

and the COST Ac&on IC1003 QUALINET.

Sensory Experience •  Consump&on of mul&media content may s&mulate also other senses –  Vision or hearing –  Olfac&on, mechanorecep&on, thermocep&on, …

•  Annota&on with metadata providing so-‐called sensory effects that steer appropriate devices capable of rendering these effects

2014/10/10 Chris&an Timmerer, Alpen-‐Adria-‐Universität, Austria 2

Sensory Experience •  Consump&on of mul&media content may s&mulate also other senses –  Vision or hearing –  Olfac&on, mechanorecep&on, thermocep&on, …

•  Annota&on with metadata providing so-‐called sensory effects that steer appropriate devices capable of rendering these effects


… giving her/him the sensa8on of being part of the par8cular mulsemedia ➪ worthwhile, informa8ve user experience

General Principle – Outline

•  General principle: there is a need for a scien8fic framework to capture, measure, quan,fy, judge, and explain the quality of (sensory) experience

•  Outline –  [How to create, delivery, consume?] – How to capture and measure? – How to quan&fy? – How to judge and explain?


How to create, delivery, consume? •  Sensory Effect Descrip&on Language (SEDL)

–  Basic building blocks to describe, e.g., light, wind, fog, vibra&on, scent –  MPEG-‐V Part 3, Sensory Informa&on: Effects, GroupOfEffects –  Adopted MPEG-‐21 DIA tools for adding &me informa&on

(synchroniza&on) •  Descrip&on conforming to SEDL :== Sensory Effect Metadata (SEM)

–  Can be associated to any kind of mul&media content (e.g., movies, music, Web sites, games)

–  Support to be included in file (MP4) and transport (M2TS) formats •  Tool support for crea&ng (annota&on tools) and consump&on

(players, Web plugins) ➜ selab.itec.aau.at •  Devices: e.g., amBX (Ambient Experience) system + SDK,

Gameskunk, Scentscape, etc.


How to capture and measure? •  Subjec&ve quality assessments

–  Methodology: based on standard methods –  Test content: different genres, manually annotated (cf. QUALINET DB)

•  Experiment I –  Aim: Demonstrate sensory effects as a vital tool for enhancing the quality of

experience depending on the actual genre •  Experiment II

–  Aim: inves&gate the rela&onship of the QoE to various video bit-‐rates of mul&media contents annotated with sensory effects.

–  Subjec&ve quality gap between video resources annotated with and without sensory effects at different bit-‐rates

•  [Experiment III] ambient lights & different color calcula&on seqngs •  Experiment IV

–  Aim: inves&gate the enhancement of the QoE and how users’ emo&ons are elicited and influenced by Web videos annotated with and without sensory effects


Experiment II: Results


Sequence Babylon A.D. Earth Duration 35s 21s Resolution 1280 x 544 1280 x 720 Motion High Low Nr. of Effects W: 7; V: 9 W: 8; V: 1 Bit-rates Kbit/s PSNR Kbit/s PSNR Low Quality 2154 38.93 2204 38.11 Medium Quality 3112 41.27 3171 40.65 High Quality 4044 42.95 4116 42.27 Highest Quality 6315 N/A 6701 N/A

Test Sequences

MOS vs. PSNR/bit-‐rate for Earth.

How to quan&fy?

•  Experiment V – Aim: towards a quality/u&lity model for QuaSE


•  S&muli with all combina&ons of sensory effects –  Vibra&on higher impact than light & wind

– Highest QoE with all effects present

•  General QuaSE model

How to judge and explain? •  Experiment VI –  Aim: understand QuaSE

•  Biosensor-‐based QoE evalua&on system


J. Donley, C. Ritz, M. Shujau, "Analysing the Quality of Experience of Mul&sensory Media from Measurements of Physiological Responses,” QoMEX2014, Singapore, Sep. 2014.

How to judge and explain?

•  Experiment VII – Aim: understand QuaSE

•  EEG Correlates of Pleasant and Unpleasant Odor Percep&on


E. Kroupi, A. Yazdani, J.-‐M. Vesin, T. Ebrahimi, "EEG Correlates of Pleasant and Unpleasant Odor Percep&on," ACM TOMM, vol. 11, no. 1s, Sep. 2014.

How to judge and explain?

•  Experiment VIII –  Aim: understand QuaSE

•  Mul&ple-‐Scent Enhanced Mul&media Synchroniza&on


N. Murray, B. Lee, Y. Qiao, and G.-‐M. Muntean, "Mul&ple-‐Scent Enhanced Mul&media Synchroniza&on," ACM TOMM, vol. 11, no. 1s, Sep. 2014.

General temporal boundaries: -‐10s to +15s are “in-‐sync”, skew values beyond are “out-‐of-‐sync”

Conclusions

•  From the need for a scien8fic framework to capture, measure, quan,fy, judge, and explain the quality of experience

•  To … – How to create, delivery, consume? – How to capture and measure? – How to quan&fy? – How to judge and explain?

•  Open issues? 2014/10/10 Chris&an Timmerer, Alpen-‐Adria-‐Universität, Austria 12

✔︎ ✔︎ ✔︎ ✔︎

Many!

Sensory Experience Lab hyp://selab.itec.aau.at/ Sozware and Services Standardiza&on Publica&ons Media Funding


References •  Markus Waltl, Chris&an Timmerer, Hermann Hellwagner, “A Test-‐Bed for Quality of Mul8media Experience Evalua8on of

Sensory Effects”, Proceedings of the First Interna&onal Workshop on Quality of Mul&media Experience (QoMEX 2009), San Diego, USA, July 29-‐31, 2009.

•  C. Timmerer, J. Gelissen, M. Waltl, H. Hellwagner, “Interfacing with Virtual Worlds”, Proceedings of the NEM Summit 2009, Saint-‐Malo, France, September 28-‐30, 2009.

•  M. Waltl, C. Timmerer H. Hellwagner, “Increasing the User Experience of Mul8media Presenta8ons with Sensory Effects”, Proceedings of the 11th Interna&onal Workshop on Image Analysis for Mul&media Interac&ve Services (WIAMIS’10), Desenzano del Garda, Italy, April 12-‐14, 2010.

•  M. Waltl, C. Timmerer, H. Hellwagner, “Improving the Quality of Mul8media Experience through Sensory Effects”, Proceedings of the 2nd Interna&onal Workshop on Quality of Mul&media Experience (QoMEX2010), Trondheim, Norway, June 21-‐23, 2010.

•  C. Timmerer, M. Waltl, B. Rainer, H. Hellwagner, “Assessing the Quality of Sensory Experience for Mul8media Presenta8ons”, Signal Processing: Image Communica8on, 2012.

•  M. Waltl, C. Timmerer, B. Rainer, H. Hellwagner, ”Sensory Effects for Ambient Experiences in the World Wide Web”, Mul&media Tools and Applica&ons (MTAP), 2012.

•  B. Rainer, M. Waltl, Eva Cheng, Muawiyath Shujau, C. Timmerer, Stephen Davis, Ian Burney, Chris&an Ritz, H. Hellwagner, ”Inves8ga8ng the Impact of Sensory Effects on the Quality of Experience and Emo8onal Response in Web Videos”, Proceedings of 4th Int’l. Workshop on Quality of Mul&media Experience (QoMEX2012), 2012.

•  M. Waltl, C. Timmerer, B. Rainer, H. Hellwagner, “Sensory Effect Dataset and Test Setups”, Proceedings of 4th Int’l. Workshop on Quality of Mul&media Experience (QoMEX2012), 2012.

•  C. Timmerer, B. Rainer, M. Waltl, “A U8lity Model for Sensory Experience”, In Proceedings of the 5th Interna&onal Workshop on Quality of Mul&media Experience (QoMEX2013), 2013.

•  J. Donley, C. Ritz, M. Shujau, “Analysing the Quality of Experience of Mul8sensory Media from Measurements of Physiological Responses,” In Proceedings of the 5th Interna&onal Workshop on Quality of Mul&media Experience (QoMEX2014), Singapore, 2014.

•  E. Kroupi, A. Yazdani, J.-‐M. Vesin, T. Ebrahimi, "EEG Correlates of Pleasant and Unpleasant Odor Percep8on," ACM TOMM, vol. 11, no. 1s, Sep. 2014.

•  N. Murray, B. Lee, Y. Qiao, and G.-‐M. Muntean, "Mul8ple-‐Scent Enhanced Mul8media Synchroniza8on," ACM TOMM, vol. 11, no. 1s, Sep. 2014.

•  G. Ghinea, C. Timmerer, W. Lin, and S. R. Gulliver. "Mulsemedia: State of the Art, Perspec8ves, and Challenges," ACM TOMM, vol. 11, no. 1s, Sep. 2014.


Thank you for your ayen&on

... ques&ons, comments, etc. are welcome …

Assoc-‐Prof. Dipl.-‐Ing. Dr. Chris&an Timmerer Klagenfurt University, Department of Informa&on Technology (ITEC)

Universitätsstrasse 65-‐67, A-‐9020 Klagenfurt, AUSTRIA chris&an.&[email protected]‐klu.ac.at

hyp://research.&mmerer.com/ Tel: +43/463/2700 3621 Fax: +43/463/2700 3699

© Copyright: Chris.an Timmerer

15 2014/10/10 Chris&an Timmerer, Alpen-‐Adria-‐Universität, Austria