This presentation was given at the final QUALINET workshop in Delft and an updated version at EUVIP'14 in Paris and follows the general principle that there is a need for a scientific framework to capture, measure, quantify, judge, and explain the quality of (sensory) experience.
Building AI-Driven Apps Using Semantic Kernel.pptx
Quality of Sensory Experience (QuaSE)
1. Slides: http://www.slideshare.net/christian.timmerer
Quality of Experience beyond Audio-
Visual: Sensory Experience (QuaSE)
Christian Timmerer
Alpen-Adria-Universität Klagenfurt (AAU) Faculty of Technical Sciences (TEWI) Department of Information
Technology (ITEC) Multimedia Communication (MMC) Sensory Experience Lab (SELab)
blog.timmerer.com selab.itec.aau.at dash.itec.aau.at
christian.timmerer@itec.aau.at
5th European Workshop on Visual Information Processing (EUVIP’14)
December 10, 2014
Acknowledgments. This work was supported in part by the European Commission in the context of the NoE INTERMEDIA (NoE
038419), the P2P-Next project (FP7-ICT-216217), the ALICANTE project (FP7-ICT-248652), the SocialSensor project (FP7-ICT-287975),
and the COST Action IC1003 QUALINET.
Change log:
- Dec’14: EUVIP’14, Paris, France
- Oct’14: QUALINET Final Workshop,
Delft, The Netherlands
2. Sensory Experience
• Consumption of multimedia content may stimulate also
other senses
– Vision or hearing [incl. emotion, sensation]
– Olfaction, mechanoreception, thermoception, …
• Annotation with metadata providing so-called sensory
effects that steer appropriate devices capable of
rendering these effects
2014/12/10 Christian Timmerer, Alpen-Adria-Universität, Austria 2
3. Sensory Experience
• Consumption of multimedia content may stimulate also
other senses
– Vision or hearing [incl. emotion, sensation]
– Olfaction, mechanoreception, thermoception, …
• Annotation with metadata providing so-called sensory
effects that steer appropriate devices capable of
rendering these effects
… giving her/him the sensation of being part
of the particular mulsemedia
➪ worthwhile, informative user experience
2014/12/10 Christian Timmerer, Alpen-Adria-Universität, Austria 3
4. General Principle – Outline
• General principle: there is a need for a
scientific framework to capture, measure,
quantify, judge, and explain the quality of
(sensory) experience
• Outline
– [How to create, deliver, consume?]
– How to capture and measure?
– How to quantify?
– How to judge and explain?
2014/12/10 Christian Timmerer, Alpen-Adria-Universität, Austria 4
5. How to create, delivery, consume?
• Sensory Effect Description Language (SEDL)
– Basic building blocks to describe, e.g., light, wind, fog, vibration, scent
– MPEG-V Part 3, Sensory Information: Effects, GroupOfEffects
– Adopted MPEG-21 DIA tools for adding time information
(synchronization)
• Description conforming to SEDL :== Sensory Effect Metadata (SEM)
– Can be associated to any kind of multimedia content (e.g., movies,
music, Web sites, games)
– Support to be included in file (MP4) and transport (M2TS) formats
• Tool support for creating (annotation tools) and consumption
(players, Web plugins) ➜ selab.itec.aau.at
• Devices: e.g., amBX (Ambient Experience) system + SDK,
Gameskunk, Scentscape, etc.
2014/12/10 Christian Timmerer, Alpen-Adria-Universität, Austria 5
6. How to capture and measure?
• Subjective quality assessments
– Methodology: based on standard methods
– Test content: different genres, manually annotated (cf. QUALINET DB)
• Experiment I
– Aim: Demonstrate sensory effects as a vital tool for enhancing the quality of
experience depending on the actual genre
• Experiment II
– Aim: investigate the relationship of the QoE to various video bit-rates of
multimedia contents annotated with sensory effects.
– Subjective quality gap between video resources annotated with and without
sensory effects at different bit-rates
• [Experiment III] ambient lights & different color calculation settings
• Experiment IV
– Aim: investigate the enhancement of the QoE and how users’ emotions are
elicited and influenced by Web videos annotated with and without sensory
effects
2014/12/10 Christian Timmerer, Alpen-Adria-Universität, Austria 6
7. Experiment II: Results
Test Sequences
Sequence Babylon A.D. Earth
Duration 35s 21s
Resolution 1280 x 544 1280 x 720
Motion High Low
Nr. of Effects W: 7; V: 9 W: 8; V: 1
Bit-rates Kbit/s PSNR Kbit/s PSNR
Low Quality 2154 38.93 2204 38.11
Medium Quality 3112 41.27 3171 40.65
High Quality 4044 42.95 4116 42.27
Highest Quality 6315 N/A 6701 N/A
MOS vs. PSNR/bit-rate for Earth.
M. Waltl, C. Timmerer H. Hellwagner, "Increasing the User Experience of Multimedia Presentations with
Sensory Effects," WIAMIS’10, Desenzano del Garda, Italy, April 2010.
2014/12/10 Christian Timmerer, Alpen-Adria-Universität, Austria 7
8. How to quantify?
• Experiment V
– Aim: towards a quality/utility model for QuaSE
• Stimuli with all
combinations of
sensory effects
– Vibration higher impact
than light & wind
– Highest QoE with all
effects present
• General QuaSE model
C. Timmerer, B. Rainer, M. Waltl, "A Utility Model for Sensory
Experience," QoMEX’13, Klagenfurt, Austria, 2013.
2014/12/10 Christian Timmerer, Alpen-Adria-Universität, Austria 8
9. How to judge and explain?
• Experiment VI
– Aim: understand QuaSE
• Biosensor-based QoE evaluation system
J. Donley, C. Ritz, M. Shujau, "Analysing the Quality
of Experience of Multisensory Media from
Measurements of Physiological Responses,”
QoMEX’14, Singapore, Sep. 2014.
2014/12/10 Christian Timmerer, Alpen-Adria-Universität, Austria 9
10. How to judge and explain?
• Experiment VII
– Aim: understand QuaSE
• EEG Correlates of Pleasant and Unpleasant Odor
Perception
E. Kroupi, A. Yazdani, J.-M. Vesin, T. Ebrahimi, "EEG Correlates of Pleasant and Unpleasant Odor
Perception," ACM TOMM, vol. 11, no. 1s, Sep. 2014.
2014/12/10 Christian Timmerer, Alpen-Adria-Universität, Austria 10
11. How to judge and explain?
• Experiment VIII
– Aim: understand QuaSE
• Multiple-Scent Enhanced Multimedia Synchronization
General temporal boundaries:
-10s to +15s are “in-sync”,
skew values beyond are “out-of-sync”
N. Murray, B. Lee, Y. Qiao, and G.-M. Muntean, "Multiple-Scent Enhanced Multimedia Synchronization,"
ACM TOMM, vol. 11, no. 1s, Sep. 2014.
2014/12/10 Christian Timmerer, Alpen-Adria-Universität, Austria 11
12. Conclusions
• From the need for a scientific framework to
capture, measure, quantify, judge, and
explain the quality of experience
• To …
– How to create, deliver, consume?
– How to capture and measure?
– How to quantify?
– How to judge and explain?
• Open issues?
✔︎
✔︎
✔︎
✔︎
Still Many!
2014/12/10 Christian Timmerer, Alpen-Adria-Universität, Austria 12
13. Open Issues / Challenges
• QoE assessment is a delicate mixture of ingredients and
choices
– Test & lab environment
– Test content
– Test methodology
– Data analysis
• (Semi-)Automatic content creation/annotation
• Towards large scale deployment
– Lessons learnt from 3D (disaster)
– 4D, 5D, xD – adding another dimension does not guarantee
success
• Holistic approach not feasible
– Need for much more specialized QuaSE models
2014/12/10 Christian Timmerer, Alpen-Adria-Universität, Austria 13
15. Sensory Experience Lab
http://selab.itec.aau.at/
Software and Services
Standardization
Publications
Media
Funding
2014/12/10 Christian Timmerer, Alpen-Adria-Universität, Austria 15
16. References
• Markus Waltl, Christian Timmerer, Hermann Hellwagner, “A Test-Bed for Quality of Multimedia Experience Evaluation of
Sensory Effects”, Proceedings of the First International Workshop on Quality of Multimedia Experience (QoMEX 2009), San
Diego, USA, July 29-31, 2009.
• C. Timmerer, J. Gelissen, M. Waltl, H. Hellwagner, “Interfacing with Virtual Worlds”, Proceedings of the NEM Summit 2009,
Saint-Malo, France, September 28-30, 2009.
• M. Waltl, C. Timmerer H. Hellwagner, “Increasing the User Experience of Multimedia Presentations with Sensory Effects”,
Proceedings of the 11th International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS’10),
Desenzano del Garda, Italy, April 12-14, 2010.
• M. Waltl, C. Timmerer, H. Hellwagner, “Improving the Quality of Multimedia Experience through Sensory Effects”,
Proceedings of the 2nd International Workshop on Quality of Multimedia Experience (QoMEX2010), Trondheim, Norway,
June 21-23, 2010.
• C. Timmerer, M. Waltl, B. Rainer, H. Hellwagner, “Assessing the Quality of Sensory Experience for Multimedia
Presentations”, Signal Processing: Image Communication, 2012.
• M. Waltl, C. Timmerer, B. Rainer, H. Hellwagner, ”Sensory Effects for Ambient Experiences in the World Wide Web”,
Multimedia Tools and Applications (MTAP), 2012.
• B. Rainer, M. Waltl, Eva Cheng, Muawiyath Shujau, C. Timmerer, Stephen Davis, Ian Burnett, Christian Ritz, H. Hellwagner,
”Investigating the Impact of Sensory Effects on the Quality of Experience and Emotional Response in Web Videos”,
Proceedings of 4th Int’l. Workshop on Quality of Multimedia Experience (QoMEX2012), 2012.
• M. Waltl, C. Timmerer, B. Rainer, H. Hellwagner, “Sensory Effect Dataset and Test Setups”, Proceedings of 4th Int’l.
Workshop on Quality of Multimedia Experience (QoMEX2012), 2012.
• C. Timmerer, B. Rainer, M. Waltl, “A Utility Model for Sensory Experience”, In Proceedings of the 5th International
Workshop on Quality of Multimedia Experience (QoMEX2013), 2013.
• J. Donley, C. Ritz, M. Shujau, “Analysing the Quality of Experience of Multisensory Media from Measurements of
Physiological Responses,” In Proceedings of the 5th International Workshop on Quality of Multimedia Experience
(QoMEX2014), Singapore, 2014.
• E. Kroupi, A. Yazdani, J.-M. Vesin, T. Ebrahimi, "EEG Correlates of Pleasant and Unpleasant Odor Perception," ACM TOMM,
vol. 11, no. 1s, Sep. 2014.
• N. Murray, B. Lee, Y. Qiao, and G.-M. Muntean, "Multiple-Scent Enhanced Multimedia Synchronization," ACM TOMM, vol.
11, no. 1s, Sep. 2014.
• G. Ghinea, C. Timmerer, W. Lin, and S. R. Gulliver. "Mulsemedia: State of the Art, Perspectives, and Challenges," ACM
TOMM, vol. 11, no. 1s, Sep. 2014.
2014/12/10 Christian Timmerer, Alpen-Adria-Universität, Austria 16