1. Are Sensory EffectsReady for the World Wide Web? Christian Timmerer, Markus Waltl, and Hermann Hellwagner Klagenfurt University (UNIKLU)  Faculty of Technical Sciences (TEWI) Department of Information Technology (ITEC)  Multimedia Communication (MMC) http://research.timmerer.com  http://blog.timmerer.com  mailto:christian.timmerer@itec.uni-klu.ac.at 20 May 2010 Acknowledgments. This work was supported in part by the European Commission in the context of the NoE INTERMEDIA (NoE 038419), the P2P-Next project (FP7-ICT-216217), and the ALICANTE project (FP7-ICT-248652).

2. Outline Background / Introduction MPEG-V Media Context and Control Concept of MPEG-V Sensory Information Sensory Effect Description Language (SEDL) Software/Hardware components: SEVino, SESim, SEMP, and amBX+SDK Are Sensory Effects Ready for the World Wide Web? How to embed SEM into Web documents? How to interpret SEM within a browser? How to synchronize with audio/video content of the Web document? How to extract color information automatically? Conclusions and Future Work 2010/05/20 2 Christian Timmerer, Klagenfurt University, Austria

3. MPEG-V: Media Context and Control (MCC) 2010/05/20 Christian Timmerer, Klagenfurt University, Austria 3 System Architecture Pt. 1: Architecture Pt. 2: Control Information Pt. 3: Sensory Information Pt. 4: Virtual World Object Characteristics Pt. 5: Data Formats for Interaction Devices Pt. 6: Common Types and Tools Pt. 7: Conformance and Reference Software http://www.chiariglione.org/mpeg/working_documents.htm#MPEG-V

4. Concept of MPEG-V Sensory Information Consumption of multimedia content may stimulate also other senses Vision or audition Olfaction, mechanoreception, equilibrioception, thermoception, … Annotation with metadata providing so-called sensory effects that steer appropriate devices capable of rendering these effects 2010/05/20 Christian Timmerer, Klagenfurt University, Austria 4 … giving her/him the sensation of being part of the particular media ➪ worthwhile, informative user experience

5. Sensory Effect Description Language (SEDL) XML Schema-based language for describing sensory effects Basic building blocks to describe, e.g., light, wind, fog, vibration, scent MPEG-V Part 3, Sensory Information Adopted MPEG-21 DIA tools for adding time information (synchronization) Actual effects are not part of SEDL but defined within the Sensory Effect Vocabulary (SEV) Extensibility: additional effects can be added easily w/o affecting SEDL Flexibility: each application domain may define its own sensory effects Description conforming to SEDL :== Sensory Effect Metadata (SEM) May be associated to any kind of multimedia content (e.g., movies, music, Web sites, games) Steer sensory devices like fans, vibration chairs, lamps, etc. via an appropriate mediation device ➪ Increase the experience of the user ➪ Worthwhile, informative user experience 2010/05/20 Christian Timmerer, Klagenfurt University, Austria 5

6. Sensory Effect Description Language (cont’d) 2010/05/20 Christian Timmerer, Klagenfurt University, Austria 6 SEM ::=[DescriptionMetadata](Declarations|GroupOfEffects|Effect|ReferenceEffect)+ Declarations ::= (GroupOfEffects|Effect|Parameter)+ GroupOfEffects ::= timestamp EffectDefinitionEffectDefinition (EffectDefinition)* Effect ::= timestamp EffectDefinition EffectDefinition ::= [activate][duration][fade][alt] [priority][intensity][position] [adaptability]

7. SEVino, SESim, SEMP, and amBX amBX (Ambient Experience) system + SDK Two fan devices, a wrist rumbler, two sound speakers, a subwoofer, two lights, and a wall washer Everything controlled by SEM descriptionsexcept light effect 2010/05/20 Christian Timmerer, Klagenfurt University, Austria 7 Annotation Tool: SEVino Simulator: SESim Player: SEMP

8. Are Sensory Effects Ready for the World Wide Web? How to embed SEM into Web documents? How to interpret/parse SEM within a browser? Directly via JavaScript Browser plug-in How to synchronize with audio/video content of the Web document? HTML5’s video and audio elements and corresponding DOM attributes 2010/05/20 Christian Timmerer, Klagenfurt University, Austria 8 <link rel="alternate" type="text/xml” href="http://server.org/sem.xml”media="light, wind, vibration"/>

9. Are Sensory Effects Ready for the World Wide Web? (cont’d) Synchronization with audio/video (cont’d) currentTimeDOM attribute: current playback position in seconds timeupdateevent: indicates that the current playback position changed as part of normal playback 2010/05/20 Christian Timmerer, Klagenfurt University, Austria 9

10. Are Sensory Effects Ready for the World Wide Web? (cont’d) How to extract color information automatically? Used for controlling ambient lights currentFramedoes neither exist nor is foreseen in HTML5 canvas element allows for manipulating the display’s content (+) Pro: extract the pixel information from the display (-) Con: needs to be added explicitly during authoring 2010/05/20 Christian Timmerer, Klagenfurt University, Austria 10

11. Conclusions Introduction to MPEG-V Sensory Effects Are Sensory Effects Ready for the World Wide Web? – Yes, they are! Future work Implementation is ongoing Subjective quality assessments 2010/05/20 Christian Timmerer, Klagenfurt University, Austria 11

12. Thank you for your attention ... questions, comments, etc. are welcome … Ass.-Prof. Dipl.-Ing. Dr. Christian Timmerer Klagenfurt University, Department of Information Technology (ITEC) Universitätsstrasse 65-67, A-9020 Klagenfurt, AUSTRIA christian.timmerer@itec.uni-klu.ac.at http://research.timmerer.com/ Tel: +43/463/2700 3621 Fax: +43/463/2700 3699 © Copyright: Christian Timmerer 12 2010/05/20 Christian Timmerer, Klagenfurt University, Austria