Augmented Reality: Connecting physical and digital worlds

1. CONNECTING PHYSICAL AND DIGITAL WORLDS MARIUS PREDA, MPEG 3DG CHAIR, INSTITUT MINES TELECOM www.slideshare.net/MariusPreda

2. From centuries, we are building two different types of worlds Physical Informational Immersion, from a human user point of view, is making abstraction of the Physical world (remove all possible connection to it) and have stimuli only from the Informational one Immersion, presence, stimuli, perception, … stories music films knowledge

3. In which world our users are? The stimuli from physical worlds are familiar or at very low intensities

4. We intentionally block our interfaces with the physical world Improving immersion feeling

5. Strategies for blocking the interfaces with the Physical world Improving immersion feeling

6. Strategies for extending the space of the Informational world stimuli Improving immersion feeling

7. How far we’ve gone with this strategy? Relatively far … Improving immersion feeling

8. Probably too far … Improving immersion feeling How far?

9. It’s the time to connect our worlds

10. Why now? • For the first time, smartphones enable “everyone” to access Informational world while sensing also the Physical world • They change our view of the environment • Enhance the rich history and meaningfulness of the real world with powerful digital elements

11. Physical & Informational Augment The Reality An invitation: become architects of a new kind of bridges

12. In AR, almost everything remains to be invented Las Vegas This place some years ago

13. In AR, almost everything remains to be invented

14. • Marketing • Tourism • Culture • Education • Serious and less serious games • … • Hardware manufacturers • Software vendors In AR, almost everything remains to be invented

15. We (and few others) already started and invite you to be part of The AR Revolution In AR, almost everything remains to be invented

16. Tidy City

17. Portal Hunt

18. Elements

19. ARQuiz

20. Augmented Books

21. Summer School games

22. TECHNOLOGY INSIGHTS MPEG-V AND MPEG ARAF

23. MPEG technologies related to AR MPEG-1/2 (AV content) 1992/4 VRML 1997 • Part 11 - BIFS: -Binarisation of VRML -Extensions for streaming -Extensions for server command -Extensions for 2D graphics - Real time augmentation with audio & video • Part 2 - Visual: - 3D Mesh compression - Face animation 1998 • Part 2 – Visual - Body animation 1999 MPEG-4 v.1 MPEG-4 v.2 First form of broadcast signal augmentation

24. MPEG technologies related to AR MPEG-4 2003 •AFX 2nd Edition: - Animation by morphing - Multi-texturing 2005 • AFX 3rd Edition - WSS for terrain and cities - Frame based animation 2007 MPEG-4 MPEG-4 • Part 16 - AFX: - A rich set of 3D graphics tools - Compression of geometry, appearance, animation • AFX 4th Edition - Scalable complexity mesh coding 2011 MPEG-4A rich set of 3D Graphics representation and compression tools

25. MPEG technologies related to AR MPEG-4 2003 •AFX 2nd Edition: - Animation by morphing - Multi-texturing 2005 • AFX 3rd Edition - WSS for terrain and cities - Frame based animation 2007 MPEG-4 MPEG-4 • Part 16 - AFX: - A rich set of 3D graphics tools - Compression of geometry, appearance, animation • AFX 4th Edition - Scalable complexity mesh coding 2011 MPEG-4 2009 • Part 25 - Compression of third-party XML (X3D, COLLADA) MPEG-4 2004 • Part 16 - X3D Interactive Profile MPEG-4

26. MPEG technologies related to AR MPEG-V - Media Context and Control 2011 • 2nd Edition: - GPS - Biosensors - 3D Camera 201x • Compression of video + depth 201x MPEG-V - 3D Video • 1st Edition - Sensors and actuators - Interoperability between Virtual Worlds • Feature-point based descriptors for image recognition 201x CDVS MPEG-U – Advanced User Interface 2012 A rich set of sensors and actuators - 3D Audio MPEG-H

27. MPEG technologies related to AR MPEG-V – Media Context and Control

28. Actuators Light Flash Heating Cooling Wind Vibration Sprayer Scent Fog Color correction Initialize color correction parameter Rigid body motion Tactile Kinesthetic Global position command Sensors Light Ambient noise Temperature Humidity Distance Atmospheric pressure Position Velocity Acceleration Orientation Angular velocity Angular acceleration Force Torque Pressure Motion Intelligent camera type Multi Interaction point Gaze tracking Wind Dust Body height Body weight Body temperature Body fat Blood type Blood pressure Blood sugar Blood oxygen Heart rate Electrograph EEG , ECG, EMG, EOG , GSR Weather Facial expression Facial morphology Facial expression characteristics Geomagnetic Global position Altitude Bend Gas MPEG technologies related to AR MPEG-V – Media Context and Control

29. • All AR-related data is available from MPEG standards • Real time composition of synthetic and natural objects • Access to • Remotely/locally stored BIFS/compressed 2D/3D mesh objects • Streamed real-time BIFS/compressed 2D/3D mesh objects • Inherent object scalability (e.g. for streaming) • User interaction & server generated scene changes • Physical context • Captured by a broad range of standard sensors • Affected by a broad range of standard actuators Main features of MPEG AR technologies

30. MPEG vision on AR: the MPEG AR Browser Point to a URL – no need to download new applications for each context The browser • Retrieves scenario from the internet • Starts video acquisition • Tracks objects • Recognizes objects from visual signatures • Recovers camera pose • Gets streamed 3D graphics • Composes new scenes • Gets inputs from various sensors • Offers optimal AR experience by constantly adapting interaction possibilities and objects from a remote server. Industry • Maximize number of customers through MPEG-compliant authoring tools and browsers • No need to develop a new application for each use case and device platform

31. MPEG vision on AR MPEG-4 + MPEG-7 + MPEG-21 + + MPEG-U+ MPEG-V = ARAF MPEG Player Compression Authoring Tool Produce Download

32. Technology insights Modeling the content chain – Creation – Consumption – The pivot : a representation formalism

33. Technology insights: the chain ARAF Browser Media Servers Service Servers User Local Sensors & Actuators Remote Sensors & Actuators MPEG ARAF Local Real World Environment Remote Real World Environment Auth. Tools

34. Technology insights: Infrastructure ARAF Browser Media Servers Service Servers User Local Sensors & Actuators Remote Sensors & Actuators MPEG ARAF Local Real World Environment Remote Real World Environment Auth. Tools

35. The ARAF browser ARAF Browser • Full scene graph representation (2D and 3D) • Static and dynamic objects and scenes • Programmable behavior (JS) • Access to sensors and cameras • Access to remote resources and services (http request and web sockets) • Support for user interaction • Support for compressed resources

36. The ARAF browser ARAF Browser ARAF Browser JavaScript Engine MPEG Totem Services WebSockets Server Communication service WebSockets Client ARAF Browser SC3DMC Decoder BBA Decoder ARAF File BIFS Stream SC3DMC Stream BBA Stream BIFS Decoder MPEG-4 Scene IndexedFaceSet SBVCAnimation Camera Image data Reference Images Image data Image data Image data AFAR Browser Reference Signal Detection Detected Images

37. MyMultimediaWorld.com • Convert media assets in a common format • Describe, index, organize, search digital assets • Compressed assets, ready for transmission A multimedia repository and processing platform supporting both creation and consumption Media & Service Servers

38. Technology insights: the formalism ARAF Player Media Servers Service Servers User Local Sensors & Actuators Remote Sensors & Actuators MPEG ARAF Local Real World Environment Remote Real World Environment Auth. Tools

39. ARAF Description Language ARAF : an XML representation Player Compression Authoring Tool Produce Download ARAF

40. A set of scene graph nodes defined in MPEG-4 • Audio, image, video, graphics, programming, communication, user interactivity, animation • Map, MapMarker, Overlay, ReferenceSignal, ReferenceSignalLocation, Cam eraCalibration, AugmentedRegion Connection to sensors defined in MPEG-V • Orientation, Position, Angular Velocity, Acceleration, GPS, Geomagnetic, Altitude, Local camera(s) Compressed media • Image, sound, video, 2D/3D graphics Augmented Reality Application Format ARAF

41. Technology insights: creation ARAF Browser Media Servers Service Servers User Local Sensors & Actuators Remote Sensors & Actuators MPEG ARAF Local Real World Environment Remote Real World Environment Auth. Tools

42. Authoring ARAF • Specific tools • Generic tools – TOTEM.Designer – TOTEM.Scout Auth. Tools

43. Specific Authoring Tools • Scenario driven approach – Tidy City, Portal Hunt, ARQuiz, AR Book, Treasure Hunt Auth. Tools

44. Towards a generic Designer • Create templates for app. objects • Fill structure with data • Create GUI • Create application logic • Export data

45. On site designer

46. Conclusions • AR – the next digital revolution • For making revolutions, one needs weapons: – MPEG technologies represent an efficient and powerful set of weapons • Current MPEG activities – Improve and enrich the weapons – Package them in consistent and easy to use tools

Augmented Reality: Connecting physical and digital worlds

Augmented Reality: Connecting physical and digital worlds

Recommandé

Contenu connexe

Tendances

Tendances(20)

Similaire à Augmented Reality: Connecting physical and digital worlds

Similaire à Augmented Reality: Connecting physical and digital worlds(20)

Dernier

Dernier(20)

Augmented Reality: Connecting physical and digital worlds