Application in Augmented and Virtual Reality
•
9 j'aime•4,173 vues
Talk given by Mark Billinghurst to Bajaj Finance Limited in India, on May 9th 2020. The talk describes AR and VR applications, example AR/VR applications in financial services, and potential research directions.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Application in Augmented and Virtual Reality
Similaire à Application in Augmented and Virtual Reality (20)
Plus de Mark Billinghurst
Plus de Mark Billinghurst (12)
Dernier
Dernier (20)
Application in Augmented and Virtual Reality
- 1. APPLICATIONS IN AUGMENTED AND VIRTUAL REALITY Mark Billinghurst mark.billinghurst@auckland.ac.nz Bajaj Finance Limited, May 9th 2020
- 2. IBM 360 - $10,000 a month to rent in 1965
- 3. The Incredible Disappearing Computer 1960-70’s Room 1970-80’s Desk 1980-90’s Lap 1990-2000’s Hand
- 6. Graphical User Interfaces • Separation between real and digital worlds • WIMP (Windows, Icons, Menus, Pointer) metaphor
- 7. Ubiquitous Computing/IoT • Embed computing and sensing in real world • Smart objects, sensors, etc..
- 8. Virtual Reality • Users immersed in Computer Generated environment • HMD, gloves, 3D graphics, body tracking
- 10. Virtual Reality •Defining Characteristics •Immersion •Interaction •Imagination Grigore Burdea (1993). Virtual Reality Technology
- 11. Augmented Reality • Virtual Images blended with the real world • See-through HMD, handheld display, viewpoint tracking, etc..
- 13. Augmented Reality •Defining Characteristics •Combines Real andVirtual Images •Interactive in real-time •Registered in 3D Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.
- 14. AR vsVR
- 15. Rekimoto, J. and Nagao, K. 1995. The world through the computer: computer augmented interaction with real world environments. Making Interfaces Invisible
- 16. From Reality toVirtual Reality Ubiquitous Computing Augmented Reality Virtual Reality Real World Virtual World
- 17. Milgram’s Mixed Reality Continuum Ubiquitous Computing Augmented Reality Virtual Reality Real World Virtual World Mixed Reality "...anywhere between the extrema of the virtuality continuum." P. Milgram and A. F. Kishino, (1994) A Taxonomy of Mixed Reality Visual Displays
- 19. Mixed Reality in 2020 • Large growing market • $11B in 2017, $19B in 2020 • Many available devices • HMD, phones, tablets, HUDs • Robust developer tools • Vuforia, WebGL, Unity, Wikitude, etc • Large number of applications • > 250K developers, > 100K mobile apps • Strong research/business communities • ISMAR, AWE, AugmentedReality.org, IEEE VR, etc
- 20. Market Growth
- 21. Market Breakdown
- 23. Using Mixed Reality in the Workplace • Blending digital/physical worlds.. • Productivity • Marketing • Training • Collaboration
- 24. Example: Productivity • MR technology for hands busy tasks • Warehouse picking (DHL, SAP) • Smart glasses show pick location • Simple user interface • Hands-free interaction • Improved performance RW VR
- 28. Example: Marketing • Add magic to company products • Snapchat lenses • Face tracking technology • Simple interaction • Animated content • Significant ROI • $750K for 224 million views • c.f. $5 million for SuperBowl ad RW VR
- 30. Example: Training • AR/VR training for difficult/dangerous tasks • Train with real objects, controls • GlobalSim Crane Simulator • Virtual Reality simulation • Train with real controls • Virtual body/hands • Realistic physics RW VR
- 32. Example: Collaboration • Use MR technology for remote collaboration • See through another’s eyes, virtual avatars • Microsoft Remote Assist • HoloLens see-through display • Remote video sharing • AR drawing/annotation • Document sharing • Group calling RW VR
- 34. FINTECH AR AND VR APP
- 35. Introducing AR/VR into the Workplace •Identify key real-world tasks • Involve spatial ability, physical interaction, etc. •Justify Return on Investment • Small technology investment leads to big return •Focus on Human Needs • User centered design •Work with Experts • Combine AR/VR experts with domain experts
- 36. Application Areas in Financial Services • Customer Experience • Customer Support • Agent Training • On-site Agent Support • Data Visualization • Meeting Support • Etc..
- 37. Enhancing Customer Experience – Westpac NZ https://www.youtube.com/watch?v=_dnKD2m_mPE
- 38. Customer Support – Virtual Agent – FidelityLabs https://www.youtube.com/watch?v=FwNsDQLiazU
- 39. Customer Support – Remote Insurance Claims https://www.youtube.com/watch?v=S59Xttnp2RY
- 40. Agent Training – Insurance Agents https://www.youtube.com/watch?v=IlSt4p5XTmY
- 41. On Site Agent Support – Building Assessment https://www.youtube.com/watch?v=a3_AtLFDisE
- 42. Data Visualization - Citibank https://www.youtube.com/watch?v=0NogltmewmQ
- 43. Meeting Support - Spatial https://www.youtube.com/watch?v=PG3tQYlZ6JQ
- 45. Key Enabling Technologies • Combines Real and Virtual Images • Display Technology • Interactive in real-time • Interaction Technologies • Registered in 3D • Tracking Technologies
- 46. • Past • Bulky Head mounted displays • Current • Handheld, lightweight head mounted • Future • Wide FOV see through • Retinal displays • Contact lens Display Technology
- 47. Wide FOV See-Through (3+ years) •Waveguide techniques • Thin, wider FOV • Socially acceptable •Pinlight Displays • LCD panel + point light sources • 110 degree FOV Lumus DK40
- 49. Retinal Displays (5+ years) • Photons scanned into eye • Infinite depth of field • Bright outdoor performance • True 3D stereo with depth modulation • Microvision (1993-) • Head mounted monochrome • MagicLeap (2013-) • Projecting light field into eye
- 50. Contact Lens (10 – 15 + years) • Contact Lens only • Unobtrusive • Significant technical challenges • Babak Parviz (2008) • Mojo Vision • https://www.mojo.vision/ • 14K ppi display • Over $100m investment http://spectrum.ieee.org/biomedical/bionics/augmented-reality-in-a-contact-lens/
- 51. Interaction • Past • Limited interaction • Viewpoint manipulation • Present • Screen based, simple gesture • Tangible interaction • Future • Natural gesture, Multimodal • Intelligent Interfaces, Sensor based
- 52. Natural Gesture (2+ years) • Freehand gesture input • Depth sensors for gesture capture • Rich two handed gestures • E.g. Microsoft Research Hand Tracker • 3D hand tracking, 30 fps, single sensor • Commercial Systems • Hololens2, Occulus, Leap Motion, Intel, etc..
- 54. Multimodal Input (5-10+ years) • Combine gesture and speech input • Gesture good for qualitative input • Speech good for quantitative input • Support combined commands • “Put that there” + pointing • E.g. HIT Lab NZ multimodal input • 3D hand tracking, speech, multimodal fusion • Complete tasks faster with MMI, less errors Billinghurst, M., Piumsomboon, T., & Bai, H. (2014). Hands in Space: Gesture Interaction with Augmented-Reality Interfaces. IEEE computer graphics and applications, (1), 77-80.
- 55. HIT Lab NZ Multimodal Input
- 56. Tracking • Past • Location-based, marker based • magnetic/mechanical • Present • Image based, hybrid tracking • Future • Ubiquitous • Model based • Environmental
- 57. Environmental Tracking (1-3+ yrs) • Environment capture • Use depth sensors to capture scene & track from model • InifinitAM (www.robots.ox.ac.uk/~victor/infinitam/) • Real time scene capture on mobiles, dense or sparse capture • Dynamic memory swapping allows large environment capture
- 59. 6D.ai AR Cloud Tracking https://www.youtube.com/watch?v=AwwU14gllS0
- 60. Wide Area Outdoor Tracking (5 + yrs) • Combine panorama’s into point cloud model (offline) • Initialize camera tracking from point cloud • Update pose by aligning camera image to point cloud • Accurate to 25 cm, 0.5 degree over very wide area Ventura, J., & Hollerer, T. (2012). Wide-area scene mapping for mobile visual tracking. In Mixed and Augmented Reality (ISMAR), 2012 IEEE International Symposium on (pp. 3-12). IEEE.
- 61. Wide Area Outdoor Tracking
- 62. Example – Holoscape Outdoor AR Game https://www.youtube.com/watch?v=Ndjdp9I6tFs
- 63. Other Research Areas • Social Acceptance • Overcome social problems with AR • Cloud Services • Cloud based storage/processing • AR Authoring Tools • Easy content creation for non-experts • Collaborative Experiences • AR teleconferencing • Etc..
- 64. Example:SocialAcceptance • People don’t want to look silly • Only 12% of 4,600 adults would be willing to wear AR glasses • 20% of mobile AR browser users experience social issues • Acceptance more due to Social than Technical issues • Needs further study (ethnographic, field tests, longitudinal)
- 66. Modern Technology Trends 1. Improved Content Capture • Move from sharing faces to sharing places 2. Increased Network Bandwidth • Sharing natural communication cues 3. Implicit Understanding • Recognizing behaviour and emotion
- 69. Empathic Computing Can we develop systems that allow us to share what we are seeing, hearing and feeling with others?
- 70. Empathy Glasses • Combine together eye-tracking, display, face expression • Implicit cues – eye gaze, face expression ++ Pupil Labs Epson BT-200 AffectiveWear
- 71. Remote Collaboration • Eye gaze pointer and remote pointing • Face expression display • Implicit cues for remote collaboration
- 73. Shared Sphere – 360 Video Sharing Shared Live 360 Video Host User Guest User
- 75. Shared Sphere System Features •AR / VR view collaboration • VR user feels that they are in AR users location •View independence • Allow one user to look around independently from another •View awareness cues • Show where the other user is looking •Hand gesture communication cues • Sharing hand gesture cues
- 76. 3D Live Scene Capture • Use cluster of RGBD sensors • Fuse together 3D point cloud
- 78. View Sharing Evolution • Increased immersion • Improved scene understanding • Better collaboration 2D 360 3D
- 79. Technology Trends • Advanced displays • Wide FOV, high resolution • Real time space capture • 3D scanning, stitching, segmentation • Natural gesture interaction • Hand tracking, pose recognition • Robust eye-tracking • Gaze points, focus depth • Emotion sensing/sharing • Physiological sensing, emotion mapping
- 80. Emotion Sensing HMDs are Coming •Looxid • EEG/Emotion sensing • Eye tracking •Emteq • EMG sensing •Neurable • EEG/BCI for VR
- 81. Technology Trends •Advanced displays •Real time space capture •Natural gesture interaction •Robust eye-tracking •Emotion sensing/sharing Empathic Tele-Existence
- 82. Empathic Tele-Existence • Move from Observer to Participant • Explicit to Implicit communication • Experiential collaboration – doing together
- 83. CONCLUSIONS
- 84. Conclusions •AR/VR in the workspace •Breaks barrier between real/digital worlds •Proven technology (display, input, sensing) •Ready for implementation •Future Research •Display, tracking, interaction, user experience •Empathic Computing