2022 COMP4010 Lecture1: Introduction to XR

1. INTRODUCTION TO XR COMP 4010 Lecture One Mark Billinghurst July 28th 2022 mark.billinghurst@unisa.edu.au

2. Who am I .. • Mark Billinghurst • Director of the Empathic Computing Lab • Univ. South Australia, Univ. of Auckland • Conducting research in: • Collaborative AR, AR/VR Interface Design, HCI • Empathic Computing • Previous worked at: • Google, Amazon, Nokia, British Telecom • MIT, Univ. of Washington, NAIST, Univ. of Canterbury • PhD from University of Washington (2002)

3. UniSA IVE - https://unisa.edu.au/research/IVE/

4. Class Logistics • Weekly lecture (2 hrs) • Thursday 10am – 12am • Room: Online • Weekly Practical (1 hr) • Thursday 9am – 10am • Assessment • 3 projects @ 20%, 30%, 40% • Class participation @ 10% • What you will need • iOS or Android phone/tablet • Access to laptop/PC for development

5. Equipment Available • VR Laboratory • 6 VR capable PCs • High end graphics cards • 3 Oculus Rift HMDs • Wide Field of View display • 1 HTC Vive HMD • Room scale tracking • IVE Equipment • 6 HoloLens2 AR displays • See through AR display • 10 Oculus Quest HMD • Self contained tracking

6. Snap Lens Studio Available from https://lensstudio.snapchat.com/

7. Unity3D Available from www.unity3d.com

8. Lecture Schedule – 13 lectures

9. Assessment • Assignment 1: Mobile AR (20%) • Develop a Snap Lens • AR Tracking and Interaction • Assignment 2: Interactive VR (30%) • Create an interactive VR scene • Oculus Quest, HTC Vive, Rift • Assignment 3: HMD Experience (40%) • Develop either VR HMD experience • Oculus Quest • Class Participation (10%) • Watch at least one short YouTube AR/VR video each week • Be prepared to present video each week

10. WHAT IS AR/VR/MR/XR?

11. 1967 – IBM 1401 – half of the computers in the world, $10,000/month to run

12. 2013 Google Glass

13. The Incredible Disappearing Computer 1960-70’s Room 1970-80’s Desk 1980-90’s Lap 1990-2000’s Hand 2010 - Head

14. Graphical User Interfaces • Separation between real and digital worlds • WIMP (Windows, Icons, Menus, Pointer) metaphor

15. Rekimoto, J. and Nagao, K. 1995. The world through the computer: computer augmented interaction with real world environments. Making Interfaces Invisible (c) Internet of Things

16. Internet of Things (IoT).. • Embed computing and sensing in real world • Smart objects, sensors, etc.. (c) Internet of Things

17. Virtual Reality (VR) • Users immersed in Computer Generated environment • HMD, gloves, 3D graphics, body tracking

18. The First VR Experience … https://www.youtube.com/watch?v=pAC5SeNH8jw

19. Virtual Reality Definition •Defining Characteristics • Sense of Immersion • User feels immersed in computer generated space • Interactive in real-time • The virtual content can be interacted with • Independence • User can have independent view and reaction to environment

20. David Zeltzer’s AIP Cube Autonomy – User can to react to events and stimuli. Interaction – User can interact with objects and environment. Presence – User feels immersed through sensory input and output channels Interaction Autonomy Presence VR Zeltzer, D. (1992). Autonomy, interaction, and presence. Presence: Teleoperators & Virtual Environments, 1(1),127-132.

21. VR Demo

22. Types of VR 2 2

23. Augmented Reality (AR) • Virtual Images blended with the real world • See-through HMD, handheld display, viewpoint tracking, etc..

24. Augmented Reality Definition •Defining Characteristics [Azuma 97] • Combines Real and Virtual Images • Both can be seen at the same time • Interactive in real-time • The virtual content can be interacted with • Registered in 3D • Virtual objects appear fixed in space Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.

25. Augmented Reality Star Wars - 1977

26. 2008 - CNN https://www.youtube.com/watch?v=thOxW19vsTg

27. https://www.youtube.com/watch?v=cCL3bFuC4IA

28. • Weak AR • Imprecise tracking • No knowledge of environment • Limited interactivity • Handheld AR • Strong AR • Very accurate tracking • Seamless integration into real world • Natural interaction • Head mounted AR Strong vs. Weak AR

29. Augmented RealityApplications

30. AR vs VR

31. From Reality to Virtual Reality Internet of Things Augmented Reality Virtual Reality Real World Virtual World

32. Milgram’s Mixed Reality (MR) Continuum Augmented Reality Virtual Reality Real World Virtual World Mixed Reality "...anywhere between the extrema of the virtuality continuum." P. Milgram and A. F. Kishino, (1994) A Taxonomy of Mixed Reality Visual Displays Internet of Things

33. Milgram’s Reality-Virtuality continuum Mixed Reality Reality - Virtuality (RV) Continuum Real Environment Augmented Reality (AR) Augmented Virtuality (AV) Virtual Environment "...anywhere between the extrema of the virtuality continuum." P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.

34. AugmentedVirtuality •VR with view of the real world

35. Extended Reality (XR) Augmented Reality Virtual Reality Real World Virtual World Mixed Reality Extended Reality Internet of Things

36. Ubiquitous Computing Continuum • Weiser Continuum

37. Milgram – Weiser Continuum

38. Mixed Reality Revisited (2019) • Interviewed experts + read papers • ”.. no universally agreed on, one-size-fits- all definition of MR” • Six notions of MR • Continuum, Synonym, Collaboration • Combination, Alignment, Strong AR Speicher, M., Hall, B. D., & Nebeling, M. (2019, May). What is mixed reality?. In Proceedings of the 2019 CHI conference on human factors in computing systems (pp. 1-15).

39. • adva

40. The Metaverse • Neal Stephenson’s “SnowCrash” (1992) ”.. When you live in a shithole, there's always the Metaverse, and in the Metaverse, Hiro Protagonist is a warrior prince.”

41. Defining the Metaverse • Real-time, 3D, Interactive, Social, Persistent AWE 2022 John Riccitiello CEO, Unity Technologies

42. Metaverse Definition (2007) • Metaverse Roadmap • http://metaverseroadmap.org/ • The Metaverse is the convergence of: • 1) virtually enhanced physical reality • 2) physically persistent virtual space

43. Metaverse Dimensions • Augmentation technologies that layer information onto our perception of the physical environment. • Simulation refers to technologies that model reality • Intimate technologies are focused inwardly, on the identity and actions of the individual or object; • External technologies are focused outwardly, towards the world at large;

44. Metaverse Components • Four Key Components • Virtual Worlds • Augmented Reality • Mirror Worlds • Lifelogging

45. Mirror Worlds • Simulations of external space/content • Capturing and sharing surroundings • Photorealistic content • Digital twins Matterport Deep Mirror Google Street View Soul Machines

46. Lifelogging • Measuring user’s internal state • Capturing physiological cues • Recording everyday life • Augmenting humans Apple Fitbit Shimmer OpenBCI

47. Sensing Immersing Augmenting Capturing

48. HISTORY OF AR/VR

49. History Timeline https://immersivelifeblog.files.wordpress.com/2015/04/vr_history.jpg

50. When anything new comes along, everyone, like a child discovering the world thinks that they’ve invented it, but you scratch a little and you find a caveman scratching on a wall is creating virtual reality in a sense. Morton Helig (Hammit 1993)

51. Early History (30,000 BC - ) The history of VR is rooted in human’s first attempts to reproduce the world around them

52. 1800’s – Capturing Reality • Panoramas (1790s) • Immersive paintings • Photography (1820-30s) • Oldest surviving photo (Niépce, 1826) • Stereo imagery (1830s) • Wheatstone (1832) • Brewster (1851) • Movies (1870s) • Muybridge (1878) • Roundhay Garden Scene (1888)

53. Stereo Viewers Wheatstone (1832) Brewster (1860)

54. Viewmaster (1939)

55. 3D Cinema Golden Era (1950-60s) • Polarized 3D projection or anaglyph (red/blue)

56. Pepper’s Ghost (1862) • Projecting onto glass to make ghost image appear on stage • Dates back to Giambattista della Porta (1584)

57. The Master Key (1901) – AR Glass "It consists of this pair of spectacles. While you wear them every one you meet will be marked upon the forehead with a letter indicating his or her character. The good will bear the letter 'G,' the evil the letter 'E.' … Thus you may determine by a single look the true natures of all those you encounter.” L. Frank Baum AR display showing if people are good or evil

58. 1900s – Interactive Experiences • Early Simulators (<1960s) • Flight simulation • Sensorama (1955) • Early HMDs (1960s) • Philco, Ivan Sutherland • Military + University Research (1970-80s) • US Airforce, NASA, MIT, UNC • First Commercial Wave (1980-90s) • VPL, Virtual i-O, Division, Virtuality • VR Arcades, Virtual Boy

59. Link Trainer (1929 – 1950s) • Flight Simulator Training • Full six degree of freedom rotation • Force feedback and motion control • Simulated instruments • Modeling common flight conditions • Over 500,000 pilots trained

60. Link Trainer Video (1966) • https://www.youtube.com/watch?v=MEKkVg9NqGM

61. Early HMD Patents

62. Early HUD (1958) F16 – Head Up Display Showing flight information over the real world

63. Sensorama (1955) • Created by Morton Heilig • Experience Theater • Multi-sensory • Visuals • Sound • Wind • Vibration • Smell • No financial support • Commercial failure

64. First HMD? (1961) • Philco Headsight – Remote Camera Viewing • Showing live camera view in monocular HMD

65. Ivan Sutherland (1960s) 6 6 Ivan Sutherland’s Head-Mounted Display (1968)

66. Sutherland Display

67. Super Cockpit (1965-80’s) • US Airforce Research Program • Wright Patterson Air Force Base • Tom Furness III • Multisensory • Visual, auditory, tactile • Head, eye, speech, and hand input • Addressing pilot information overload • Flight controls and tasks too complicated • Research only • big system, not safe for ejecting

68. LEEP Optics (1979) • Large Expanse, Extra Perspective optics • Developed by Eric Howlett • Lens design for extremely wide field of view • High resolution in centre, lower resolution in periphery • 90o direct FOV, 140o corneal FOV • Used as basis for most VR HMDs

69. LEEP Optics Design

70. The Data Glove (1981-82) • Precursor, Sayre Glove • Univ. of Illinois, 1977 • Thomas Zimmerman (1982) • Fiber optic bend sensors • Detecting finger bending • Commercialized by VPL • Mattel PowerGlove (1989)

71. VPL DataGlove Demo • https://www.youtube.com/watch?v=fs3AhNr5o6o

72. NASA VIEW/VIVED (1981-86) • Early HMD (McGreevy Humphries) • LCD “Watchman” displays • VIEW (Scott Fisher) • Polhemus tracker • LEEP-based HMD • 3D audio (Convolvotron) • DataGlove gesture input • Simple graphics

73. VPL Research (1985 – 1999) • First Commercial VR Company • Jaron Lanier, Jean-Jacques Grimaud • Provided complete systems • Displays, software, gloves, etc • DataGlove, EyePhone, AudioSphere

74. The University of North Carolina at Chapel Hill (1980s-1990s) 7 6 Head-Mounted Displays Tracking, Haptics, Applications

75. University of Washington (1989 - ) • Human Interface Technology Laboratory (HIT Lab) • Founded by Tom Furness III • Many AR/VR Innovations • Virtual Retinal Display • ARToolKit AR Tracking library • GreenSpace shared VR experience • VR and pain care • VR and Education

76. First Industrial Use of AR (1990’s) • 1992: Tom Caudell at Boeing coined the term “AR.” • Wire harness assembly application begun • Lead by Tom Caudell, and David Mizell

77. CAVE (1992) • Projection VR system • 3-6 wall stereo projection, viewpoint tracking • Developed at EVL, University of Illinois Chicago • Commercialized by Mechdyne Corporation(1996) C. Cruz-Neira, D. J. Sandin, T. A. DeFanti, R. V. Kenyon and J. C. Hart. "The CAVE: Audio Visual Experience Automatic Virtual Environment", Communications of the ACM, vol. 35(6), 1992, pp. 64–72.

78. CAVE Demo Video • https://www.youtube.com/watch?v=aKL0urEdtPU

79. Desktop VR - 1995 • Expensive - $150,000+ • 2 million polys/sec • VGA HMD – 30 Hz • Magnetic tracking

80. Mobile/Wearable Systems (1995) • 1995 Navicam (Rekimoto) • Handheld AR • 1997 Touring Machine (Feiner) • Backpack AR, GPS, see-through display • 1998 Tinmith (Thomas, UniSA) • Outdoor gaming, CAD

81. Virtual Reality was HOT! .. In 1995..

82. Rise of Commercial VR Companies • W Industries/Virtuality (1985 - 97) • Location based entertainment • Virtuality VR Arcades • Division (1989 – 1998) • Turn key VR systems • Visual programming tools • Virtual i-O (1993 -1997) • Inexpensive gamer HMDs • Sense8 (1990 - 1998) • WorldToolKit, WorldUp • VR authoring tools

83. Dactyl Nightmare • https://www.youtube.com/watch?v=L60wgPuuDpE

84. Overview of VR in the 1990’s

85. Development of AR Tools • 1996 CyberCode (Rekimoto) • First matrix code tracking • 1999 ARToolKit (Kato & Billinghurst) • Open source tracking library

86. Tracking Demos https://www.youtube.com/watch?v=TqGAqAFlGg0 ARToolKit Cybercode

87. • April 2007 Computer World • VRVoted 7th on list of 21 biggest technology flops • MS Bob #1

88. Lessons Learned • Don’t believe the hype • Not everything is better inVR • Many factors determine technology acceptance • Human Centered Design/Design for users • Need to move from Demo to Production • Profitable niche markets first • Follow the money

89. Mobile Phone AR (2005) • Mobile Phones • camera • processor • display • AR on Mobile Phones • Simple graphics • Optimized computer vision • Collaborative Interaction

90. AR Advertising (HIT Lab NZ 2007) • Txt message to download AR application (200K) • See virtual content popping out of real paper advert • Tested May 2007 by Saatchi and Saatchi

91. Wellington Zoo Demo https://www.youtube.com/watch?v=edTjuXcce_c

92. Eye of judgement (2007) • Sony Playstation3 game • First AR console game • Over 300,000 copies sold • Used Eye camera + tracking cards

93. 2007 - AR Reaches Mainstream • MIT Technology Review • March 2007 • One of the 10 most exciting technologies • Economist • Dec 6th 2007 • Reality, only better

94. Google Searches for AR • Cross over in 2009, with more interest in AR than VR

95. 2008 - Browser BasedAR • Flash + camera + 3D graphics • ARToolKit ported to Flash • High impact • High marketing value • Large potential install base • 1.6 Billion web users • Ease of development • Lots of developers, mature tools • Low cost of entry • Browser, web camera

96. Demo: GE Smart Grid • https://www.youtube.com/watch?v=vJO_AZkCL9U

97. 2008: Location Aware Phones Nokia Navigator Motorola Droid

98. Outdoor Information Overlay • Mobile phone based • Tag real world locations • GPS + Compass input • Overlay graphics on live video • Applications • Travel guide, Advertising, etc. • Wikitude, Metaio, Layar, etc.. • iOS/Android, Public API released

99. Layar Demo (2008) • https://www.youtube.com/watch?v=b64_16K2e08

100. AR in Magazines (2009- ) • Esquire Magazine • Dec 2009 issue • 12 pages AR content • Many Others • Wired • Colors • Red Bull • etc.

101. Esquire Demo

102. Google Glass (2011 - )

103. Google Glass Demo

104. VR Second Wave (2010 - ) • Palmer Luckey • HMD hacker • Mixed Reality Lab (MxR) intern • Oculus Rift (2011 - ) • 2012 - $2.4 million kickstarter • 2014 - $2B acquisition FaceBook • $350 USD, 110o FOV

105. The Oculus Kickstarter Video • https://www.youtube.com/watch?v=aNSYscbxFAw

106. Desktop VR in 2016 • Graphics Desktop • $1,500 USD • >4 Billion poly/sec • $600 HMD • 1080x1200, 90Hz • Optical tracking • Room scale

107. Oculus Rift Sony Morpheus HTC/Valve Vive 2016 - Rise of Consumer HMDs

108. HTC Vive • Room scale tracking • Gesture input devices

109. Example Vive App – Tilt Brush • https://www.youtube.com/watch?v=ijukZmYFX-0

110. VR2GO (2013) • MxR Lab • 3D print VR viewer for mobiles • Open source hardware + software • http://projects.ict.usc.edu/mxr/diy/vr2go/ • Early Mobile VR viewer

111. Google Cardboard • Released 2014 (Google 20% project) • >5 million shipped/given away • Easy to use developer tools + =

112. Multiple Mobile VR Viewers Available

113. Epson Moverio BT-300 ▪ Stereo see-through display ($700) ▪ 1280 RGB x 720 pixels, 23 degree FOV, 30Hz, 69g ▪ Android Powered, separate controller ▪ VGA camera, GPS, gyro, accelerometer

114. Smart Glasses Available

115. Social Mobile Camera AR Apps (2015 - ) • SnapChat - Lenses, World Lenses • Cinco de Mayo lens > 225 million views • Facebook - Camera Effects • Google – Word Lens/Translate

116. Hololens (2016) • Integrated system – Windows • Stereo see-through display • Depth sensing tracking • Voice and gesture interaction • Note: Hololens2 coming September 2019

117. ARKit/ARcore (2017) • Visual Inertial Odometry (VIO) systems • Mobile phone pose tracked by • Camera (Visual), Accelerometer & Gyroscope (Intertial) • Features • Plane detection, lighting detection, hardware optimisation • Links • https://developer.apple.com/arkit/ • https://developers.google.com/ar/

118. • https://www.youtube.com/watch?v=SgAnnwl2VB8 ARKIT3 Demo

119. MagicLeap ML-1 (2018) • Bi-Focal Display – two focus planes • Horizontal FoV of 40o, vertical FoV of 30o, diagonal value of 50o • 1280×960 resolution, Eye-tracking • Separate display and computer • Nvidia "Parker" Tegra X2 CPU, 8GB RAM, 128 GB storeage • 6 DOF handheld controller, magnetic tracking

120. History Summary • 1960’s – 80’s: Early Experimentation • 1980’s – 90’s: Basic Research • Tracking, displays • 1995 – 2005: Tools/Applications • Interaction, usability, theory • 2005 - : Commercial Applications • Mobile, Games, Medical, Industry

121. QUESTIONS ?

122. Resources • Azuma, R. T. (1997). A survey of augmented reality. Presence: teleoperators & virtual environments, 6(4), 355-385. • Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., & MacIntyre, B. (2001). Recent advances in augmented reality. IEEE computer graphics and applications, 21(6), 34-47. • Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems, 77(12), 1321-1329. • Billinghurst, M., Clark, A., & Lee, G. (2015). A Survey of Augmented Reality. Foundations and Trends in Human–Computer Interaction, 8(2-3), 73-272. • Speicher, M., Hall, B. D., & Nebeling, M. (2019, May). What is mixed reality?. In Proceedings of the 2019 CHI conference on human factors in computing systems (pp. 1-15).

123. www.empathiccomputing.org @marknb00 mark.billinghurst@auckland.ac.nz

2022 COMP4010 Lecture1: Introduction to XR

Vous êtes en train de lire un aperçu.

Consultez-les par la suite

2022 COMP4010 Lecture1: Introduction to XR

Plus De Contenu Connexe

Similaire à 2022 COMP4010 Lecture1: Introduction to XR (20)

Plus par Mark Billinghurst (20)

Plus récents (20)