2022 COMP4010 Lecture2: Perception

1. PERCEPTION COMP 4010 Lecture Two Mark Billinghurst August 4rd 2022 mark.billinghurst@unisa.edu.au

2. REVIEW

3. The Incredible Disappearing Computer 1960-70’s Room 1970-80’s Desk 1980-90’s Lap 1990-2000’s Hand 2010 - Head

4. Rekimoto, J. and Nagao, K. 1995. The world through the computer: computer augmented interaction with real world environments. Making Interfaces Invisible (c) Internet of Things

5. Internet of Things (IoT).. • Embed computing and sensing in real world • Smart objects, sensors, etc.. (c) Internet of Things

6. Virtual Reality (VR) • Users immersed in Computer Generated environment • HMD, gloves, 3D graphics, body tracking

7. Augmented Reality (AR) • Virtual Images blended with the real world • See-through HMD, handheld display, viewpoint tracking, etc..

8. AR vs VR

9. Milgram’s Mixed Reality (MR) Continuum Augmented Reality Virtual Reality Real World Virtual World Mixed Reality "...anywhere between the extrema of the virtuality continuum." P. Milgram and A. F. Kishino, (1994) A Taxonomy of Mixed Reality Visual Displays Internet of Things

10. Extended Reality (XR) Augmented Reality Virtual Reality Real World Virtual World Mixed Reality Extended Reality Internet of Things

11. Metaverse Components • Four Key Components • Virtual Worlds • Augmented Reality • Mirror Worlds • Lifelogging

12. Ivan Sutherland (1960s) 1 2 Ivan Sutherland’s Head-Mounted Display (1968)

13. Super Cockpit (1965-80’s) • US Airforce Research Program • Wright Patterson Air Force Base • Tom Furness III • Multisensory • Visual, auditory, tactile • Head, eye, speech, and hand input • Addressing pilot information overload • Flight controls and tasks too complicated • Research only • big system, not safe for ejecting

14. VPL Research (1985 – 1999) • First Commercial VR Company • Jaron Lanier, Jean-Jacques Grimaud • Provided complete systems • Displays, software, gloves, etc • DataGlove, EyePhone, AudioSphere

15. First Industrial Use of AR (1990’s) • 1992: Tom Caudell at Boeing coined the term “AR.” • Wire harness assembly application begun • Lead by Tom Caudell, and David Mizell

16. Desktop VR - 1995 • Expensive - $150,000+ • 2 million polys/sec • VGA HMD – 30 Hz • Magnetic tracking

17. Mobile/Wearable Systems (1995) • 1995 Navicam (Rekimoto) • Handheld AR • 1997 Touring Machine (Feiner) • Backpack AR, GPS, see-through display • 1998 Tinmith (Thomas, UniSA) • Outdoor gaming, CAD

18. Rise of Commercial VR Companies • W Industries/Virtuality (1985 - 97) • Location based entertainment • Virtuality VR Arcades • Division (1989 – 1998) • Turn key VR systems • Visual programming tools • Virtual i-O (1993 -1997) • Inexpensive gamer HMDs • Sense8 (1990 - 1998) • WorldToolKit, WorldUp • VR authoring tools

19. Mobile Phone AR (2005) • Mobile Phones • camera • processor • display • AR on Mobile Phones • Simple graphics • Optimized computer vision • Collaborative Interaction

20. 2008 - Browser Based AR • Flash + camera + 3D graphics • ARToolKit ported to Flash • High impact • High marketing value • Large potential install base • 1.6 Billion web users • Ease of development • Lots of developers, mature tools • Low cost of entry • Browser, web camera

21. 2008: Location Aware Phones Nokia Navigator Motorola Droid

22. VR Second Wave (2010 - ) • Palmer Luckey • HMD hacker • Mixed Reality Lab (MxR) intern • Oculus Rift (2011 - ) • 2012 - $2.4 million kickstarter • 2014 - $2B acquisition FaceBook • $350 USD, 110o FOV

23. Desktop VR in 2016 • Graphics Desktop • $1,500 USD • >4 Billion poly/sec • $600 HMD • 1080x1200, 90Hz • Optical tracking • Room scale

24. Oculus Rift Sony Morpheus HTC/Valve Vive 2016 - Rise of Consumer HMDs

25. Social Mobile Camera AR Apps (2015 - ) • SnapChat - Lenses, World Lenses • Cinco de Mayo lens > 225 million views • Facebook - Camera Effects • Google – Word Lens/Translate

26. Hololens (2016) • Integrated system – Windows • Stereo see-through display • Depth sensing tracking • Voice and gesture interaction • Note: Hololens2 coming September 2019

27. ARKit/ARcore (2017) • Visual Inertial Odometry (VIO) systems • Mobile phone pose tracked by • Camera (Visual), Accelerometer & Gyroscope (Intertial) • Features • Plane detection, lighting detection, hardware optimisation • Links • https://developer.apple.com/arkit/ • https://developers.google.com/ar/

28. History Summary • 1960’s – 80’s: Early Experimentation • 1980’s – 90’s: Basic Research • Tracking, displays • 1995 – 2005: Tools/Applications • Interaction, usability, theory • 2005 - : Commercial Applications • Mobile, Games, Medical, Industry

29. THE BUSINESS OF AR/VR

30. Why 2022 won’t be like 1996 • It’s not just VR anymore • Huge amount of investment • Inexpensive hardware platforms • Easy to use content creation tools • New devices for input and output • Proven use cases – no more Hype! • Most important: Focus on User Experience

31. Example: Pokemon GO Killer Combo: brand + social + mobile + geo-location + AR

32. Pokemon GO Effect • Fastest App to reach $500 million in Revenue • Only 63 days after launch, > $1 Billion in 6 months • Over 500 million downloads, > 25 million DAU • Nintendo stock price up by 50% (gain of $9 Billion USD)

33. Augmented Reality in 2022 • Large growing market • > $13Billion USD in 2021 • Many available devices • HMD, phones, tablets, HUDs • Robust developer tools • Vuforia, ARToolKit, Unity, Wikitude, etc • Large number of applications • > 150K developers, > 100K mobile apps • Strong research/business communities • ISMAR, AWE conferences, AugmentedReality.org, etc

34. Large Growing Industry

35. Conclusion • AR/VR has a long history • > 50 years of HMDs, simulators • Key elements for were in place by early 1990’s • Displays, tracking, input, graphics • Strong support from military, government, universities • First commercial wave failed in late 1990’s • Too expensive, bad user experience, poor technology, etc • We are now in second commercial wave • Better experience, Affordable hardware • Large commercial investment, Significant installed user base • Will XR be a commercial success this time?

36. PERCEPTION

37. What is Reality?

38. How do We Perceive Reality? • We understand the world through our senses: • Sight, Hearing, Touch, Taste, Smell (and others..) • Two basic processes: • Sensation – Gathering information • Perception – Interpreting information

39. Simple Sensing/Perception Model

40. Goal of Virtual Reality “.. to make it feel like you’re actually in a place that you are not.” Palmer Luckey Co-founder, Oculus

41. Creating the Illusion of Reality • Fooling human perception by using technology to generate artificial sensations • Computer generated sights, sounds, smell, etc

42. Reality vs. Virtual Reality • In a VR system there are input and output devices between human perception and action

43. Example Birdly - http://www.somniacs.co/ • Create illusion of flying like a bird • Multisensory VR experience • Visual, audio, wind, haptic

44. Birdly Demo https://www.youtube.com/watch?v=gHE6H62GHoM

45. PRESENCE

46. Presence .. “The subjective experience of being in one place or environment even when physically situated in another” Witmer, B. G., & Singer, M. J. (1998). Measuring presence in virtual environments: A presence questionnaire. Presence: Teleoperators and virtual environments, 7(3), 225-240.

47. Immersion vs. Presence • Immersion: describes the extent to which technology is capable of delivering a vivid illusion of reality to the senses of a human participant. • Presence: a state of consciousness, the (psychological) sense of being in the virtual environment. • So Immersion, defined in technical terms, is capable of producing a sensation of Presence • Goal of VR: Create a high degree of Presence • Make people believe they are really in Virtual Environment Slater, M., & Wilbur, S. (1997). A framework for immersive virtual environments (FIVE): Speculations on the role of presence in virtual environments. Presence: Teleoperators and virtual environments, 6(6), 603-616.

48. How to Create Strong Presence? • Use Multiple Dimensions of Presence • Create rich multi-sensory VR experiences • Include social actors/agents that interact with the user • Have environment respond to the user • What Influences Presence • Vividness – ability to provide rich experience (Steuer 1992) • Using Virtual Body – user can see themselves (Slater 1993) • Internal factors – individual user differences (Sadowski 2002) • Interactivity – how much users can interact (Steuer 1992) • Sensory, Realism factors (Witmer 1998)

49. Five Key Technical Requirements for Presence • Persistence • > 90 Hz refresh, < 3 ms persistence, avoid retinal blur • Optics • Wide FOV > 90 degrees, comfortable eyebox, good calibration • Tracking • 6 DOF, 360 tracking, sub-mm accuracy, no jitter, good tracking volume • Resolution • Correct stereo, > 1K x 1K resolution, no visible pixels • Latency • < 20 ms latency, fuse optical tracking and IMU, minimize tracking loop http://www.roadtovr.com/oculus-shares-5-key-ingredients-for-presence-in-virtual-reality/

50. Example: UNC Pit Room • Key Features • Training room and pit room • Physical walking • Fast, accurate, room scale tracking • Haptic feedback – feel edge of pit, walls • Strong visual and 3D audio cues • Task • Carry object across pit • Walk across or walk around • Dropping virtual balls at targets in pit • http://wwwx.cs.unc.edu/Research/eve/walk_exp/

51. Typical Subject Behaviour • Note – from another pit experiment • https://www.youtube.com/watch?v=VVAO0DkoD-8

52. Richie’s Plank • https://www.youtube.com/watch?v=4M92kfnpg-k

53. Why do people behave like this? • Presence can be decomposed into two dimensions (Slater 2009): • “Place Illusion” (PI): being in the place depicted in the VR environment • perception in VR matches natural sensorimotor input • Plausibility Illusion (Psi): the events in the VR environment are actually occurring • VR environment responds to user actions • When both PI and Psi are high, people respond realistically to events in the VR Slater, M. (2009). Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1535), 3549-3557. Presence = PI + Psi + ??

54. Slater, M., Banakou, D., Beacco, A., Gallego, J., Macia-Varela, F., & Oliva, R. (2022). A Separate Reality: An Update on Place Illusion and Plausibility in Virtual Reality. Frontiers in Virtual Reality, 81. Four Illusions of Presence (Slater 2022) • Place Illusion: being in the place • Plausibility Illusion: events are real • Body Ownership: seeing your body in VR • Copresence/Social Presence: other people are in VR

55. Social Presence • What makes a Person appear real? • Interactivity • Visual appearance • Audio cues • Touch • Contextual cues • Etc.. Oh, C. S., Bailenson, J. N., & Welch, G. F. (2018). A systematic review of social presence: Definition, antecedents, and implications. Frontiers in Robotics and AI, 5, 114.

56. Object Presence • What makes an object appear real? • Touch/Haptic feedback • Appearance • Lighting • Audio cues • Occlusion • Etc..

57. Benefits of High Presence • Leads to greater engagement, excitement and satisfaction • Increased reaction to actions in VR • People more likely to behave like in the real world • E.g. people scared of heights in real world will be scared in VR • More natural communication (Social Presence) • Use same cues as face-to-face conversation • Note: The relationship between Presence and Performance is unclear

58. Measuring Presence • Presence is very subjective so there is a lot of debate among researchers about how to measure it • Subjective Measures • Self report questionnaire • University College London Questionnaire (Slater 1999) • Witmer and Singer Presence Questionnaire (Witmer 1998) • ITC Sense Of Presence Inventory (Lessiter 2000) • Continuous measure • Person moves slider bar in VE depending on Presence felt • Objective Measures • Behavioural • reflex/flinch measure, startle response • Physiological measures • change in heart rate, skin conductance, skin temperature Presence Slider

59. PERCEPTION

60. Motivation • Understand: In order to create a strong sense of Presence we need to understand the Human Perception system • Stimulate: We need to be able to use technology to provide real world sensory inputs, and create the VR illusion VR Hardware Human Senses

61. Senses • How an organism obtains information for perception: • Sensation part of Somatic Division of Peripheral Nervous System • Integration and perception requires the Central Nervous System • Five major senses (but there are more..): • Sight (Opthalamoception) • Hearing (Audioception) • Taste (Gustaoception) • Smell (Olfacaoception) • Touch (Tactioception)

62. Relative Importance of Each Sense • Percentage of neurons in brain devoted to each sense • Sight – 30% • Touch – 8% • Hearing – 2% • Smell - < 1% • Over 60% of brain involved with vision in some way

63. Other Lessor Known Senses.. • Proprioception = sense of body position • what is your body doing right now • Equilibrium = balance • Acceleration • Nociception = sense of pain • Temperature • Satiety = state of being fed or gratified to or beyond capacity • Thirst • Micturition = amount of CO2 and Na in blood

64. Sight

65. The Human Visual System • Purpose is to convert visual input to signals in the brain

66. The Human Eye • Light passes through cornea and lens onto retina • Photoreceptors in retina convert light into electrochemical signals

67. Photoreceptors – Rods and Cones • Retina photoreceptors come in two types, Rods and Cones • Rods – 125 million, periphery of retina, no colour detection, night vision • Cones – 4-6 million, center of retina, colour vision, day vision

68. Human Horizontal and Vertical FOV • Humans can see ~135 o vertical (60 o above, 75 o below) • See up to ~ 210 o horizontal FOV, ~ 115 o stereo overlap • Colour/stereo in centre, black and white/mono in periphery

69. Vergence + Accommodation • saas

70. Vergence/Accommodation Demo • https://www.youtube.com/watch?v=p_xLO7yxgOk

71. Vergence-Accommodation Conflict • Looking at real objects, vergence and focal distance match • In VR, vergence and accommodation can miss-match • Focusing on HMD screen, but accommodating for virtual object behind screen

72. Visual Acuity Visual Acuity Test Targets • Ability to resolve details • Several types of visual acuity • detection, separation, etc • Normal eyesight can see a 50 cent coin at 80m • Corresponds to 1 arc min (1/60th of a degree) • Max acuity = 0.4 arc min

73. Stereo Perception/Stereopsis • Eyes separated by IPD • Inter pupillary distance • 5 – 7.5cm (avge. 6.5cm) • Each eye sees diff. image • Separated by image parallax • Images fused to create 3D stereo view

74. Depth Perception • The visual system uses a range of different Stereoscopic and Monocular cues for depth perception Stereoscopic Monocular eye convergence angle disparity between left and right images diplopia eye accommodation perspective atmospheric artifacts (fog) relative sizes image blur occlusion motion parallax shadows texture Parallax can be more important for depth perception! Stereoscopy is important for size and distance evaluation

75. Common Depth Cues

76. Depth Perception Distances • i.e. convergence/accommodation used for depth perception < 10m

77. Properties of the Human Visual System • visual acuity: 20/20 is ~1 arc min • field of view: ~200° monocular, ~120° binocular, ~135° vertical • resolution of eye: ~576 megapixels • temporal resolution: ~60 Hz (depends on contrast, luminance) • dynamic range: instantaneous 6.5 f-stops, adapt to 46.5 f-stops • colour: everything in CIE xy diagram • depth cues in 3D displays: vergence, focus, (dis)comfort • accommodation range: ~8cm to ∞, degrades with age

78. Creating the Perfect Illusion Cuervo, E., Chintalapudi, K., & Kotaru, M. (2018, February). Creating the perfect illusion: What will it take to create life-like virtual reality headsets?. In Proceedings of the 19th International Workshop on Mobile Computing Systems & Applications (pp. 7-12). • Technology to create life-like VR HMDs • Compared to current HMDs • 6 − 10× higher pixel density • 20 − 30× higher frame rate

79. Comparison between Eyes and HMD

80. Hearing

81. Anatomy of the Ear

82. Auditory Thresholds • Humans hear frequencies from 20 – 22,000 Hz • Most everyday sounds from 80 – 90 dB

83. Sound Localization • Humans have two ears • localize sound in space • Sound can be localized using 3 coordinates • Azimuth, elevation, distance

84. Sound Localization https://www.youtube.com/watch?v=FIU1bNSlbxk

85. Sound Localization (Azimuth Cues) Interaural Time Difference

86. HRTF (Elevation Cue) • Pinna and head shape affect frequency intensities • Sound intensities measured with microphones in ear and compared to intensities at sound source • Difference is HRTF, gives clue as to sound source location

87. Accuracy of Sound Localization • People can locate sound • Most accurately in front of them • 2-3° error in front of head • Least accurately to sides and behind head • Up to 20° error to side of head • Largest errors occur above/below elevations and behind head • Front/back confusion is an issue • Up to 10% of sounds presented in the front are perceived coming from behind and vice versa (more in headphones) BUTEAN, A., Bălan, O., NEGOI, I., Moldoveanu, F., & Moldoveanu, A. (2015). COMPARATIVE RESEARCH ON SOUND LOCALIZATION ACCURACY IN THE FREE-FIELD AND VIRTUAL AUDITORY DISPLAYS. InConference proceedings of» eLearning and Software for Education «(eLSE)(No. 01, pp. 540-548). Universitatea Nationala de Aparare Carol I.

88. Touch

89. Haptic Sensation • Somatosensory System • complex system of nerve cells that responds to changes to the surface or internal state of the body • Skin is the largest organ • 1.3-1.7 square m in adults • Tactile: Surface properties • Receptors not evenly spread • Most densely populated area is the tongue • Kinesthetic: Muscles, Tendons, etc. • Also known as proprioception

90. Cutaneous System • Skin – heaviest organ in the body • Epidermis outer layer, dead skin cells • Dermis inner layer, with four kinds of mechanoreceptors

91. Mechanoreceptors • Cells that respond to pressure, stretching, and vibration • Slow Acting (SA), Rapidly Acting (RA) • Type I at surface – light discriminate touch • Type II deep in dermis – heavy and continuous touch Receptor Type Rate of Acting Stimulus Frequency Receptive Field Detection Function Merkel Discs SA-I 0 – 10 Hz Small, well defined Edges, intensity Ruffini corpuscles SA-II 0 – 10 Hz Large, indistinct Static force, skin stretch Meissner corpuscles RA-I 20 – 50 Hz Small, well defined Velocity, edges Pacinian corpuscles RA-II 100 – 300 Hz Large, indistinct Acceleration, vibration

92. Spatial Resolution • Sensitivity varies greatly • Two-point discrimination Body Site Threshold Distance Finger 2-3mm Cheek 6mm Nose 7mm Palm 10mm Forehead 15mm Foot 20mm Belly 30mm Forearm 35mm Upper Arm 39mm Back 39mm Shoulder 41mm Thigh 42mm Calf 45mm http://faculty.washington.edu/chudler/chsense.html

93. Proprioception/Kinaesthesia • Proprioception (joint position sense) • Awareness of movement and positions of body parts • Due to nerve endings and Pacinian and Ruffini corpuscles at joints • Enables us to touch nose with eyes closed • Joints closer to body more accurately sensed • Users know hand position accurate to 8cm without looking at them • Kinaesthesia (joint movement sense) • Sensing muscle contraction or stretching • Cutaneous mechanoreceptors measuring skin stretching • Helps with force sensation

94. AR TECHNOLOGY

95. Augmented Reality Definition •Combines Real and Virtual Images •Both can be seen at the same time •Interactive in real-time •The virtual content can be interacted with •Registered in 3D •Virtual objects appear fixed in space

96. Augmented Reality technology •Combines Real and Virtual Images •Needs: Display technology •Interactive in real-time •Needs: Input and interaction technology •Registered in 3D •Needs: Viewpoint tracking technology

97. Example: MagicLeap ML-1 AR Display •Display • Multi-layered Waveguide display •Tracking • Inside out SLAM tracking •Input • 6DOF wand, gesture input

98. MagicLeap Display • Optical see through AR display • Overlay graphics directly on real world • 40o x 30o FOV, 1280 x 960 pixels/eye • Waveguide based display • Holographic optical element • Very thin physical display • Two sets of waveguides • Different focal planes • Overcomes vergence/accommodation problem • Eye tracking for selecting focal plane • Separate CPU/GPU unit

99. AR Vergence and Accommodation • Fixed focal distance for OST displays • Accommodation conflict between real and virtual object

100. Tracking • Inside out tracking • Sensors on the user’s head • Using multiple sensors • Time of Flight Depth Sensor • IR dot projector • Wide angle cameras • Internal accelerometer (IMU) • Creates 3D model of real world • Tracks from model

101. Spatial Mapping (Hololens)

102. Input • Multiple input methods • Handheld Controller • Multiple buttons, trackpad input • 6 DOF magnetic tracking • Eye gaze • Integrated eye tracking • Hand tracking • Natural hand input

103. Hand Tracking

104. Eye Tracking

105. 1: AR DISPLAYS

106. AR Display Technologies • Classification (Bimber/Raskar 2005) • Head attached • Head mounted display/projector • Body attached • Handheld display/projector • Spatial • Spatially aligned projector/monitor

107. Bimber, O., & Raskar, R. (2005). Spatial augmented reality: merging real and virtual worlds. CRC press. DisplayTaxonomy

108. HEAD MOUNTED DISPLAYS

109. Types of Head Mounted Displays Occluded See-thru Multiplexed

110. Optical see-through Head-Mounted Display Virtual images from monitors Real World Optical Combiners

111. ViewThrough Optical See-Through HMD

112. Optical Design - Birdbath ▪ Reflect off beam splitter

113. Optical Design – Curved Mirror ▪ Reflect off free-space curved mirror

114. Example: Meta2 • https://www.youtube.com/watch?v=e1W29w63W4g

115. Optical Design - Prism

116. Epson Moverio BT-300 ▪ Stereo see-through display ($700) ▪ 1280 RGB x 720 pixels, 23 degree FOV, 30Hz, 69g ▪ Android Powered, separate controller ▪ VGA camera, GPS, gyro, accelerometer

117. Optical Design - Waveguide • Use prisms/grating elements

118. Lumus Display

119. • https://www.youtube.com/watch?v=G2MtI7asLcA

120. Example: Sony Smart EyeGlasses https://www.youtube.com/watch?v=kYPWaMsarss

121. Hololens Waveguide Display

122. AR HMDs • Microsoft HoloLens2 - $3,500 USD • Wearable computer, 47 degree FOV • Waveguide displays, optical see-through • Vuzix Blade - $1000 USD • 30 degree FOV, optical see-through • Self contained, Monocular, Android OS • Epson BT 30C - $499 USD • 25 degree FOV, see-through • Tethered display, USB-C connector

123. Pros and Cons of Optical see-throughAR • Pros • Simpler design (cheaper) • Direct view of real world • No eye displacement • Socially acceptable (glasses form factor) • Cons • Difficult to occlude real world • Image washout outdoors/bright lights • Wide field of view challenging • Can’t delay the real world

124. Video see-through HMD Video cameras Monitors Graphics Combiner Video

125. ViewThrough aVideo See-Through HMD

126. Example: Varjo XR-1 • Wide field of view • 87 degrees • High resolution • 1920 x 1080 pixel/eye • 1440 x 1600 pixel insert • Low latency stereo cameras • 2 x 12 megapixel • < 20 ms delay • Integrated Eye Tracking

127. Varjo XR-1 Image Quality

128. • https://www.youtube.com/watch?v=L0sg-3EGbZs

129. Handheld AR • Camera + display = handheld AR • Mobile phone/Tablet display

130. Pros and Cons ofVideo See-ThroughAR • Pros • True occlusion • Digitized image of real world • Registration, calibration, matchable time delay • Wide FOV is easier to support • Cons • Larger, bulkier hardware • Can’t see real world with natural eyes

131. Multiplexed Display Virtual Image ‘inset’ into Real World

132. Example:Google Glass

133. ViewThrough Google Glass

134. See-Through Display Taxonomy Example Products Binocular See-Through Displays Monocular Optical See-Through Video See-Through Stereoscopic Overlays Monoscopic Overlays Single Camera Dual Camera Monoscopic Overlays Stereoscopic Overlays Stereoscopic Overlays Monoscopic Overlays Video See-Through E.g.: smartphone- or tablet-based hand-held AR Also: Google Glass in VST mode E.g.: Lumos DK-40 E.g.: Microsoft HoloLens, Epson Moverio BT-200, Vuzix STAR 1200XLD E.g.: Trivisio ARVision E.g.: Vuzix iWear VR920 with iWear CamAR Possible, but no clear advantage E.g.: Canon COASTAR, Vuzix Wrap 1200DXAR Optical See-Through E.g.: Microvision Nomad, DigiLens DL40, TacEye ST, Vuzix M2000AR

135. More on Head Mounted Displays • Karl Guttag Blog - https://kguttag.com/

136. HANDHELD AR

137. Handheld AR • Camera + display = handheld AR • Mobile phone/Tablet display • Video see-through AR

138. User Perspective Rendering for AR

139. User-Perspective Hand-Held Display Handheld display with device perspective Handheld display with user perspective Image: Domagoj Baričević

140. https://www.youtube.com/watch?v=z0nVgk1OxSc

141. SPATIALAUGMENTED REALITY

142. SpatialAugmented Reality • Project onto irregular surfaces • Geometric Registration • Projector blending, High dynamic range • Book: Bimber, Rasker “Spatial Augmented Reality”

143. Lightform • Depth sensor + projector • Create 3D model of space • Deform image mapping • Content creation tools

144. • https://www.youtube.com/watch?v=AUJNxNkwEy0

145. Steerable Projector Image: Claudio Pinhanez, IBM Research Everywhere Projector Display A steerable, tracked projector can display images anywhere

146. Head Mounted Projector • NVIS P-50 HMPD • 1280x1024/eye • Stereoscopic • 50 degree FOV • www.nvis.com

147. HMD vs.HMPD Head Mounted Display Head Mounted Projected Display

148. Tilt5 - https://www.tiltfive.com/ • Stereo head worn projectors • Interactive wand • Roll-able retro-reflective sheet • Designed for shared interaction

149. • Retroreflective roll-able mat incident light diffusion retro-reflection reflection Lambertian reflector (e.g. unfinished wood) Mirror reflector Retro-reflector

150. • https://www.youtube.com/watch?v=gNnBX1cW3L4

151. OTHER AR DISPLAYS

152. Video MonitorAR Video cameras Monitor Graphics Combiner Video Stereo glasses

153. Examples

154. Magic Mirror AR Experience • See AR overlay of an image of yourself

155. • https://www.youtube.com/watch?v=Mr71jrkzWq8&t=2s

156. OtherTypes ofAR Display • Audio • spatial sound • ambient audio • Tactile • physical sensation • Haptic • virtual touch

157. Haptic Input • AR Haptic Workbench • CSIRO 2003 – Adcock et. al.

158. Phantom • SensableTechnologies (www.sensable.com) • 6 DOF Force Feedback Device

159. AR Haptic Interface • Phantom, ARToolKit, Magellan

160. Olfactory Display MetaCookie: An olfactory display is combined with visual augmentation of a plain cookie to provide the illusion of a flavored cookie (chocolate, in the inset). Image: Takuji Narumi

161. • https://www.youtube.com/watch?v=3GnQE9cCf84

162. www.empathiccomputing.org @marknb00 mark.billinghurst@unisa.edu.au

2022 COMP4010 Lecture2: Perception

Vous êtes en train de lire un aperçu.

Consultez-les par la suite

2022 COMP4010 Lecture2: Perception

Plus De Contenu Connexe

Diaporamas pour vous (20)

Similaire à 2022 COMP4010 Lecture2: Perception (20)

Plus par Mark Billinghurst (13)

Plus récents (20)