2022 COMP4010 Lecture3: AR Technology

1. AR TECHNOLOGY COMP 4010 Lecture Three Mark Billinghurst August 11th 2022 mark.billinghurst@unisa.edu.au

2. REVIEW

3. How do We Perceive Reality? • We understand the world through our senses: • Sight, Hearing, Touch, Taste, Smell (and others..) • Two basic processes: • Sensation – Gathering information • Perception – Interpreting information

4. Simple Sensing/Perception Model

5. Reality vs. Virtual Reality • In a VR system there are input and output devices between human perception and action

6. Presence .. “The subjective experience of being in one place or environment even when physically situated in another” Witmer, B. G., & Singer, M. J. (1998). Measuring presence in virtual environments: A presence questionnaire. Presence: Teleoperators and virtual environments, 7(3), 225-240.

7. Slater, M., Banakou, D., Beacco, A., Gallego, J., Macia-Varela, F., & Oliva, R. (2022). A Separate Reality: An Update on Place Illusion and Plausibility in Virtual Reality. Frontiers in Virtual Reality, 81. Four Illusions of Presence (Slater 2022) • Place Illusion: being in the place • Plausibility Illusion: events are real • Body Ownership: seeing your body in VR • Copresence/Social Presence: other people are in VR

8. Senses • How an organism obtains information for perception: • Sensation part of Somatic Division of Peripheral Nervous System • Integration and perception requires the Central Nervous System • Five major senses (but there are more..): • Sight (Opthalamoception) • Hearing (Audioception) • Taste (Gustaoception) • Smell (Olfacaoception) • Touch (Tactioception)

9. The Human Visual System • Purpose is to convert visual input to signals in the brain

10. Comparison between Eyes and HMD

11. Sound Localization • Humans have two ears • localize sound in space • Sound can be localized using 3 coordinates • Azimuth, elevation, distance

12. Haptic Sensation • Somatosensory System • complex system of nerve cells that responds to changes to the surface or internal state of the body • Skin is the largest organ • 1.3-1.7 square m in adults • Tactile: Surface properties • Receptors not evenly spread • Most densely populated area is the tongue • Kinesthetic: Muscles, Tendons, etc. • Also known as proprioception

13. Proprioception/Kinaesthesia • Proprioception (joint position sense) • Awareness of movement and positions of body parts • Due to nerve endings and Pacinian and Ruffini corpuscles at joints • Enables us to touch nose with eyes closed • Joints closer to body more accurately sensed • Users know hand position accurate to 8cm without looking at them • Kinaesthesia (joint movement sense) • Sensing muscle contraction or stretching • Cutaneous mechanoreceptors measuring skin stretching • Helps with force sensation

14. Augmented RealityTechnology • Combines Real and Virtual Images • Needs: Display technology • Interactive in real-time • Needs: Input and interaction technology • Registered in 3D • Needs: Viewpoint tracking technology

15. AR Display Technologies • Classification (Bimber/Raskar 2005) • Head attached • Head mounted display/projector • Body attached • Handheld display/projector • Spatial • Spatially aligned projector/monitor

16. Types of Head Mounted Displays Occluded See-thru Multiplexed

17. Optical see-through Head-Mounted Display Virtual images from monitors Real World Optical Combiners

18. Video see-through HMD Video cameras Monitors Graphics Combiner Video

19. Handheld AR • Camera + display = handheld AR • Mobile phone/Tablet display

20. SpatialAugmented Reality • Project onto irregular surfaces • Geometric Registration • Projector blending, High dynamic range • Book: Bimber, Rasker “Spatial Augmented Reality”

21. 2: AR TRACKING

22. AR RequiresTracking and Registration • Registration • Positioning virtual object wrt real world • Fixing virtual object on real object when view is fixed • Calibration • Offline measurements • Measure camera relative to head mounted display • Tracking • Continually locating the user’s viewpoint when view moving • Position (x,y,z), Orientation (r,p,y)

23. REGISTRATION AND CALIBRATION

24. Coordinate Systems Local object coordinates Global world coordinates Eye coordinates Model transformation • Track for moving objects, if there are static objects as well View transformation • Track for moving objects, if there are no static objects • Track for moving observer Perspective transformation • Calibrate offline • For both camera and display

25. Spatial Registration

26. The Registration Problem • Virtual and Real content must stay properly aligned • If not: • Breaks the illusion that the two coexist • Prevents acceptance of many serious applications t = 0 seconds t = 0.5 second

27. Sources of Registration Errors •Static errors • Optical distortions (in HMD) • Mechanical misalignments • Tracker errors • Incorrect viewing parameters •Dynamic errors • System delays (largest source of error) • 1 ms delay = 1/3 mm registration error

28. Reducing Static Errors •Distortion compensation • For lens or display distortions •Manual adjustments • Have user manually alighn AR andVR content •View-based or direct measurements • Have user measure eye position •Camera calibration (video AR) • Measuring camera properties

29. View Based Calibration (Azuma 94)

30. Uncalibrated Calibrated The Benefit of Calibration

31. Dynamic errors • Total Delay = 50 + 2 + 33 + 17 = 102 ms • 1 ms delay = 1/3 mm = 33mm error Tracking Calculate Viewpoint Simulation Render Scene Draw to Display x,y,z r,p,y Application Loop 20 Hz = 50ms 500 Hz = 2ms 30 Hz = 33ms 60 Hz = 17ms

32. Reducing dynamic errors (1) •Reduce system lag •Faster components/system modules •Reduce apparent lag •Image deflection •Image warping

33. Reducing System Lag Tracking Calculate Viewpoint Simulation Render Scene Draw to Display x,y,z r,p,y Application Loop Faster Tracker Faster CPU Faster GPU Faster Display

34. ReducingApparent Lag Tracking Update x,y,z r,p,y Virtual Display Physical Display (640x480) 1280 x 960 Last known position Virtual Display Physical Display (640x480) 1280 x 960 Latest position Tracking Calculate Viewpoint Simulation Render Scene Draw to Display x,y,z r,p,y Application Loop

35. Reducing dynamic errors (2) • Match video + graphics input streams (video AR) • Delay video of real world to match system lag • User doesn’t notice • Predictive Tracking • Inertial sensors helpful Azuma / Bishop 1994

36. PredictiveTracking Time Position Past Future Can predict up to 80 ms in future (Holloway) Now

37. PredictiveTracking (Azuma 94)

38. TRACKING

39. Frames of Reference • Word-stabilized • E.g., billboard or signpost • Body-stabilized • E.g., virtual tool-belt • Screen-stabilized • Heads-up display

40. Tracking Requirements • Augmented Reality Information Display • World Stabilized • Body Stabilized • Head Stabilized Increasing Tracking Requirements Head Stabilized Body Stabilized World Stabilized

41. Tracking Technologies § Active • Mechanical, Magnetic, Ultrasonic • GPS, Wifi, cell location § Passive • Inertial sensors (compass, accelerometer, gyro) • Computer Vision • Marker based, Natural feature tracking § Hybrid Tracking • Combined sensors (eg Vision + Inertial)

42. Tracking Types Magnetic Tracker Inertial Tracker Ultrasonic Tracker Optical Tracker Marker-Based Tracking Markerless Tracking Specialize dTracking Edge-Based Tracking Template- BasedTracking Interest Point Tracking Mechanical Tracker

43. MechanicalTracker •Idea: mechanical arms with joint sensors •++: high accuracy, haptic feedback •-- : cumbersome, expensive Microscribe

44. MagneticTracker • Idea: coil generates current when moved in magnetic field. Measuring current gives position and orientation relative to magnetic source. • ++: 6DOF, robust • -- : wired, sensible to metal, noisy, expensive Flock of Birds (Ascension)

45. InertialTracker • Idea: measuring linear and angular orientation rates (accelerometer/gyroscope) • ++: no transmitter, cheap, small, high frequency, wireless • -- : drifts over time, hysteresis effect, only 3DOF IS300 (Intersense) Wii Remote

46. UltrasonicTracker • Idea: time of Flight or phase-Coherence SoundWaves • ++: Small, Cheap • -- : 3DOF, Line of Sight, Low resolution, Affected by environmental conditons (pressure, temperature) Ultrasonic Logitech IS600

47. Global Positioning System (GPS) • Created by US in 1978 • Currently 29 satellites • Satellites send position + time • GPS Receiver positioning • 4 satellites need to be visible • Differential time of arrival • Triangulation • Accuracy • 5-30m+, blocked by weather, buildings etc.

48. Mobile Sensors • Inertial compass • Earth’s magnetic field • Measures absolute orientation • Accelerometers • Measures acceleration about axis • Used for tilt, relative rotation • Can drift over time

49. OPTICAL TRACKING

50. Why Optical Tracking for AR? • Many AR devices have cameras • Mobile phone/tablet, Video see-through display • Provides precise alignment between video and AR overlay • Using features in video to generate pixel perfect alignment • Real world has many visual features that can be tracked from • Computer Vision well established discipline • Over 40 years of research to draw on • Old non real time algorithms can be run in real time on todays devices

51. Common AR Optical Tracking Types • Marker Tracking • Tracking known artificial markers/images • e.g. ARToolKit square markers • Markerless Tracking • Tracking from known features in real world • e.g. Vuforia image tracking • Unprepared Tracking • Tracking in unknown environment • e.g. SLAM tracking

52. Visual Tracking Approaches • Marker based tracking with artificial features • Make a model before tracking • Model based tracking with natural features • Acquire a model before tracking • Simultaneous localization and mapping • Build a model while tracking it

53. Marker tracking • Available for more than 10 years • Several open source solutions exist • ARToolKit, ARTag, ATK+, etc • Fairly simple to implement • Standard computer vision methods • A rectangle provides 4 corner points • Enough for pose estimation!

54. Demo: ARToolKit

55. Marker Based Tracking: ARToolKit http://www.artoolkit.org

56. Tracking challenges inARToolKit False positives and inter-marker confusion (image by M. Fiala) Image noise (e.g. poor lens, block coding / compression, neon tube) Unfocused camera, motion blur Dark/unevenly lit scene, vignetting Jittering (Photoshop illustration) Occlusion (image by M. Fiala)

57. Other MarkerTracking Libraries

58. Marker Target Identification • More targets or features à more easily confused • Must be as unique as possible • Square markers • 2D barcodes with error correction • E.g., 6x6=36 bits (2 orientation, 6-12 payload, rest for error correction) • Marker tapestries 59

59. But - You can’t cover world with ARToolKit Markers!

60. Markerless Tracking Magnetic Tracker Inertial Tracker Ultrasonic Tracker Optical Tracker Marker-Based Tracking Markerless Tracking Specialized Tracking Edge-Based Tracking Template-Based Tracking Interest Point Tracking • No more Markers! èMarkerless Tracking Mechanica l Tracker

61. Natural Feature Tracking • Use Natural Cues of Real Elements • Edges • Surface Texture • Interest Points • Model or Model-Free • No visual pollution Contours Features Points Surfaces

62. TextureTracking

63. Natural Features • Detect salient interest points in image • Must be easily found • Location in image should remain stable when viewpoint changes • Requires textured surfaces • Alternative: can use edge features (less discriminative) • Match interest points to tracking model database • Database filled with results of 3D reconstruction • Matching entire (sub-)images is too costly • Typically interest points are compiled into “descriptors” Tracking 64 Image: Gerhard Reitmayr Image: Martin Hirzer

64. Tracking by Keypoint Detection • This is what most trackers do… • Targets are detected every frame • Popular because tracking and detection are solved simultaneously Keypoint detection Descriptor creation and matching Outlier Removal Pose estimation and refinement Camera Image Pose Recognition

65. What is a Keypoint? • Invariant visual feature • Different detectors possible • For high performance use the FAST corner detector • Apply FAST to all pixels of your image • Obtain a set of keypoints for your image • Describe the keypoints Rosten, E., & Drummond, T. (2006, May). Machine learning for high-speed corner detection. In European conference on computer vision (pp. 430-443). Springer Berlin Heidelberg.

66. FAST Corner Keypoint Detection

67. Example:FAST Corner Detection https://www.youtube.com/watch?v=vEkHoYpMD3Y

68. Descriptors • Describe the Keypoint features • Can use SIFT • Estimate the dominant keypoint orientation using gradients • Compensate for detected orientation • Describe the keypoints in terms of the gradients surrounding it Wagner D., Reitmayr G., Mulloni A., Drummond T., Schmalstieg D., Real-Time Detection and Tracking for Augmented Reality on Mobile Phones. IEEE Transactions on Visualization and Computer Graphics, May/June, 2010

69. Detection and Tracking Detection Incremental tracking Tracking target detected Tracking target lost Tracking target not detected Incremental tracking ok Start + Recognize target type + Detect target + Initialize camera pose + Fast + Robust to blur, lighting changes + Robust to tilt Tracking and detection are complementary approaches. After successful detection, the target is tracked incrementally. If the target is lost, the detection is activated again

70. Demo: Vuforia Texture Tracking https://www.youtube.com/watch?v=1Qf5Qew5zSU

71. Edge Based Tracking • Example: RAPiD [Drummond et al. 02] • Initialization, Control Points, Pose Prediction (Global Method)

72. Demo: Edge Based Tracking

73. Line Based Tracking • Visual Servoing [Comport et al. 2004]

74. Marker vs.Natural FeatureTracking • Marker tracking • Usually requires no database to be stored • Markers can be an eye-catcher • Tracking is less demanding • The environment must be instrumented • Markers usually work only when fully in view • Natural feature tracking • A database of keypoints must be stored/downloaded • Natural feature targets might catch the attention less • Natural feature targets are potentially everywhere • Natural feature targets work also if partially in view

76. Model BasedTracking • Tracking from 3D object shape • Example: OpenTL - www.opentl.org • General purpose library for model based visual tracking

77. Demo: OpenTL Model Tracking

78. Demo: OpenTL Face Tracking

79. Vuforia Model Tracker • Uses pre-captured 3D model for tracking • On-screen guide to line up model

80. Model Tracking Demo https://www.youtube.com/watch?v=6W7_ZssUTDQ

82. Tracking from an Unknown Environment • What to do when you don’t know any features? • Very important problem in mobile robotics - Where am I? • SLAM • Simultaneously Localize And Map the environment • Goal: to recover both camera pose and map structure while initially knowing neither. • Mapping: • Building a map of the environment which the robot is in • Localisation: • Navigating this environment using the map while keeping track of the robot’s relative position and orientation

83. Parallel Tracking and Mapping Tracking Mapping New keyframes Map updates + Estimate camera pose + For every frame + Extend map + Improve map + Slow updates rate Parallel tracking and mapping uses two concurrent threads, one for tracking and one for mapping, which run at different speeds

84. Parallel Tracking and Mapping Video stream New frames Map updates Tracking Mapping Tracked local pose FAST SLOW Simultaneous localization and mapping (SLAM) in small workspaces Klein/Drummond, U. Cambridge

85. Visual SLAM • Early SLAM systems (1986 - ) • Computer visions and sensors (e.g. IMU, laser, etc.) • One of the most important algorithms in Robotics • Visual SLAM • Using cameras only, such as stereo view • MonoSLAM (single camera) developed in 2007 (Davidson)

86. Example:Kudan MonoSLAM

87. How SLAMWorks • Three main steps 1. Tracking a set of points through successive camera frames 2. Using these tracks to triangulate their 3D position 3. Simultaneously use the estimated point locations to calculate the camera pose which could have observed them • By observing a sufficient number of points can solve for both structure and motion (camera path and scene structure).

88. Evolution of SLAM Systems • MonoSLAM (Davidson, 2007) • Real time SLAM from single camera • PTAM (Klein, 2009) • First SLAM implementation on mobile phone • FAB-MAP (Cummins, 2008) • Probabilistic Localization and Mapping • DTAM (Newcombe, 2011) • 3D surface reconstruction from every pixel in image • KinectFusion (Izadi, 2011) • Realtime dense surface mapping and tracking using RGB-D

89. Demo:MonoSLAM

90. LSD-SLAM (Engel 2014) • A novel, direct monocular SLAM technique • Uses image intensities both for tracking and mapping. • The camera is tracked using direct image alignment, while • Geometry is estimated as semi-dense depth maps • Supports very large-scale tracking • Runs in real time on CPU and smartphone

91. Demo:LSD-SLAM

92. Applications of SLAM Systems • Many possible applications • Augmented Reality camera tracking • Mobile robot localisation • Real world navigation aid • 3D scene reconstruction • 3D Object reconstruction • Etc.. • Assumptions • Camera moves through an unchanging scene • So not suitable for person tracking, gesture recognition • Both involve non-rigidly deforming objects and a non-static map

93. Hybrid Tracking Combining several tracking modalities together

94. Combining Sensors andVision • Sensors • Produces noisy output (= jittering augmentations) • Are not sufficiently accurate (= wrongly placed augmentations) • Gives us first information on where we are in the world, and what we are looking at • Vision • Is more accurate (= stable and correct augmentations) • Requires choosing the correct keypoint database to track from • Requires registering our local coordinate frame (online- generated model) to the global one (world)

95. OutdoorARTracking System You, Neumann,Azuma outdoor AR system (1999)

96. Types of Sensor Fusion • Complementary • Combining sensors with different degrees of freedom • Sensors must be synchronized (or requires inter-/extrapolation) • E.g., combine position-only and orientation-only sensor • E.g., orthogonal 1D sensors in gyro or magnetometer are complementary • Competitive • Different sensor types measure the same degree of freedom • Redundant sensor fusion • Use worse sensor only if better sensor is unavailable • E.g., GPS + pedometer • Statistical sensor fusion www.augmentedrealitybook.org Tracking 97

97. Example: Outdoor Hybrid Tracking • Combines • computer vision • inertial gyroscope sensors • Both correct for each other • Inertial gyro • provides frame to frame prediction of camera orientation, fast sensing • drifts over time • Computer vision • Natural feature tracking, corrects for gyro drift • Slower, less accurate

98. Robust OutdoorTracking • HybridTracking • ComputerVision, GPS, inertial • Going Out • Reitmayr & Drummond (Univ. Cambridge) Reitmayr, G., & Drummond, T. W. (2006). Going out: robust model-based tracking for outdoor augmented reaity. In Mixed and Augmented Reality, 2006. ISMAR 2006. IEEE/ACM International Symposium on (pp. 109-118). IEEE.

99. Handheld Display

100. Demo: Going Out Hybrid Tracking

101. ARKit – Visual Inertial Odometry • Uses both computer vision + inertial sensing • Tracking position twice • Computer Vision – feature tracking, 2D plane tracking • Inertial sensing – using the phone IMU • Output combined via Kalman filter • Determine which output is most accurate • Pass pose to ARKit SDK • Each system compliments the other • Computer vision – needs visual features • IMU - drifts over time, doesn’t need features

102. ARKit –Visual Inertial Odometry • Slow camera • Fast IMU • If camera drops out IMU takes over • Camera corrects IMU errors

103. ARKit Demo • https://www.youtube.com/watch?v=dMEWp45WAUg

104. Conclusions • Tracking and Registration are key problems • Registration error • Measures against static error • Measures against dynamic error • AR typically requires multiple tracking technologies • Computer vision most popular • Research Areas: • SLAM systems, Deformable models, Mobile outdoor tracking

105. www.empathiccomputing.org @marknb00 mark.billinghurst@unisa.edu.au

2022 COMP4010 Lecture3: AR Technology

Vous êtes en train de lire un aperçu.

Consultez-les par la suite

2022 COMP4010 Lecture3: AR Technology

Plus De Contenu Connexe

Similaire à 2022 COMP4010 Lecture3: AR Technology (20)

Plus par Mark Billinghurst (20)

Plus récents (20)