Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
FAR: Fachgebiet Augmented Reality Technische Universität München
Head Mounted Displays
How to realize ultimate AR experien...
FAR: Fachgebiet Augmented Reality
Prof. Gudrun Klinker
2/52
Industrial Augmented Reality
3/52
Head-Mounted Displays (HMDs)
Optical See-Through (OST-HMD)
Non Optical See-Through
4/52
OST-HMD in science fiction
From Ironman, 2008 ©Marvel Entertainment
5/52
Everything is perfect in Science Fiction
 World sensing
 Image recognition
 Eye tracking
6/52
7/52
Arianespace, 2009Le Voyage dans la Lune, 1902
1902 2009
8/52
What is the current status
of the OST-HMD technology?
What’s still missing?
9/52
Hugo Gernsback, TV Glasses
1963
10/52
Ivan Sutherland, The Sword of Damocles
1968
11/52
2013-
12/52
13/52
14/52
OST-HMDs Overwrite the reality
NVIS nVisor ST60
15/52
Related Work | AR X-ray vision
Avery, Ben, Christian Sandor, and Bruce H. Thomas. "Improving spatial perception for augmen...
Related Work | Visual Assistance
Hwang, Alex D., and Eli Peli. "An augmented-reality edge enhancement application for Goog...
Issues in OST-HMDs
18/52
Design Factor (non-OST)
Toshiba, 2006 Oculus Rift DK2, 2014
Photo from IFIXIT
19/52
Design Factor
Google Glass, 2013
Photo by Mikepanhu
20/52
Wide Field of View
“Pinlight Displays: Wide Field of View Augmented Reality Eyeglasses
using Defocused Point Light Sources...
Wide Field of View
Issues on OST-HMD
“Pinlight Displays: Wide Field of View Augmented Reality Eyeglasses
using Defocused P...
Consistency issues in OST-HMDs
Temporal
[Zheng ’14] [Didier ’05]
Visual
[Kiyokawa ’01] [Liu ’08] [Lee ’09]
Spatial
[Azuma ...
Temporal Inconsist. in OST-HMD
“How fast is fast enough? : a study of the effects of latency in
direct-touch pointing task...
Temporal Inconsist. in OST-HMD
Digital Light Processing Projector
“Minimizing Latency for Augmented Reality Displays: Fram...
Visual Consistency
Occlusion, Depth, Color, Shadow, etc..
Lee and woo, 2009
Kiyokawa et al. 2003
Liu et al. 2008
26/52
Occlusion (physical “Z-buffer”)
27/52
Depth (eye accommodation)
Liu et al. 2008
28/52
Grubert, J., Tümler, J., & Mecke, R. (2008). Untersuchungen zur optimierung der see-through-
kalibrierung fuer mobile augm...
Spatial Calibration of OST-HMDs
30/52
Eye-HMD Calibration
3D
2D
31/52
P is a Black Box
Find 3D-2D Projection:
3D2D
32/52
2D
Say, is a Perspective Projection
33/52
Find 3D-2D Projection:
In the Eye of the Beholder…
= K*[R t]
34/52
Manual alignment
– Intensive user interaction
– User-dependent noise
Predictive tracking for augmented reality
RT Azuma - ...
Manual alignment
SPAAM: Single Point Active Alignment Method
– Medium user interaction
– User-dependent noise
Single point...
SPAAM (Manual Calibration) video
37/52
Idea: user-perspective rendering
Spindler, Martin, Wolfgang Büschel, and Raimund Dachselt. "Use your head: tangible window...
Utilizes 3D Eye Localization
–Interaction-free, thus do not bother users
–More accurate than a realistic SPAAM setup
Autom...
1. Estimate a 2D iris ellipse
– Iris detector + Fitting by RANSAC
2. Back project it to 3D circle
Robust real-time pupil t...
Estimated 3D Eye Position
[m]
>10 cm
41/52
Wearable eye-trackers
Pupil Labs Tobii
42/52
OST-HMD + eye trackers?
43/52
OST-HMD + eye trackers?
A compact, eye-tracked optical see-through head-mounted display
H Hua, C Gao - Proc. SPIE, 2012 - ...
Pupil Center Cornea Reflection
Villanueva ‘08
Cornea reflection
45
HMD Screen resembles a plane
HMD
Image
Screen
46
Corneal Reflection
47
Detected Checkerboard in 2D
Detected grid corners
48
Scene Reflection in the Eye
49
Detected and Projected
Corners
Reprojected Points (3D  2D)
Original Image
50
Estimated Cornea Position
Limbus tracking Corneal Imaging
Plopski, Alexander, et al. "Corneal-imaging calibration for
opti...
Optical See-Through HMDs are hard fun
• Consistency issues
(Spatial, visual, temporal, etc.)
• Eye sensing will be a hot t...
Appendix
53/52
An ideal eye-HMD system
Eye
Image source
Environment
Combiner
(a half mirror)
54/52
HMD-dependent parameters
User-dependent parameters
Image by Soerfm
Photo by Laitr Keiows
55
• Shipments of smart watches&glasses
–130 million by 2018
•10 times than 2013
Dawn of a Commercial Wearable Devices
56/52
Innovega iOptik
57/52
Simple
No user interaction
Accurate
 better than Degraded manual
calibrations
Summary of INDICA
Calibration of OST-HMDs u...
Eye Position Estimation
E
<
C1
Eyeball
sphere
C3
Eyeball
orbit
g2
g3
g1
C2
59
Prochain SlideShare
Chargement dans…5
×

Head Mounted Displays: How to realize ultimate AR experiences?

2 286 vues

Publié le

Slides from a talk I gave at:
Wissens-Austausch-Workshop: The 3rd Workshop on Visualisierung großer Datenmengen in der Wissenschaft (VisDa3? WAW), DLR@Oberpfaffenhofen, Germany, Jun. 9 - 10, 2015.

For more detail of my work, see:
http://campar.in.tum.de/Main/YutaItoh

Publié dans : Technologie
  • Soyez le premier à commenter

  • Soyez le premier à aimer ceci

Head Mounted Displays: How to realize ultimate AR experiences?

  1. 1. FAR: Fachgebiet Augmented Reality Technische Universität München Head Mounted Displays How to realize ultimate AR experiences? Yuta Itoh (TUM) Workshop Visualisierung großer Datenmengen, 9th June 2015 1
  2. 2. FAR: Fachgebiet Augmented Reality Prof. Gudrun Klinker 2/52
  3. 3. Industrial Augmented Reality 3/52
  4. 4. Head-Mounted Displays (HMDs) Optical See-Through (OST-HMD) Non Optical See-Through 4/52
  5. 5. OST-HMD in science fiction From Ironman, 2008 ©Marvel Entertainment 5/52
  6. 6. Everything is perfect in Science Fiction  World sensing  Image recognition  Eye tracking 6/52
  7. 7. 7/52
  8. 8. Arianespace, 2009Le Voyage dans la Lune, 1902 1902 2009 8/52
  9. 9. What is the current status of the OST-HMD technology? What’s still missing? 9/52
  10. 10. Hugo Gernsback, TV Glasses 1963 10/52
  11. 11. Ivan Sutherland, The Sword of Damocles 1968 11/52
  12. 12. 2013- 12/52
  13. 13. 13/52
  14. 14. 14/52
  15. 15. OST-HMDs Overwrite the reality NVIS nVisor ST60 15/52
  16. 16. Related Work | AR X-ray vision Avery, Ben, Christian Sandor, and Bruce H. Thomas. "Improving spatial perception for augmented reality x-ray vision." Virtual Reality Conference, 2009. VR 2009. IEEE. IEEE, 2009. 16/52
  17. 17. Related Work | Visual Assistance Hwang, Alex D., and Eli Peli. "An augmented-reality edge enhancement application for Google Glass." Optometry & Vision Science 91.8 (2014): 1021-1030. 17/52
  18. 18. Issues in OST-HMDs 18/52
  19. 19. Design Factor (non-OST) Toshiba, 2006 Oculus Rift DK2, 2014 Photo from IFIXIT 19/52
  20. 20. Design Factor Google Glass, 2013 Photo by Mikepanhu 20/52
  21. 21. Wide Field of View “Pinlight Displays: Wide Field of View Augmented Reality Eyeglasses using Defocused Point Light Sources” Maimone et al., TOG’14 21/52
  22. 22. Wide Field of View Issues on OST-HMD “Pinlight Displays: Wide Field of View Augmented Reality Eyeglasses using Defocused Point Light Sources” Maimone et al., TOG’14 22/52
  23. 23. Consistency issues in OST-HMDs Temporal [Zheng ’14] [Didier ’05] Visual [Kiyokawa ’01] [Liu ’08] [Lee ’09] Spatial [Azuma ’95] [Tuceryan ’00] [Genc ’02] Social Image from Google Glass: Don't Be A Glasshole | Mashable 23/52
  24. 24. Temporal Inconsist. in OST-HMD “How fast is fast enough? : a study of the effects of latency in direct-touch pointing tasks” Jota et al. CH’13 “latencies down to 2.38 ms are required to alleviate user perception when dragging” 24/52
  25. 25. Temporal Inconsist. in OST-HMD Digital Light Processing Projector “Minimizing Latency for Augmented Reality Displays: Frames Considered Harmful” Zheng et al. ISMAR’14 25/52
  26. 26. Visual Consistency Occlusion, Depth, Color, Shadow, etc.. Lee and woo, 2009 Kiyokawa et al. 2003 Liu et al. 2008 26/52
  27. 27. Occlusion (physical “Z-buffer”) 27/52
  28. 28. Depth (eye accommodation) Liu et al. 2008 28/52
  29. 29. Grubert, J., Tümler, J., & Mecke, R. (2008). Untersuchungen zur optimierung der see-through- kalibrierung fuer mobile augmented reality assistenzsysteme. Michael Schenk, Hrsg, 6(7). Spatial Calibration of OST-HMDs 29/52
  30. 30. Spatial Calibration of OST-HMDs 30/52
  31. 31. Eye-HMD Calibration 3D 2D 31/52
  32. 32. P is a Black Box Find 3D-2D Projection: 3D2D 32/52
  33. 33. 2D Say, is a Perspective Projection 33/52
  34. 34. Find 3D-2D Projection: In the Eye of the Beholder… = K*[R t] 34/52
  35. 35. Manual alignment – Intensive user interaction – User-dependent noise Predictive tracking for augmented reality RT Azuma - 1995 - ronaldazuma.com 35/52
  36. 36. Manual alignment SPAAM: Single Point Active Alignment Method – Medium user interaction – User-dependent noise Single point active alignment method (SPAAM) for optical see-through HMD calibration for AR M Tuceryan, N Navab - Augmented Reality, 2000.(ISAR 2000). …, 2000 - ieeexplore.ieee.org Practical solutions for calibration of optical see-through devices Y Genc, M Tuceryan, N Navab - … of the 1st International Symposium on …, 2002 - dl.acm.org N times 36/52
  37. 37. SPAAM (Manual Calibration) video 37/52
  38. 38. Idea: user-perspective rendering Spindler, Martin, Wolfgang Büschel, and Raimund Dachselt. "Use your head: tangible windows for 3D information spaces in a tabletop environment."Proceedings of the 2012 ACM international conference on Interactive tabletops and surfaces. ACM, 2012. 38/52
  39. 39. Utilizes 3D Eye Localization –Interaction-free, thus do not bother users –More accurate than a realistic SPAAM setup Automated alignement Yuta Itoh, and Gudrun Klinker. "Interaction-free calibration for optical see-through head-mounted displays based on 3d eye localization." 3D User Interfaces (3DUI), 2014 IEEE Symposium on. IEEE, 2014. 39/52
  40. 40. 1. Estimate a 2D iris ellipse – Iris detector + Fitting by RANSAC 2. Back project it to 3D circle Robust real-time pupil tracking in highly off-axis images L Świrski, A Bulling, N Dodgson - … of the Symposium on Eye Tracking …, 2012 - dl.acm.org 3D Eye Position Estimation Image-based Eye Pose and Reflection Analysis for Advanced Interaction Techniques andScene Understanding C Nitschke, A. Nakazawa, H. Takemura - CVIM,[ …, 2011 - ci.nii.ac.jp 40/52
  41. 41. Estimated 3D Eye Position [m] >10 cm 41/52
  42. 42. Wearable eye-trackers Pupil Labs Tobii 42/52
  43. 43. OST-HMD + eye trackers? 43/52
  44. 44. OST-HMD + eye trackers? A compact, eye-tracked optical see-through head-mounted display H Hua, C Gao - Proc. SPIE, 2012 - ebooks.spiedigitallibrary.org 44/52
  45. 45. Pupil Center Cornea Reflection Villanueva ‘08 Cornea reflection 45
  46. 46. HMD Screen resembles a plane HMD Image Screen 46
  47. 47. Corneal Reflection 47
  48. 48. Detected Checkerboard in 2D Detected grid corners 48
  49. 49. Scene Reflection in the Eye 49
  50. 50. Detected and Projected Corners Reprojected Points (3D  2D) Original Image 50
  51. 51. Estimated Cornea Position Limbus tracking Corneal Imaging Plopski, Alexander, et al. "Corneal-imaging calibration for optical see-through head-mounted displays." Visualization and Computer Graphics, IEEE Transactions on 21.4 (2015): 481-490. 51
  52. 52. Optical See-Through HMDs are hard fun • Consistency issues (Spatial, visual, temporal, etc.) • Eye sensing will be a hot topic • Big players => big markets Takeaways: 52/52
  53. 53. Appendix 53/52
  54. 54. An ideal eye-HMD system Eye Image source Environment Combiner (a half mirror) 54/52
  55. 55. HMD-dependent parameters User-dependent parameters Image by Soerfm Photo by Laitr Keiows 55
  56. 56. • Shipments of smart watches&glasses –130 million by 2018 •10 times than 2013 Dawn of a Commercial Wearable Devices 56/52
  57. 57. Innovega iOptik 57/52
  58. 58. Simple No user interaction Accurate  better than Degraded manual calibrations Summary of INDICA Calibration of OST-HMDs using 3D eye position 58/52
  59. 59. Eye Position Estimation E < C1 Eyeball sphere C3 Eyeball orbit g2 g3 g1 C2 59

×