Augmented and Virtual Reality (AR / VR) continue to buzz hotly as contenders for the next wave of personal computing. Tech-industry pundits predict enormous future markets reaching hundreds of billions of dollars based on speculation that AR will replace the smartphone. But while VR has a clear entry to market in immersive gaming (and potentially social VR), there are many challenges to AR beyond just making the hardware lightweight and comfortable. What applications are important enough for the mass population to wear a device on their head most of the time? Or can pervasive displays obviate the need for wearable AR headsets? How will we overcome the uncanny valley of non-realistic avatars? Are light-field optics necessary to align digital and physical objects at the same focal distance? And what will it take to make 3D interaction as easy and reliable as mouse and trackpads for 2D? This presentation will explore the challenges and some promising directions to really make AR become a replacement for the smart phone.

1. www.huawei.com
HUAWEI TECHNOLOGIES CO., LTD.
AR We Kidding
Ourselves?
Bo Begole
See TechCrunch article

2. #MUM18
‫ﺷﻜﺮﺍﹰ‬ً‫جزيال‬

3. HUAWEI TECHNOLOGIES CO., LTD.
Business projections
for AR/VR
Source : Goldman Sachs and IDC
TOTAL
HW
$170B TOTAL
SW
$35B

4. 5 //
Motion sensing Location sensing (alarms/doors) Wearable sensing
Context sensing user ID Conversational speech
Learn behaviors,
recommend actions
PROACTIVE
SYSTEMS
REACTIVE
SYSTEM-
INITIATED
Smart appliance displays Tabletop displaysHuge TV displays Wall-sized
Curved Flexible FoldableRoll-up Natural origami-capable
Kiosk displays Theater screens Jumbotron E-billboards
Digital Ads
displays Coordinated displays
TV PCs
Home displays (wall,
immersive, etc.)
Smart appliancesProjectors Appliances
Laptop PDA Smart phone/tabletTablet/Ultra PC
Smart wearables
(watch, HMD, etc.)
Keyboard/mouse Stylus pen Touch/tactile screen Tangible Interfaces Multi modal
Speech Gaze trackingGestures Activity & Intention Sensing
HOLISTIC
INTERACTION
DETERMINISTIC
PROBABILISTIC
DISPLAYS
EVERYWHERE
PRIVATE
PUBLIC
SHARED
NEW
FORMS
SIZES
MATERIALS
CURRENT FUTURE
TECHNOLOGY TRENDS
DEVICES
FOR EVERY
PURPOSE
+
MACHINE
COGNITION
& ACTION
Proactive
personal
services,
devices
VR/AR HMDs
Holographic

5. Mixed Reality: Presenting Digital Information
Augmented Reality
(AR)
Virtual Reality
(VR)
Mixed Reality
(MR)
AR bases reality as the main focus, the virtual
information is presented over the reality
VR bases virtual data as the main focus, having the
user immerse into the middle of the synthetic
reality virtual environment,.
MR attempts to merge the physical and virtual
information seamlessly so that the user does not greatly
distinguish the sources.
* Milgram Bridge
Physical
World
Cyber
World

6. Mixed Reality: Presenting Digital Information
Augmented Reality
(AR)
Virtual Reality
(VR)
Mixed Reality
(MR)
* Milgram Bridge
Physical
World
Cyber
World
[Sutherland 1968] [Weiser, 1980s]

7. Since 2015: Lots of sizzle …
Page 8

8. And make no mistake, VR will sell millions
Page 9

9. Screens have grown more
connected, interactive and
intuitive as they have moved
closer.
The viewer is more aware of
the experience.
The experience is more
aware of the viewer.  Responsive Media

10. Detection technologies exist but need to be integrated,
directed by behavior models:
- We can predict intention before user is even aware to ask.
- Respond to intentions with action and information.
Gaze
[Cohen et al. ‘03]Expression
Vocal tone,
laughter
Temp, pulse
[Daugman ‘94]
Pupil dilation
[Vogel &
Balakrishnan ‘04]
facial affect
limbs,
Head,
body pose,
orientation,
& proximity
Brain
EEG
Muscles, EMG Hyperspectral
Many indicators of a person’s state of mind

11. Page 12
Synthetic movies
and fantasy
sports that keep
the intensity high!

12. Responsive Media: The Experience is Aware of the Viewer
Intention Recognition Technologies
Detect human body, activities, emotions,
gaze, physiology
Mixed-Initiative Interaction Engine
Machine can initiate an interaction, not just
the human
Convert intention to responses
Assess response effectiveness
User Behavior Model
Combines personal patterns, preferences
and contexts to tailor interaction to
individual
Dynamic content framework
Map intentions to content segments to
achieve the objective
Tailorable to structure of engagements
across multiple applications
Far-reaching research and invention of
future interaction paradigm across
devices and applications

13. Page 15

14. AR could kill the Smartphone
Page 17

15. PARC | 18

16. AR technologies are not as simple as VR
• Not a simple “cardboard” HMD
• Visual fidelity is poor
– See-through optics diminish the real world
– Non-opaque virtual objects (real world
bleeds through)
– Low resolution
– Mismatching brightness, lighting direction,
shadows
– Mismatching alignment and depth between
virtual and real

17. HUAWEI TECHNOLOGIES CO., LTD. Page 20
… let’s assume the technologies will work
Retinal Displays (Micro LED) Light Field

18. Will you wear a heavy, hot headset for these?
Page 21

19. Gratuitous Spatialization
Page 23

20. The return of Bob?
Page 24

21. Gratuitous Spatialization
Page 25

22. Several Clearly
Valuable Use Cases:
• Surgery
• Emergency response
• Aircraft and vehicle
operation
– (but these are “autonomizing”)
• Complex machinery
assembly and repair

23. Entervise: Remote Expertise
PARC | 27
Augmented Media
Collaboration
Server
• Expert technicians support less-experienced technicians
• Variety of technology investigated for this purpose
• Collaboration server – experts see what field users see
• Can annotate on video with shared multi-user voice chat
Field user Expert user
“Webex for the real world.”

24. Hands-free AR for hands-on learning
• Supporting just-in-time,
life-long learning
• E.g., “Pianolens”
Page 28

25. What could supplant the smart phone?
• 3D media in physical spaces (sports, teleconferencing)?
• Augmented information about media and the world?
– Navigation info in front of me
– Search results while I talk
– Foreign language translation in realtime, currency conversion
• Reminders only when desired, not interrupting
• Enhance the world with delightful visuals (Cinematic Reality)
PARC | 29
Is this enough to wear a hot, heavy, occluding HMD?

26. Changing the way we communicate ...
Holoportation… with lively avatars…
and/or more
“beautiful” people

27. HUAWEI TECHNOLOGIES Co., Ltd.
Light field AR – 4D Television
real view: empty room
View responds to user position.
Light field ARVR: Real-time Insertion of Captured Light
Field into Current 3D Environment
Characteristics of our approach:
• Light weight rendering enables real time
performance on mobile device
• Allow users to move freely in space and perceive
full 360-degree view of inserted contents on mobile
devices and/or immersive displays
Responds to user head and body position naturally
View is controlled by broadcaster

28. What could supplant the smart phone?
• 3D media in physical spaces (sports, teleconferencing)?
• Augmented information about media and the world?
– Supplemental information
– Search results while I talk
– Navigation info in front of me, foreign language translation, currency conversion, …
• Reminders only when desired, not interrupting
• Enhance the world with delightful visuals (Cinematic Reality)
PARC | 32
Is this enough to wear a hot, heavy, occluding HMD?

29. Augmented Media
 Automated Content Recognition to
find related information
 TV shows: actors, characters,
synopses, back stories, …
 Movies: soundtracks, products
 Sports: realtime stats, histories
 News: other sources, related stories
 Social network activity
 Information on demand commercials
 Personal notifications
Page 33
SeeSpace InAir
ExpectLabs MindMeld

30. What could supplant the smart phone?
• 3D media in physical spaces (sports, teleconferencing)?
• Augmented information about media and the world?
– Supplemental information
– Search results while I talk
– Navigation info in front of me, foreign language translation, currency conversion, …
• Contextual reminders only when desired, not interrupting
• Enhance the world with delightful visuals (Cinematic Reality)
PARC | 34
Is this enough to wear a hot, heavy, occluding HMD?

31. What could supplant the smart phone?
• 3D media in physical spaces (sports, teleconferencing)?
• Augmented information about media and the world?
– Supplemental information
– Search results while I talk
– Navigation info in front of me, foreign language translation, currency conversion, …
• Contextual reminders only when desired, not interrupting
• Enhance the world with delightful visuals (Cinematic Reality)
PARC | 35
Is this enough to wear a hot, heavy, occluding HMD?

32. PARC | 36

33. PARC | 37

34. What could supplant the smart phone?
• 3D media in physical spaces (sports, teleconferencing)?
• Augmented information about media and the world?
– Supplemental information
– Search results while I talk
– Navigation info in front of me, foreign language translation, currency conversion, …
• Contextual reminders only when desired, not interrupting
• Enhance the world with delightful visuals (Cinematic Reality)
PARC | 38
Is this enough to wear a hot, heavy, occluding HMD?

35. PARC | 39
“We estimate we can sell up to 80% of
an individual’s visual field before
inducing seizures.” – Ready Player One

36. Humans are better robots than robots
Self-maintaining, self
fueling, self-replicating,
self-cleaning, self-
storing, self-teaching
But maybe a little too
autonomous
If only humans could be
directed in realtime …

37. “Digitize Your Workforce”
• Digitize the work force

38. Takeaways
• Everyone needs a phone to stay in touch, but not
everyone needs to wear glasses
• AR has a few high-value use cases, but niche
– Surgery, emergency, field service
– Hands-free for hands-on education
• … and many nice-to-haves that smartphones do
pretty well
See TechCrunch article

39. 43
WE HAVE MET
THE ROBOTS &
THEY ARE US.