A talk from the Design Track at AWE USA 2018 - the World's #1 XR Conference & Expo in Santa Clara, California May 30- June 1, 2018. Michael Hoffman (Object Theory): Mixed Reality Use Cases and Challenges Today, four years in, Object Theory has spoken to many hundreds of innovators and business leaders across a dozen industries to hear the types of 3D business problems they wish to solve with mixed reality. In this process clear patterns have emerged in the types of use cases that match well with mixed reality’s unique advantages. http://AugmentedWorldExpo.com

1. Mixed Reality
Use Cases and Challenges
Michael Hoffman
Co-founder / Partner
Object Theory
michael@objecttheory.com

2. As a trusted advisor,
Object Theory
helps F1000 clients
strategically identify, design and build
XR solutions
that provide real business value
utilizing today’s XR devices

3. Mixed Reality Benefits
• Hands-free – Increase information access efficiency while performing tasks
• Shared 3D context – Accurately convey focus & intent to reduce misunderstandings
• Stereoscopic vision – Improve comprehension of subject matter
• True scale – Accurately experience & evaluate spatial relationships and appearance
• Spatial mapping – Reduce mistakes by proximity of information to objects/environment
• 3D sound – Increase sense of presence, and accurately direct attention to focal points
• Data visualization – Improve comprehension of complex geospatial data sets
• High fidelity 3D objects –
• Reduce need for expensive 3D print & paint
• Reduce dependency on expensive equipment and supplies
• Safely train for dangerous tasks
• 3D environments – Leverage spatial memory to improve retention & navigation
Quantifiable business value from these unique features & benefits:

4. Mixed Reality Use Cases
Remote Collaboration
Product Design
Annotated Environment
Premium Selling
Soft Skills Training
Hard Skills Training
Space Planning
Situation Planning
Experience Invisible Data
Medical Imaging
Remote Expert
Infinite Screen
Education
Expert Coaching
Automated Guidance
Art and Creativity
Entertainment

5. Remote Collaboration
The Business Challenge
When dispersed teams collaborate on 3D problems, it’s
difficult to establish an accurate shared context and convey
intent with existing 2D tools such as screen sharing, which
can result in costly mistakes.
Mixed Reality Solution
Collaborators feel a sense of presence, plus virtual
geospatial annotations and markers can accurately convey
intent to establish a more accurate shared context resulting
in more informed decisions and fewer mistakes due to
miscommunication.
Why Mixed Reality?
• Gaze indicates focus and intent
• 3D Spatial sound provides cues and sense of presence
• Stereoscopic vision improves comprehension
• Geospatial annotations provide critical context
• Ability to see content while conversing/collaborating
Potential Challenges
• 3D audio chat requires reliable internet connection

6. Product Design
The Business Challenge
When designing products, it is difficult to understand how all
of its components interplay and how it will appear as a final
product. Furthermore, efficiently collaborating between
different design disciplines to find the best outcomes is
challenging using existing 2D means.
Mixed Reality Solution
Mixed Reality makes it possible for design collaborators to
establish an accurate shared context of a product under
design for efficient, accurate decision making and faster
iteration cycle times.
Why Mixed Reality?
• Stereoscopic vision
• Accurate, true scale visualization of product
• Ability to see content while conversing/collaborating
• Efficient ability to try alternative designs
Potential Challenges
• Color fidelity not as good as calibrated monitors
• Large model geometries

7. Annotated Environment (BIM / IoT)
The Business Challenge
BIM data and environmental sensor data is collected by the
terabytes, yet most of that data is never utilized. Only a
fraction of the potential value is extracted.
Mixed Reality Solution
With ML/AI + MR, the collected data can be analyzed for
patterns an then the insights can be presented geosnatially
right where an issues need addressing. Further, the
relationships between the metadata and the physical plant
are much easier to glean from observation.
Why Mixed Reality?
• Virtual on Physical
• Ability to see data while performing tasks
• Stereoscopic vision for better insights
Potential Challenges
• Not recommended yet for outdoor use cases
• Need to use markers when navigating large facilities.

8. Premium Selling
The Business Challenge
It is difficult to sell expensive products, services and
experiences that are hard to put in front of a client or hard to
explain in detail.
Mixed Reality Solution
Telling the rich story of an offering in 3D makes it more
personal, relatable, understandable and memorable
Why Mixed Reality?
• Stereoscopic vision to better visualize
• Alignment to physical to reveal hidden features
• Spatial story telling is more compelling and memorable
Potential Challenges
• It takes 5 - 15 minutes to acclimate to mixed reality

9. Soft skills training
The Business Challenge
It is expensive to train soft skills that require understanding
nuance and adept reaction to varying external stimuli and
behaviors.
Mixed Reality Solution
The combination of AI+MR reduces training cost by making it
possible to recreate realistic interactive soft skill scenarios,
automate the AI/ML assessment of performance and
generation of precise feedback.
Why Mixed Reality?
• Stereoscopic vision for more realistic scenarios
• Alignment to physical further increases realism
• Gaze indicates trainee focus and intent
• Positional info, camera and mics feed data to cognitive
services
• Hands free enables using actual soft skill devices
Potential Challenges
• No significant challenges

10. Hard skills training
The Business Challenge
It is expensive to train skills to a large workforce that require
learning a wide variety of steps and operations, particularly in
high-turnover environments.
Mixed Reality Solution
Creation of self-guided, interactive skills training tutorials
substantially reduces the labor cost of repeatedly training the
same skill.
Why Mixed Reality?
• Hands free
• Stereoscopic vision for better, faster comprehension
• Alignment to physical enables practicing on real
equipment
• Information close to where you need it
Potential Challenges
• Creating 3D interactive content for each skill can be
expensive

11. Space Planning
The Business Challenge
Experiencing the feel and function of a space
under design is a critical step in the design
process to reduce mistakes, yet is so costly to
achieve that it’s rarely done.
Mixed Reality Solution
Mixed Reality makes it possible to efficiently
experience and even modify a non-existent room
design at true scale.
Why Mixed Reality?
• True size and scale
• Stereoscopic vision
• Ability to see content while conversing/collaborating
• Align virtual changes to physical environment
Potential Challenges
• True scale immersive scenarios can benefit from wider
FOV, particularly for new users

12. Situation Planning
The Business Challenge
Planning and monitoring complex geospatial scenarios
demands comprehensive understanding of complex
information across a dispersed team of experts. It is difficult
to provide all relevant information to create a comprehensive
and accurate shared context.
Mixed Reality Solution
Mixed reality enables effective collaboration across
dispersed teams while interacting with a wide variety of
geospatially accurate 3D information and supplementary 2D
information.
Why Mixed Reality?
• Geospatially accurate information
• Stereoscopic vision
• Ability to see content while conversing/collaborating
Potential Challenges
• Content creation can be expensive
• Geometry complexity
• WiFi security (for Defense use cases)

13. Experience Invisible Data
The Business Challenge
Data analysis of 3D data sets and simulations of geospatial
information on 2D screens makes it difficult to comprehend
the full extent of insights that can be gained.
Mixed Reality Solution
With Mixed Reality, simulations and visualizations of
collected sensor data or simulations can be played and
animated geospatially and can be aligned with physical
objects related to the data.
Why Mixed Reality?
• Stereoscopic vision
• 3D in 3D
• Align virtual to physical
• Ability to see content while conversing/collaborating
• Experience the inaccessible
Potential Challenges
• No major challenges

14. Medical Imaging
The Business Challenge
Surgeons are constantly evaluating 3D problems on 2D
screens. Missing critical insights and longer assessment
times can lead to worse outcomes for patients.
Mixed Reality Solution
Hands-free access to all relevant 2D and 3D information
exactly where it’s needed enables more accurate and
efficient assessment for better outcomes.
Why Mixed Reality?
• Hands free
• Stereoscopic vision for better, faster insights
• Alignment to physical reduces errors
• Information proximity without physical interference
• Instant access to vast amounts of 2D & 3D information
Potential Challenges
• Accurate alignment to body below 1-2mm difficult
• Regulatory requirements for operating room
• Weight and comfort for long-term use

15. Remote Expert
The Business Challenge
Flying subject matter experts onsite to support field operations
is expensive and requires too many experts on staff for peak
load.
Mixed Reality Solution
A smaller team of centralized experts can effectively assist
many field technicians as needed at lower travel and labor cost.
Why Mixed Reality?
• Hands free enables performing tasks while being assisted
• Geospatial indicators/annotations increases accuracy
• Stereoscopic vision increases comprehension
Potential Challenges
• Requires reliable internet connection for audio & video
• Safety compliance requirements
• Outdoor use
• Safety compliance requirements
• WiFi security (for Defense use cases)

16. Infinite Screen
The Business Challenge
Many tasks require analyzing huge amounts of 2D and 3D
information. We are constrained by the realistic number of
pixels we can surround ourselves with.
Mixed Reality Solution
Mixed Reality enables augmenting existing screen real
estate with a nearly infinite amount of additional 2D panels
as well as 3D content for a vast increase in available
information for analysis and decision making.
Why Mixed Reality?
• Can interact with physical and virtual world simultaneously
• Not constrained by limits of 2D monitors
Potential Challenges
• Today’s devices don’t yet run the sophisticated apps
professionals depend on.

17. Education
The Business Challenge
It is difficult and expensive to teach concepts and techniques
that consume expensive supplies, depend on expensive
equipment, or can be dangerous.
Mixed Reality Solution
For at least the early stages of the learning process,
simulating concepts and techniques is a less dangerous and
expensive alternative, and can reduce dependency on
access to expensive equipment.
Why Mixed Reality?
• Hands free
• Stereoscopic vision for better, faster comprehension
• Alignment to physical enables practicing with some or all
physical equipment, or even props.
• Information close to where you need it
• Instant access to vast amounts of 2D & 3D information
• Ability to see content while conversing/collaborating
Potential Challenges
• Creating a large variety of interactive 3D content can be

18. Expert Coaching
The Business Challenge
Improving a fine motor skill such as the perfect golf swing is
extremely difficult and requires deep expertise.
Mixed Reality Solution
IoT sensors plus mixed reality enables accurate capture and
visualization of a wide variety of information for more
accurate assessment and improvement of technique.
Why Mixed Reality?
• Hands free
• Alignment to physical to see relationships and variance
• 3D visualization for accurate assessment and comparison
of 3D movements
Potential Challenges
• Outdoor use not recommended

19. Automated Guidance
The Business Challenge
Utilizing only physical props, providing all useful information
to guide an audience through an experience creates traffic
bottlenecks, unwanted visual clutter and limits storytelling
possibilities.
Mixed Reality Solution
Mixed reality increases expressiveness and efficiency while
eliminating bottlenecks. Any amount of interactive 2D, 3D
and spatial audio content can be situated geospatially around
points of interest to convey useful information in compelling
ways that maximize expressiveness, comprehension and
memory.
Why Mixed Reality?
• Stereoscopic vision for better, faster comprehension
• Information proximity & alignment increases context
• Hands free
• Instant access to vast amounts of 2D & 3D information
Potential Challenges
• Outdoor use not recommended

20. Art and Creativity
The Business Challenge
Monetizing entertainment is becoming increasingly difficult.
Theater and sports arena attendance continues to drop.
Mixed Reality Solution
Mixed Reality adds interactivity and extends the average
time spent consuming entertainment. This increases the
opportunities to monetize it.
Why Mixed Reality?
• Can augment existing means of consuming
entertainment.
• Can move and interact with environment while
experiencing content.
Potential Challenges
• MR not yet a ubiquitous consumer device.

21. Entertainment
The Business Challenge
Monetizing entertainment is becoming increasingly difficult.
Theater and sports arena attendance continues to drop.
Mixed Reality Solution
Mixed Reality adds interactivity and extends the average
time spent consuming entertainment. This increases the
opportunities to monetize it.
Why Mixed Reality?
• Can augment existing means of consuming
entertainment.
• Can move and interact with environment while
experiencing content.
Potential Challenges
• MR not yet a ubiquitous consumer device.

22. Addressing Today’s Challenges
• Safety: 3rd Party HoloLens hard hats (Trimble)
• Typing: Device assisted authentication
• Placement/Location:
• Fiducial markers
• Manual logical alignment
• Global anchor sharing
• Attention: 3D audio, narrative and visual cues
• Static 3D Content:
• Expert manual optimization
• Level-of-detail (LODs)
• Smart use of geometry culling
• FOV:
• Start small for context, then enlarge
• Attention directors and audio cues
• Dynamic 3D Content: Simplygon automation
• Material fidelity: Hand optimized shaders
• Input:
• Hand controllers and clickers
• Smart phones as companion input devices
• All day:
• USB battery packs
• Device swap
• Spectator View for professional videos / photos
• Spatial mapping / tracking:
How Object Theory addresses today’s MR challenges:

23. With thoughtful experience
design, diligent optimization and
smart engineering,
today’s mixed reality devices
deliver real business value
for a wide variety of use cases.

24. Thank you!
Michael Hoffman
Co-founder / Partner
Object Theory
michael@objecttheory.com
linkedin.com/in/mthoffman
@objecttheory