A talk from the Main Stage at AWE USA 2019 - the World's #1 XR Conference & Expo in Santa Clara, California May 29-31, 2019.
Jim Heppelmann (PTC): The New Essential Capabilities of an AR Strategy
Jim Heppelmann, CEO of PTC and co-author of A Manager’s Guide to Augmented Reality, will present the findings of his latest research collaboration with Prof. Michael Porter of the Harvard Business School. As the Augmented Reality (AR) market matures, there are a growing number of capabilities for industrial companies like the real-time capture of front-line processes from the expert’s point-of-view. With these new capabilities comes a set of choices around the strategy, content, and delivery of AR experiences. In this case-study driven keynote, Jim will present a framework of essential questions to capitalize on the promise of AR in the enterprise today.
5. 5
Digital Design
Review
Maintenance
Work
Instructions
Operator &
Assembly Work
Instructions
Performance
Dashboards
Augmented
Service
Manuals
Service
Inspection &
Verification
Remote Expert
Guidance
Augmented
Operator
Manual
Job-Specific
Training
Virtual Product
Companions
Manufacturing Service & Support Operate Train
Survey of PTC customers, January 2019. n = 134
6%
3%
6%
5%5%
6%
5%
4%
6%
5%
Product
Development
Sales &
Market
ADOPTION OF AR USE CASES ACROSS FUNCTIONS
15. 15
ENTERPRISE AR FRAMEWORKContent
3D REF How will you map physical objects and the setting?
INFORMATION What information do you need?
SOURCE How will you source your content?
CONFIGURATION How will you configure your information?
AUTHORING How will you author the experience?
None Surface Marker Object Spatial
Text Images Sound Video Holograms Logic AI Voice AI
Media
Haptics
Purpose
Built
IoT/OT
Data
Simulation
Human
Assistance AI
Hard
Coded
File
System
Content
Server PLM/BIM
Code Author Capture Ad Hoc
Integrate
IoT/OT
Repurposed
Content
Strategy
REALITY Which type of reality will your experience utilize?
CAPABILIITIES What AR capabilities will you leverage?
SENSES What human senses will you stimulate?
DEVICE Which type of device will you use?
PLATFORM What platform will you support?
Assisted Virtual
Visualize Instruct Guide Interact Program Sense Recognize
Sight Sound Touch
Monocular Mobile Projection HUD HMD
Proprietary iOS Android
Windows
UWP
Cross
Platform
Multi
Platform
Augmented
Hard
Coded
Delivery
MAPPING How will you place content in the physical setting?
LOCALIZATION How will you track location and content?
INITIALIZATION How will you initiate your experience?
TRANSPORT How will you send info from device to user?
FEEDBACK How will you provide feedback to the user?
None Unregistered Waypoint Surface Re-tetherable
Manual GPS Beacons Image Object Terrain Triangulation AI
None Manual Marker Wireless Triangulate
Cached Downloaded Streaming
None Touchscreen Voice Gesture Virtual
Screen
Eye
Tracking
Location
IoT
Sensors
Marker Object
Hard
Coded
16. 16
TYPES OF AR
REALITY Which type of reality will your experience utilize?
Strategy
Assisted Virtual
CAPABILIITIES What AR capabilities will you leverage? Visualize Instruct Guide Interact Program Sense Recognize
SENSES What human senses will you stimulate? Sight Sound Touch
DEVICE Which type of device will you use? Monocular Mobile Projection HUD HMD
PLATFORM What platform will you support? Proprietary iOS Android
Windows
UWP
Cross
Platform
Multi
Platform
AugmentedAugmented
Visualize Instruct
Sight
HMD
Windows
UWP
17. 17
DEVICES & PLATFORMS
REALITY Which type of reality will your experience utilize?
Strategy
CAPABILIITIES What AR capabilities will you leverage?
SENSES What human senses will you stimulate?
DEVICE Which type of device will you use?
PLATFORM What platform will you support?
Assisted Virtual
Visualize Instruct Guide Interact Program Sense Recognize
Sight Sound Touch
Augmented
Monocular Mobile Projection HUD HMD
Proprietary iOS Android HoloLens
Cross
Platform
Multi
Platform
18. 18
AUTHOR AND ORIENT
Marker Spatial Targets Object recognition
3D REF How will you map physical objects and the setting?
Content
None Surface Marker Object Spatial
INFORMATION What information do you need? Text Images Sound Video Holograms Logic AI Voice
SOURCE How will you source your content? Purpose
Built
IoT/OT
Data
Simulation
Human
Assistance
AI
CONFIGURATION How will you configure your information?
Hard
Coded
File
System
Content
Server PLM/BIM
AUTHORING How will you author the experience? Code Author Capture Ad Hoc Integrate
IoT/OT
Repurposed
Content
Marker
Sound Holograms
Repurposed
Content
IoT/OT
Data
Content
Server
Code Author
19. 19
CONTENT
Haptics AI Digital Thread
3D REF How will you map physical objects and the setting?
Content
None Surface Marker Object Spatial
INFORMATION What information do you need? Text Images Sound Video Holograms Logic AI Voice
AI
Media
Haptics
SOURCE How will you source your content? Purpose
Built
IoT/OT
Data
Simulation
Human
Assistance
AI
CONFIGURATION How will you configure your information?
Hard
Coded
File
System
Content
Server PLM/BIM
AUTHORING How will you author the experience? Code Author Capture Ad Hoc Integrate
IoT/OT
Repurposed
Content
Marker
Sound Holograms
Repurposed
Content
IoT/OT
Data
Content
Server
Code Author
20. 20
None Unregistered Waypoint Surface Re-tetherable
Manual GPS Beacons Image Object Terrain Triangulation AI
Marker Object
INTERACTION
Triangulate Gesture Eye tracking
MAPPING How will you place content in the physical setting?
Delivery
LOCALIZATION How will you track location and content?
INITIALIZATION How will you initiate your experience? None Manual Hard
Coded
Marker Wireless Triangulate
TRANSPORT How will you send info from device to user? Cached Downloaded Streaming
FEEDBACK How will you provide feedback to the user? None Touchscreen Voice Gesture Virtual
Screen
Eye
Tracking
Location
IoT
Sensors
21. 21
PLACE AND TRACK
Waypoint Beacons
MAPPING How will you place content in the physical setting?
Delivery
None Unregistered Waypoint Surface Re-tetherable
LOCALIZATION How will you track location and content? Manual GPS Beacons Image Object Terrain Triangulation AI
INITIALIZATION How will you initiate your experience? None Manual Hard
Coded
Marker Wireless Triangulate
TRANSPORT How will you send info from device to user? Cached Downloaded Streaming
FEEDBACK How will you provide feedback to the user? None Touchscreen Voice Gesture Virtual
Screen
Eye
Tracking
Location
IoT
Sensors
Marker Object
22. 22
DeliveryContent
ENTERPRISE AR FRAMEWORKStrategy
MAPPING How will you place content in the physical setting?
LOCALIZATION How will you track location and content?
INITIALIZATION How will you initiate your experience?
TRANSPORT How will you send info from device to user?
FEEDBACK How will you provide feedback to the user?
3D REF How will you map physical objects and the setting?
INFORMATION What information do you need?
SOURCE How will you source your content?
CONFIGURATION How will you configure your information?
AUTHORING How will you author the experience?
REALITY Which type of reality will your experience utilize?
CAPABILIITIES What AR capabilities will you leverage?
SENSES What human senses will you stimulate?
DEVICE Which type of device will you use?
PLATFORM What platform will you support?
None Unregistered Waypoint Surface Re-tetherable
Manual GPS Beacons Image Object Terrain Triangulation AI
None Manual Hard
Coded
Marker Wireless Triangulate
Cached Downloaded Streaming
None Touchscreen Voice Gesture Virtual
Screen
Eye
Tracking
Location
IoT
Sensors
Marker Object
None Surface Marker Object Spatial
Text Images Sound Video Holograms Logic AI Voice AI
Media
Haptics
Purpose
Built
IoT/OT
Data
Simulation
Human
Assistance AI
Hard
Coded
File
System
Content
Server PLM/BIM
Code Author Capture Ad Hoc
Integrate
IoT/OT
Repurposed
Content
Assisted Virtual
Visualize Instruct Guide Interact Program Sense Recognize
Sight Sound Touch
Monocular Mobile Projection HUD HMD
Proprietary iOS Android
Windows
UWP
Cross
Platform
Multi
Platform
Augmented
23. 23
MAPPING How will you place content in the physical setting?
Delivery
None Unregistered Waypoint Surface Re-tetherable
LOCALIZATION How will you track location and content? Manual GPS Beacons Image Object Terrain Triangulation AI
INITIALIZATION How will you initiate your experience? None Manual Hard
Coded
Marker Wireless Triangulate
TRANSPORT How will you send info from device to user? Cached Downloaded Streaming
FEEDBACK How will you provide feedback to the user? None Touchscreen Voice Gesture Virtual
Screen
Eye
Tracking
Location
IoT
Sensors
Marker Object
3D REF How will you map physical objects and the setting?
Content
None Surface Marker Object Spatial
INFORMATION What information do you need? Text Images Sound Video Holograms Logic AI Voice AI
Media
Haptics
SOURCE How will you source your content? Purpose
Built
IoT/OT
Data
Simulation
Human
Assistance AI
CONFIGURATION How will you configure your information?
Hard
Coded
File
System
Content
Server PLM/BIM
AUTHORING How will you author the experience? Code Author Capture Ad Hoc
Integrate
IoT/OT
Repurposed
Content
MARKETING FURNITURE
REALITY Which type of reality will your experience utilize?
Strategy
Assisted Virtual
CAPABILIITIES What AR capabilities will you leverage? Visualize Instruct Guide Interact Program Sense Recognize
SENSES What human senses will you stimulate? Sight Sound Touch
DEVICE Which type of device will you use? Monocular Mobile Projection HUD HMD
PLATFORM What platform will you support? Proprietary iOS Android
Windows
UWP
Cross
Platform
Multi
Platform
Augmented
24. 24
OPERATING EQUIPMENT
MAPPING How will you place content in the physical setting?
Delivery
None Unregistered Waypoint Surface Re-tetherable
LOCALIZATION How will you track location and content? Manual GPS Beacons Image Object Terrain Triangulation AI
INITIALIZATION How will you initiate your experience? None Manual Hard
Coded
Marker Wireless Triangulate
TRANSPORT How will you send info from device to user? Cached Downloaded Streaming
FEEDBACK How will you provide feedback to the user? None Touchscreen Voice Gesture Virtual
Screen
Eye
Tracking
Location
IoT
Sensors
Marker Object
3D REF How will you map physical objects and the setting?
Content
None Surface Marker Object Spatial
INFORMATION What information do you need? Text Images Sound Video Holograms Logic AI Voice AI
Media
Haptics
SOURCE How will you source your content? Purpose
Built
IoT/OT
Data
Simulation
Human
Assistance AI
CONFIGURATION How will you configure your information?
Hard
Coded
File
System
Content
Server
PLM/BIM
AUTHORING How will you author the experience? Code Author Capture Ad Hoc
Integrate
IoT/OT
Repurposed
Content
REALITY Which type of reality will your experience utilize?
Strategy
Assisted Virtual
CAPABILIITIES What AR capabilities will you leverage? Visualize Instruct Guide Interact Program Sense Recognize
SENSES What human senses will you stimulate? Sight Sound Touch
DEVICE Which type of device will you use? Monocular Mobile Projection HUD HMD
PLATFORM What platform will you support? Proprietary iOS Android
Windows
UWP
Cross
Platform
Multi
Platform
AugmentedAugmented
Visualize Instruct
Sight
HMD
Windows
UWP
Image
Manual
Cached
Voice
Object
Marker
Sound Holograms
Repurposed
Content
IoT/OT
Data
Content
Server
Code Author
25. 25
MAPPING How will you place content in the physical setting?
Delivery
None Unregistered Waypoint Surface Re-tetherable
LOCALIZATION How will you track location and content? Manual GPS Beacons Image Object Terrain Triangulation AI
INITIALIZATION How will you initiate your experience? None Manual Hard
Coded
Marker Wireless Triangulate
TRANSPORT How will you send info from device to user? Cached Downloaded Streaming
FEEDBACK How will you provide feedback to the user? None Touchscreen Voice Gesture Virtual
Screen
Eye
Tracking
Location
IoT
Sensors
Marker Object
3D REF How will you map physical objects and the setting?
Content
None Surface Marker Object Spatial
INFORMATION What information do you need? Text Images Sound Video Holograms Logic AI Voice AI
Media
Haptics
SOURCE How will you source your content? Purpose
Built
IoT/OT
Data
Simulation
Human
Assistance AI
CONFIGURATION How will you configure your information?
Hard
Coded
File
System
Content
Server PLM/BIM
AUTHORING How will you author the experience? Code Author Capture Ad Hoc
Integrate
IoT/OT
Repurposed
Content
ASSEMBLING COMPONENTS
REALITY Which type of reality will your experience utilize?
Strategy
Assisted Virtual
CAPABILIITIES What AR capabilities will you leverage? Visualize Instruct Guide Interact Program Sense Recognize
SENSES What human senses will you stimulate? Sight Sound Touch
DEVICE Which type of device will you use? Monocular Mobile Projection HUD HMD
PLATFORM What platform will you support? Proprietary iOS Android
Windows
UWP
Cross
Platform
Multi
Platform
Augmented
26. 26
MAPPING How will you place content in the physical setting?
Delivery
None Unregistered Waypoint Surface Re-tetherable
LOCALIZATION How will you track location and content? Manual GPS Beacons Image Object Terrain Triangulation AI
INITIALIZATION How will you initiate your experience? None Manual Hard
Coded
Marker Wireless Triangulate
TRANSPORT How will you send info from device to user? Cached Downloaded Streaming
FEEDBACK How will you provide feedback to the user? None Touchscreen Voice Gesture Virtual
Screen
Eye
Tracking
Location
IoT
Sensors
Marker Object
3D REF How will you map physical objects and the setting?
Content
None Surface Marker Object Spatial
INFORMATION What information do you need? Text Images Sound Video Holograms Logic AI Voice AI
Media
Haptics
SOURCE How will you source your content? Purpose
Built
IoT/OT
Data
Simulation
Human
Assistance AI
CONFIGURATION How will you configure your information?
Hard
Coded
File
System
Content
Server PLM/BIM
AUTHORING How will you author the experience? Code Author Capture Ad Hoc
Integrate
IoT/OT
Repurposed
Content
CAPTURING MANUFACTURING EXPERTISE
REALITY Which type of reality will your experience utilize?
Strategy
Assisted Virtual
CAPABILIITIES What AR capabilities will you leverage? Visualize Instruct Guide Interact Program Sense Recognize
SENSES What human senses will you stimulate? Sight Sound Touch
DEVICE Which type of device will you use? Monocular Mobile Projection HUD HMD
PLATFORM What platform will you support? Proprietary iOS Android
Windows
UWP
Cross
Platform
Multi
Platform
Augmented
27. 27
MAPPING How will you place content in the physical setting?
Delivery
None Unregistered Waypoint Surface Re-tetherable
LOCALIZATION How will you track location and content? Manual GPS Beacons Image Object Terrain Triangulation AI
INITIALIZATION How will you initiate your experience? None Manual Hard
Coded
Marker Wireless Triangulate
TRANSPORT How will you send info from device to user? Cached Downloaded Streaming
FEEDBACK How will you provide feedback to the user? None Touchscreen Voice Gesture Virtual
Screen
Eye
Tracking
Location
IoT
Sensors
Marker Object
3D REF How will you map physical objects and the setting?
Content
None Surface Marker Object Spatial
INFORMATION What information do you need? Text Images Sound Video Holograms Logic AI Voice AI
Media
Haptics
SOURCE How will you source your content? Purpose
Built
IoT/OT
Data
Simulation
Human
Assistance AI
CONFIGURATION How will you configure your information?
Hard
Coded
File
System
Content
Server PLM/BIM
AUTHORING How will you author the experience? Code Author Capture Ad Hoc
Integrate
IoT/OT
Repurposed
Content
TRAINING OPERATORS JUST-IN-TIME
REALITY Which type of reality will your experience utilize?
Strategy
Assisted Virtual
CAPABILIITIES What AR capabilities will you leverage? Visualize Instruct Guide Interact Program Sense Recognize
SENSES What human senses will you stimulate? Sight Sound Touch
DEVICE Which type of device will you use? Monocular Mobile Projection HUD HMD
PLATFORM What platform will you support? Proprietary iOS Android
Windows
UWP
Cross
Platform
Multi
Platform
Augmented
28. 28
MAPPING How will you place content in the physical setting?
Delivery
None Unregistered Waypoint Surface Re-tetherable
LOCALIZATION How will you track location and content? Manual GPS Beacons Image Object Terrain Triangulation AI
INITIALIZATION How will you initiate your experience? None Manual Hard
Coded
Marker Wireless Triangulate
TRANSPORT How will you send info from device to user? Cached Downloaded Streaming
FEEDBACK How will you provide feedback to the user? None Touchscreen Voice Gesture Virtual
Screen
Eye
Tracking
Location
IoT
Sensors
Marker Object
3D REF How will you map physical objects and the setting?
Content
None Surface Marker Object Spatial
INFORMATION What information do you need? Text Images Sound Video Holograms Logic AI Voice AI
Media
Haptics
SOURCE How will you source your content? Purpose
Built
IoT/OT
Data
Simulation
Human
Assistance AI
CONFIGURATION How will you configure your information?
Hard
Coded
File
System
Content
Server PLM/BIM
AUTHORING How will you author the experience? Code Author Capture Ad Hoc
Integrate
IoT/OT
Repurposed
Content
GUIDING WAREHOUSE WORKERS
REALITY Which type of reality will your experience utilize?
Strategy
Assisted Virtual
CAPABILIITIES What AR capabilities will you leverage? Visualize Instruct Guide Interact Program Sense Recognize
SENSES What human senses will you stimulate? Sight Sound Touch
DEVICE Which type of device will you use? Monocular Mobile Projection HUD HMD
PLATFORM What platform will you support? Proprietary iOS Android
Windows
UWP
Cross
Platform
Multi
Platform
Augmented
29. 29
MAPPING How will you place content in the physical setting?
Delivery
None Unregistered Waypoint Surface Re-tetherable
LOCALIZATION How will you track location and content? Manual GPS Beacons Image Object Terrain Triangulation AI
INITIALIZATION How will you initiate your experience? None Manual Hard
Coded
Marker Wireless Triangulate
TRANSPORT How will you send info from device to user? Cached Downloaded Streaming
FEEDBACK How will you provide feedback to the user? None Touchscreen Voice Gesture Virtual
Screen
Eye
Tracking
Location
IoT
Sensors
Marker Object
3D REF How will you map physical objects and the setting?
Content
None Surface Marker Object Spatial
INFORMATION What information do you need? Text Images Sound Video Holograms Logic AI Voice AI
Media
Haptics
SOURCE How will you source your content? Purpose
Built
IoT/OT
Data
Simulation
Human
Assistance AI
CONFIGURATION How will you configure your information?
Hard
Coded
File
System
Content
Server PLM/BIM
AUTHORING How will you author the experience? Code Author Capture Ad Hoc
Integrate
IoT/OT
Repurposed
Content
SERVICING MED DEVICES
REALITY Which type of reality will your experience utilize?
Strategy
Assisted Virtual
CAPABILIITIES What AR capabilities will you leverage? Visualize Instruct Guide Interact Program Sense Recognize
SENSES What human senses will you stimulate? Sight Sound Touch
DEVICE Which type of device will you use? Monocular Mobile Projection HUD HMD
PLATFORM What platform will you support? Proprietary iOS Android
Windows
UWP
Cross
Platform
Multi
Platform
Augmented
30. 30
MAPPING How will you place content in the physical setting?
Delivery
None Unregistered Waypoint Surface Re-tetherable
LOCALIZATION How will you track location and content? Manual GPS Beacons Image Object Terrain Triangulation AI
INITIALIZATION How will you initiate your experience? None Manual Hard
Coded
Marker Wireless Triangulate
TRANSPORT How will you send info from device to user? Cached Downloaded Streaming
FEEDBACK How will you provide feedback to the user? None Touchscreen Voice Gesture Virtual
Screen
Eye
Tracking
Location
IoT
Sensors
Marker Object
3D REF How will you map physical objects and the setting?
Content
None Surface Marker Object Spatial
INFORMATION What information do you need? Text Images Sound Video Holograms Logic AI Voice AI
Media
Haptics
SOURCE How will you source your content? Purpose
Built
IoT/OT
Data
Simulation
Human
Assistance AI
CONFIGURATION How will you configure your information?
Hard
Coded
File
System
Content
Server PLM/BIM
AUTHORING How will you author the experience? Code Author Capture Ad Hoc
Integrate
IoT/OT
Repurposed
Content
SUPPORTING FIELD OPERATORS
REALITY Which type of reality will your experience utilize?
Strategy
Assisted Virtual
CAPABILIITIES What AR capabilities will you leverage? Visualize Instruct Guide Interact Program Sense Recognize
SENSES What human senses will you stimulate? Sight Sound Touch
DEVICE Which type of device will you use? Monocular Mobile Projection HUD HMD
PLATFORM What platform will you support? Proprietary iOS Android
Windows
UWP
Cross
Platform
Multi
Platform
Augmented
31. 31
CLOSING THOUGHTS
• AR is the most natural way to deliver digital information to front-line workers
– More effective training and guidance to deal with turnover and an expected millions
shortage of skilled front-line workers
• AR enables people, as IoT enables products
– Allows human work to be monitored, guided, and optimized
– Significantly enhancing worker productivity through better collaboration with
machines
• Connecting people to the internet, via AR, counterbalances the threat from
automation by empowering human workers
• Both IoT and AR change the competitive environment, requiring new strategic
choices and organizational models