Synapse Product Development Electrical Engineers, Doug Bruey and Michael Ciuffo presented a technical workshop at SXSW 2017.
This year, Valve opened up licensing of SteamVR Tracking, the world’s best 3D tracking solution and the breakthrough technology behind the HTC Vive. Now anyone can develop devices to operate in VR. Since 2014, Synapse has provided engineering support for Valve's VR development and now delivers training courses for all new licensees.
Synapse is thrilled to bring this training to SXSW as a condensed workshop; providing a behind the scenes look into the technology with hands-on experience in tracked object design and integration; delivering a solid foundation in the theory of SteamVR Tracking and an understanding of the hardware & software experience needed to design tracked objects.
Sneak Peek: Designing Tracked Objects for Steam VR
1. 1 | S X S W 2 0 1 7
DESIGNING TRACKED
DOUG BRUEY | MICHAEL CIUFFO
OBJECTS FOR
2. 2 | S X S W 2 0 1 7
■ Product development for the
world’s leading companies
■ 150+ multi-disciplinary engineers
■ Cambridge Consultants
• Global parent company
• 500+ Engineers and innovators
SYNAPSE
SEATTLE
SAN FRANCISCO
ORLANDO
BOSTON
NEW DELHI
CAMBRIDGE
HONG KONG
SINGAPORE
TOKYO
3. 3 | S X S W 2 0 1 7
Valve is an entertainment software and technology company founded in 1996. In addition to creating several of
the world's most award-winning games, Valve is also a developer of leading-edge technologies including the
Source® game engine and Steam®, the premier online gaming platform.
4. OCT 2014
Synapse started
working with Valve
on their vr system
HISTORY
4 | S X S W 2 0 1 7
JUNE 2015
Developer release
of the base
stations
APRIL 2016
HTC Vive released
AUG 2016
Valve announces royalty
free licensing of
SteamVR™ tracking
SEPT 2016
SteamVR™ tracking
reference design
released
2017
Develop firmware and
assist partners
integrating SteamVR™
tracking into their
products
SEPT 2016 - TODAY
SteamVR™ tracking
training courses at
synapse
5. • What is SteamVR™ Tracking?
• The problem it solves in VR and AR applications
• How it works to track the pose of an object
• How to make objects that track well using the technology
• How to get started using the technology
• Answer as many questions and possible
WE’LL COVER
5 | S X S W 2 0 1 7
7. Z-Axis
z
7 | S X S W 2 0 1 7
X-Axis
Y-Axis
x
POINT
(x,y,z)
y
POSITION
POSE TRACKING
8. STATE OF THE ART
8 | S X S W 2 0 1 7
IMUs
(Inertial Measurement Units)
■ Gyroscopes and accelerometers
track orientation
External Cameras
(Pose Tracking)
Internal Cameras
(Inside-out Pose Tracking)
■ Popular in AR
■ Depth Perception & Room
Mapping
SteamVR™ Tracking
(Pose Tracking)
9. STEAMVR™ TRACKING
9 | S X S W 2 0 1 7
■ Room scale
■ Security/privacy
■ Low latency & bandwidth
■ Absolute position
■ Eliminates challenges of camera-based tracking
■ Track to submillimeter accuracy
15. ■ Optical receivers detect the reference signals from the base station
■ Photodiode converts IR light to current
■ Transimpedance amplifier converts current to voltage
■ Envelope detector removes the modulation frequency
■ FPGA connects to all sensors to timestamp the arrival of reference signals
OPTICAL RECEIVERS
1 5 | S X S W 2 0 1 7
17. ■ The angle to all sensors creates a solution set for the position of the object
■ SteamVR™ matches the measured angles to the known object geometry
■ There must only be one solution to the problem!
SOLVING THE SYSTEM
1 7 | S X S W 2 0 1 7
19. DEMO
1 9 | S X S W 2 0 1 7
ONE TO FOUR SENSORS
20. ■ Reference signal jitter
• Motor jitter
• Timestamp quantization
• Laser modulation
■ Minimized by design
• Base station design
• Object design
■ Sources of error place requirements on object design
SOURCES OF ERROR
2 0 | S X S W 2 0 1 7
21. ■ Baseline increases the time between laser hits
■ Sources of error are angular
■ The limit of the system is a minimum detectable angle
■ More baseline accommodates the same angle at a greater distance
■ Need baseline to overcome translation error
CONSEQUENCES OF ERROR | TRANSLATION ERROR
2 1 | S X S W 2 0 1 7
22. ■ Baseline increases the time between laser hits
■ Sources of error are angular
■ The limit of the system is a minimum detectable angle
■ More baseline accommodates the same angle at a greater distance
■ Need baseline to overcome translation error
OVERCOMING TRANSLATION ERROR | SUFFICIENT BASELINE
2 2 | S X S W 2 0 1 7
23. ■ Rotation orthogonal to a plane yields significant displacement
■ Rotation in the plane yields much smaller displacement per degree rotation
■ Error dominates the small change in distance
CONSEQUENCES OF ERROR | ROTATION ERROR
2 3 | S X S W 2 0 1 7
24. ■ Baseline amplifies the effect of rotation
■ Pose tracking detects
• Roll
• Pitch
• Yaw
■ Detecting all three means amplifying all three
■ Need baseline in X, Y and Z axes to overcome rotation error
OVERCOMING ROTATION ERROR | BASELINE IN 3 AXES
2 4 | S X S W 2 0 1 7
25. ■ Four visible sensors (minimum)
■ Maximize the distance between sensors
■ Maximize baseline in three axes
TRACKING PERFORMANCE DRIVERS
2 5 | S X S W 2 0 1 7
d
x
z
y2
3
1
4
28. ■ Objects that track well have geometries designed for optimal
sensor placement!
■ Simulation lets us verify performance early in the process.
SENSOR PLACEMENT CRITERIA
2 8 | S X S W 2 0 1 7
29. X, Y, Z coordinates,
facing direction
2 9 | S X S W 2 0 1 7
±60° from normal Shadows cast by the
sensor object
Shadows cast by
nearby objects
39. DEMO
3 9 | S X S W 2 0 1 7
HMD DESIGN REFINEMENTS
40. ■ Consumer products need to look and feel great
■ Products also need to perform, especially VR products
INDUSTRIAL DESIGN CHALLENGES
MINIATURIZATION
4 0 | S X S W 2 0 1 7
FLAT SURFACES RIGHT ANGLES LOW PROFILE CURVED SURFACES
Translation error? Translation and rotation error?Sensor viewing angle?Rotation error? Sensor covering?
41. MECHANICAL DESIGN CHALLENGES
4 1 | S X S W 2 0 1 7
■ Facets on shapes improve performance
• How many slides in the mold?
■ Sensors facing in all directions
• Multiple parts to facilitate ejection
42. ■ Sensor interconnect
■ FPC design challenge
ELECTRICAL DESIGN CHALLENGES
4 2 | S X S W 2 0 1 7
43. RECOMMENDATIONS
4 3 | S X S W 2 0 1 7
■ Collaborate between engineering and
industrial design early in the process
■ Teach industrial designers and product
visionaries about sensor placement
■ Use the constraints of sensor placement as a
seed for creating unique, compelling designs
■ Reduce risk early
• Use the simulation tools in the HDK to validate design choices
• Prototype shapes using rapid prototyping techniques and evaluation hardware
• See your object track in SteamVR™ before investing in tooling
44. ■ Laser based pose tracking system
■ Solves the problem in VR and AR applications
■ Tracks the pose of an object through….
■ Design objects that track well using the technology
• Four visible sensors (minimum)
• Maximize the distance between sensors
• Maximize baseline in three axes
■ Use the software to simulate & refine design
■ Collaboration between ID, ME, EE, & SW is key to successful design
SUMMARY | STEAMVR™ TRACKING
4 4 | S X S W 2 0 1 7
46. ■ VR Accessories
■ AR Products
■ Tracking Systems
WHAT ARE YOU GOING TO MAKE?
4 6 | S X S W 2 0 1 7
47. The information contained herein is confidential and proprietary and may not be reproduced or distributed without the consent of Synapse Product Development
QUESTIONS?
4 7 | S X S W 2 0 1 7
Notes de l'éditeur
Doug and Ciuffo
Electrical Engineers at Synapse
Involved in SteamVR development since May 2015
Teach the SteamVR Tracking course at Synapse
Fun facts?
Ciuffo - Vive early adopter, Cloudlands VR Minigolf expert
Doug - Still trying to master Joust
Synapse locations in Seattle (HQ), San Francisco, Orlando, and Hong Kong
150+ Engineers and innovators
Our parent company and close partners in product development; Cambridge Consultants has locations in Boston, Cambridge, New Delhi, Singapore, and Tokyo
500+ Engineers and innovators
Valve was a software company and Synapse helped them enter the hardware space.
In addition to creating several of the world's most award-winning games, Valve is also a developer of leading-edge technologies including the Source® game engine and Steam®, the premier online gaming platform.
Valve/Synapse, SteamVR Tracking, the HTC Vive, and you (the licensee)...
Outline the story
Synapse started working with Valve on their VR system in Oct, 2014
Collaborated on base stations leading up to the developer release in June, 2015
HTC was already integrating the tech into the HTC Vive which was released in April, 2016
Valve announced royalty free licensing of SteamVR Tracking in Aug, 2016
Developed the SteamVR Tracking reference design released in Sept, 2016
Organized and hosted SteamVR Tracking training courses Sept, 2016 - Today
And actually, the material we are presenting today is a compressed version of the introduction day of that 2 ½ day course
Continue to develop firmware and assist partners integrating SteamVR tracking into their products
Provide a technical overview of SteamVR Tracking
What is SteamVR Tracking?
The problem it solves in VR and AR applications
How it works to track the pose of an object
How to make objects that track well using the technology
How to get started using the technology
Answer as many questions and possible
Pose Tracking, What is it?
Orientation (Roll, Pitch, and Yaw)
Lets you look around
Position (X, Y, and Z)
Lets you move your head to the side, crouch down and look under a table, and walk around a room
VR and AR systems employ a variety of sensors and algorithms to achieve pose tracking
IMUs (Inertial Measurement Units), gyroscopes and accelerometers track orientation
Cannot track location.
Stuck in a seat.
Rollercoaster looking down example.
Samsung GearVR, or Google Daydream
Internal Cameras (‘inside-out’ pose tracking), popular in AR
Not good enough for VR. Too much lag. Not accurate enough.
Microsoft HoloLens
External Cameras (pose tracking)
Cameras aren’t as accurate.
Limited field of view.
PlayStation VR or Oculus Rift
SteamVR™ Tracking (pose tracking)
Sub-mm tracking space
Large tracking space.
HTC Vive
Room scale- That’s actually our VR set up in our Seattle office. Julie is able to move around the large room
Anecdote - People sometimes don’t immediately realize they can walk around and interact with objects, but they can.
Security/privacy
Cameras have extra information not necessary for VR.
Low latency & Low bandwidth
Less information to capture makes it easy to process more quickly and easier to move around the room.
Absolute position
Unlike IMU solutions, positions are locked to 3D space.
Eliminates depth of field and resolution challenges of camera-based tracking
Resolution
Depth of field
Track the pose of an object in a 5m x 5m x 5m volume with submillimeter accuracy.
What is the latency?
“the time it takes sound from my voice to hit audience”
Explain Frames/second
MENTION WE’RE PASSING OUT BASE STATIONS
Sync Blinker (IR LEDs)
X and Y (IR Lasers/Motors)
Synchronization
Wired
Optical
SteamVR Tracking
Optical sensors- Dimples on the Vive
Tracking core electronics
Two base stations reduce shadows
Requires synchronization
Synchronization
Wired
60 Hz clock from master
Balanced wired connection
Original method
Optical
Sync blink
Detected by optical receiver
Better user experience
Maybe mention multi-camera systems like motion capture systems (common misconception)
Mention that base station we hand out has no optical sync
Ciuffo slide. Grab a flashlight and the laser beam splitter.
Sync blinks and laser hits are timestamped by the tracked object
The difference between the sync blink and laser hit gives the angle
End result: Object is somewhere along a ray.
IR light is modulated…ambient light
Optical receivers detect the reference signals from the base station
Photodiode converts IR light to current
Transimpedance amplifier converts current to voltage
Envelope detector remove the modulation frequency
FPGA connects to all sensors to timestamp the arrival of reference signals
The SteamVR software is doing this math in the background
Relative locations of sensors are fixed on the device.
The angle from the base station to each sensor is known
SteamVR™ matches the measured angles to the known object geometry
This example has some ambiguity. Will result in bad performance.
Adding another sensor here makes only one position/orientation the correct one.
Blown up
Single sensor
X-Y is good
No Z
No rotation
Two sensors
More rotation info (left to right)
Ambiguity of distance and rotation
Must have another sensors
Three sensors
In line
No more information than 2 sensors
First rule of sensor placement “avoid collinear sensors!”
In plane
Now can tell between rotation and translation
Still don’t know which way we’re rotating
Four sensors
In plane
No better than 3 sensors.
Still don’t know which way we’re rotating
Out of plane
Finally, have absolute pose
AFTER DEMO: Rearrange object for small baseline example.
0.8 seconds over the course of a day.
Toggle Demo: Noise
Translation Error
As distance increases
Tangential velocity increases
Time between sensors decreases
Error begins to dominate
Limits the maximum radius from the base station
How could we reduce this error?
Baseline increases the time between laser hits
Sources of error are angular
The limit of the system is a minimum detectable angle
More baseline accommodates the same angle at a greater distance
Need baseline to overcome translation error
Toggle Demo: Sufficient Baseline
Rotation Error
Rotation orthogonal to a plane yields significant displacement
Rotation in the plane yields much smaller displacement per degree rotation
Error dominates the small change in distance
How could we reduce this error?
Baseline amplifies the effect of rotation
Pose tracking detects
Roll
Pitch
Yaw
Detecting all three means amplifying all three
Need baseline in X, Y and Z axes to overcome rotation error
Demo:
Make sure to rotate normal to plane before rotating in plane
Viewing Angle with Large object
Allude to need for sensors to be at angles:
Tracking doesn’t work from behind.
Tracking doesn’t really work from the side either
Need to hand off to other sensors.
Mention self-occlusion
Doug spitballs while toggle over to HTC Vive controller
Talk about the wings- probably added to create that sufficient baseline in a 3rd axes
Toggle back to slideshow
Launch HMD-Designer viewer project of X-frame.
Objects that track well have geometries designed for optimal sensor placement!
Simulation lets us verify performance early in the process.
Sensor positions and normals
Position = X, Y, Z coordinates
Normal = facing direction
Sensor field of view
±60° from normal
Self occlusion
Shadows cast by the sensor object
Obstacle occlusion
Shadows cast by nearby objects
Model hands, heads, handles, accessories, etc.
Number or Visible Sensors
Rotation Error
Translation Error
Initial Pose Possible? - Explain
Simple demo object:
# of visible sensors
Rotation error
Translation error
Initial pose possible
Bad HMD design
Show sensors & new shape & complexity
(don’t go into each for the simulation outputs, Doug goes through those next)
Initial indicator of placement quality
Object boots or not from a given pose
MENTION WE’RE PASSING OUT REFERENCE HARDWARE
Refined Object:
Mention it is the actual reference hardware, and that we’re passing them out
Consumer products need to look and feel great
Products also need to perform, especially VR products
Appealing design features:
Miniaturization: What about translation error?
Flat surfaces: What about rotation error?
Right angles: What about sensor viewing angle?
Low profile: What about translation and rotation error?
Curved surfaces: What about sensor covering?
Facets on shapes improve performance
How many slides in the mold?
Sensors facing in all directions
Multiple parts to facilitate ejection
Sensor interconnect
20 - 32 sensors
Distributed over the surface of an object
Four wires to each sensor
All sensors at different angles
FPC design challenge
Circuit size
Panel efficiency
Interconnect density
MENTION: I see a lot of nervous faces in the room. We do this all the time, and it’s not that difficult if you start early knowing the challenges..
Collaborate between engineering and industrial design early in the process
Teach industrial designers and product visionaries about sensor placement
Use the constraints of sensor placement as a seed for creating unique, compelling designs
Reduce risk early
Use the simulation tools in the HDK to validate design choices
Prototype shapes using rapid prototyping techniques and evaluation hardware
See your object track in SteamVR before investing in tooling
Laser based pose tracking system
Solves the problem in VR and AR applications
Tracks the pose of an object through….
Design objects that track well using the technology
Four visible sensors (minimum)
Maximize the distance between sensors
Maximize baseline in three axes
Use the software to simulate & refine design
Collaboration between ID, ME, EE, & SW is key to successful design
Become a SteamVR Tracking Licensee
Just a couple weeks ago, Valve made the HDK(hardware development kit) available, so now you can access the ….
Also, We’ll still be offering classes throughout the year at Synapse, a 2.5 day class that extends and covers
Generating and simulating sensor placement
Writing object config files
You’d walk away with the reference hardware (get the dev kit)
Testing and calibrating objects
Integrating objects with SteamVR
Troubleshooting tracking problems
Hands on experience
Synapse is helping half a dozen companies customize hardware for their needs
Different tracking systems out there, there’s no one system that is right for every scenario
If you are looking for room scale, submillimeter, highly precise pose tracking system for the most immersive experience possible, that’s where Steam VR shines
HTC has done it, LG just announced they’re doing it, there’s a hundred people who have come who our class who have done it
And valve has supplied the software
So you can do it too
Automotive, cinema, aviation, AR