An presentation used for explaining the basics of building a HoloLens application using functionality like 3D objects, spatial sound, gaze, gestures, speech, spatial mapping and occlusion.
2024: Domino Containers - The Next Step. News from the Domino Container commu...
Holo basics develop your mixed reality hololens app with unity and visual studio
1. DEVELOP YOUR MIXED REALITY APP FOR HOLOLENS
WITH UNITY AND VISUAL STUDIO
HOLOBASICS
2. “HoloLens Evangelist”
At ETTU I am responsible architecture related business for our project services
department at ETTU. I’m involved as architect at different companies like Planon,
PGGM, Gemeente Amersfoort and others
Alexander Meijers
ABOUT ME
Solutions Architect for ETTU
Founder of the Mixed Reality User Group
MixUG: http://www.mixug.nl
Email: a.meijers@ettu.nl or alexander@appzinside.com
Twitter: @ameijers
Blog: http://www.appzinside.com
5. HOLOBASICS 5
VIRTUAL REALITY
Generation of realistic images, sounds and other
sensations that replicate a real environment or create an
imaginary setting
An immersive experience created entirely from
computer-generated Content. Also similar to 360 degree
video
6. HOLOBASICS 6
AUGMENTED REALITY
Direct or indirect view of a physical, real-world
environment whose elements are augmented by
computer-generated sensory input such as sound, video,
graphics or GPS data
An overlay of content on the real world where that
content is not anchored to or part of it
7. HOLOBASICS 7
MIXED REALITY
Merging of real and virtual worlds to produce new
environments and visualizations where physical and
digital objects co-exist and interact in real time.
Overlay of synthetic content on the real world that is
anchored to and interacts with the real world
9. HOLOBASICS 9
HISTORY OF HOLOLENS
Codenamed Project HoloLens
Chief inventor Alex Kipman
Official title is technical fellow
He dreamed up Kinect in at the end of 2007
and set a vision which incorporated HoloLens
It is the start of a transforming world
In the new reality, sensors will be anywhere
A visual computing platform controlled by speech and gesture
Satya Nadella - “The emergence of
the next computing interface”
Alex Kipman
10. HOLOBASICS 10
MICROSOFT HOLOLENS
SPECIFICATIONS
Windows 10 device based on 32 bit architecture
64GB flash
2GB memory
1GB Holographic Processor Unity (HPU)
First of its kind
Device is more powerfull than a laptop
No overheating due to warm air flows to the sides
2-3 hours active and 2 weeks standby
Weight 579g
11. HOLOBASICS 11
SENSORS, OPTICS AND
SPEAKERS
Environment
camera
Depth
camera
Video
camera
Spatial Sound
With
speakers
High definition
lenses
12. HOLOBASICS 12
MICROSOFT HOLOLENS
First Holographic computer running on Windows 10
Contains a CPU, GPU and HPU
Human understanding
Gesture and voice input
Gaze tracking
Spatial sound
Uses wi-fi to connect and has no wires attached
Design for comfort
13. HOLOBASICS 13
MICROSOFT HOLOLENS
SPECIFICATIONS
Contains depth camera
Field of vision that spans 120 by 120 degrees
18 sensors flooding the device with terabytes of data
every second
Tricks your brain into perceiving holographic images
Light engine in which light particles are bouncing million times
Photons enter the two lenses
Ricochet some between layers of blue, green and red glass
Hitting back of the eye
21. HOLOBASICS 21
DEVELOPMENT TOOLS
Visual Studio 2017
UWP workload
Game development with Unity workload
Windows 10 SDK (version 1511 or later)
There is no separate SDK for HoloLens
Unity 5.6 or Unity 2017.1
Be aware! - Unity 2017.2 beta for immersive headsets
HoloLens emulator
HoloLens Emulator (build 10.0.14393.1358)
Hyper-V
Contains DirectX project templates for Visual Studio
HoloLens device
23. HOLOBASICS 23
BUILD LIFECYCLE OF A
HOLOLENS PROJECT
Create Unity Project
Configure HoloLens settings
Create scene
Visual Studio
Unity HoloLens
Configure build settings
Build and generate Visual Studio
project
Open project with Visual Studio
Pair with HoloLens
Build & deploy Visual Studio project
Start Application
Test & debug
Monitor
Unity
24. HOLOBASICS 24
DEVELOPMENT FOR
HOLOLENS
HoloLens Device
Developer mode needs to be turned on
Pair the device with your development machine
Retrieve IP address of HoloLens Device
Ask Cortana
Settings Network & Internet Advanced options
28. HOLOBASICS 28
UNITY
GAMEOBJECT SCRIPT
• Derived from MonoBehaviour
• Contains methods which are called by Unity through the a
broadcast messaging system
• void Awake() – Called when class is instantiated
• void Start() – Called when object is enabled once before
the first Update()
• void Update() – Called once per frame
• More methods are available
• FixedUpdate, LateUpdate, OnGUI, OnDisable, OnEnabled
29. HOLOBASICS 29
UNITY
MESSAGE SYSTEM
public void BroadcastMessage(string methodName, object parameter = null, SendMessageOptions options =
SendMessageOptions.RequireReceiver);
• Used for calling methods on GameObjects
• Calls the methods also on its Children
• Examples
• this.BroadcastMessage(“OnSelect”);
• focusedObject.BroadcastMessage(“OnSelect”);
30. HOLOBASICS 30
UNITY
COMPONENTS
• Transform
• Position of object
• Rotation of object
• Scale (size) of object
• Mesh Renderer & Mesh Filter
• Renders the object
• Mostly done using a mesh
• Mesh Collider
• Determines the collision for the object
• Script
• Perform actions on the object
• Handle messages on an object
33. HOLOBASICS 33
COORDINATE SYSTEM
HOLOLENS AND UNITY
forward (x, y, z)Camera (0, 0, 0)
Rotation (x, y, z)
Camera (1, -1, 4)
Rotation (x, y, z)
Hologram
Object Pinned
HoloLens
moved to
new location
View direction
Gaze
34. HOLOBASICS 34
HOLOGRAMS
AND PINNING
Device can project a hologram into a
room and keep it locked in position
Called pinnnig
Objects are not moving relative to you
but you can move around the object
Allows you to view the object from any angle
36. HOLOBASICS 36
PINNED HOLOGRAM
SUMMARY
Basics of Unity and 3D objects
Building an app using prefabs, components and script
Build lifecycle of a HoloLens application using Unity and Visual Studio
Pinned hologram robot in the room continue facing the HoloLens wearer
38. HOLOBASICS 38
SPATIAL SOUND
OVERALL
• Simulates 3D sound using direction,
distance and environmental
simulations
• Above, below, behind, to the side, etc.
• Attach sound to holographic objects
• Works also when object is not in line of
sight
• Used to draw attention
• Audio engine in HoloLens
• CPU and memory considerations
• 10-12 spatial sound voices
39. HOLOBASICS 39
SPATIAL SOUND
TYPES
• Gaze mixing
• Highlighting objects
• Play a sound on the object to get the users attention
• Audio Haptics
• Reactive audio for touchless interactions
• Play a sound when user selects an object or when his hands appear inside
the gesture frame
• Immersion
• Ambient sounds surrounding the user
• Support your scene with background sounds or music
40. HOLOBASICS 40
SPEECH
CORTANA
• Start with “Hey Cortana”
• Some thing you can say
• Move <app name> here
• Take a picture
• Start or stop recording
• Call <contact>
• How much battery do i have left?
• Restart or Shutdown
• Cortana can be turned on/off
42. HOLOBASICS 42
PINNED HOLOGRAM
SPATIAL SOUND
Unity and audio sources
Creating a spatial sound with an audio source
Explained how the speakers of HoloLens adjust based on the location of the sound
44. HOLOBASICS 44
GESTURES
GAZE
• The direction of the HoloLens pointing is called the
gaze
• Hits an object or part of the environment
• Uses a Vector3 based value to define its so called
“forward” direction
45. HOLOBASICS 45
GESTURES
GESTURE RECOGNIZER
• Recognize input by tracking the position of either or both hands
• Gesture frame (Frustum)
• Recognized input from hands
• Bloom
• Press, hold and release
• Following your hand
• Gestures
• Hold
• Manipulation
• Navigation
46. HOLOBASICS 46
GESTURES
GESTURE RECOGNIZER
• Connect commands to gestures
• GestureRecognizer gestures = new GestureRecognizer();
• gestures.TappedEvent += (source, tapCount, ray) => { … };
• gestures.StartCapturingGestures();
• Events
• Tapped
• Select press and release
• Hold (Started, Completed, Canceled)
• Select press beyond system hold threshold
• Navigation (Started, Updated, Completed, Canceled)
• Select press with relative movement
• Velocity based continuous scrolling or zooming
• Virtual Joystick
• Manipulation (Started, Updated, Completed, Canceled)
• Select press with absolute movement
• Move, resize or rotate hologram
47. HOLOBASICS 47
GESTURES
SPEECH RECOGNIZER
• Use words or sentences to control your environment
• English language only at the moment
• Connect commands to spoken text
• Dictionary<string, System.Action> keywords = new Dictionary<string,
System.Action>();
• keywords.Add(“some text”, () => { … };
• KeywordRecognizer keywords = new
KeywordRecognizer(keywords.Keys.ToArray());
• keywordRecognizer.OnPhraseRecognized += OnPhraseRecognized;
• keywordRecognizer.Start();
• OnPhraseRecognized( PhaseRecognizedEventArgs args);
• Invoke the action
• keywords[args.text].Invoke();
48. “DEMONSTRATION OF GESTURES BASED ON
HAND AND SPEECH TO INTERACT WITH
HOLOGRAM”
“HoloBasics by Alexander Meijers”
49. HOLOBASICS 49
PINNED HOLOGRAM
GESTURES WITH HAND AND SPEECH
Implemented code for handling the gaze and cursor
Implemented code for adding gestures
Implemented code for adding speech
Control the robot by shooting at it with a click gesture
Control the scene by using words and sentences like “reset”
51. HOLOBASICS 51
SPATIAL PERCEPTION
HOW DOES IT WORK?
• Spatial perception and mapping
• Rooms
• Different ways of detecting and working with spatial
perception
• Device scans your environment and builds a digital model in
real time
• It allows HoloLens to see different surfaces, walls and ceiling
• Meshes & planes
• Possible to simulate a physical space
• E.g. Projecting a terrain over your floor
Spatial perception is the
ability to be aware of
your relationships with
the environment around
you and with yourself
52. HOLOBASICS 52
SPATIAL MAPPING AND
UNDERSTANDING
Device scans your environment and builds a
digital model in real time
It allows HoloLens to see different surfaces, walls
and ceiling
Possible to simulate a physical space
E.g. Projecting a terrain over your floor
53. “DEMONSTRATION OF LIVE SPATIAL MAPPING
AND UNDERSTANDING BY HOLOLENS”
“HoloBasics by Alexander Meijers”
54. HOLOBASICS 54
PINNED HOLOGRAM
SPATIAL MAPPING UNDERSTANDING
Windows Device Portal connected to the HoloLens device
Mapping the room by walking around
Show mapping in the Windows Device Portal
55. HOLOBASICS 55
SPATIAL MAPPING AND
UNDERSTANDING
Spatial Mapping components in Unity
Spatial Mapping Renderer script
Spatial Mapping Collider script
Use an empty GameObject in your scene
Spatial Mapping render state
None
Visualization
Occlusion
Cursor on spatial mapping
SpatialMappingCollider.layer = 31;
Cursor uses Raycast with layer mask 1<< 31
Rendering material
Occlusion material
Visual material
57. HOLOBASICS 57
PINNED HOLOGRAM
SPATIAL MAPPING
Using the component SpatialMappingRenderer and SpatialMappingCollider in Unity
Show spatial mapping by using the HoloDeck material/shader
59. HOLOBASICS 59
OCCLUSION
CULLING
Process of determining which surfaces and parts are not visible from a certain viewpoint
Create a more naturally feeling with mixed reality
Holograms can be occluded by
Other holograms
Real-life objects in your environment which are spatially mapped
Culling enforces the HoloLens not to (partially) render the object
To speed up the rendering process
Objects which are too far away are left out the view (frustum)
Use Spatial Mapping Renderer
Set Occlusion material
Set SpatialMappingRenderer.renderState = RenderState.Occlusion;
60. “DEMONSTRATION OF SPATIAL MAPPING AND
OCCLUSION WITH HOLOGRAM FOR TRUE
MIXED REALITY”
“HoloBasics by Alexander Meijers”
61. HOLOBASICS 61
PINNED HOLOGRAM
SPATIAL MAPPING AND OCCLUSION
Using the component SpatialMappingRenderer and SpatialMappingCollider in Unity
Change the way spatial mapping is done for None, Visualization and Occlusion
Placed another robot behind a real-life object and explain occlusion
62. HOLOBASICS 62
WRAP-UP
TAKAWAYS
Unity and Visual Studio are the tools when it comes to developing mixed reality apps for HoloLens
Know your tools like the Windows Device Portal
Understand physics when creating an app for HoloLens
Lot’s of stuff to make your life easy for gestures, speech, spatial mapping and other
Understand the basics and how it works
Lot of stuff available on GitHub like the MixedReality Toolkit to make your life easier
63. THANKS FOR WATCHING!
HoloBasics code on gitHub - https://github.com/ameijers/HoloBasics
Kampenringweg 45b, Gouda
0182-686 000
a.meijers@ettu.nl / alexander@appzinside.com
Contact me:
facebook.com/alexandermeijers.5
@ameijers
Follow me on: