2. NICE TO MEET YOU
www.slideshare.net/MatteoValoriani
https://it.linkedin.com/in/matteovaloriani
http://fifthingenium.com/blog
https://github.com/mvaloriani
mvaloriani at gmail dot com
@MatteoValoriani
Matteo Valoriani
CEO of FifthIngenium
PhD at Politecnico of Milano
Speaker and Consultant
3. How develop HoloGraphic App
• Device
• Setup and SDK
• My first App
HoloToolKit Unity
Input: Gaze / Tap / Voice
Q&A?
AGENDA
14. Cross Platform
HoloLens Support
WYSIWYG Editor
Quick Interaction Times
Powerful Asset Pipeline
Easy to lean API’s
Programming Language: C#
Wrapper and dlls support: C++
UNITY 3D
16. Walk forward, back, left, and right - Use the W,A,S, and D keys on your keyboard, or the left stick on an Xbox
controller.
Look up, down, left, and right - Click and drag the mouse, use the arrow keys on your keyboard, or the right
stick on an Xbox controller.
Air tap gesture - Right-click the mouse, press the Enter key on your keyboard, or use the A button on an Xbox
controller.
Bloom gesture - Press the Windows key or F2 key on your keyboard, or press the B button on an Xbox
controller.
Hand movement for scrolling - Hold the Alt key, hold the right mouse button, and drag the mouse up / down,
or in an Xbox controller hold the right trigger and A button down and move the right stick up and down.
HOLOLENS EMULATOR INPUT
18. Download package from:
https://github.com/Microsoft/Holo
Toolkit-Unity/releases
HOW TO USE HOLOTOOLKIT
Clone GitHub rep:
https://github.com/Microsoft/Holo
Toolkit-Unity.git
Open the folder you just cloned in
Unity.
Assets -> Export Package…
1. Open or create your project in Unity.
2. Import the package: Assets -> Import Package -> Custom Package
3. Create a new Scene: File -> New Scene
4. Remove the default objects in the scene and add
the HoloLensCamera.prefab (found under HoloToolkit/Input/Prefabs).
20. Raycast determines point of attention.
Intersect gaze with Holograms
Gesture based on gaze
Intersect with spatial map
Visual & Audio queues
Manage Cursor
WHAT WE CAN DO WITH GAZE
21. Cursor Indicates the Gaze Direction.
Cursor can hug surfaces it intersects.
Non-directional cursors often work best.
Arrow can lead user to action.
Cursor can be temporarily hidden.
GAZE CURSOR
23. INPUT MANAGER HOLOTOOLKIT
5. Add the InputManager.prefab (found under HoloToolkit/Input/Prefabs) as
a child to your new 'Managers' Object.
6. Add an Event System to your scene by right click on 'Managers' object in
your scene Hierarchy: UI -> Event System.
7. Building Your Project for HoloLens: HoloToolkit -> Build Window -> Build
Visual Studio SLN
8. Open SLN and Deploy to the emulator or device.
24. HoloLens apps built in Unity now use a global input manager
Game objects that want to consume input events can implement one or
many input interfaces, such as:
• IFocusable for focus enter and exit. The focus can be triggered by the user's gaze or any
other gaze source.
• IHoldHandle for the Windows hold gesture.
• IInputHandler for source up and down. The source can be a hand that tapped, a clicker
that was pressed, etc.
• IInputClickHandler for source clicked. The source can be a hand that tapped, a clicker
that was pressed, etc.
• IManipulationHandler for the Windows manipulation gesture.
• INavigationnHandler for the Windows navigation gesture.
• ISourceStateHandler for the source detected and source lost events.
• ISpeechHandler for voice commands.
EXTENSIBLE INPUT SYSTEM HOLOLENS
26. GAZE & HOLOTOOLKIT
1. Navigate to HoloToolkit –> Input and drag
the GazeManager.cs script on to Mangers object in the
object Hierarchy.
2. Navigate to HoloToolkit –> Prefabs –> Input, and add the cursor
into the object hierarchy
CursorManager class takes Cursor GameObjects
and shows the appropriate Cursor when a
Hologram is hit. This hit is calculated by
the GazeManager Raycast.
31. Programs that run on the GPU
Written in C like language (CG, HLSL, GLSL)
Used for lighting, effects, computation, etc
SHADERS
DX11 Rendering Pipeline
38. TIR PRISM COMBINERS
Prisms and beam splitters: Prisms are crystals
that bend and redirect light. Beam splitters use
similar technology to split light and send it in
two directions simultaneously. The reflected
light is projected directly into the user’s retina.
Google Glass uses a prism to redirect the image
into the eye.
Mirrors: The basis for many optical
instruments, mirrors can be used to redirect
and focus light. Depending on how they are
designed and manufactured, they can transmit
light from one direction and reflect light from
another. Osterhout Design Group (ODG) uses a
mirror with a special coating in its R-7
smartglasses.
39. GENERAL CONCEPT OF WAVE GUIDE
Waveguides: These devices channel light along a path as in
an optical fiber, and they are used widely in
telecommunications and electronics. In smartglasses,
waveguides direct light from tiny displays housed in the
temples of the glasses toward the lenses in front of the eye.
Vuzix was the first to use waveguides in 2013.
41. DIFFRACTIVE EXTRACTION - EXIT PUPIL EXPANSION
Nanometer wide structures or gratings are placed on the
surface of the waveguide at the location where we want to
extract an image. The grating effectively creates an
interference pattern that diffracts the light out and even
enlarges the image. This is known as SRG or surface relief
grating.
EPE literally means making an image bigger (expanding it) so
it covers as much of the exit pupil as possible, which means
your eye plus every area your pupil might go to as you rotate
your eyeball to take in your field of view (about a 10mm x
8mm rectangle or eye box).
Original work by Nokia on 1D EPE waveguide grating conbiners (1995)
43. LIGHT FIELD
This term is defined as the amount of light flowing in every direction through
every point in space.4 It is emerging as an alternative method for displaying 3-D
objects that appear more realistic than those created by providing different left
and right images in a stereoscopic display.
Magic Leap states it is using light field technology in its smartglasses.
46. TSMC-fabricated 28 nm co-processor.
24 Tensilica DSP cores (12 clusters)
65 million logic gates (used 50%)
8 MB of SRAM
1GB DRAM
1 Trillion Operation per second
Sensor aggregator with gesture and environment processing
200x over software implementation
Low Power (<10 Watts)
HOLOGRAPHIC PROCESSING UNIT V 1.0
12 mm
12mm
50. COMMERCIAL OPTIONS
Kiosk mode. With HoloLens kiosk mode, you can limit which apps to run to
enable demo or showcase experiences.
Mobile Device Management (MDM) for HoloLens. Your IT department can
manage multiple HoloLens devices simultaneously using solutions like
Microsoft InTune. You will be able to manage settings, select apps to install
and set security configurations tailored to your organization's need.
Identity. Azure Active Directory and next generation credentials with
PIN unlock.
Windows Update for Business. Controlled operating system updates to
devices and support for long term servicing branch.
Data security. BitLocker data encryption and secure boot is enabled on
HoloLens to provide the same level of security protection as any other
Windows device.
Work access. Anyone in your organization can remotely connect to the
corporate network through a virtual private network on a HoloLens.
HoloLens can also access Wi-Fi networks that require credentials.
Windows Store for Business. Your IT department can also set up an
enterprise private store, containing only your company’s apps for your
specific HoloLens usage. Securely distribute your enterprise software to
selected group of enterprise users
