HoloLens Programming Tutorial: AirTap & Spatial Mapping

1. AirTap & Spatial Mapping
HoloLens Programming Tutorial:

2. Development Tools
Development environment：Unity
 Game engine and development environment for multi-platform
 You can visually place CGs and add functions by using GUI
 Interactive behavior can be described in C# or javascript
 HoloLens is supported from Unity 5.5
Development support tool:：HoloToolkit
 Toolkit to support the development of HoloLens compatible systems
 You can use AirTap and cursor behavior, spatial mapping easily
 Various sample programs are provided
 Also provides functions such as moving viewpoint on Unity editor
It is possible to develop without HoloLens or emulator
2

3. Goal of this tutorial
YouTube → https://youtu.be/_BmbL4yw8R0
3

4. First of all, the basic operation of Unity
4

5. Create a project
(1) Extract HoloToolkit-Unity-master.zip to any directory
(2) After starting Unity, click OPEN at the upper right of the screen
OPEN
5

6. Create a project
Open HoloToolkit-Unity-masterfolder and click [folder selection]
HoloToolkit-Unity-master
Select folder
HoloToolkit-Unity-master
6

7. Unity operation screen (overview)
Game space design area
Name list of objects
placed in space
List of Asset (CG, program, etc.) added to the project 7

8. Let's add CG to the game space
Right click
3D Object→Cube
8

9. Let's run
Click and run
Click to Exit
Space seen from
a camera
9

10. Adjust the position, orientation and size of the object
Click and select
move rotate zoom in / out
10

11. Let's change the perspective of Scene
[←][→] -Move left and right
[↑] [↓] -Zoom in/out
[Alt] + drag -Rotation
Other thingshttp://goo.gl/Lq1ILT
→ http://goo.gl/Lq1ILT
11

12. Designation of CG position/ size using numerical values
Click
Change Position to 0 0 1.2 Change all scale to 0.15
※ The unit of position and size is meter
Inspector(≒Detailed Information
12

13. Editing CG detailed information :color setting (1/3)
②Right click
③ Create
Make detailed material to set colour and texture
④ Material
①Assets
13

14. Editing CG detailed information: color setting (2/3)
Edit colour (Albedo) of the created material
(2) Click on the white area next
to Albedo and select a color
(1) Select created material
14

15. Editing CG detailed information: colour setting (3/3)
Open the material of the Cube to assign the created material
(2) Top of Materials▼
(1)Cube
(3) Drag and drop to
Element 0 review
15

16. Check
It is OK if the colour is reflected. You can change the colour later.
16

17. Setting the influence of gravity
Add Component
選択
RigidBody
17

18. Operation check and fine adjustment
Let's run it
When Cube looks smaller, change Position of MainCame to 0, 0, 0
18

19. Creation of ground
Click right
3D Object→Plane
19

20. Adjust the ground position
Plane
Change Position to 0 -0.8 0
20

21. Save Scene
Scene Name
 Open the dialog with [Ctrl] + [s] and save with your favourite name
 Frequently save during content creation
21

22. Development using HoloToolKit
22

23. Setting HoloLens Camera
Delete Main Camera
23

24. Adding HoloLens Camera
(1) Assets → HoloToolkit → Input → Prefabs
(2)Drag & Drop HoloLens Camera
24

25. Check
Light & Left by[←] [→]
Forward & Backword by[↑] [↓]
Up & down by[Q] [E]
Rotation by right click + mouse move
25

26. Supplement: Adjust movement / rotation sensitivity
(1) Select operation content
KeyboardXZTranslation
(2) Sensitivity Scale to 2
(3) Axis Type to Keyboard Arrows
Amount of movement when pushing keys :↑↓← → , will become less than the default
26

27. Spatial recognition using SpatialMapping
Remove Plane
27

28. Spatial recognition using SpatialMapping
Assets → HoloToolkit → SpatialMapping → Prefabs
Drag & drop SpatialMapping
28

29. Operation
Space is recognized by actual HoloLens, but not by PC
29

30. Importing a space model (official sample)
② Assets → HoloToolkit → SpatialMapping → Tests → Meshes
③ Drag & drop SRMesh into Room Model
① Spatial Mapping
30

31. Importing a space model (official sample)
(1)Run
The model of the room is displayed
More simplified one provided instead of more complexed one 31

32. A room for use
32

33. Import of space model (for study group)
Assets
→ Import New Asset
→ isit.obj
Import spatial model (isit.obj) prepared as sample
33

34. Import of space model
(2)SpatialMapping
(3) Drag & drop into Room Model
(1)Assets
34

35. Change appearance of mesh
Click button next to
Surface Material SpatialMappingWireframe
Before After
35

36. Create and drop Cube at the same time as AirTap
What to do next
36

37. Add function to manage input operations such as AirTap
(1)HoloToolkit → Input → Prefabs
(2)Drag & drop Input Manager
37

38. Check motion
Run
(1) Press [Shift] to use hand
(2) Left-click emulates AirTap
38

39. Create a script to receive AirTap
Right click on
blank of herarchy
Create Empty
Create an empty object to attach script
39

40. Create a script to receive AirTap
(1)GameObject
(2)Add Component
40

41. Create a script to receive AirTap
New Script
Scriot name
e.g.RcvAirTap
Create and Add
Added
41

42. Open script file by VisualStudio
Assets
Double click on
script (RcvAirTap)
42

43. Writing a script
using HoloToolkit.Unity.InputModule;
public class RcvAirTap : MonoBehaviour
{
// Start function is executed only once for initialization
void Start () {"R
obot Kyle");
}
// Update function is executed every frame
void Update () {
}
}
First, load InputModule and implement IInputClickHandler
, IInputClickHandler
43

44. Writing a script
public class RcvAirTap : MonoBehaviour , IInputClickHandler
{
// Start function is executed only once for initialization
void Start () {
}
// Update function to be executed every frame
void Update () {
}
}
•Move the cursor to llnputClickHandler,
Select lmplement interface from hint
44

45. Check the status quo
using HoloToolkit.Unity.InputModule;
public class RcvAirTap : MonoBehaviour , IInputClickHandler
{
//Called up function when AirTapped
public void OnInputClicked(InputEventData eventData)
{
throw new NotImplementedException();
}
void Start () {
}
void Update () {
}
}
throw new NotImplementedException();
Delete this line as unnecessary
45

46. What to do next
InputManager knows AirTap is done or not
Previous script is written here
Notify GameObject when InputManager recognizes AirTap
46

47. Receive AirTap
public class RcvAirTap : MonoBehaviour , IInputClickHandler
{
//Called up function when AirTapped
public void OnInputClicked(InputEventData eventData)
{
Debug.Log("AirTap!");
}
void Start () {
InputManager.Instance.
PushFallbackInputHandler(gameObject);
}
void Update () {
}
}
Register so that AirTap will
be notified to gameObject
with this script attached
47

48. Check motion
Run
AirTap! To be displayed
Console tab
48

49. Prefabrication of Edited Cube
(1) Assets
(2) Drag & drop Cube
49

50. What to do next
Cube prefab
Create a Cube (Prefab) in space for each AirTap
50

51. Generation of cube using script
public class RcvAirTap : MonoBehaviour , IInputClickHandler
{
//Variables to handle Cube prefab
public GameObject original;
public void OnInputClicked(InputEventData eventData)
{
// Instantiate cube using Cube prefab
GameObject cube = GameObject.Instantiate(original);
//Convert the position 1.2 m ahead from your
//viewpoint into the position in real space
cube.transform.position =
Camera.main.transform.TransformPoint(0, 0, 1.2f);
Debug.Log("AirTap!");
}
/*Omitted below*/
51

52. Assign Cube Prefab with variable of script
(1) Assets
(2) GameObject
Associate variable(=original) of script with Cube Prefab
Drag & drop Cub into Original
52

53. Check
Run
Generate cubes as many times as AirTapped
53

54. Limit the number of cubes
//container holding several cubes
List<GameObject> list = new List<GameObject>();
public void OnInputClicked(InputEventData eventData)
{
GameObject cube = GameObject.Instantiate(original);
cube.transform.position =
Camera.main.transform.TransformPoint(0, 0, 1.2f);
if (list.Count == 10)
{
Destroy(list[0]); //Delete the oldest cubes from space
list[0] = null;
list.RemoveAt(0); // Remove element in list 0
}
list.Add(cube);
}
54

55. Let's run on HoloLens: Confirm Quality
Edit
Project Settings
Quality
Fastest
55

56. Run on HoloLens: Build Setting
(1)Edit
(2)Project Settings
(3)Player
(4)Name ProductName
for yours
(5)Windows Store App
56

57. Run on HoloLens: Build Setting
Name Short
namefor yours
icon Other Settings
Virtual Reality
Supported
Publishing Settings
Name Package
Namefor yours
57

58. Run on HoloLens: Build Setting
File
Build Settings
58

59. Run on HoloLens: Build
Add OpenScenes
Windows Store
Switch Platform
・ SDK
Universal 10
・ Target device
HoloLens
・ UWP Build Type
D3D
Finally Build
59

60. Run on HoloLens: Create a folder for Build
Name App
New folder
Select folder
60

61. Project
yourname.sln
Appfolder
61

62. Run on a real machine: Deploy to HoloLens
x86
Click ▼
Remote
(In case of
Wifiapplication)
62

63. Run on a real machine: Deploy to HoloLens
In put HoloLens IP
Click Select
63

64. Run on a real machine: Deploy to HoloLens
Debug
Start Without
Debugging
64

65. Check motion
65

66. Hide the mesh
Spatial Mapping Smatial Mapping Manager
Click 〇
Select Spatial Mapping Occlusion for material
66

67. Increase the display range
HoloLens Camera
Change Nearto 0.3
Decreasing the value of the camera's Near, will work out
even in case of approaching to some extent
67

68. Import images pasted on Cube
[Assets]→
[Import New Asset]
→ Select the image you brought up
68

69. Material setting change
Click on the material (New Material) assigned to Cube
69

70. Material setting change
(1) Shader
(2) Unlit
(3) Texture
70

71. Paste texture
Drag and drop images
71

72. Completion
72

73. Any questions?
Please Contact Me!
Name：
Takashi Yoshinaga
Affiliation：
Institute of Systems, Information
Technologies and Nanotechnologies
Title：
(1) Researcher
(2) Microsoft MVP for Windows Development
taka.yoshinaga @tks_yoshinaga
YoshinagaTakashi

74. 74
I’m also waiting for your suggestions on English expressions :)