Programming with kinect v2

1. Matteo Valoriani

2. Nice to Meet You
Matteo Valoriani
PhD at Politecnico of Milano
CEO of Fifth Element
Speaker and Consultant
Microsoft MVP for Kinect
Intel Software Innovator: RealSense
email: mvaloriani@gmail.com
twitter: @MatteoValoriani
linkedin: https://it.linkedin.com/in/matteovaloriani
2

3. Usage Scenarios
TRAININGRETAIL HEALTHCARETHERAPY EDUCATION

4. You have to be a
magician…
… or at least a
good illusionist

5. “Any sufficiently
advanced
technology is
indistinguishable
from magic”
(Arthur C. Clarke)

6. Sensor Anatomy

7. Kinect 2 - Specs
3D DEPTH SENSOR
RGB CAMERA
MULTI-ARRAY MIC
Hardware:
Depth resolution:
512×424
RGB resolution:
1920×1080 (16:9)
FrameRate:
60 FPS
Latency:
60 ms

8. Kinect Adapter

9. Kinect 2 vs Kinect 1

10. Version 1 Version 2
Depth range 0.4m → 4.0m 0.4m → 4.5m
Color stream 640×480 1920×1080
Depth stream 320×240 512×424
Infrared stream None 512×424
Type of Light Light coding ToF
Audio stream 4-mic array 16 kHz 4-mic array 48 kHz
USB 2.0 3.0
# Bodies Traked 2 (+4) 6
# Joints 20 25
Hand Traking External tools Yes
Face Traking Yes Yes+Expressions
FOV 57° H 43° V 70° H 60° V
Tilt Motorized Manual

11. System / Software Requirements
OS Windows 8, 8.1, Embedded 8, Embedded 8.1 (x64)
CPU Intel Core i7 3.1GHz (or higher)
RAM 4GB (or more)
GPU DirectX 11 supported
USB USB 3.0
Compiler Visual Studio 2012, 2013 (Supported Express)
Language Native (C++), Managed (C#,VB.NET), WinRT (C#,HTML)
Other Unity Pro (Add-in), Cinder, openFrameworks (wrapper)

12. DEMO
Kinect Evolution

13. Accessing Kinect Data Sources

14. High-level architecture

15. Basic Flow of Programming
Sensor Stream Frame Data
Sensor Source Reader Frame Data
Kinect for Windows SDK v1
Kinect for Windows SDK v2
 Source independent to each Data
(e.g. ColorSource, DepthSource, InfraredSource, BodyIndexSource, BodySource, …)
 Doesn’t depend on each other Source
(e.g. Doesn't need to Depth Source when retrieve Body Data)

16. • In “New Project” create a new Windows Store app
• Enable Microphone and Webcam capabilities
• Add a reference to Microsoft.Kinect
• Use the Microsoft.Kinect namespace in your code
Creating a new store app using Kinect

17. Represents a single physical sensor
Always valid: when device is disconnected no more frame are
generated.
Use IsAviable Property to verify if the device is connected
The KinectSensor class
this KinectSensor
this
// Make the world a better place with Kinect
this

18. Sources

19. Give access to frames
– Events
– Polling
Multiple readers may be
created on a single source
Readers can be paused
Readers
InfraredFrameReader reader =
sensor.InfraredFrameSource.OpenReader();
reader.FrameArrived +=
InfraredReaderFrameArrived;
...

20. Frame references
void InfraredFrameReader
InfraredFrameArrivedEventArgs
using (InfraredFrame frame =
args.FrameReference.AcquireFrame())
{
if (frame != null)
{
// Get what you need from the frame
}
}
}
Sent in frame event args
AcquireFrame gives access to the actual frame

21. • Gives access to the frame data
– Make a local copy or access the underlying buffer directly
• Contains metadata for the frame
– e.g. Color: format, width, height, etc.
• Important: Minimize how long you hold onto the frame
– Not Disposing frames will cause you to not receive more frames
Frames

22. • Allows the app to get a matched set of frames from multiple
sources on a single event
• Delivers frames at the lowest FPS of the selected sources
MultiSourceFrameReader
MultiSourceFrameReader MultiReader =
Sensor.OpenMultiSourceFrameReader(FrameSourceTypes.Color |
FrameSourceTypes.BodyIndex |
FrameSourceTypes.Body);
var frame = args.FrameReference.AcquireFrame();
if (frame != null) {
using (colorFrame = frame.ColorFrameReference.AcquireFrame())
using (bodyFrame = frame.BodyFrameReference.AcquireFrame())
using (bodyIndexFrame = frame.BodyIndexFrameReference.AcquireFrame()){
//
}
}

23. System
Kinect
Driver
Kinect SDK
Application
Kinect
Driver
Kinect SDK
Application
Kinect
Service
Kinect SDK
Application
Kinect SDK
Application
Kinect for Windows v1 Kinect for Windows v2

24. Connection
Kinect for Windows v1 Kinect for Windows v2
PCPC

25. Example Multiple Connection
PCPC PC
Hub
Server

26. DEMO
Display frame in a Windows Store App

27. Data Source Details
Physical image sensors, ColorFrameSource,
InfraredFrameSource, DepthFrameSource,
BodyIndexFrameSource, BodyFrameSource, AudioFrameSource

28. Kinect Data Sources

29. • 30 or 15 fps, based on lighting
conditions
ColorFrameSource

30. InfraredFrameSource

31. – 16-bit distance in millimeters from the
sensor’s focal plane
DepthFrameSource

32. – 0 to 5: Index of the corresponding body,
as tracked by the body source
– > 5: No tracked body at that pixel
BodyIndexFrameSource
255
0 1

33. • Range is 0.5-4.5 meters
• Frame data is a collection of Body objects each
with 25 joints
– Each joint has position in 3D space and an
orientation
• Up to six simultaneous bodies
• 30fps
• Hand State on 2 bodies
• Lean
BodyFrameSource

34. Skeleton vs Body

35. Coordinate System
 ColorSpace (Coordinate System of the Color
Image)
– … Color
 DepthSpace (Coordinate System of the
Depth Data)
– … Depth, Infrared, BodyIndex
 CameraSpace (Coordinate System with the
origin located the Depth Sensor)
– … Body (Joint)

36. • Three coordinate systems
• Coordinate mapper provides conversions between each system
• Convert single or multiple points
Coordinate mapping
Name Applies to Dimensions Units Range Origin
ColorSpacePoint Color 2 pixels 1920x1080 Top left corner
DepthSpacePoint Depth,
Infrared,
Body index
2 pixels 512x424 Top left corner
CameraSpacePoint Body 3 meters – Infrared/depth
camera

37. DEMO
Handle body frames and coordinate mapping

38. Kinect Studio
Recording and Playback

39. Recordable Data Sources
Infrared
13 MB/s
Depth
13 MB/s
BodyFrame
BodyIndex
Color
120 MB/s
Audio
32 KB/s
Legend
Record/Play
Record Only

40. DEMO
Kinect Studio 2.0
Recording | Playback

41. Gesture Recognition
Heuristic + Machine Learning Approaches

42. DEMO
Programmatic detection of a gesture/pose

43. Gesture Builder
• New tool, shipping with v2 SDK
• Organize data using projects and solutions
• Give meaning to data by tagging gestures
• Build gestures using machine learning technology
– Adaptive Boosting (AdaBoost) Trigger
• Determines if player is performing gesture
– Random Forest Regression (RFR) Progress
• Determines the progress of the gesture performed by player
• Analyze / test the results of gesture detection
• Live preview of results

44. Visual Gesture Builder
Your Application

45. DEMO
Machine Learning Demo

46. Heuristic
• Gesture is a coding problem
• Quick to do simple
gestures/poses (hand over head)
• ML can also be useful to find
good signals for Heuristic
approach
Machine Learning (ML) with G.B.
• Gesture is a data problem
• Signals which may not be easily
human understandable (progress
in a baseball swing)
• Large investment for production
• Danger of over-fitting, causes you
to be too specific – eliminating
recognition of generic cases
Gesture Recognition

47. Hand Pointer Gestures

48. Hand pointer gestures
Engagement Targeting
Press
Panning/Zoom

49. DEMO
Controls Basics – XAML
UserViewer | Engagement | Targeting | Press

50. Demo
Building ControlsBasics-XAML from Scratch
• File New Project > Grid App
• Add Reference to Microsoft.Kinect.Xaml.Controls
• Enable Microphone + Camera capabilities for app
• Add KinectRegion as container for rootFrame in
App.xaml.cs
• Run it!

51. Window
Frame
Page
Window
Frame
Page
Grid
KinectRegion
Kinect
UserViewer
Adding Hand Cursor Support

52. System Engagement Manual Engagement
• Hands over head
• In certain location of room
• Etc…
“Engagement” for hand cursor
KinectRegion.SetOnePersonSystemEngagement()
KinectRegion.SetTwoPersonSystemEngagement()
KinectRegion.SetOnePersonManualEngagement(…)
KinectRegion.SetTwoPersonManualEngagement(…)

53. DEMO
PHiZ (Physical Interaction Zone) & KinectPointerPoint

54. PHIZ – Physical Interaction Zone
X / Y - For Cursor Rendering
KinectPointerPoint->Position->X
– Between 0.0 and 1.0
KinectPointerPoint->Position->Y
– Between 0.0 and 1.0
X / Y – Allows offscreen info
KinectPointerPoint->Properties->UnclampedPosition->X
KinectPointerPoint->Properties->UnclampedPosition->Y
Z
KinectPointerPoint->Properties->HandReachExtent (raw)
KinectPointerPoint->Properties->PressExtent (affected by user/UI)
Correlating PointerPoint to NUI Data
KinectPointerPoint->Properties->BodyTrackingId
KinectPointerPoint->Properties->HandType
KinectPointerPoint->Properties->BodyTimeCounter

55. Press
Recommended Minimum Size:
• 208 x 208 (in 1080p resolution)
• Press attraction towards center
• Larger buttons will just
attract away from the edge
Adapting to smaller visual sizes:
• Make large-size hit testable
• Set KinectRegion.KinectPressInset
(Thickness) to non-visible part of button
• Attraction region is smaller

56. Cursors

57. Demo
Controls Basics – XAML
Scroll
Zoom

58. Interacting with Controls (in Preview release)
• UI Frameworks
– XAML (Store)
– WPF
• DirectX (Store)
ControlsBasics-
XAML
Microsoft.Kinect.
Xaml.Controls
Microsoft.Kinect.Toolkit.Input
WindowsPreview.Kinect
ControlsBasics-
WPF
Microsoft.Kinect.
Wpf.Controls
Microsoft.Kinect
ControlsBasics-
DX
KinectCoreWindow.PointerMoved,
KinectPointerPoint, KinectGestureRecognizer
InputPointerManager, PressableModel,
ManipulatableModel
Controls Layer – Cursor rendering, Hit Test,
Route PointerPoint via InputPointerManager
App (host KinectRegion, etc…)

59. Interacting with Content
• Navigating a camera through a 3d space
• Rotating a 3d object
• “Scrubbing” a video

60. Kinect Resources
• General Info & Blog -> http://kinectforwindows.com
• Purchase Sensor -> http://aka.ms/k4wv2purchase
• v2 Preview SDK -> http://aka.ms/k4wv2previewsdk
• Developer Forums -> http://aka.ms/k4wv2forum

61. Q&A
Tutto il nateriale di questa sessione sul sito
http://www.dotnetcampus.it
@MatteoValoriani

62. ©2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, Office, Azure, System Center, Dynamics and other product names are or may be registered trademarks and/or trademarks in the
U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft
must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after
the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.