Project Prague e’ un SDK disponibile tra i progetti di Cognitive Services Labs di Microsoft che consente agli sviluppatori e ai progettisti di UX di incorporare controlli basati sui gesti nelle proprie app. Permette di definire gesture in maniera dichiarativa impostando dei vincoli in un linguaggio semplice e consentendo, grazie anche alle camere RealSense, di creare un’esperienza utente più naturale. Sessione tenuta durante IoTDay Catania del 11/12/2017

1. #IoTDayAQ17

2. Project Prague & RealSense:
il potere nelle mani!!
Massimo Bonanni
Paranormal Developer, with the head in the Cloud and all the REST in
microservices!
massimo.bonanni@microsoft.com
@massimobonanni

3. Gesture scenarios.....

4. “… the idea of gesture input
will become mainstream –
both when working far
away from the screen
(TV, big meeting room)
and when working at your
desk (PC).”
Microsoft’s Focused Dreams to Survive and Thrive,
Bill Gates, January 2016

6. What is Project Prague?
Is an easy-to-use SDK that helps
developers to incorporate gesture-
based controls into their apps.
Use a machine learning algorithm
called DFE (Discriminative Ferns
Ensemble) to obtain the hand
skeleton.
The gesture detection is implemented
using a straight forward heuristic, with
very few manually tuned parameters.

7. what do I need?
Intel® RealSense™ SR300
https://aka.ms/IntelRS300
Intel® RealSense™ F200

8. Requirements
Category Recommended Minimal
CPU Intel® Core™ i7 series, 8 logical cores Intel® Core™ i5 series, 4 logical cores
Free RAM 2GB or more 1GB
Operating System Windows 10 with Creator Update installed Windows 10
Brand Gesture Detection Range Experience
Intel® RealSense™ SR300 camera 20-60 [cm] Best
Intel® RealSense™ F200 camera 20-60 [cm] Good
Kinect for Windows v2 60-110 [cm] Good
http://aka.ms/gestures/download

9. Hand pose, Hand Motion & Gestures

10. Palm Pose
Finger Poses
Fingertip Placement
Relations
+
+
Fingertip Distance
Relations+
HandPose
=
Direction: {left, right, up, down,
backward, forward}
Flexion: {opened, openedStretched,
folded, foldedTucked}
RelativePlacement: {left, right,
above, below,
behind, inFornt}
RelativeDistance: {notTouching, touching}
Direction: {left, right, up, down,
backward, forward}
Orientation: {left, right, up, down,
backward, forward}
Hand Pose
A hand pose refers to a snapshot of the hand at a given moment and contains
a complete description of the state of the palm and the fingers in that
snapshot

11. palm facing {forward}, orientation {up}
{index, thumb} are {touching}
{middle, ring, pinky} are {above}
{index, thumb}
{index} is {above} the {thumb}
{left} hand
{thumb, index, middle, ring, pinky}
are {opened}
Pinch pose
Hand Pose

12. <svg>
<path d="M 100 350 q 50 -50 100 0
q 50 -50 100 0 q 50 -50 100"/>
</svg>
HandMotion
= MotionScript
Upward,
Downward,
Right,Left,
ClockwiseArcRightUpward, ClockwiseArcRightDownward,
ClockwiseArcLeftUpward, ClockwiseArcLeftDownward,
CounterClockwiseArcLeftDownward, CounterClockwiseArcLeftUpward,
CounterClockwiseArcRightDownward, CounterClockwiseArcRightUpward,
For example, the SVG curve above can be simply described by:
ClockwiseArcRightUpward,
ClockwiseArcRightDownward,
ClockwiseArcRightUpward,
ClockwiseArcRightDownward,
ClockwiseArcRightUpward,
ClockwiseArcRightDownward,
DiagonalLeftUpward,
DiagonalRightDownward,
DiagonalRightUpward,
DiagonalLeftDownward,
Hand Motion
As you move your hand, your fingertips trace a curve through space: this
curve is a "hand motion"

13. A Gesture is a finite state machine that its states are either Pose objects,
Motion objects or other Gesture object
Basic templates:
A gesture should be Un/Registered with respect to the application context.
Only registered gestures trigger ( ).
Pose Motion
Pose
Pose Pose
Gesture =
Gesture
Gesture
Pose
Motion
Gesture

14. hand facing {forward}, orientation {up}
{index, thumb} are {touching}
{index} is {above} {thumb}
{left, right} hand
{thumb, index} are {opened}
Pinch Retract NotPinchingSlingshot =
{index, thumb} are {opened}
{left, right} hand
{index, thumb} are {notTouching}
{index, thumb} are {opened}
{left, right} hand
{index, thumb} are {notTouching} move {backward}
{left, right} hand
NotPinching
Slingshot Example

15. Developers, developers, developers….

16. Gestures Service
The Gestures Service provides
gesture detection as a local
service for client applications.
After you install Project Prague,
the Gestures Service will be
launched every time your machine
starts.

17. Demo
Project Prague SDK and Tools

18. Using Gesture Service
The detection is performed in the Microsoft.Gestures.Service.exe
process that acts as a service for gesture detection.
You have to use the GesturesServiceEndpoint class in order to communicate
with the service.
Reference the following assemblies:
• Microsoft.Gestures.dll
• Microsoft.Gestures.Endpoint.dll
• Microsoft.Gestures.Protocol.dll

19. Create a gesture

20. Detect a gesture
You need to hook up the event indicating gesture detection to the appropriate
handler:
You can detect when the single pose is triggered by the user:

21. Declare a Gesture as XAML

22. Skeleton tracking
Project Prague also provides API for the skeleton tracking.
For each hand:
Position of the fingertips
Fingers direction
Palm position
Palm orientation

23. V1 Released July ‘17
Runtime
• Supported sensors: Intel RealSense SR300 & F200, Kinect v2
• Gestures support injected into
• Windows Shell, PowerPoint, Skype, YouTube and Photos App
• Wolfly Unity Game
• Camera Gesture-triggered overlays (can be used in skype)
Docs
• Great Overview + tutorials
Samples @Github
• .NET
• Unity

24. Demo
Using Project Prague SDK

25. Cognitive Service Labs
https://labs.cognitive.microsoft.com/
Trials have both rate
limit, in terms of
transactions per
second or minute, and
a monthly usage cap.

26. Key takeaways
Human Interaction is the new challenge.
Project Prague may be one of the solution.
In the roadmap will be different kind of camera (e.g. IR
camera)
Keep in mind: Project Prague is an experimental project.
If we’ll use it, it will become a supported project.

27. Feedback & Questions
Grazie!!!!!
Massimo Bonanni
massimo.bonanni@microsoft.com
@massimobonanni
https://www.linkedin.com/in/massimobonanni/
#ProjectPragueCT

28. Project Prague Reference
Project Prague Web Site
https://labs.cognitive.microsoft.com/en-us/project-prague
Project Prague documentation
https://docs.microsoft.com/en-us/gestures/
Twitter/mail account
@ProjectPrague / gestures@Microsoft.com
Project Prague GitHub
https://github.com/Microsoft/Gestures-Samples
Toward Realistic Hands Gesture Interface: Keeping it Simple for Developers and Machines [PDF]
https://aka.ms/TRHGI
Sample GitHub
https://github.com/massimobonanni/ProjectPragueSamples