Présenté par Xavier Hallade le 24 mars lors du meetup Intel Developers à Paris. http://www.meetup.com/Intel-Technologies-Meetup/events/221027251/

1. First steps with the Intel® RealSense SDK™
Xavier Hallade, Developer Evangelist, Intel Corporation

2. 2
FRONT-FACING
Intel RealSense 3D Camera F200 Intel RealSense 3D Camera R200
Intel RealSense Snapshot R100
REAR-FACING
Intel® RealSense™ Cameras
Redefining how we interact with our devices

3. 3
Learning & Edutainment
Immersive Collaboration
3D Capture, Share & Recreate
Interact Naturally
Front Facing Depth Camera Usages & Experiences
Gaming & Play
Navigate content and perform simple
actions with robust and comfortable
gestures or voice
Enhance gaming experience
with natural language and
intuitive gestures, AR and
head tracking
Blend physical and digital worlds for a more immersive
and engaging learning & edutainment experience
Scan, share, edit, augment, or 3D print objects
and people
Mimic real-life interactions in immersive
virtual spaces and easily share or create
digital content
Intel Confidential

4. 4
Intel® RealSense™ F200 3D Camera
3D 2D
 Platforms : 4th and 5th Generation Intel® Core Processor based
Notebooks, 2 in 1 and All in Ones
 Usage : Indoors user facing ; 0.2 – 1.2m
 OS: Windows* 8.1
 Technology: Coded Light
Depth: VGA 60fps
RGB: 1080p 30fps
* Other Names and Brands may be claimed as the property of others

5. Using the Intel® RealSense SDK™ for Windows*
With C++, C#, JavaScript, and frameworks
*Other names and brands may be claimed as the property of others.

6. 6
SDK Installation Folder
C:Program Files (x86)IntelRSSDK
• bin: compiled samples
• contrib: metaio* toolbox
and nuance* language packs
• doc: the documentation
• framework: samples and wrappers for
C#/JavaScript/Unity/Java/processing
• props: VS2013 property sheets
• sample: C++ samples

7. 7
Hands
Face
Speech
Environment
7
Intel® RealSense™ SDK for Windows*
Understands 4 basic types of input - Standalone or various permutations
Categories of Input Capabilities Features
Hands • Hand and Finger Tracking
• Gesture Recognition
• 22-point Hand and Finger Tracking
• Static Poses and Dynamic Gestures
Face • Face Detection and
Tracking
• Multiple Face Detection and tracking
• 78-point Landmark Detection (facial features)
• Face Recognition and Facial Expressions
• Emotion Detection
• Pulse Estimator (new in R2)
Speech • Speech Recognition and
Synthesis
• Command and Control
• Dictation
• Text to Speech
Environment • Segmentation
• 3D Scanning
• Augmented Reality
• Background Segmentation
• 3D Object / Face / Room Scanning (new in R2
• 2D/3D Object Tracking
**Roadmap Notice: All products, computer systems, dates and figures specified are preliminary based on current expectations, and are subject to change without notice.
*Other names and brands may be claimed as the property of others.

8. 8
8
Intel® RealSense™ SDK Gold R1 for Windows*
Required Hardware A system with a minimum of a 4th generation Intel® Core™ processor including
an Intel® RealSense™ 3D camera (or a peripheral camera)
Required OS Microsoft Windows* 8.1 64-bit Desktop Mode
Supported Programming
Languages
C++, C#, Java, JavaScript
Supported IDE Microsoft* Visual Studio 2010-2013 with service pack 1 or newer
Supported Development
Tools
-Microsoft* .NET 4.0 Framework for C# development
-Unity* PRO 4.1.0 or later for Unity game development
-Processing* 2.1.2 or higher for Processing framework development
-Java* JDK 1.7.0_11 or higher for Java development
- Chromium/IE/Firefox in their latest version for HTML5 development
The Intel® RealSense™ SDK for Windows does not support Intel® RealSense™ Snapshot, support for that is coming soon.**
**Roadmap Notice: All products, computer systems, dates and figures specified are preliminary based on current expectations, and are subject to change without notice.
*Other names and brands may be claimed as the property of others.

9. 9
9
9
Web / HTML5
Run Intel® RealSense™ technology in a browser
HTML5 / JavaScript
SDK Local
Web-Socket Server
Intel® RealSense™
SDK runtime
Operating System
Web Socket Communication
127.0.0.1:4181
Internet Browser
Trigger face and gesture events, just like in C# applications
* Other names and brands may be claimed as the property of others
HTML5/JavaScript
w/ SDK WebSocket server
Unity* Web Player
Currently supported in R2:
Hand, Blob, Face Tracking, and
Speech Recognition.

10. 10
1010
Productivity extension for Unity* games
Add Intel® RealSense™ technology to your games
• A set of scripts that provides
configurable actions/rules based
on the capabilities provided in
the SDK.
• Programming: Associate your
game objects with the action
scripts.
• Programming: Create rules.
Associate your game objects
with rules.
* Other names and brands may be claimed as the property of others
 Drag and drop programming
 Write less code!

11. 11
11
Unity Toolkit Concepts
The toolkit consist of three main concepts
• Actions – MonoBehavior scripts. Actions define certain behavior on game objects.
• Triggers – Define a family of interactions that can be implemented in several ways such as
a Tracking Trigger. This trigger defines all interactions that track a certain object in the real
world such as a hand or face. Each interaction is implemented as a Rule. Thus each trigger
has it’s own set of rules.
• Rules – Are specific SDK code that implements certain Unity events, actions and data
types. You can use rules in one or multiple triggers. When a rule is part of a trigger, the
rule and it’s configuration parameters are shown in the Unity Editor’s UI under the specific
trigger(s).
When attaching more than one rule to a trigger, the rules are used in an OR relationship.
This means that if one of the rules fires, the trigger is triggered with the firing rule’s value.
The order of which they display in the Unity Editor’s UI is the order of priority.

12. Using the PXC[M]SenseManager

13. 14
14
Using the PXC[M]SenseManager
PXCSenseManager *psm = PXCSenseManager::CreateInstance();
psm->EnableXXX(); //XXX can be: 3DScan, 3DSeg, Emotion, Face, Hand, Stream, Tracker… not Speech
PXCXXXModule xxxAnalyzer = psm->QueryXXX();
//for Face and Hand:
PXCXXXData xxxData = xxxAnalyzer->CreateOutput();
PXCXXXConfiguration *xxxConfig = xxxAnalyzer->CreateActiveConfiguration();
xxxConfig->Query/Set/Disable/Enable/SubscribeYYY(…);
xxxConfig->ApplyChanges();
psm->Init();
//if using callbacks set with xxxConfig->SubscribeYYY()
psm->StreamFrames(/*blocking=*/true); //non-blocking will create a thread.
//else, in update loop:
psm->AcquireFrame(/*all=*/false,/*blocking=*/true);
xxxAnalyzer->QueryYYY(); // for Face and Hand: xxxData->Update(); xxxData->QueryYYY(); xxxIsAlertFired();
psm->ReleaseFrame();
//closing:
xxxConfig->Release(); psm->Release();

14. 15
Coordinate System
x
y
z
x
y
World coordinates Image coordinates

15. EnableFace()

16. 17
17
What does the “Face” modality offer?
• Best for sensing natural expressions, emotions and engagement
• SDK provides APIs for:
• Face Detection
• Head Orientation
• Expressions Tracking
• Landmark Tracking
• Emotion Recognition
• Pulse Estimation (new in R2)

17. 18
Face Detection
18
• Accurate 3D detection and tracking
• Up to 4 faces with marked
rectangles for face boundaries
• Developers can choose which 4
faces to detect
• Only one face has landmarks
• Works up to 1.2 meters

18. 19
Expression
19
EXPRESSION_HEAD_(TILT_LEFT|TURN_LEFT|UP|…)
EXPRESSION_(SMILE|KISS|MOUTH_OPEN|TONGUE_OUT)
EXPRESSION_EYES_(TURN_LEFT|UP|CLOSED_LEFT|…)
EXPRESSION_BROW_(RAISER_LEFT|LOWERER_LEFT|…)
PXCFaceData::Face *trackedFace = outputData->QueryFaceByIndex(i);
PXCFaceData::ExpressionsData *expressionData = trackedFace->QueryExpressions();
if(expressionData!=NULL)
{
if (expressionData->QueryExpression(PXCFaceData::ExpressionsData::EXPRESSION_MOUTH_OPEN,
&expressionResult)){
//use expressionResult.intensity
}
}

19. 20
Poses
20
• Detect how the user’s head is oriented
PXCFaceData::Face *trackedFace = outputData->QueryFaceByIndex(i);
PXCFaceData::PoseData *poseData = trackedFace->QueryPose();
PXCFaceData::PoseEulerAngles angles;
if(poseData!=NULL && poseData->QueryPoseAngles(&angles)){ // or QueryPoseQuaternion
//uses angles.yaw, angles.pitch, angles.roll
}

20. 21
Landmark tracking
21
• 3D tracking of 78 facial landmark
points
• Available in image and world
coordinates
• Supports avatar creation, emotion
recognition and facial animation
• Access to a subset of landmark
regions

21. 22
22
Retrieving Face Landmark Data (C#)
pxcU16 numOfFaces = outputData->QueryNumberOfDetectedFaces();
for (pxcU16 i = 0; i < numOfFaces; i++)
{
PXCFaceData::Face *trackedFace = outputData->QueryFaceByIndex(i);
PXCFaceData::LandmarksData* landmarkData = trackedFace->QueryLandmarks();
if (landmarkData != NULL)
{
pxcI32 numPoints = landmarkData->QueryNumPoints();
PXCFaceData::LandmarkPoint* landmarkPoints = new
PXCFaceData::LandmarkPoint[numPoints];
landmarkData->QueryPoints(landmarkPoints);
break;
}
}

22. EnableEmotion()

23. 24
Emotion Detection
24
• Supports 6 primary emotions – Anger, Disgust, Fear, Joy, Sadness, Surprise
• Support 3 sentiments: Postive, Neutral, Negative
• Returns intensity (0 to 1.0) and evidence (on a log scale)
int numFaces = ft.QueryNumFaces();
for (int i = 0; i < numFaces; i++) {
PXCMEmotion.EmotionData data;
if (ft.QueryEmotionData(i, PXCMEmotion.Emotion.EMOTION_PRIMARY_JOY, out data) >=
pxcmStatus.PXCM_STATUS_NO_ERROR) {
//use data.evidence and data.intensity
}
}

24. Demos

25. EnableHand()

26. 27
27
Hand Tracking Modes
TRACKING_MODE_FULL_HAND
VGA or HVGA (faster tracking)
• closest
• left most
• right most
• top most
• bottom most
• center
TRACKING_MODE_EXTREMITIES
Other information: palm orientation, hand openness, mass center…

27. 28
Retrieving closest extremity Data (C++)
PXCHandData::IHand* handData;
PXCHandData::ExtremityData extremityData;
pxcI32 numOfHands = outputData->QueryNumberOfHands();
for (pxcI32 i = 0; i < numOfHands; i++)
{
if (outputData->QueryHandData(PXCHandData::ACCESS_ORDER_BY_TIME, i, handData) ==
PXC_STATUS_NO_ERROR)
{
handData->QueryExtremityPoint(PXCHandData::ExtremityType::EXTREMITY_CLOSEST,
extremityData);
}
}

28. 29
29
Gestures
spreadfingers fist tap thumb_down thumb_up two_fingers_pinch_open
v_sign full_pinch swipe click (new in R2) wave
GestureData.state:
GESTURE_STATE_START
GESTURE_STATE_IN_PROGRESS
GESTURE_STATE_END
Retrieve:
IsGestureFired(name, GestureData)
OnFiredGesture(GestureData)
Enable:
PXCHandConfiguration::
EnableGesture(const pxcCHAR
*name, pxcBool continuous);

29. Demos

30. Enable<Other features>()

31. 32
Using other features from the SDK
• Use PXC[M]SenseManager the same way for 3d segmentation, scan, etc.
• PXCSpeechRecognition/Synthesis are used without PXCSenseManager
• Look at all the samples and the documentation !

32. Speech Modality

33. 34
Speech Modality
• Speech Recognition
• Grammar
• List
• JSGF
• Dictation
• Speech Synthesis

34. 35
Recognizing pre-defined words (C#)
PXCMAudioSource source = session.CreateAudioSource();
source.SetVolume(0.2f);
PXCMAudioSource.DeviceInfo dinfo;
source.QueryDeviceInfo(out dinfo);
source.SetDevice(dinfo);
PXCMSpeechRecognition sr;
session.CreateImpl<PXCMSpeechRecognition>(out sr);
string[] cmds = { "one", "two", "three", "stop" };
sr.BuildGrammarFromStringList(1, cmds, null);
sr.SetGrammar(1);
PXCMSpeechRecognition.Handler handler = new PXCMSpeechRecognition.Handler();
handler.onRecognition = OnRecognition;
sr.StartRec(source, handler);
while (run) {System.Threading.Thread.Sleep(500);}
sr.StopRec();
sr.Dispose();
source.Dispose();

35. 36
Get Started Now!
 Reserve Your Developer Kit now (includes peripheral camera, the
latest SDK, and other software updates)
Provided by Intel
 Free Intel® RealSense™ SDK for Windows, (Version 2014, Gold)
 Intel® RealSense™ Developer Kit camera available for purchase
 Huge opportunity to reach customers with integrated 3D camera
 Works with languages/frameworks/engines developers already use
 High-level APIs for NUI beginners. Low-level APIs for NUI experts
intel.com/realsense/sdk
Seize the Opportunity
36

36. 37
INFORMATION IN THIS DOCUMENT IS PROVIDED “AS IS”. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR
OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. INTEL ASSUMES NO
LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO THIS
INFORMATION INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE,
MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.
Software and workloads used in performance tests may have been optimized for performance only on Intel
microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer
systems, components, software, operations and functions. Any change to any of those factors may cause the results
to vary. You should consult other information and performance tests to assist you in fully evaluating your
contemplated purchases, including the performance of that product when combined with other products.
Copyright © 2014 , Intel Corporation. All rights reserved. Intel, the Intel logo, Xeon, Core, VTune, Cilk, and
RealSense are trademarks of Intel Corporation in the U.S. and other countries.
Optimization Notice
Intel’s compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel
microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the
availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent
optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are
reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the
specific instruction sets covered by this notice.
Notice revision #20110804
Legal Disclaimer & Optimization Notice
Copyright© 2014, Intel Corporation. All rights reserved.
*Other brands and names are the property of their respective owners.

37. Thank you

38. Backup

39. 40
Utility Classes
Smoother
 Online data smoothing algorithms
 See PXCSmoother (replacing PXCDataSmoothing)
Rotation
 Conversion between different representations
 See PXCRotation
Point Converter
 Convert from tracked coordinates to user coordinates
 Image  2D (e.g. screen)
 Camera  3D world (e.g. Unity game)
 See PXCPointConverter

40. 41
Blob Tracking
Track anything in front of the camera
No “hand” classification
Similar to Face/Hand modules
Up to 4 blobs
Different access orders