Designing Apps for Intel RealSense Technology

2
Designing Apps for Intel® RealSense™
Technology: User Experience Guidelines
with Examples for Windows®
Kevin Arthur – Senior User Experience Researcher, Intel Corporation
Meghana Rao – Developer Evangelist, Intel Corporation
SFTS018

3
Agenda
• F200 Short Range Camera
• F200 UX Guidelines for Laptop and
All-In-One Experiences
• F200 SDK tips
• R200 Long Range Camera
• R200 UX Guidelines for Tablet and
Two-In-One Experiences
• R200 SDK tips
• Summary and Q&A
F200
R200

4
Agenda
• F200 SDK tips
• R200 SDK tips
• Summary and Q&A
F200
R200

5
F200 Short Range Camera
• Integrated into Laptop
and All-In-One Devices
• Front, user-facing
configuration
F200

6
F200 Features
• Features available with the
Intel® RealSense™ SDK
- Hand tracking and gestures
- Face recognition and animation
- Background replacement
- 3D scanning
- Speech
F200

7
Agenda
• F200 SDK tips
• R200 SDK tips
• Summary and Q&A
F200
R200

8
UX Guidelines Learned from F200 Apps
• Dozens of rounds of 1-on-1
usability testing with
prototypes
• Co-design with select
software vendors
Selected apps from the Intel® RealSense™ App Showcase
https://appshowcase.intel.com/realsense/
F200

9
What the Camera Sees:
Know the Effective Range and Capture Volume
Effective Range for Gestures 0.2 - 0.6 m
Effective Range for Face Tracking 0.35 - 1.2 m
RGB Resolution Up to 1080p at 30 fps
Depth Resolution
Up to 640x480 at 60
fps
F200

10
Provide Visual Feedback About Range
Distance Prompts
World Diagrams
F200

11
Provide Continuous Feedback When Needed
User Viewport
Move It! by Umajin
F200

12
Provide Continuous Feedback When Needed
User Overlay
Kagura by Shikumi Design
F200

13
User Input: Choose the Right Technology
Hand Gestures
• Fun and engaging for
short experiences
• Natural and
unencumbered
Mouse
• Precise, fast, and efficient
• Comfortable for long-term
use
Understand the strengths of input technologies. Use gestures to enhance, not replace.
Touch
• Simple and direct
F200

14
Follow Gesture Best Practices to Minimize Fatigue
• Design for left-right or arc gesture motion
• Avoid actions that require your users to lift their
hand above the height of their shoulder
• Design for breaks, and break up activities into small,
short actions
• Do not require many repeating gestures
• Do not require precise input, like entering text or
selecting small items
• Continuous and low-risk actions are best for
gesture
F200

15
Agenda
• F200 SDK tips
• R200 SDK tips
• Summary and Q&A
F200
R200

16
APIs To Detect Capture Volume
• Developers should use device neutral functions to obtain Field of View (FOV)
frustrum information to help guide users through visual feedback
• Device interface – QueryColorFieldofView and QueryDepthFieldofView
functions allow you to obtain the Depth and Color capture volumes
Code:
F200

17
APIs To Detect Capture Volume
Other Device interface functions of interest:
• PXCMCapture.Device. QueryDepthSensorRange – returns depth sensor range in mm
• PXCMCapture.Device. QueryColorFocalLengthMM – returns color sensor focal length in mm
• PXCMCapture.Device. QueryDepthFocalLengthMM – returns depth sensor focal length in mm
F200

18
Alerts to Indicate Face Out of the Field of View
• PXCMFaceData.AlertData.AlertType
• AlertType enumerator itemizes alert events
Alert Type Description
ALERT_NEW_FACE_DETECTED A new face is detected
ALERT_FACE_NOT_DETECTED There is no face in the scene.
ALERT_FACE_OUT_OF_FOV The face is out of camera field of
view.
ALERT_FACE_BACK_TO_FOV The face is back to field of view.
ALERT_FACE_LOST Face tracking is lost
F200

19
Coding Face Alerts in Your App
F200

20
Contd…Alert Handling
}
}
F200

21
Alerts to Indicate Hands Out of FOV
Use PXCMHandData.TrackingStatusType
Alert Name Description
ALERT_HAND_OUT_OF_BORDERS A tracked hand is outside of a 2D bounding box or 3D
bounding cube defined by the user
ALERT_HAND_INSIDE_BORDERS A tracked hand has moved back inside the 2D bounding box
or 3D bounding cube defined by the user.
ALERT_HAND_TOO_FAR A tracked hand is too far to the camera
ALERT_HAND_TOO_CLOSE A tracked Hand is too close to the camera.
ALERT_HAND_DETECTED A tracked hand is identified and its mask is available
ALERT_HAND_NOT_DETECTED A previously detected hand is lost, either because it left the
field of view or because it is occluded.
And more… Refer to the documentation
F200

22
Coding the Hand Alerts in Your App
F200

23
Contd…Alert Handling
F200

24
World Space to Screen Space Mapping –
Projection Interface
• The Projection interface provides functions to map or project among color,
depth and world coordinates
- MapColorToDepth
- MapDepthToColor
- ProjectCameraToColor
- ProjectCameraToDepth
- ProjectColorToCamera
- ProjectDepthToCamera
F200

25
Adjusting UI to Different Screen Resolutions – Unity
To make the App go Full-screen in windowed/unwindowed mode, whatever the
resolution of the display:
• Pre-compilation, in player’s settings, make sure the "Use Direct3D 11" box is
checked
• Set it to use ONLY specific aspect ratios (16:9 and 16:10 and matching children)
Result - when the ‘resolution selection screen’ appears:
• If the “Windowed” check box is UNchecked, it will always start at Full Screen
• If it’s checked, it will start windowed but can be resized/maximized
F200

26
APIs to Detect Speed and Precision
• Joint Speed – Enabling joint speed allows developers to obtain the
absolute/average speed of motion of each joint
- PXCMHandConfiguration.EnableJointSpeed
- Disabled by default to preserve CPU and memory resources. Only enable as necessary.
• Hand jitters can hamper tracking. Use PXC[M]Smoother to reduce jitter
F200

27
APIs to Detect Speed and Precision
• Use PXCMHandData.TrackingStatusType to obtain tracking status for speed
- TRACKING_STATUS_HIGH_SPEED
• If using hand gestures to control the OS using Touchless Controller, use
the PXC[M]TouchlessController member functions to set pointer and scroll
sensitivity:
- SetPointerSensitivity
- SetScrollSensitivity
• For face detection, fast movements may lead to lost tracking. Use
PXCMFaceData.AlertData.AlertType – ALERT_FACE_LOST to determine if
tracking is lost
F200

28
Bounding Boxes
• Provide clear visual cues to the user on the source and destination areas
for object of interaction
• Bounding boxes provide a rough boundary for user action thus
alleviating the need for high precision
• Hand bounding boxes:
- PXCMHandData.IHand.QueryBoundingBoxImage
- Use PXCMHandData.AlertType – ALERT_HAND_OUT_OF_BORDERS to detect if
hand is out of the bounding box
• Face bounding boxes:
- PXCMFaceData.DetectionData.QueryBoundingRect
F200

29
Agenda
• F200 SDK tips
• R200 SDK tips
• Summary and Q&A
F200
R200

30
R200 Long Range Camera
• Integrated into Tablets
and Two-In-One
Detachable Devices
• Rear, world-facing
configuration
R200

31
R200 Features
• Features available with the
Intel® RealSense™ SDK
- 3D scanning
- Depth-enabled photo and video
- Measurement
- Gaming and play with mixed and
augmented reality
 Scene perception module
 Camera tracking and localization
 Mesh reconstruction
- Face detection and tracking
- Speech (Windows® SDK only)
R200

32
Video – ToyZ Game Prototype
• Shows real-time scene
perception for collision
and occlusion (no pre-
scan)
• By Shachar Oz, Omek
Studio at Intel
R200

33
Video – Procedural Island Prototype
• Shows scan as part of
“capture and play”
• Illustrates procedural
shaders and set
dressing
• By Eddy Ortega, Garrett
Stevens, Perceptual
Computing at Intel
R200

34
Agenda
• F200 SDK tips
• R200 SDK tips
• Summary and Q&A
F200
R200

35
R200 Range and Capture Volume
R200
Effective
Range
0.6 - 3.5 m
indoors
RGB
Resolution
Up to 1080p
at 30 fps
Depth
Resolution
Up to 640x480
at 60 fps

36 36
3D Scanning: Plan for the Scene
Small object capture: Instruct users
to place object on flat surface and
move around the object to capture it
from side and top views
Person capture: The subject must remain
still while the user walks around them and
captures side views
Scene capture: User will scan the tablet
around the room as if taking a panoramic
photo
Focus 3D scanning applications on one of these 3 scenarios, as the interaction
flow will differ according to the size and surfaces being captured
R200

37
User Prompts
• Give simple instructions to start
• Preview the subject or area that will be captured
R200

38
User Prompts
• Tell user if tracking is lost
• Give distance cues
R200

39
Augmented Reality:
Give People a Reason to Move, or They Won’t
• Tablet as window into a virtual
space vs. tablet as fixed screen
• Address with motion hints –
explicitly with instructions or
implicitly with content
Google Spotlight Stories “Windy Day”
R200

40
Minimize Fatigue from Holding the Device
Active Camera Mode: Tiring Inactive Camera Mode: Less tiring
Support both active and inactive camera modes
R200

41
Choose the Right Mode for the Activity
Active Camera Mode
Touch interaction less
comfortable, less precise.
Best for short capture
interactions.
Inactive Camera Mode
Touch interaction more
comfortable, more precise.
Best for extended play
interactions.
Touch Zones Touch Zones
R200

42
Two Styles of Mixed-Reality Games
R200

43
Plan for the Scene and Camera Qualities
• Understand the camera limitations.
Depth data is less accurate on
- Very bright areas
- Clear glass
- Black surfaces
• Give relevant feedback when problems
are detected
• Fail gracefully and don’t prevent play
R200

44
Agenda
• F200 SDK tips
• R200 SDK tips
• Summary and Q&A
F200
R200

45
APIs to Obtain R200 Camera Device Information
• The Device interface provides the following functions:
API Description
QueryDSLeftRightCropping
SetDSLeftRightCropping
Access to the left/right image cropping mode.
QueryDSLeftRightExposure
QueryDSLeftRightExposureInfo
SetDSLeftRightAutoExposureSetDSLeftRightExposure
SetDSLeftRightExposure
Access to the left/right stream exposure value.
QueryDSLeftRightGain
QueryDSLeftRightGainInfo
SetDSLeftRightGain
Access to the left/right stream gain adjustment.
QueryDSMinMaxZ
SetDSMinMaxZ
Access to the distance value range.
R200

46
Checking Scene Quality
• Use PXC[M]ScenePerception:CheckSceneQuality to check whether the
stream quality is suitable to start scene perception
• Returns a value between 0 and 1. Higher the return value, better the
scene
• Code:
R200

47
Checking Tracking Accuracy
• Use PXCScenePerception::TrackingAccuracy to obtain tracking accuracy
data
- The TrackingAccuracy enumerator itemizes the tracking accuracy definitions
Name Description
HIGH High tracking accuracy
LOW Low tracking accuracy
MED Median tracking accuracy
FAILED Tracking failed
R200

48
Setting Voxel Resolution in Your Application
• Use PXC[M]ScenePerception:SetVoxelResolution/QueryVoxelResolution
• The PXC[M]VoxelResolution enumerator itemizes the supported voxel
resolutions
Name Description
LOW_RESOLUTION The low voxel resolution. Use this resolution in
a room-sized scenario (4/256m).
MED_RESOLUTION The median voxel resolution. Use this resolution
in a table-top-sized scenario (2/256m).
HIGH_RESOLUTION The high voxel resolution. Use this resolution in
a object-sized scenario (1/256m).
R200

49
3D Scanning Tracking Alerts
• Use PXC3DScan::AlertEvent for 3D Scanning tracking alerts
Name Description
ALERT_IN_RANGE The scanning object is in the right range
ALERT_TOO_CLOSE The scanning object is too close to the camera.
Prompt the user to move the object away from the
camera
ALERT_TOO_FAR The scanning object is too far away from the
camera. Prompt the user to move the object closer
ALERT_TRACKING The scanning object is in good tracking
ALERT_LOST_TRACKING Lost tracking on the scanning object
R200

50
Agenda
• F200 SDK tips
• R200 SDK tips
• Summary and Q&A
F200
R200

51
Summary and Next Steps
• Understand what the camera sees and give the user effective feedback
• Choose the right technology: Know when and when not to use gestures
• Understand your users and do iterative usability testing!
• Check out the UX guidelines and videos and send us feedback
- software.intel.com/articles/realsense-ux-design-guidelines
R200F200

52
Day 1 Prize
Completing an Online Session Evaluation by 10am
Tomorrow Automatically Enters You in a Drawing to Win!
Day 2 Prize
Intel® Compute Stick (20)
Copies of the complete sweepstakes rules are available at the Info Desk
Winners will be notified by email
You will receive an email with a link to the online evaluation prior to the end of this session.
Day 3 Prize
Microsoft* Surface* 3 (6) Dell Venue 10 7000 Series (4)

55
Legal Notices and Disclaimers
Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com,
or from the OEM or retailer.
No computer system can be absolutely secure.
Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual
performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and
benchmark results, visit http://www.intel.com/performance.
Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may affect future
costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction.
This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice.
Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps.
Statements in this document that refer to Intel’s plans and expectations for the quarter, the year, and the future, are forward-looking statements that involve a
number of risks and uncertainties. A detailed discussion of the factors that could affect Intel’s results and plans is included in Intel’s SEC filings, including the annual
report on Form 10-K.
The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current
characterized errata are available on request.
No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.
Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether
referenced data are accurate.
Intel, RealSense, and the Intel logo are trademarks of Intel Corporation in the United States and other countries.
*Other names and brands may be claimed as the property of others.
© 2015 Intel Corporation.

56
Risk FactorsThe above statements and any others in this document that refer to plans and expectations for the second quarter, the year and the future are forward-
looking statements that involve a number of risks and uncertainties. Words such as "anticipates," "expects," "intends," "plans," "believes," "seeks,"
"estimates," "may," "will," "should" and their variations identify forward-looking statements. Statements that refer to or are based on projections, uncertain
events or assumptions also identify forward-looking statements. Many factors could affect Intel's actual results, and variances from Intel's current
expectations regarding such factors could cause actual results to differ materially from those expressed in these forward-looking statements. Intel
presently considers the following to be important factors that could cause actual results to differ materially from the company's expectations. Demand for
Intel's products is highly variable and could differ from expectations due to factors including changes in business and economic conditions; consumer
confidence or income levels; the introduction, availability and market acceptance of Intel's products, products used together with Intel products and
competitors' products; competitive and pricing pressures, including actions taken by competitors; supply constraints and other disruptions affecting
customers; changes in customer order patterns including order cancellations; and changes in the level of inventory at customers. Intel's gross margin
percentage could vary significantly from expectations based on capacity utilization; variations in inventory valuation, including variations related to the
timing of qualifying products for sale; changes in revenue levels; segment product mix; the timing and execution of the manufacturing ramp and associated
costs; excess or obsolete inventory; changes in unit costs; defects or disruptions in the supply of materials or resources; and product manufacturing
quality/yields. Variations in gross margin may also be caused by the timing of Intel product introductions and related expenses, including marketing
expenses, and Intel's ability to respond quickly to technological developments and to introduce new products or incorporate new features into existing
products, which may result in restructuring and asset impairment charges. Intel's results could be affected by adverse economic, social, political and
physical/infrastructure conditions in countries where Intel, its customers or its suppliers operate, including military conflict and other security risks, natural
disasters, infrastructure disruptions, health concerns and fluctuations in currency exchange rates. Results may also be affected by the formal or informal
imposition by countries of new or revised export and/or import and doing-business regulations, which could be changed without prior notice. Intel
operates in highly competitive industries and its operations have high costs that are either fixed or difficult to reduce in the short term. The amount, timing
and execution of Intel's stock repurchase program could be affected by changes in Intel's priorities for the use of cash, such as operational spending,
capital spending, acquisitions, and as a result of changes to Intel's cash flows or changes in tax laws. Product defects or errata (deviations from published
specifications) may adversely impact our expenses, revenues and reputation. Intel's results could be affected by litigation or regulatory matters involving
intellectual property, stockholder, consumer, antitrust, disclosure and other issues. An unfavorable ruling could include monetary damages or an
injunction prohibiting Intel from manufacturing or selling one or more products, precluding particular business practices, impacting Intel's ability to design
its products, or requiring other remedies such as compulsory licensing of intellectual property. Intel's results may be affected by the timing of closing of
acquisitions, divestitures and other significant transactions. A detailed discussion of these and other factors that could affect Intel's results is included in
Intel's SEC filings, including the company's most recent reports on Form 10-Q, Form 10-K and earnings release.
Rev. 4/14/15

Designing Apps for Intel RealSense Technology

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Designing Apps for Intel RealSense Technology

Similaire à Designing Apps for Intel RealSense Technology (20)

Dernier

Dernier (20)

Designing Apps for Intel RealSense Technology