Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Using Intel’s RealSense to create
games with Natural User Interfaces
Justin Link, Chronosapien Interactive
Overview
RealSense Overview
• Camera types
• System requirements
• SDK features
• Getting data
• Unity toolkit
Our Use Cas...
RealSense Overview
Camera Types
F200 R200 SR300
RealSense Overview
System Requirements
• 4th generation (or later) Intel® Core™ processor
• 8GB free hard disk space
• Mic...
RealSense Overview
SDK Features
• Hand tracking/gesture recognition
• Face tracking/recognition
• 3D Scanning
• Background...
RealSense Overview
Getting Data
• The first step for getting any data from the RealSense SDK is creating a
session instanc...
RealSense Overview
Getting Data
• Next you can create an instance of the module you want to use and
(optionally) configure...
RealSense Overview
Getting Data
• Once the module is created and set up, you can poll data:
https://software.intel.com/sit...
RealSense Overview
The Unity Toolkit
• Easy to use
• Quick to set up
• Covers a lot, but has limited functionality compare...
Hand trackingin space between
Hand Tracking in Space Between
What is Space Between?
• Underwater themed game that lets you play as five different sea
cr...
Hand Tracking in Space Between
Getting Hand Data from RealSense
Let’s check out some of our code
Hand Tracking in Space Between
Challenges
User understanding of the technology
• Tracking has bounds
• Tracking requires a...
Hand Tracking in Space Between
Challenges
“Gorilla Arm”, or Consumed Endurance
• Constant motion fatigues users’ arms afte...
Hand Tracking in Space Between
Challenges
Data is noisy
• Can create false positives
• Limits precise actions
• Can look v...
Hand Tracking in Space Between
What We Did
• Cursor for every game to always show the tracked hand/head position
• Game in...
Hand Tracking in Space Between
What We Did
Let’s jump into Space Between
VoiceRecognition in The Risen
Voice Recognition in The Risen
What is The Risen?
• A tech demo we created for the RealSense App Challenge
• Let’s players...
Voice Recognition in The Risen
Getting Speech Data from RealSense
Let’s check out some of our code
Voice Recognition in The Risen
Challenges
Recognition is based on the player’s
environment
• Loud environments kill recogn...
Voice Recognition in The Risen
Challenges
Recognition is slower than players’
expectations
• RealSense uses a local soluti...
Voice Recognition in The Risen
Challenges
Recognition is often inaccurate
• False positives
• True negatives
Voice Recognition in The Risen
What We Did
• Step by step tutorial
• UI feedback for when speech was detected
• UI feedbac...
Voice Recognition in The Risen
What We Did
Let’s check out The Risen
User Segmentation in our Photo booth
User Segmentation in our Photo booth
What is The Photo booth?
• A photo booth that puts users into a virtual environment w...
User Segmentation in our Photo booth
Getting Segmentation Data from RealSense
Let’s check out some of our code
User Segmentation in our Photo booth
Challenges
Segmentation quality is highly dependent
on lighting
• Be careful with ext...
User Segmentation in our Photo booth
Challenges
Segmentation often cuts off parts of the
user
• This is a result of the me...
User Segmentation in our Photo booth
Challenges
Segmentation only supports a single user
• This is mostly true, but someti...
User Segmentation in our Photo booth
Challenges
Color/lighting of the user often does not
match the virtual environment
• ...
User Segmentation in our Photo booth
What We Did
• Used bright lights to illuminate the user
• Used curtains/walls to bloc...
Voice Recognition in The Risen
What We Did
Let’s jump into the photo booth
Wrapping Up
Take Away’s
• Users must be trained to understand how the camera works
• User feedback is critical to make up ...
Wrapping Up
References
• RealSense SDK download page - https://software.intel.com/en-us/intel-realsense-sdk/download
• Rea...
Wrapping Up
Contact
• http://www.chronosapien.com/
• https://www.reddit.com/r/chronosapien
• @jlink__ + @_chronosapien
• j...
Thanks for Watching!
Time for QA!
Prochain SlideShare
Chargement dans…5
×

Using intel's real sense to create games with natural user interfaces justin link

1 413 vues

Publié le

As technology advances, more sophisticated ways of interfacing with it are emerging. Even though new tech strives to make our apps more intuitive and easy to use, designing interfaces for those apps is not quite as straight forward. We’ve learned a few rules and “gotchas” when working with gesture cameras that can help to make apps that use them easy and fun to use.

In this talk Justin described:
1. Different data types you can get from Intel® RealSense™ and how to get them
2. Designing an interface for a gesture camera
3. Using your hands, face, and voice as an interface

Publié dans : Technologie
  • Soyez le premier à commenter

Using intel's real sense to create games with natural user interfaces justin link

  1. 1. Using Intel’s RealSense to create games with Natural User Interfaces Justin Link, Chronosapien Interactive
  2. 2. Overview RealSense Overview • Camera types • System requirements • SDK features • Getting data • Unity toolkit Our Use Cases • Getting the data • Challenges • What we did Space Between The Risen Photo booth
  3. 3. RealSense Overview Camera Types F200 R200 SR300
  4. 4. RealSense Overview System Requirements • 4th generation (or later) Intel® Core™ processor • 8GB free hard disk space • Microsoft Windows* 8.1 or 10 OS 64-bit
  5. 5. RealSense Overview SDK Features • Hand tracking/gesture recognition • Face tracking/recognition • 3D Scanning • Background removal (user segmentation) • Object tracking • Speech recognition
  6. 6. RealSense Overview Getting Data • The first step for getting any data from the RealSense SDK is creating a session instance: https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?pxcsession.html
  7. 7. RealSense Overview Getting Data • Next you can create an instance of the module you want to use and (optionally) configure it: https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?doc_hand_initializing_and_configuring.html
  8. 8. RealSense Overview Getting Data • Once the module is created and set up, you can poll data: https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?doc_hand_implementing_the_main_loop.html
  9. 9. RealSense Overview The Unity Toolkit • Easy to use • Quick to set up • Covers a lot, but has limited functionality compared to the SDK • Great for prototyping https://www.youtube.com/watch?v=SK50ncrYq4w
  10. 10. Hand trackingin space between
  11. 11. Hand Tracking in Space Between What is Space Between? • Underwater themed game that lets you play as five different sea creatures or machines • Uses RealSense to let players control characters with their hands and head • Unique input for each character
  12. 12. Hand Tracking in Space Between Getting Hand Data from RealSense Let’s check out some of our code
  13. 13. Hand Tracking in Space Between Challenges User understanding of the technology • Tracking has bounds • Tracking requires a line of sight • Tracking is based on silhouettes
  14. 14. Hand Tracking in Space Between Challenges “Gorilla Arm”, or Consumed Endurance • Constant motion fatigues users’ arms after about 60 seconds • Might be ok for exercise, but isn’t that fun
  15. 15. Hand Tracking in Space Between Challenges Data is noisy • Can create false positives • Limits precise actions • Can look visually unappealing
  16. 16. Hand Tracking in Space Between What We Did • Cursor for every game to always show the tracked hand/head position • Game instructions to tell players what is being tracked and how it’s used • Depth feed of the player during gameplay • Instructions/tutorials accessible at all times
  17. 17. Hand Tracking in Space Between What We Did Let’s jump into Space Between
  18. 18. VoiceRecognition in The Risen
  19. 19. Voice Recognition in The Risen What is The Risen? • A tech demo we created for the RealSense App Challenge • Let’s players play as a skeleton lord who commands skeletons with their voice • Let’s players be IN the game via user segmentation • Let’s players use their hands to help their skeleton minions
  20. 20. Voice Recognition in The Risen Getting Speech Data from RealSense Let’s check out some of our code
  21. 21. Voice Recognition in The Risen Challenges Recognition is based on the player’s environment • Loud environments kill recognition • Quiet environments may prevent talking
  22. 22. Voice Recognition in The Risen Challenges Recognition is slower than players’ expectations • RealSense uses a local solution • Expectations are based on cloud solutions
  23. 23. Voice Recognition in The Risen Challenges Recognition is often inaccurate • False positives • True negatives
  24. 24. Voice Recognition in The Risen What We Did • Step by step tutorial • UI feedback for when speech was detected • UI feedback for speech too loud or too quiet • Microphone volume slider
  25. 25. Voice Recognition in The Risen What We Did Let’s check out The Risen
  26. 26. User Segmentation in our Photo booth
  27. 27. User Segmentation in our Photo booth What is The Photo booth? • A photo booth that puts users into a virtual environment without the need for green screen • Uses gestures to control the app • Has print and share functionality
  28. 28. User Segmentation in our Photo booth Getting Segmentation Data from RealSense Let’s check out some of our code
  29. 29. User Segmentation in our Photo booth Challenges Segmentation quality is highly dependent on lighting • Be careful with external IR sources • Dark environments create grainy color feeds
  30. 30. User Segmentation in our Photo booth Challenges Segmentation often cuts off parts of the user • This is a result of the method of getting a depth image • Hair almost always gets lost • Reflective materials can be invisible
  31. 31. User Segmentation in our Photo booth Challenges Segmentation only supports a single user • This is mostly true, but sometimes people can trick the software into thinking they are one “blob”
  32. 32. User Segmentation in our Photo booth Challenges Color/lighting of the user often does not match the virtual environment • Not necessarily a fault of the SDK, but still a challenge when making an app
  33. 33. User Segmentation in our Photo booth What We Did • Used bright lights to illuminate the user • Used curtains/walls to block external light sources • Used hats to hide chopping users’ hair off • Used a plugin to control segmentation color, brightness, and contrast
  34. 34. Voice Recognition in The Risen What We Did Let’s jump into the photo booth
  35. 35. Wrapping Up Take Away’s • Users must be trained to understand how the camera works • User feedback is critical to make up for the loss of haptic feedback • Giving users control over the camera’s properties can improve recognition and overall UX • This is new tech – it has many quirks and working through them will be a big part of the work you do
  36. 36. Wrapping Up References • RealSense SDK download page - https://software.intel.com/en-us/intel-realsense-sdk/download • RealSense Documentation - https://software.intel.com/sites/landingpage/realsense/camera- sdk/v1.1/documentation/html/index.html • User Feedback for NUI’s with RealSense P1 - https://software.intel.com/en-us/articles/user- feedback-for-natural-user-interfaces-with-intel-realsense-technology-part-1 • User Feedback for NUI’s with RealSense P2 - https://software.intel.com/en-us/articles/user- feedback-for-natural-user-interfaces-with-intel-realsense-technology-part-2 • “Gorilla Arm”, or Consumed Endurance - https://pdfs.semanticscholar.org/d852/e9749467b06f6cd516ea3f6c894408dea6d4.pdf • Space Between on the RealSense App Showcase - https://appshowcase.intel.com/en- us/realsense/space-between?cam=f200
  37. 37. Wrapping Up Contact • http://www.chronosapien.com/ • https://www.reddit.com/r/chronosapien • @jlink__ + @_chronosapien • jlink@chronosapien.com • rclark@chronosapien.com
  38. 38. Thanks for Watching! Time for QA!

×