1. Ing. Matteo Valoriani
matteo.valoriani@studentpartner.com
KINECT Programming

2. Gesture
• What is a gesture?
• An action intended to communicate feelings or intentions
• What is “Gesture Detection” or “Gesture Recognition”?
• Computer’s ability to understand human gestures as input
• First used in 1963 with pen-based input device
• What is it used for?
• Mouse movements, Handwriting recognition, Sign language,
recognition, Touch screen input, Kinect
KINECT Programming

3. Interaction metaphors
• Depend by the tasks
• Important aspect in design of UI
Cursors (hands tracking): Avatars (body tracking):
Target an object Interaction with virtual space
KINECT Programming

4. The shadow/mirror effect
Shadow Effect: Mirror Effect:
• I see the back of my avatar • I see the front of my avatar
• Problems with Z movements • Problem with mapping left/right
movements
KINECT Programming

5. KINECT Programming

6. IR Emitter User Interaction
Game mindset ≠ UI mindset
Challenging = fun Challenging = easy and effective
KINECT Programming

7. Gesture semantically fits user task
Abstract Meaningful
KINECT Programming

8. User action fits UI reaction
1 2 3 4 5
System’s UI feedback relates to the user’s physical
movement
KINECT Programming

9. User action fits UI reaction
5 61 72 83 94 10
5
System’s UI feedback relates to the user’s physical
movement
KINECT Programming

10. Gestures family-up
1 2 3 4 5
Each gesture feels related and cohesive
with entire gesture set
KINECT Programming

11. Handed gestures
1 2 3 4 5
Different gesture depending on hand: only left hand
can do gesture A
KINECT Programming

12. Repeting Gesture?
Will users want/need to perform the proposed gesture
repeatedly?
KINECT Programming

13. Repeting Gesture?
Will users want/need to perform the proposed gesture
repeatedly?
KINECT Programming

14. Number of Hands
1 2 3 4 5
One-handed gestures are preferred
KINECT Programming

15. Symmetrical two-handed gesture
Two hand gesture should be symmetrical
KINECT Programming

16. Gesture payoff
1 2 3 4 5
Interactions requiring more work and effort should
have a higher payoff
KINECT Programming

17. Fatigue kills gesture
Fatigue is the start of downward that kills gesture
Fatigue increase messiness  poor performance 
frustration  bad UX
KINECT Programming

18. Gorilla Arm problem
Gorilla arm problem: try to put the hand up for 10
minutes…
KINECT Programming

19. Confortable positions
KINECT Programming

20. User Posture
User posture may affect design of a gesture
KINECT Programming

21. The challenges
• Physical variable
• Environment
• Recognizing intent
• Input variability
KINECT Programming

22. KINECT Programming

23. Heuristics
• Experience-based techniques for problem solving, learning, and
discovery
Cost
• Cost effective
• Helps reconstruct missing
information
• Helps compute outcome of
a gesture
Gesture
Heuristics Machine Learning Complexity
KINECT Programming

24. Define What Constitutes a Gesture
• Some players have more energy (or enthusiasm) than
others
• Some players will “optimize” their gestures
• Most players will not perform the gesture precisely as
intended
KINECT Programming

25. Select the Right Triggers
• Use skeleton view to analyze whole skeleton behavior
• Use joint view to isolate and analyze specific joints and
axis behavior
• Use data sheet view: to get the real numbers
• Not all joints are needed
• Player location in the play area can cause some joints to
become occluded
KINECT Programming

26. Define Key Stages of a Gesture
• Determine
• When the gesture begins
• When the gesture ends
• Determine other key stages
• Changes in motion direction
• Pauses
• …
• You could simply signal that the gesture has been completed, or
• You could keep a progress, or
• You could use distinct states
KINECT Programming

27. Determine the Type of Outcome
• Definite gesture • Continuous gesture
• Contact or release • Frequency
point • Amplitude
• Direction
• Initial velocity
KINECT Programming

28. Run a Detection Filter Only When Necessary
• Define clear context for when a gesture is expected
• Provide clear feedback to the player
• Run the gesture filter when the context warrants it
• Cancel the gesture if context changes
KINECT Programming

29. Causes of Missing Information
• Self Occlusion
• Side poses
• Player’s position in play space
• Obstacles
• Other players
• Furniture
• Outside the camera’s field of view
• Left or right (easy to fix)
• Top or bottom (hard to avoid)
KINECT Programming

30. KINECT Programming

31. class GestureRecognizer {
public Dictionary<JointType, List<Joint>> skeletonSerie = new Dictionary<JointType, List<Joint>>() {
{ JointType.AnkleLeft, new List<Joint>()}, { JointType.AnkleRight, new List<Joint>()},
{ JointType.ElbowLeft, new List<Joint>()}, { JointType.ElbowRight, new List<Joint>()},
{ JointType.FootLeft, new List<Joint>()}, { JointType.FootRight, new List<Joint>()},
{ JointType.HandLeft, new List<Joint>()}, { JointType.HandRight, new List<Joint>()},
{ JointType.Head, new List<Joint>()}, { JointType.HipCenter, new List<Joint>()},
{ JointType.HipLeft, new List<Joint>()}, { JointType.HipRight, new List<Joint>()},
{ JointType.KneeLeft, new List<Joint>()}, { JointType.KneeRight, new List<Joint>()},
{ JointType.ShoulderCenter, new List<Joint>()}, { JointType.ShoulderLeft, new List<Joint>()},
{ JointType.ShoulderRight, new List<Joint>()},
{ JointType.Spine, new List<Joint>()},
Key Value
{ JointType.WristLeft, new List<Joint>()},
{ JointType.WristRight, new List<Joint>()} AnkleLeft <Vt1, Vt2, Vt3, Vt4,..>
}; AnkleRight <V , V , V , V ,..>
t1 t2 t3 t4
protected List<DateTime> timeList; ElbowLeft <Vt1, Vt2, Vt3, Vt4,..>
private static List<JointType> typesList = new List<JointType>() {JointType.AnkleLeft, JointType.AnkleRight,
JointType.ElbowLeft, JointType.ElbowRight, JointType.FootLeft, JointType.FootRight, JointType.HandLeft,
JointType.HandRight, JointType.Head, JointType.HipCenter, JointType.HipLeft, JointType.HipRight, JointType.KneeLeft,
JointType.KneeRight, JointType.ShoulderCenter, JointType.ShoulderLeft, JointType.ShoulderRight, JointType.Spine,
JointType.WristLeft, JointType.WristRight };
//... continue
}
KINECT Programming

32. const int bufferLenght=10;
public void Recognize(JointCollection jointCollection, DateTime date) {
timeList.Add(date);
foreach (JointType type in typesList) {
skeletonSerie[type].Add(jointCollection[type]);
if (skeletonSerie[type].Count > bufferLenght) {
skeletonSerie[type].RemoveAt(0);
}
}
startRecognition();
}
List<Gesture> gesturesList = new List<Gesture>();
private void startRecognition() {
gesturesList.Clear();
gesturesList.Add(HandOnHeadReconizerRT(JointType.HandLeft,
JointType.ShoulderLeft));
// Do ...
}
KINECT Programming

33. Boolean isHOHRecognitionStarted;
DateTime StartTimeHOH = DateTime.Now;
private Gesture HandOnHeadReconizerRT (JointType hand, JointType shoulder) {
// Correct Position
if (skeletonSerie[hand].Last().Position.Y > skeletonSerie[shoulder].Last().Position.Y + 0.2f) {
if (!isHOHRecognitionStarted) {
isHOHRecognitionStarted = true;
StartTimeHOH = timeList.Last();
}
else {
double totalMilliseconds = (timeList.Last() - StartTimeHOH).TotalMilliseconds;
// time ok?
if ((totalMilliseconds >= HandOnHeadMinimalDuration)) {
isHOHRecognitionStarted = false;
return Gesture.HandOnHead;
Alternative: count
} number of
}
occurrences
}
else {//Incorrect Position
if (isHOHRecognitionStarted) {
isHOHRecognitionStarted = false;
}
}
return Gesture.None; }
KINECT Programming

34. How to notify a gesture?
• Synchronous Solution:
• Return gesturesList to GUI
• Asynchronous Solution:
• Use Event
public delegate void HandOnHeadHadler(object sender, EventArgs e);
public event HandOnHeadHadler HandOnHead;
private Gesture HandOnHeadReconizerRTWithEvent(JointType hand,
JointType shoulder) {
Gesture g = HandOnHeadReconizerRT(hand, shoulder);
if (g == Gesture.HandOnHead) {
if (HandOnHead != null) HandOnHead(this, EventArgs.Empty);
}
return g;
}
KINECT Programming

35. KINECT Programming

36. const float SwipeMinimalLength = 0.08f;
const
const
float SwipeMaximalHeight = 0.02f;
int SwipeMinimalDuration = 200;
∆x too small or ∆y too
const
const
int SwipeMaximalDuration = 1000;
int MinimalPeriodBetweenGestures = 0;
big  shift start
private Gesture HorizzontalSwipeRecognizer(List<Joint> positionList) {
int start = 0; ∆x > minimal lenght
for (int index = 0; index < positionList.Count - 1; index++) {
if ((Math.Abs(positionList[0].Position.Y - positionList[index].Position.Y) > SwipeMaximalHeight) ||
Math.Abs((positionList[index].Position.X - positionList[index + 1].Position.X)) < 0.01f) {
start = index; }
∆t in the accepted range
if ((Math.Abs(positionList[index].Position.X - positionList[start].Position.X) > SwipeMinimalLength)) {
double totalMilliseconds = (timeList[index] - timeList[start]).TotalMilliseconds;
if (totalMilliseconds >= SwipeMinimalDuration && totalMilliseconds <= SwipeMaximalDurati {
if (DateTime.Now.Subtract(lastGestureDate).TotalMilliseconds > MinimalPeriodBetweenGestures)
{
lastGestureDate = DateTime.Now;
if (positionList[index].Position.X - positionList[start].Position.X < 0)
return Gesture.SwipeRightToLeft;
else
return Gesture.SwipeLeftToRight;
}
}
}
} return Gesture.None; }
KINECT Programming

37. public delegate void SwipeHadler(object sender, GestureEventArgs e);
public event SwipeHadler Swipe;
private Gesture HorizzontalSwipeRecognizer(JointType jointType) {
Gesture g = HorizzontalSwipeRecognizer(skeletonSerie[jointType]);
switch (g) {
case Gesture.None:
break;
case Gesture.SwipeLeftToRight:
if (Swipe != null) Swipe(this, new GestureEventArgs("SwipeLeftToRight"));
break;
case Gesture.SwipeRightToLeft:
if (Swipe != null) Swipe(this, new GestureEventArgs("SwipeRightToLeft"));
break;
default:
break;
}
return g;
}
...
Personalized EventArgs
public class GestureEventArgs : EventArgs
{
public string text;
public GestureEventArgs(string text) { this.text = text; }
}
KINECT Programming

38. Performance
• Skeleton processing is an expensive operation.
• Use VS2010 Performance Tool
KINECT Programming

39. KINECT Programming

40. Pros & Cons
PROs
• Easy to understand
• Easy to implement (for simple gestures)
• Easy to debug
Recommendation
Use for simple gestures
CONs • Hand wave
• Challenging to choose best values for
parameters • Head movement
• Doesn’t scale well for variants of same
gesture
• Gets challenging for complex gestures
• Challenging to compensate for latency
KINECT Programming

41. KINECT Programming

42. Gesture Definition
Define gesture as weighted network
• Simple neural network
• Simple algorithmic gestures as input nodes
• Use fuzzy logic, i.e. probabilities, not Booleans
HeadAboveBaseLine 1
LeftKneeAboveBaseLine 2  Jump?
3
RightKneeAboveBaseLine
KINECT Programming

43. Abstract Neuron
x1
1
x2 2
f (i 1 ixi )
n
f
n
xn
KINECT Programming

44. Perceptron
• Simple network using weighted threshold elements
P1
1
P2 2
  iPi  
n
i 1
n
Pn
KINECT Programming

45. Example
HandAboveElbow AND HandInFrontOfShoulder
Hand.y
HandAboveElbow
1
Elbow.y
(HandAboveElbow * 1) +
2 (HandInFrontOfShoulder * 1) >= 2
1
Hand.z HandInFrontOfShoulder
Shoulder.z
KINECT Programming

46. Example
HandAboveElbow OR HandInFrontOfShoulder
Hand.y
HandAboveElbow
1
Elbow.y
(HandAboveElbow * 1) +
1 (HandInFrontOfShoulder * 1) >= 1
1
Hand.z HandInFrontOfShoulder
Shoulder.z
KINECT Programming

47. Network Definition for Detector
• Similar to perceptron
• Normalize using weights
• Use probabilities, not Booleans
P1
1
P2 2
 iPi
n
 i 1
 
 i
n
n i 1
Pn
KINECT Programming

48. Surely This Will Suffice?
HeadAboveBaseLine
0.3
LeftKneeAboveBaseLine 0.1
0.1 0.8 Jump?
RightKneeAboveBaseLine
0.5
LegsStraightPreviouslyBent
• But due to noise, still many false positives
• How can we reduce false positives?
KINECT Programming

49. And We’re Done!
HeadAboveBaseLine 0.3
LeftKneeAboveBaseLine 0.1
RightKneeAboveBaseLine
0.1 0.8
LegsStraightPreviouslyBent 0.5
1
HeadBelowBaseLine
2 Jump?
1
AND
LeftKneeBelowBaseLine 1
OR NOT 1
RightKneeBelowBaseLine 1 -1
1 0
LeftAnkleBelowBaseLine 1
1
RightAnkleBelowBaseLine
1
BodyFaceUpwards
KINECT Programming

50. But Wait, If We Know For Sure…
HeadAboveBaseLine 0.3 HeadFarAboveBaseLine
0.1 1
LeftKneeAboveBaseLine
RightKneeAboveBaseLine
0.1 0.8 OR
1
LegsStraightPreviouslyBent 0.5 1 Jump?
HeadBelowBaseLine
2
1
AND
LeftKneeBelowBaseLine 1 1
OR NOT
RightKneeBelowBaseLine 1 -1
1 0
LeftAnkleBelowBaseLine 1
1
RightAnkleBelowBaseLine
1
BodyFaceUpwards
KINECT Programming

51. Implementation Overview
• Update height baseline values
• Update input nodes, i.e. algorithmic gestures
• Evaluate each node in network
• Calculate probability of gesture
KINECT Programming

52. Pros
• Neural networks well understood
• Introduced in 1940’s
• Learning algorithm can be used to find optimum
• Parameters, weights, and thresholds
• Complex gestures can be detected
• Scale well for variants of same gesture
• Nodes can be reused in different gestures
• Easy to visualize as node graph
• Good CPU performance
• 0.095 ms to execute Jump Detector
KINECT Programming

53. Cons
• Lots of parameters, weights, and thresholds
• Small changes can have dramatic changes in results
• Very time consuming to choose manually
• Not easy to debug
• Is the code wrong or are parameters not optimal
• Challenging to compensate for latency
KINECT Programming

54. Recommendation
• Use for more complex gestures
• Jump, duck, punch
• Break complex gestures into collection of simple
gestures
• Use learning algorithm
• Debug visualization is essential
KINECT Programming

55. KINECT Programming

56. Gesture Definition
• Define gesture as pre-recorded animations
• Motion capture animations
• Record different people doing same gesture
• Each person doing same gesture multiple times
KINECT Programming

57. Exemplar
• Definition: ideal example to compare against
• Pre-recorded animations are exemplars
KINECT Programming

58. Exemplar Matching
• Need to compare skeleton frames
• Define error metric for skeleton
• Angular difference for each joint in local space
• Peak Signal to Noise Ratio for whole skeleton
1
MSE   Distancei2
0.3 N
PSNR  10 * log10 ( MAX 2 / MSE )
KINECT Programming

59. Exemplar Matching
• Search for best matching frames
• Best matching frame has strongest signal
• Different classifiers can be used
• K-Nearest
• Dynamic Time Warping (DTW)
• Hidden Markov Models (HMM)
KINECT Programming

60. Exemplar Matching
25
20
15
10 PSNR
5
0
1 2 3 4 5 6 7 8
KINECT Programming

61. Pros
• Works well for context-sensitive gesture detection
• Works well for animation blending
• Very complex gestures can be detected
• DTW allows for different speeds
• Can compensate for latency
• Can scale for variants of same gesture
• Just need more resources
• Easy to visualize exemplar matching
KINECT Programming

62. Cons
• Requires lots of resources to be robust
• Multiple recordings of multiple people for one
gesture
• i.e. requires lots of CPU and memory
• K-Nearest
• 1.5 ms for 16 exemplar matches
• DTW
• 5 ms for 16 exemplar matches
KINECT Programming

63. Example
• 10 Gestures, 10 People, 5 times = 500 Exemplars
• K-Nearest 180
• 46 ms 160
• DTW 140 K-Nearest
120
• 156 ms
100 DTW
• Weighted network 80
• 1 ms 60 Weighted
40 Network
20
0
KINECT Programming

64. Recommendation
• Use for context-sensitive gesture detection
• Use for complex gestures
• Dancing, fitness exercises
• Use when reducing latency is critical
• Optimize by reducing exemplar matches
• Preprocess exemplar data with key frames
• Use context of game
• Use another fast method first
• Implement debug visualization
KINECT Programming

65. KINECT Programming

66. Building Great Gesture Detection
Data Collection
Development
Testing
KINECT Programming

67. Data Collection
Jump
Identify Gestures Punch
At least depth & skeleton 1. Exemplar
2. Sequence of same gesture
Record Gestures 3. General (actual game play)
Old, young, male, female,
overweight, handedness
Meta data per recording, tag
Tag Gesture Recordings start/stop events for each
gesture
Use custom tool,or export to
Excel
Someone other than tagger
Verify Gesture Tagging should verify correctness
Backup & Share
KINECT Programming

68. Development
Phase 1 – Exemplar Data Tagged Gesture
Phase 2 – Sequence Data Recordings
Phase 3 – General Data Filter Joints Normalize
Skeleton
Parameters
Gesture Debug
Weights
Detector Visualization
Thresholds
Result Verification
Machine Learning
Algorithm Error
KINECT Programming

69. Testing
Live Camera Tagged Gesture
Stream Recordings
Filter Joints
Normalize Skeleton
Parameters
Weights Gesture
Thresholds Detector
Human Verification
Result
Verification
Feels No Data Error
Robust? Collection
KINECT Programming

70. Takeaways
• A system, not just a detector
• Detector is small component
• Invest equally in other components
• Manage data
• You’ll have lots of it!
• Most valuable component
• Tagging correctly is essential
• Collect real user data
KINECT Programming

71. References
• “A Brief History of Human Computer Interaction Technology” – Brad A. Myers
• “Neural Networks – A Systematic Introduction” – Raúl Rojas
• “A Gesture Processing Framework for Multimodal Interaction in Virtual Reality” – Marc E. Latoschik
• Gamefest 2010 – “Gesture Recognition” – Lewey Geselowitz & J. McBride
• Kinect Developer Summit 2011 – “Inside Kinect Skeletal Tracking Deep Dive” – Zsolt Mathe
KINECT Programming