The document discusses hand gesture detection for human-computer interaction. It describes several existing approaches to hand gesture detection using devices like cameras, sensors and markers. The proposal is to develop a new algorithm for detecting hand gestures to control applications like a GPS or robot using a webcam without markers. The algorithm will involve image resizing, skin filtering, contour detection, convex hull analysis and finger estimation to recognize gestures.

1. “Hand sign detection”
David J. Rios
Debate and Conversation in English
DCM-FIME-UANL
november, 2011

2. What is augmented reality?
Add information to the reality we see to know
more about the things.

3. Applications

4. what is Human Computer
Interaction?
HCI is the way a person interacts with a
computer, it can be with a mouse, a
keyboard, the screen, touchpads, trackballs,
and also using gestures.

5. What is gesture detection?
Use a computer to understand the gestures
being made by a person, it can be done
using computer vision, sensors, markers,...

6. Hand gesture interaction

10. Sixthsense

11. Related work
● Hirobe et al. [4] have created a mobile
interface that tracks the image of the fingers
and allows to type on a keyboard in the air and
3D drawings.

12. ● Finger counter [6] Using a webcam, that
counts fingers and interprets specific hand
gestures as input to a system.

13. ● E. Kollorz et. al. [7] clasification of gestures
performed of the image projections on the
axes x and y. It uses a Photonic-Mixer-
Device (PMD) 3-D sensor

14. Related work
camera sensor makers signs following position
Kinect infrared Infrared no no yes no
light
Elliptics no ultrasound no 5 yes no
Sixthsense webcam no yes 7 yes yes
Hirobe high- FPGA no no yes yes
frame-rate board*
Finger count webcam no no 4 no yes
Kollortz 3-D Photonic- no 12 no yes
Mixer-
Device
Proposal webcam no no 6 yes no

15. Introduction
● The focus is on the hand gestures
detection and augmented reality area .
● We want to develop and implement new
algorithms for detecting hand gestures for
natural interaction with computer systems
and augmented reality.
● We want to develop an augmented reality
prototype to display maps and directions
where natural interaction methods based
on hand gestures are implemented.

16. Image resize
We reduce the input image to 160 x 120
pixels faster processing.
● cv.Resize(source,destination,cv.CV_INTER_
LINEAR)

17. Skin color filter
● Split the image on three channels R,G,B
cv.Split(image,R, G, B, None)
● Clasify skin and not-skin pixels [5,10]
R (x, y) > 95 and G (x, y) > 40 and B (x, y) > 20 and
max{R (x, y), G (x, y), B (x, y)} − min{R (x, y), G (x, y),
B (x, y)} > 15 and
R (x, y) − G (x, y)| > 15 and R (x, y) > G (x, y) and R
(x, y) > B (x, y)
● Create binary image

19. Border detection
contour = cv.FindContours(binary, storage,
cv.CV_RETR_EXTERNAL ,
cv.CV_CHAIN_APPROX_SIMPLE)
Convex hull
hull = cv.ConvexHull2 (contour, storage,
cv.CV_CLOCKWISE, 1)
Defects
defects = cv.ConvexityDefects(contour, hull, storage)

20. Contour Convex
hull
Defects

21. Finger estimation
● Using the number of defects and the depth
of those defects e can estimate the number
of fingers.
Average depth:
= i ∑
d di
n i=0... n

22. Recognized signs

24. Hand gesture detection procedure

26. Current work
We create a computer program capable of
detect hand signs and use the sign detected
to comunicate with other devices like a GPS
or a LEGO robot

27. Applications
● GPS control device:

28. Video:
http://www.youtube.com/watch?v=lt_yv4lRMxw

29. ● Robot control:

30. Video:
http://www.youtube.com/watch?v=dTS1ejXeOk4

31. References
[1] R. Azuma, A Survey of Augmented Reality Presence: Teleoperators
and Virtual Environments, pp. 355–385, August 1997.
[2] ARToolKit {http://www.hitl.washington.edu/artoolkit/}
[3] Ebling, Maria R. and Caceres, Ramon, "Gaming and Augmented
Reality Come to Location-Based Services", IEEE Pervasive Computing,
vol. 9, pp. 5-6, 2010.
[4] TERAJIMA , K., T. KOMURO y M. I SHIKAWA, Fast finger
tracking system for in-air typing interface , en CHI ’09: Proceedings of
the 27th international conference extended abstracts on Human factors
in computing systems, ACM, New York, NY, USA, pages. 3739–3744,
2009.
[5] Mahmoud, Tarek M., "A New Fast Skin Color Detection
Technique", vol. World Academy of Science, Engineering and
Technology, no. 43, 2008

32. [6] CRAMPTON , S. C. y M. B ETKE, Counting fingers in real time: A
webcam-based human-computer interface game applications , en
Proceedings of the Conference on Universal Access in Human-Computer
Interaction, pages. 1357–1361, 2003.
[7] Eva Kollorz, Jochen Penne, Joachim Hornegger, and Alexander Barke.
2008. Gesture recognition with a Time Flight camera. Int. J. Intell. Syst.
Technol. Appl. 5, 3/4 (November 2008), 334-343.
[8] Stergiopoulou, E. and Papamarkos, N., "Hand gesture recognition using
a neural network shape fitting technique", Engineering Applications of
Artificial Intelligence, vol. 22, no. 8, pp. 1141--1158, 2009.
[9] Vassili, Vladimir Vezhnevets and Sazonov, Vassili and Andreeva, Alla,
"A Survey on Pixel-Based Skin Color Detection Techniques", in in Proc.
Graphicon-2003, 2003, pp. 85—92.
[10] Mahmoud, Tarek M., "A New Fast Skin Color Detection Technique",
vol. World Academy of Science, Engineering and Technology, no. 43,
2008.