High Profile Call Girls Nagpur Meera Call 7001035870 Meet With Nagpur Escorts
Gesture recognition document
1. GESTURE RECOGNITION
Abstract:
Gesture recognition is the process of understanding and
interpreting meaningful movements of the hands, arms, face,
or sometimes head. It is of great need in designing an
efficient human-computer interface. The technology has
been in study in recent years because of its potential for
application in user interfaces.Gesture recognition is one of
the main areas of research for the engineers and scientists.
It is the natural way of human machine interaction. Today the
industry is working on different application to make
interactions more easy, natural and convenient without
wearing any extra device. There are many key top people in
gesture recognition systems who have marketed their
solution for various issues. The key vendors in gesture
recognition space include Cognitec-Systems, eyeSight
Mobile Ltd.,GestureTek Technologies Inc., Omek Interactive
Ltd., PointGrab Ltd., PrimeSense Ltd., and SoftKinetic Inc.
Some more to name are Intel Corp., Qualcomm Inc., and
Thalmic Labs Inc...
In this paper we present what gesture recognition technology
is all about and conceptualizing an ideal approach for
gesture recognition.
NIKHIL RAJ
4
2. CONTENTS
S.N TITLE PAGE
1 Introduction...................................................... 1
2 Gesture types................................................... 2
3 Input devices.................................................... 2
4 Algorithms........................................................ 5-9
o D model-based algorithms
o Skeletal-based algorithms
o Appearance-based models
5 Architecture..................................................... 10
6 Application Development................................. 11
5 Challenges....................................................... 12-13
o "Gorilla arm"
7 Market Trend.................................................... 13
8 Factors driving the need of solution................. 13
9 Upcoming New Technology............................. 15
10 Construction..................................................... 16
11 Advantage and Disadvantage.......................... 17
11 Conclusion........................................................ 18
12 References........................................................ 19
5
3. INTRODUCTION
Gesture recognition
It is a topic in computer science and language technology
with the goal of interpreting human gestures via
mathematical algorithms. Gestures can originate from any
bodily motion or state but commonly originate from the face
or hand. Current focuses in the field include emotion
recognition from face and hand gesture recognition. Many
approaches have been made using cameras and computer
vision algorithms to interpret sign language. However, the
identification and recognition of posture, gait, proxemics, and
human behaviors is also the subject of gesture recognition
techniques. Gesture recognition can be seen as a way for
computers to begin to understand human body language,
thus building a richer bridge between machines and humans
than primitive text user interfaces or even GUIs (graphical
user interfaces), which still limit the majority of input to
keyboard and mouse.
Gesture recognition enables humans to communicate with
the machine (HMI) and interact naturally without any
mechanical devices. Using the concept of gesture
recognition, it is possible to point a finger at the computer
screen so that the cursor will move accordingly. This could
potentially make conventional input devices such as mouse,
keyboards and even touch-screens redundant.Gesture
recognition can be conducted with techniques from computer
vision and image processing.
The literature includes ongoing work in the computer vision
field on capturing gestures or more general human pose and
movements by cameras connected to a computer.
6
4. Gesture recognition and pen computing:
Pen computing reduces the hardware impact of a system
and also increases the range of physical world objects
usable for control beyond traditional digital objects like
keyboards and mice. Such implements can enable a new
range of hardware not requiring monitors. This idea may lead
to the creation of holographic display. The term gesture
recognition has been used to refer more narrowly to non-
text-input handwriting symbols, such as inking on a graphics
tablet, multi-touch gestures, and mouse gesture recognition.
This is computer interaction through the drawing of symbols
with a pointing device cursor.
Gesture types
In computer interfaces, two types of gestures are
distinguished. We consider online gestures, which can also
be regarded as direct manipulations like scaling and rotating.
In contrast, offline gestures are usually processed after the
interaction is finished; e. g. a circle is drawn to activate a
context menu.
Offline gestures: Those gestures that are processed
after the user interaction with the object. An example is
the gesture to activate a menu.
Online gestures: Direct manipulation gestures. They
are used to scale or rotate a tangible object.
Input devices
The ability to track a person's movements and determine
what gestures they may be performing can be achieved
through various tools. Although there is a large amount of
research done in image/video based gesture recognition,
7
5. there is some variation within the tools and environments
used between implementations.
Wired gloves. These can provide input to the computer
about the position and rotation of the hands using
magnetic or inertial tracking devices. Furthermore,
some gloves can detect finger bending with a high
degree of accuracy (5-10 degrees), or even provide
haptic feedback to the user, which is a simulation of the
sense of touch. The first commercially available hand-
tracking glove-type device was the DataGlove, a glove-
type device which could detect hand position,
movement and finger bending. This uses fiber optic
cables running down the back of the hand. Light pulses
are created and when the fingers are bent, light leaks
through small cracks and the loss is registered, giving
an approximation of the hand pose.
Depth-aware cameras. Using specialized cameras
such as structured light or time-of-flight cameras, one
can generate a depth map of what is being seen
through the camera at a short range, and use this data
to approximate a 3D representation of what is being
seen. These can be effective for detection of hand
gestures due to their short range capabilities.
Stereo cameras. Using two cameras whose relations
to one another are known, a 3d representation can be
approximated by the output of the cameras. To get the
cameras' relations, one can use a positioning reference
such as a lexian-stripe or infrared emitters. In
combination with direct motion measurement (6D-
Vision) gestures can directly be detected.
Controller-based gestures. These controllers act as
an extension of the body so that when gestures are
performed, some of their motion can be conveniently
captured by software. Mouse gestures are one such
8
6. example, where the motion of the mouse is correlated
to a symbol being drawn by a person's hand, as is the
Wii Remote or the Myo, which can study changes in
acceleration over time to represent gestures. Devices
such as the LG Electronics Magic Wand, the Loop and
the Scoop use Hillcrest Labs' Freespace technology,
which uses MEMS accelerometers, gyroscopes and
other sensors to translate gestures into cursor
movement. The software also compensates for human
tremor and inadvertent movement. AudioCubes are
another example. The sensors of these smart light
emitting cubes can be used to sense hands and fingers
as well as other objects nearby, and can be used to
process data. Most applications are in music and sound
synthesis, but can be applied to other fields.
Single camera. A standard 2D camera can be used for
gesture recognition where the resources/environment
would not be convenient for other forms of image-based
recognition. Earlier it was thought that single camera
may not be as effective as stereo or depth aware
cameras, but some companies are challenging this
theory. Software-based gesture recognition technology
using a standard 2D camera that can detect robust hand
gestures, hand signs, as well as track hands or fingertip
at high accuracy has already been embedded in
Lenovo’s Yoga ultrabooks, Pantech’s Vega LTE
smartphones, Hisense’s Smart TV models, among other
devices.
9
7. Algorithms
Different ways of tracking and analyzing gestures exist, and
some basic layout is given is in the diagram above. For
example, volumetric models convey the necessary
information required for an elaborate analysis, however they
prove to be very intensive in terms of computational power
and require further technological developments in order to be
implemented for real-time analysis. On the other hand,
appearance-based models are easier to process but usually
lack the generality required for Human-Computer Interaction.
Depending on the type of the input data, the approach for
interpreting a gesture could be done in different ways.
However, most of the techniques rely on key pointers
represented in a 3D coordinate system. Based on the
relative motion of these, the gesture can be detected with a
high accuracy, depending of the quality of the input and the
algorithm’s approach. In order to interpret movements of the
body, one has to classify them according to common
properties and the message the movements may express.
For example, in sign language each gesture represents a
word or phrase. The taxonomy that seems very appropriate
for Human-Computer Interaction has been proposed by
10
8. Quek in "Toward a Vision-Based Hand Gesture Interface".He
presents several interactive gesture systems in order to
capture the whole space of the gestures:
1. Manipulative;
2. Semaphoric;
3. Conversational.
Some literature differentiates 2 different approaches in
gesture recognition: a 3D model based and an appearance-
based. The foremost method makes use of 3D information of
key elements of the body parts in order to obtain several
important parameters, like palm position or joint angles. On
the other hand, Appearance-based systems use images or
videos for direct interpretation.
A real hand (left) is interpreted as a collection of vertices and
lines in the 3D mesh version (right), and the software uses
their relative position and interaction in order to infer the
gesture.
11
9. 3D model-based algorithms:The 3D model approach can
use volumetric or skeletal models, or even a combination of
the two. Volumetric approaches have been heavily used in
computer animation industry and for computer vision
purposes. The models are generally created of complicated
3D surfaces, like NURBS or polygon meshes. The drawback
of this method is that is very computational intensive, and
systems for live analysis are still to be developed. For the
moment, a more interesting approach would be to map
simple primitive objects to the person’s most important body
parts ( for example cylinders for the arms and neck, sphere
for the head) and analyse the way these interact with each
other. Furthermore, some abstract structures like super-
quadrics and generalised cylinders may be even more
suitable for approximating the body parts. The exciting thing
about this approach is that the parameters for these objects
are quite simple. In order to better model the relation
between these, we make use of constraints and hierarchies
between our objects.
The skeletal version (right) is effectively modelling the hand
(left). This has fewer parameters than the volumetric version
and it's easier to compute, making it suitable for real-time
gesture analysis systems.
Skeletal-based algorithms:Instead of using intensive
processing of the 3D models and dealing with a lot of
12
10. parameters, one can just use a simplified version of joint
angle parameters along with segment lengths. This is known
as a skeletal representation of the body, where a virtual
skeleton of the person is computed and parts of the body are
mapped to certain segments. The analysis here is done
using the position and orientation of these segments and the
relation between each one of them( for example the angle
between the joints and the relative position or orientation)
Advantages of using skeletal models:
Algorithms are faster because only key parameters are
analyzed.
Pattern matching against a template database is
possible
Using key points allows the detection program to focus
on the significant parts of the body
These binary silhouette(left) or contour(right) images
represent typical input for appearance-based algorithms.
They are compared with different hand templates and if they
match, the correspondent gesture is inferred.
13
11. Appearance-based models
These models don’t use a spatial representation of the body
anymore, because they derive the parameters directly from
the images or videos using a template database. Some are
based on the deformable 2D templates of the human parts of
the body, particularly hands. Deformable templates are sets
of points on the outline of an object, used as interpolation
nodes for the object’s outline approximation. One of the
simplest interpolation function is linear, which performs an
average shape from point sets, point variability parameters
and external deformators. These template-based models are
mostly used for hand-tracking, but could also be of use for
simple gesture classification.
A second approach in gesture detecting using appearance-
based models uses image sequences as gesture templates.
Parameters for this method are either the images
themselves, or certain features derived from these. Most of
the time, only one ( monoscopic) or two ( stereoscopic )
views are used.
14
12. Architecture
The solution should act as middleware to form glue between
platform and application. A sensor collect and sends raw
data to the solution. The solution is a middleware that sits
between the platform and application. It receives raw data
from the sensor,processes and send high level data to
application various vision based gesture application for
market such as TV, Automobile, healthcare and many others
can be built on top of the solution.
15
13. Application Development
The solution effectiveness is judged by the solution usage.
The aim of the solution will be to allow developers to easily
create a variety of natural gesture control applications
such as games, consumer electronics, and car
infotainment solution.
Let us consider the application example of car
infotainment solution to illustrate how effectively this ideal
solution will be used. Automobile manufacturers are
benefitted with gesture recognition as the technology is
adding more value to their offerings. Intuitive car
infotainment solutions enable the user to explore maps,
toggle menus and radio stations using simple gesture
control.
16
14. Challenges
There are many challenges associated with the accuracy
and usefulness of gesture recognition software. For image-
based gesture recognition there are limitations on the
equipment used and image noise. Images or video may not
be under consistent lighting, or in the same location. Items in
the background or distinct features of the users may make
recognition more difficult.
The variety of implementations for image-based gesture
recognition may also cause issue for viability of the
technology to general usage. For example, an algorithm
calibrated for one camera may not work for a different
camera. The amount of background noise also causes
tracking and recognition difficulties, especially when
occlusions (partial and full) occur. Furthermore, the distance
from the camera, and the camera's resolution and quality,
also cause variations in recognition accuracy.
In order to capture human gestures by visual sensors, robust
computer vision methods are also required, for example for
hand tracking and hand posture recognition or for capturing
movements of the head, facial expressions or gaze direction.
"Gorilla arm" :"Gorilla arm" was a side-effect of vertically
oriented touch-screen or light-pen use. In periods of
prolonged use, users' arms began to feel fatigue and/or
discomfort. This effect contributed to the decline of touch-
screen input despite initial popularity in the 1980s. In order to
measure arm fatigue and the gorilla arm side effect,
researchers developed a technique called Consumed
Endurance.
17
15. Market Trend
The market is changing rapidly due to evolving technology
and more and more OEM’s are moving towards gesture
recognition technology adoption. As per a report published
by Markets and Markets, the gesture recognition market is
estimated to grow at a healthy CAGR from 2013 till 2018 and
is expected to cross $15.02 billion by the end of these five
years. Analysts forecast the Global Gesture Recognition
market to grow at a CAGR of 29.2 percent over the period
2013-2018.
If we talk in terms of industry, then currently consumer
electronics application contributes to more than 99% of the
global gesture recognition market. As per the report
published, the Healthcare application is expected to emerge
as a significant market for gesture recognition technologies
over the next five years. The automotive application for
gesture recognition is expected to be commercialized in
2015.
Factors driving the need for a Solution
There are many solutions provided by top vendors which
cater to the business need of gesture recognition. Each
solution comes with one or more shortcomings. The
solutions available are closely coupled with the underlying
hardware. It allows almost no flexibility for using it with any
other hardware. This is one of the major factors which makes
it impossible to port gesture recognition application
developed on one platform to another.
Solutions available in the market today come up with a list of
pre-defined cameras. The camera interface is either
integrated into the solution or is supported by the solution.
Gesture recognition applications built using these solutions
18
16. are bound to use the specified set of cameras. The
applications are highly dependent on camera capability and
the application development is limited by the set of features
camera is offering.
There could be a situation where the hardware (camera) is
already in place but the solution required to build the gesture
recognition application does not support the camera which
results in extra hardware cost. Today’s solutions are highly
dependent on camera and do not allow the flexibility to use
any other camera.
Gesture recognition is spanning its wings across various
industries like automobile, healthcare, media etc. Solutions
available today are targeting specific market area. The
leading vendors are offering solutions which cater to the
need of specific OEM. If the solution is designed for
developing gesture recognition application for automobile, it
will have the SDK designed specifically for automobile
gesture recognition applications. If a solution is targeting
healthcare, it will have gesture detectors specifically for
developing healthcare gesture recognition applications.
19
17. Upcoming New Technologies
The Sixth Sense Device:-
SixthSense is a wearable gestural interface device
developed by Pranav Mistry, a PhD student in the Fluid
Interfaces Group at the MIT Media Lab. IT is similar to
Telepointer, a neckworn projector/camera system developed
by Media Lab student Steve Mann (which Mann originally
referred to as "Synthetic Synesthesia of the Sixth Sense").
The SixthSense prototype is comprised of a pocket projector,
a mirror and a camera. The hardware components are
coupled in a pendant like mobile wearable device. Both the
projector and the camera are connected to the mobile
computing device in the user‟s pocket. The projector projects
visual information enabling surfaces, walls and physical
objects around us to be used as interfaces; while the camera
recognizes and tracks user's hand gestures and physical
objects using computer-vision based techniques. The
software program processes the video stream data captured
by the camera and tracks the locations of the colored
markers (visual tracking fiducials) at the tip of the user‟s
fingers using simple computer-vision techniques. The
movements and arrangements of these fiducials are
interpreted into gestures that act as interaction instructions
for the projected application interfaces. The maximum
number of tracked fingers is only constrained by the number
of unique fiducials, thus SixthSense also supports multi-
touch and multi-user interaction. The SixthSense prototype
implements several applications that demonstrate the
usefulness, viability and flexibility of the system. The map
application lets the user navigate a map displayed on a
nearby surface using hand gestures, similar to gestures
supported by Multi-Touch based systems, letting the user
zoom in, zoom out or pan using intuitive hand movements.
20
18. The drawing application lets the user draw on any surface by
tracking the fingertip movements of the user‟s index finger.
SixthSense also recognizes user‟s freehand gestures
(postures). For example, the SixthSense system implements
a gestural camera that takes photos of the scene the user is
looking at by detecting the „framing‟ gesture. The user can
stop by any surface or wall and flick through the photos
he/she has taken. SixthSense also lets the user draw icons
or symbols in the air using the movement of the index finger
and recognizes those symbols as interaction instructions. For
example, drawing a magnifying glass symbol takes the user
to the map application or drawing an „@‟ symbol lets the
user check his mail. The SixthSense system also augments
physical objects the user is interacting with by projecting
more information about these objects projected on them.
Construction and Working: -
The SixthSense prototype comprises a pocket projector, a
mirror and a camera contained in a pendant like, wearable
device. Both the projector and the camera are connected to
amobile computing device in the user‟s pocket. The projector
projects visual information enabling surfaces, walls and
physical objects around us to be used as interfaces; while
the camera recognizes and tracks user's hand gestures and
physical objects using computer-vision based techniques.
The software program processes the video stream data
captured by the camera and tracks the locations of the
colored markers (visual tracking fiducials) at the tips of the
user‟s fingers. The movements and arrangements of these
fiducials are interpreted into gestures that act as interaction
instructions for the projected application interfaces.
SixthSense supports multi-touch and multi-user interaction.
21
19. Advantage
Replaces large I/P devices.
Also usefull for Physically handicapped person.
It provides a simple, usable and interesting user
interface and satisfies the need for more freedom in a
human computer interaction environment.
It is considered as a powerful tool for computers to
begin to understand human body language.
It is widely used in various application areas since it
gives the user a new experience of feeling.
Disadvantages
Very costly.
22
20. Conclusion
According to the Markets and Markets analysis, the growth of
gesture recognition is going to be huge. So, we have a huge
opportunity to play in this technology. The end user is now
moving towards a whole new path of human machine
interaction. This is creating a demand for enabling gesture
recognition in every facet of market.
The solution we are proposing will have a mammoth place in
gesture recognition market. Using the solution, we can
develop easily and quickly gesture recognition applications
for various industries. The solution will aim to develop
business tie-ups with major OEMs.
23
