1. Implementing Gesture Recognition
The first question that aroused my curiosity was, ‘What exactly is a ‘gesture?’ According
to a dictionary, gesture is a sequence of postures that are connected by motions over a
short period of time. I did not find that really helpful. I’ll explain it in an easier way—a
gesture is a form of non-verbal or non-vocal communication. So your messages are
communicated by visible bodily actions like waving, swiping, facial movements or even
pointing to the sky.
What is gesture recognition
This technology enables devices to interpret human gestures using sensors, whose data
is then processed using mathematical algorithms. The system is built to identify specific
human gestures, and then use them either to convey information or to control a device.
The gestures used have to be intuitive, simple and universally acceptable to ensure they
are easily adopted by users.
Gesture recognition makes computers more accessible for the physically-challenged and
also makes interaction more natural in the 3D virtual world or in gaming. In today’s
world, gesture recognition is an evolving technology that can be seen all around us. The
latest phones, computers, gaming consoles, and even TVs now feature it.
How it works
The human body’s movements are read by an imaging sensor like a camera, which then
sends that data to a computer. The computer uses the gestures captured as inputs,
processes this input to understand what was gestured, and then sends the answer as a
command to the currently running application or control device.
Let us look at an example of a person clapping his hands in front of a camera. When this
clapping gesture data is processed by the connected computer, it could produce the
sound of cymbals, as programmed. The specialty of this technology is that users do not
have to wear any special equipment or attach any device to their bodies, as is the case
with some alternative technologies.
Sensing the gestures
When you gesture at your computer or mobile device, it obviously needs a way to ‘see’
or ‘sense’ what you are doing. This is where sensors come in. There are a number of
sensing technologies that can be considered, but we’ll stick to the popular ones here.
Accelerometers. Also known as inertial or acceleration sensors, these are used to detect
the changes in force resulting from the fall, motion, tilt, shock, positioning or vibration of
the body they are connected to, with respect to time. These measure linear acceleration,
so if the device is in a free fall, the accelerometer will give a ‘0’ value for acceleration.
Application. In the case of a smartphone, the direction of the screen changes as soon as
we tilt or turn the phone. The direction in which we hold the phone (horizontal or
vertical) is sensed by an accelerometer and the information in the form of electrical
signals is sent to the screen, which automatically adjusts itself either in landscape or in
portrait mode. So, here, contact gesture recognition plays an important role.
Accelerometers have paved the way for many games to be developed for smartphones,
especially for some of those sports simulation games that let you use the phone as a
steering wheel or a fishing pole. To get an idea of the range of accelerometers that you
can select from, refer to Table I.

3. Gyroscopes. Gyroscopes, also popularly known as gyros, have had a major impact in the
world of gesture recognition. Their ability to measure the orientation of a device enables
them to measure both vertical and horizontal rotation. This has brought smoothness and
preciseness to the operation of a mobile device. The next time you hold a phone on its
side, and the screen transforms into landscape mode automatically, you know which
component to thank.
Application. In the case of a smartphone or a tablet, adding a gyro and an accelerometer
allows the device to sense motion on six axes including roll, pitch and yaw—which helps
you fly your airplane in a flight simulator game on a phone. So if accelerometers have
given more business to game developers, gyros have helped them be more innovative.
Bend sensors. These are also known as flex sensors. As we know, even finger
movements are considered as gestures, so bend sensors help us to identify those
gestures by recognising finger postures. They work on the principle that resistance
increases with an increase in the bend.
Data gloves. Gesture recognition technology has not only been implemented in
smartphones and tablets, but also in television sets and computers. In order to
implement hand gesture recognition on these devices, data gloves have been developed.
These gloves identify the finger and hand gestures in real time scenario. These are wired
either to a TV set or to a computer and then worn by the person operating the device.
These act as an input device and eliminate the use of a mouse or a keyboard.

4. Application. The bend sensors are placed on the joints of all the fingers in the data glove,
so the more the fingers are bent, the more is the resistance which is introduced in the
device for further action.
Infrared proximity sensors. These have the ability to detect the presence of nearby
objects without any physical contact. They emit an electromagnetic field and then
identify the changes either in the field or in the return signal, which they detect after
reflection from an object that has got close to the sensor. There are many types of
proximity sensors; however, infrared proximity sensors play a very important role in
gesture recognition. As soon as these infrared sensors recognise the movement of a
body in front of them, they pass a signal to the device for further action.
Application. Many of us have seen a wash basin with an automated tap. The tap turns on
as soon as it senses some hand movement under it. This is because of the presence of
infrared proximity sensors that detect your hand movement. Here’s a fun thing to try
out—if you keep your hand under the tap without any movement, the tap would not
allow the water to fall. This is because if you keep your hand immobile, the sensor does
not sense a changing field and thus thinks that your hand is not in front of it any more.
Capacitive proximity sensors. They support gesture recognition enabling robust, easy-to-
use and feature-rich user touch interfaces on a device—these are what make up your
capacitive touch screen. Such sensors allow users to operate the device just by the touch
of their skin or finger rather than by applying pressure. Some of the newer models have
superior sensors that do not even require a touch—hovering above the screen is enough
to get their attention.
Application. This has helped to bring about a new era of devices with superior touch
screens that allow multiple points of input, and better durability. What’s more, capacitive
screens allow the device to have a glass or some other rigid protection on top of the
touch screen, thus improving the device’s ergonomics.
Touch-sensing controllers, such as the CapSense from Cypress, have specific technical
improvements like water tolerant capacitive touch buttons, which eliminates false
touches under moist conditions. “They also allow capacitive proximity sensing for up to
30cm, which is instrumental in enabling features such as ‘wake-on approach,’ and can
even replace the IR sensors mentioned in the previous section,” explains Anbarasu
Samiappan, product apps manager, Cypress Semiconductor.

5. Microsoft’s Digits prototype is part of an effort to bring gesture-based
control in a mobile device. This wrist-worn sensor uses IR laser, LEDs and
camera to recognise the fingers movement (Image courtesy: Microsoft)
What’s coming up in the future
I am pretty sure we will see something like Tom Cruise’s sorcery in ‘Minority Report’
within a couple of decades. But until that happens, the field is taking great strides
forward.
Researchers have been working on new kinds of hardware to compute gestures,
including FPGAs. “Apart from basic components to capture the images, an FPGA-based
gesture recognition system handling large computational complexity of optical flow to
perform parallel processing is a low-cost solution. Using an FPGA system, gesture
recognition can be implemented at 30 frames per second, and the system software can
subsequently schedule all tasks during processing,” explains P. Chow Reddy, manager
(R&D), Power Division, ICOMM Tele Pvt Ltd.
Apart from the focus in consumer electronics, gesture recognition seems to have
matured enough to be noticed and implemented in industrial settings. The BMW Group
has demonstrated a gesture-driven system that allows a quality assurance worker to
“…examine and document flaws in a component simply by pointing.” Ford had earlier
announced its own gesture-based ambitions in the form of ‘a complete virtual factory.’
Industry aside, even Microsoft Office 2013 has gesture recognition built into it.
At Electronics Rocks‘13 conference, one of our speakers Hemant Surale showcased a
cool smart glove product that he had designed. Hemant says that the present scenario in
gesture recognition seems that development in this area is at its peak. “Most of the
devices comes with their own proprietary SDKs for application developers. In addition
there are many emerging startups developing effective way to interact with day to day
equipments. But, Internet of things will present new challenges with respect to
integration of such systems into day to day life of users.” Hemant is also an active
member of the IoTBLR community. IoTBLR is a community based in Bangalore, which is
excited about connecting disparate devices to one another via the web to create Internet
of Things products. Any engineer can find them on facebook at
facebook.com/groups/IoTBLR
The field of gesture recognition seems to be gearing up for quite a future, and design
houses working on this technology better get ready for the competition that comes with
any high-profile industry.