1. Diksha Jain, Meera Mali, Ojas Milind Kale, Sanjeedha Sanofer Raja, Shagun Juneja
CMPE Department, San Jose State University, CA
Abstract: This paper surveys the idea of
augmented reality, which is the incorporation of
digital data such as online information and
graphics with the user's environment in real
time. It makes the end-user’s world more
interactive and digitally manipulable. It is
proving as an extremely demanding technology
these days. This paper presents the concepts and
designs behind building up this technology and
the characteristics of applications using
augmented reality. It focuses on the related
work or the different cases of Augmented
Reality applications using this technology as a
part of different creative ways. It demonstrates
the historical viewpoint of the technology and
the procedure to augment the reality. This
paper summarizes the current problems with
the existing systems and the future areas
requiring further research. The objective of this
survey paper is to provide a general layout, fact
and findings and current analysis on the
ongoing projects based on Augmented Reality
with some future enhancements.
Index Terms: 3-D Graphics with real-world,
augmented reality, project Tango, augmented real
world.
I.INTRODUCTION
Nowadays, researchers and engineers are extracting
graphics out of the TV screen or computer display
and integrating them into real-world environments
[1]. This new innovation, called augmented reality,
blurs the line between what's real and what's
computer-generated by enhancing what we see,
hear, feel and smell. Augmented reality is not just
like a real life but better than it. In a general term,
Augmented Reality is a combination of a real scene
viewed by the end-user and a virtual scene
generated by a computer that augments the scene
with some additional information and meaning to a
real object or a place. It involves AR technology
i.e. addition of computer vision and object
recognition which helps in making a simulated
environment whose components are augmented by
computer-based sensory inputs such as sound,
graphics, video, or GPS data. As a result, this
innovative technology functions by improvising
one’s present aspect of reality.
Unlike virtual reality, which makes a totally
artificial environment, augmented reality uses the
current existing environment and overlays new data
on top of it. The idea behind this is straightforward
and direct: take a real-life scene, or (better) a video
of a scene, and include some kind of logical
information to it. This is how augmented reality
works and thus helps in identifying the things
which are around you, recognizing the people in
the scene and the places you want to visit.
Milgram characterized a continuum of real to
virtual environments, where augmented reality is a
part of “Mixed Reality.” He defined the term
Augmented Virtuality to recognize systems which
are mostly synthetic in nature with some real world
imagery included just like texture mapping video
on the virtual objects.
A. How Does Augmented Reality Work?
The webcam connected to the computer is
responsible for capturing the photos or videos
which captures the images or videos in the
traditional manner (by taking lots of photos
successively i.e. one after the other.) When a
“marker” is placed in front of the webcam, it
recognizes the “marker” and collects the
information or pattern encoded in it and sends this
data to the computer. The computer perceives the
data and overlays the marker with a picture such
that it seems to the viewer that the image has
appeared by magic. The computer is capable
enough to track the size and continuous movement
AUGMENTED REALITY: INTEGRATING THE
DIGITAL WORLD WITH REAL WORLD

2. of the image which means if the marker is brought
closer to the webcam, the image will get magnified.
If the marker is tilted towards left, the image will
also behave in the same manner and thus it appears
to be tilting in the left direction. This procedure is
like sports broadcasts on television, such as
swimming events, a line is powerfully added over
the paths to demonstrate the virtual position of the
present record holder as a race continues.
B. Why Did Augmented Reality Come?
Augmented Reality is a live view of the world from
an indirect point of access. When 2D and 3D
photos can give only a static view of the live world,
people were not satisfied with only a two or a three
dimension. Business people wanted a methodology
to lure more shoppers to their products. They
wanted the products and the specs of the products
to be more reachable to the consumers without
actually buying their products. Medical
professionals need to get more pragmatic contour
of the patient when they are in remote operation.
Gamers need to indulge themselves to the gaming
environment and they want to be the characters
themselves. The main idea for all the people is to
create more realistic and tactile view of the
environment and the object placed in the real
world. The imaginary world of reality must be
interactive and digitally manipulable. This need
provokes the creation of an augmented reality
environment.
II. RELATED WORK
Augmented Reality (AR) has rapidly turned out to
be very well-known and widely used technology
and thus become an expression that trips easily off
technologists' lips; yet we've been seeing various
versions of it for a long while. AR has come quite
close to a wide range of engineers and this
technology is sufficiently effective to make use of
it by making large number of people possessed by
it. The arrival of powerful smartphones and
computers having built-in video capabilities
increases the use of AR and thus people don’t have
to wait for the AR effects as they do with
television. They can simply be overlaid onto real
life. Stepping one step ahead in the same like
Apple's iPhone, and phones based on Google’s
Android as an operating system, they both are
capable of overlaying data or information on top of
an image or video.
A. What Are Some Examples?
1) Intel X-ray-like glasses:
One of the project which is based on AR is Intel X-
ray-like glasses which permit wearers to ‘see
inside’ objects. Smart augmented reality helmet
allows wearers to overlays maps, schematics and
thermal pictures to viably see through walls,
funnels, pipes and other solid objects. Intel has
launched a set of glasses incorporated with a
helmet which used RealSense 3D camera for
providing x-ray-like vision such that viewer can
view inside the object.
This technology makes use of a 3D high-definition
and infrared camera, an infrared laser projector and
software that is used to customize the camera
usable for a particular application. Unlike other
devices, which have been developed for consumers
or end-user’s use, this device is designed and
developed with industrial use in mind. It will allow
the wearer or the person wearing this gadget to
effectively view the actual workings of the objects
using real-time information, such as wiring graphs,
diagrams, charts etc. It will be helpful for the
viewer to find out the problem areas, which needs
further maintenance or needs to get fixed.
Fig 1: A wearable helmet to see the x-ray like vision
.
This gadget makes use of RealSense systems,
which are combined with Intel’s latest Core m7
processor, a 360-degree sensor array, and Daqri’s
computer vision, and tracking system. By making
use of such a high tech objects, it claims to be the
“most powerful augmented reality wearable
device” available in the market.
This device will be useful in various domains such
as aerospace, oil, construction and gas industries.

3. This product of Intel will be available in the market
for purchase in the first quarter of this year i.e.
2016.
2) Google’s Project Tango + Lenovo
Another venture working on AR is Google’s
project Tango. Project Tango consolidates 3D
motion tracking with depth sensing to give mobile
device an ability to know where it is and how
travels through space.
With project Tango, developers or engineers can
develop applications that investigate physical space
around the user, including exact routing without
GPS, windows into virtual 3D worlds, estimation
of spaces, and diversions that know where they are
present in the room and what’s around them [2].
We can understand the meaning of above sentence
as visualizing the actual real world with close eyes.
Google’s Project Tango and Lenovo are bounding
together to make the world’s first smart phone
powered by Project Tango innovation. The device,
will allow the users to experience the world in a
more innovative manner, which was never possible
through a smartphone.
This device will be going to launch in summer
2016 and will be available in the market for end-
user for purchasing.
Fig 2: Sensors used in Project Tango
This technology works by adjoining various inputs
from the pre-installed range of sensors and
processing these inputs into usable information
very rapidly. These sensors consist of an infrared
camera and infrared emitter, which emits radars
and helps in absorbing the reflected light.
A wide-angle camera adds visualization clues
about the current location. This system also
consists of varieties of highly functional
accelerometers, barometers and gyroscopes.
For this technology, Google offers three APIs- first
one for game developers, second one for using Java
to integrate Tango into various mobile apps, and
third one for the apps, which have their own
visualization centric engine to provide
compatibility to Tango.
III. HISTORICAL PERSPECTIVE
Over the past few decades, augmented reality-the
artificial view of the physical world has matured
from a scientist's plaything to a part of daily life.
In 1647 Rene Descartes, a French mathematician
and scientist, mused upon the difference between
what is real and what is constructed in our mind
through a series of thought experiments about
reality, as shown in Fig.3
Fig 3: The Proposed Idea In 1647
Ivan Sutherland, a computer scientist and Harvard
Associate Professor and his student Bob Sproull
invented the first augmented reality head-mounted
display system. As shown in Fig 4, the system was
suspended from a ceiling for the user to experience
the graphics, which was fed by computer programs.
It was primitive both in terms of user interface and
realism. The graphics that comprised the virtual
environment created were very simple wireframe
rooms. Depending on the position of the user's
gaze, the perspective was showed to the user. It
was attached to a mechanical arm suspended from
the ceiling, which tracked the head movements.
The translucence nature of the system where the
unit was partially see-through along with its other

4. features makes it the perfect predecessor to
augmented reality technology.
Fig.4 First Augmented Reality System in 1968.
In 1990, Tom Caudell, a Boeing researcher
introduced the term "augmented reality". In 1994,
Julie Martin took augmented reality to television by
creating the Theater production "dancing in
Cyberspace”. It featured dancers and acrobats
manipulating virtual objects in real time, which
were projected into the same performance plane
and physical space.
In 1997, Ronald T. Azuma's research brought light
into the different uses of augmented reality in
medical, manufacturing, research, mechanical
operation and entertainment by publishing "A
Survey of Augmented Reality".
In 2004, a group of German researchers projected
the idea of see-through augmented reality to mobile
phones via a weak perspective projection camera
model and an OpenGL rendering pipeline [3].
Later in the 2000s, Hirokazu Kato from Nara
Institute of Technology developed the ARToolKit -
an open-source computer-tracking library for
creation of strong augmented reality applications. It
combines virtual graphics with real life, and uses
video tracking to overlap computer graphics on a
video camera in turn resolving the two problems of
viewpoint tracking and virtual object interaction.
To this day the world is seeing augmented reality
growing and applied in many domains and it is
continuing to create revolutions in computer
interface design.
IV. HOW TO AUGMENT THE REALITY
There are three basic criteria for augmenting the
real world scenario
1) User Augmentation
The User uses a wearable technology to get the feel
of the reality surrounding them as in fig 1. The
wearable computing is actually a physical device,
usually worn on the head or hands to gain
knowledge about the physical objects [4].
The earliest device used was a head-mounted
display, which allowed the used to view and feel
artificially, designed objects and become immersed
in a world of virtual reality. Then, data gloves were
introduced to control slides and videos during
presentations. The presentation movements can be
controlled by specialized gestures. Some
augmentation allowed users to view through the
objects just like a mother feeling the baby inside
her during ultrasound. Knowledge- Based
Augmented Reality for Maintenance Assistance
(KARMA) was designed to help the technicians
see through the relevant repaired parts and correct
the damage in real-time. These devices involve
coupling of the images and the view of the physical
world.
Fig 5: A wearable headset consisting the sensor for augmented
reality
2) Physical Object Augmentation
Reality is infused into the physical objects by
transforming the object’s reality by embedding the
input, output and the computational devices
associated with the object.
In the early 70s, Lego/LOGO language laid the
basic foundation for augmentation of the physical

5. objects. The language was designed to help the
kids play with the electronic blocks. The blocks
contained a type of sensor, which helped to control
the construction. Other sensors and objects can be
added when necessary. This approach paved the
way for ubiquitous computing which detects
specialized objects by sensors placed all over the
place. Another example would be the use of active
badges, which help support collaborative activities,
like sharing documents, reminding events and
meetings.
3) Environment Augmentation
Instead of affecting the user or the object, the
knowledge of the environment surrounding the
user or the object is provided by devices. The
devices capture the information about the user’s
interactions and display the information unto
objects.
Imagine a user sitting at a place and controlling the
movements of an object by directing commands to
move the object. Sensors are used to detect the
gesture of the users. At the earliest period, video
camera and a close-up camera is used to capture
the images, which have now been replaced with
sophisticated cameras and sensors. In all these
cases of augmenting the environment, special
devices are not needed to visualize the reality.
A. Components of AR
1) Camera Data
The live-data capture of the augmented reality is
done by camera. Camera feed is the essential part
in generating the query for the technology. Using
the camera lens, the image for which the
information is to be populated can be captured as in
fig 2. For example, by pointing the camera to an
object, knowledge can be gained about the details
of the object. Camera APIs are used to analyze the
frame data. Methods such as setPreviewDisplay()
and setPreviewCallback() are used to display the
camera data.
Fig 6: An augmented Reality Camera setup to capture the live
feed
2) Location Data
Only the camera data would not suffice. There also
rises a demand to determine the location of the
device and the user accessing the device. The
location API can help to listen to the location and
populate the events based on the location mostly
from the live feed. OpenCV libraries are used for
the same to augment data. The location also
depends on the tagged information or the marker
info.
3) Sensor Data
Sensors are useful to determine the connection
between the user’s gestures and the objects [5]. The
orientation of the device plays a major role in
synchronization of the camera with the live feed.
Sensor APIs and methods such as
Sensor.TYPE_MAGNETIC_FIELD,Sensor.TYPE
_ACCELEROMETER help to calculate the
accurate position of the devices. The use of sensors
allow the user to move the device in any direction
and also experience live changes in the screen with
relation to the reality. For example, Google Maps
or the Street view helps to explore the live data
according to the device orientation.
4) Graphics Overlay
The concept of augmenting the reality is to add
layers above the camera feed data, [5] which makes
the user to experience live scenarios. Each frame
from the camera feed is overlaid and a frame is
drawn corresponding to each screen. Depending on
the selected 2D or 3D, WebGL packages, the
frames are rendered.

6. 5) Storage
The data for augmentation comes from the
database, which might be stored locally or on
remote servers. For Example, SQLite database is
used for quick and easy search. For web-based data
access, web services are used which takes the use
of methods such as HTTP or XML parsing.
B. Open Technologies
1) WebGL
WebGL is a JavaScript API for 2D/3D graphics
rendering. It is run on any compatible browsers. It
utilizes the effects of physical motion and image
processing with additional effects to render the
complex image on the canvas. WebGL is based on
OpenGL standards and HTML5 Canvas elements
and can be accessed by using the Document Object
Model in JavaScript hierarchy.
var canvas = document.getElementById
(‘mycanvas’);
var gl = canvas.getContext(‘webgl2’);
It does not require additional installation
capabilities and has sufficient interfaces and
extensions to render images accurately.
2) ThreeJS
ThreeJS is a JavaScript API to create and display
3D graphics using WebGL on a web browser. It
does not need any plugins. The 3D graphics can be
rendered by creating camera, objects, lights and
other materials. The scene to be rendered can be
decided by the usage of HTML5 canvas, WebGL
or SVG.
var renderer = new THREE.WebGLRenderer();
3) WebRTC
WebRTC enables audio/video streaming and data
sharing between the server and the client. WebRTC
provides conferencing facilities which aims to
exchange information between peers without the
inclusion of any third party network. It uses
JavaScript and net stream API’s. There is also a
data stream API that helps in real-time text and file
transfer.
4) Awe.js
Awe.js is a JavaScript library for rendering
augmented reality. It also aids in creating and
manipulation of the objects using JavaScript.
Awe.js uses Three.js in addition with JavaScript to
create the effect in the browser. It provides four
scenarios to experience Augmented Reality –
geo_ar for mapping compass points, grift_ar for
making environments compatible with Oculus Rift,
leap_ar for leap motion control and marker_ar to
create experiments on AR.
5) ARToolKit
ARToolKit is a library for constructing the
Augmented Reality applications. It solves the main
problem of determining the viewpoint of the user.
This toolkit uses vision algorithms to calculate the
camera positions and physical markers in the real-
time. It defines the calibration of the device to
position the viewport for the device.
6) Blend4Web
It is a web-oriented 3D engine for authoring and
interactive rendering of the scenes on the context of
3D graphics and audio in the web. It is tightly
coupled with the Blender, which is a 3D animation
tool using WebGL rendering. It also has debugging
and optimization features.
7) BeyondAR
BeyondAR is a framework designed for augmented
reality on smartphones and tablets. It deals with
geolocation based Augmented Reality on android
environment and gaming. The base concept is to
use OpenGL libraries for the scene rendering.
C. Next Steps
1) Localization and Registration
There arises a need to devise a way to localize a
device when performing outdoor navigation.
Earlier systems used Global Positioning System for
Outdoor usage. This GPS was used as a compass
for sensing the direction and also for position
measurements. However, the accuracy of the GPS
is still questionable and it creates large problems
for registration of virtual objects since GPS signals
deteriorate in urban surroundings and building
blockages. Similarly, there is a drift on the sensor
data when disturbed by shadowing. For a large
algorithm computation, there exists only a small
mobile computing processor, which also poses a
problem. Hence for good localization criteria,
image-matching and geo-location sensors must be
working in coordination to provide a robust
platform.

7. 2) User Evaluation
The user prototype must be evaluated to provide
guidance to application development in augmented
reality [6]. Evaluation methods for user prototypes
for mobile devices are a field, which is emerging.
As of now, the prototypes are concentrated only
upon a small arena. The user evaluation must be
done for large-scale use and cover a wider arena of
social networking applications.
3) Application Development
The application areas of developing augmented
reality are more and so are the challenges for
establishing the hardware and the software for
building the environment. For example, to design a
geo localization augmented reality, the device has
to plan for city exploration, urban locations,
tagging the unknown places, getting the culture
information, 3D reconstruction of sites and urban
maintenance.
V. ISSUES WITH AUGMENTED REALITY
With this evolving technology, there are some
issues or problems, which restricts this technology
to be used in every domain.
1) Profiling
As the name suggests this includes the profiling of
person it may include his/ her personal data as well
as some other information. This typically includes
Face recognition techniques, geo-location with co-
ordination of data from augmented reality. The
combination of these technology will lead in
integration of Offline as well as online live of the
person. With this progress comes the drawback or
concern, which states that, a person now walking
will no longer be just a physical Human being, but
he/she will be walking with his digital profile as
well. And this can lead in myriads of problem for
the person some of them would be, getting solicited
by the society or a group of people because of
certain believes or Political or Religious stands. Or
maybe singled away for additional Security check
on public transport stations, such as Airports, Train
stations etc.
2) Unauthorized Advertising
Advertising your product is the need of today’s
market, and augmented reality can play vital part in
this. Companies these days are thinking about the
possibilities of using simple objects and physical
spaces around. Each of them can possibly use as
the digital ads board which can hold ads onto them
real-time. The dark side of this possibility is, when
you think about the private belongings and
intellectual properties being used for the
advertising purpose without prior permission can
lead into a big problem of “Unauthorized
advertising”. If the advertising using augmented
reality does not come with the inbuilt controls
which prevent them from using any private or
intellectual property, this can be the biggest
concern in augmented media domain.
3) Physical danger
One of the biggest concerns of the augmented
reality. Think of the scenario where mobile phones
are currently distracting while driving, now think
about the augmented windshield providing you
continuous feeds of driving directions along with
more data of what is happening around you,
providing you live feed of current Football game,
highlights of the presidential election debate and
what not. Moving ahead imagine crossing a street
busy with heavy traffic in unfamiliar neighborhood,
while feeding you with tweets and snap-chats from
your friends and simultaneously giving you
information about nearest restaurant along with
some advertise. Thus, if the augmented reality is
not used smartly can lead into chaos.
4) Spam
When there is an opportunity for advertising there
will be an opportunity for spamming. Deceptive
advertising can lead customers to buy thing they
don’t really need. Augmented reality can depict a
product bigger and better than actual. Although
there are lot of legitimate companies that are
harvesting your shared data, more than them, there
are scammers who have all your data that can be
used to lure you in too good to be true augmented
offer in real world.
VI. APPLICATION & FUTURE SCOPE
With the bulk of possibilities that Augmented
opens for the human kind, there is at least one
application of Augmented in almost every field. To
name a few, the military uses augmented reality to
help men and women making repairs in the field.
The gaming industry is moving games outside like
the old days equipped with wearable headgear of
course.

8. 1) Navigation
Navigation using augmented reality is a real world
example of how this technology can make human
life easier and better. It makes use of Global
Positioning system hand in hand with augmented
reality, to make it easier to travel from your
position to your destination. “Wikitude Drive” is
one of the examples of this technology. It uses GPS
along with phones camera to depict the exact
picture of what is lying in front of your vehicle.
This is the 21st
century map.
Fig 7: Camera along with the Car’s GPS to show the road
ahead.
Fig 7 shows the camera is used along with the
Car’s GPS to see the road ahead. It is also
providing many other information such as distance,
speed probable reach time etc. [7].
2) Sightseeing
The use of technology in the domain of tourism
industry. To think of a best scenario, think of
tourist for the museum. The ability to augment a
live view of all the scenes in a museum with all the
statistics of facts and figures is a natural use of the
technology. Their interactive kiosk solution allows
guests to interact with the display in 3D.
Also out in the real world, sightseeing has been
enhanced using augmented reality. Using
augmented reality, tourists can walk through
historic sites and see facts and figures representing
all the live data. These applications use GPS and
image recognition technology to look up data from
an online database. There could be another
advantage of this technology that a picture can
show up how the exact same place would look like
200 years ago.
3) Military
One of the most productive use of augmented
reality is in Military domain. The Heads-Up
Display (HUD) is the typical example of
augmented reality in military applications of the
technology. The term "heads-up" comes from the
fact that the pilot doesn't have to look down at the
aircraft's instrumentation to get the data they need.
It exhibits a transparent display that is positioned
directly in the fighter pilots view. Data typically
displayed to the pilot includes altitude above the
ground, airspeed and the description of the horizon
line in addition to other critical data.
The Head-Mounted Display (HMD) is used by
ground troops as well. Critical data such as enemy
location can be presented to the soldier within their
line of sight [8].
4) Medical
There are many interesting advantages of
augmented reality in medical application. Medical
students use this technology to practice surgery in a
more controlled environment. Visualizations helps
in explaining and understanding the complex
medical conditions to various patients. AR helps in
reducing the risk of an operation by providing the
surgeon with an improvised sensory perception.
This technology can also be combined with MRI or
X-ray systems to bring everything into a single
view for the Doctor. Neurosurgery takes the first
position when it comes to surgical applications of
augmented reality. The ability to show the image of
the brain in 3D on top of the patient's actual
anatomy is very important for the surgeon. Since
the brain is somewhat fixed compared to other
parts of the body, thus the registration of exact
coordinates can be achieved for the brain. But
concern still exists surrounding the movement of
tissue during surgery, might affect the predefined
coordinates to move a little bit.
VII. CONCLUSION
In this paper, we have described the concept of
augmented reality, which can be widely deployed
on handheld devices and at consumer level.
Furthermore, we have showed the working of
augmented reality and the requirements, which are
essential to develop location-based, augmented
reality experiences that can be flourished on a
broader or global level. In the future, various
domains such as Education, E-commerce and

9. digital marketing can make use of augmented
reality for further development.
VIII REFERENCES
[1]. Ronal T. Azuma, Tele operators and Virtual
Environments 6, 4 (August 1997), 355-385.
[2]. Adrian Kingsley-Hughes, August 19, 2014.
Inside Google's Project Tango tablet.
http://www.zdnet.com/insidegoogles-project-tango-
tablet-7000032746/.
[3]. M. Mohring; Bauhaus Univ., Weimar,
Germany; C.Lessig; O. Bimber: Video see-through
AR on consumer cell-phones. 2, 5 (Nov. 2004),
252-253
[4] Azuma, R.T., A Survey of Augmented Reality.
Presence: Teleoperators and Virtual Environments
6, 4 (Aug 1997), 355–385, 1997
[5] MacIntyre B., Gandy M., Dow S., and Bolter
J.D., DART: A Toolkit for Rapid Design
Exploration of Augmented Reality Experiences,
User Interface Software and Technology
(UIST’04), Sante Fe, New Mexico, 2004
[6] Wagner D., Mulloni A., Langlotz T., and
Schmalstieg D., Real-time Panoramic Mapping and
Tracking on Mobile Phones. In Proc. of Virtual
Reality’10, pp. 211–218, 2010
[7].Burnett, G. E. A Road-Based Evaluation of a
Head-Up Display for Presenting Navigation
Information. In Proc. HCI Int’l Conference (2003).
[8] E. Foxin, M. Harrington: WearTrack: A Self-
Referenced Head and Tracker for Wearable
Computers and Portable VR, Proceeding of the
Forth International Symposium on Wearable
Computers, pp. 155-162, 2000. M. Young, the
Technical Writer's Handbook. Mill Valley, CA:
University Science, 1989.