Augmented Reality (AR) superimposes computer-generated graphics onto the user's view of the real world. Today, AR applications can occasionally be found in the late phases of the product development process (PDP), such as in training and in maintenance. Head-mounted displays (HMDs) are typically used for AR applications. HMDs still show deficiencies in terms of ergonomics and image quality. Looking at the development of VR technology, it can be stated that VR began its triumphal procession only after replacing HMDs by immersive projection technology (IPT) such as PowerWalls, CAVEs, Virtual Tables. Projection-based Augmented Reality enhances such environments towards augmented reality applica- tions. We believe that this concept opens new application possibilities for AR – especially in the early phases of the product development process. In this paper we will describe the concept of Projection-based Augmented Reality (PBAR), introduce current hardware prototypes and discuss applications of PBAR configurations within the following areas: augmented design review, hybrid assembling simulation, hybrid modeling/sketching and visual inspection of augmented physical parts.

1. Projection-Based Augmented Reality
... in Engineering Applications
André Stork / Oliver Bimber / R. de Amicis
Fraunhofer-IGD, Darmstadt
Fraunhofer CRCG, Providence
GRIS, TU Darmstadt
The Magic Mirror
M.C. Escher, 1946

2. PBAR – Projection-based Augmented Reality
André Stork, Oliver Bimber,
R. de Amicis
Overview
• Motivation
• Introduction
• The Extended Virtual Table
• Engineering applications
• Conclusions and future work
Still Life with Mirror
M.C. Escher, 1934

3. PBAR – Projection-based Augmented Reality
André Stork, Oliver Bimber,
R. de Amicis
Motivation
projection-based
(SSP: CAVEs, domes, etc. /
workbenches, walls, etc.)
projection-based
(SSP: CAVEs, domes, etc. /
workbenches, walls, etc.)
HMDs
(opaque)
screen-based
(fish-tank)
VR ...
screen-based
(video-mixing)
HMDs
(see-through or video-mixing)
... AR
Mixed Reality (MR)
Augmented
Reality (AR)
Real
Environment
Augmented
Virtuality (AV)
Virtual
Environment
Virtuality Continuum (VC)
Paul
Milgram

4. PBAR – Projection-based Augmented Reality
André Stork, Oliver Bimber,
R. de Amicis
Motivation
Why are projection-based VR systems attractive?
• scalable
 resolution
 FOV/deg. of
immersion
• integratable into real
environment
• light weight/no glasses
• focal length
• reduced motion-
sickness (reduced
swimming effects)
technological reasons ergonomic reasons economic reasons
• well established and
applied
• wide variety of
systems on market
• application-specific
functionality (not all-
purpose)

5. PBAR – Projection-based Augmented Reality
André Stork, Oliver Bimber,
R. de Amicis
Motivation
What could be the benefits of a projection-based AR approach?
Adoption of the technological, ergonomic and economic advantages
of well-established projection-based VR systems
 technology
 concepts
 applications
 experience
 etc.
Possibility of a VR/AR combination
Higher acceptance Hand with Reflecting Sphere
M.C. Escher, 1935

6. PBAR – Projection-based Augmented Reality
André Stork, Oliver Bimber,
R. de Amicis
Motivation
What are the main limitations of projection-based VR systems ?
• Front/rear-projection systems cause occlusion problems
• Semi-immersive systems are constrained to a limited viewing space
(window violation)
What would be the main limitations of projection-based AR
systems ?
• Mobility (devices are usually stationary)
• Cost (usually higher)
• Multi-user (support a limited number of users simultaneously)
• Not all-purpose (have application-specific functionality)
• Interactivity

7. PBAR – Projection-based Augmented Reality
André Stork, Oliver Bimber,
R. de Amicis
Introduction
What is our approach ?
• Integration of mirror-beam splitters (half-silvered mirrors) into
today’s projection-based VR systems
• Mirror-beam splitters with:
 different geometry (e.g. planar, convex, concave)
 different configurations (e.g. single or multiple inter-playing mirrors)
• Projection system
 non-immersive, semi-immersive, immersive
• View-dependent off-axis rendering (single/multiple viewer)

8. PBAR – Projection-based Augmented Reality
André Stork, Oliver Bimber,
R. de Amicis
Introduction
What are the challenges ?
Still Life with Reflecting Sphere
M.C. Escher, 1934
• Real-time stereoscopic rendering
 mirror/view dependent scene transformation
(affine/non-affine)
 correct illumination
 correct depth-handling
• Distortion correction
 optical (e.g. projector-geometry, refraction,
mirror flexion, curved optics)
 non-optical (e.g. tracking)
• Appropriate interaction

9. PBAR – Projection-based Augmented Reality
André Stork, Oliver Bimber,
R. de Amicis
The Extended Virtual Table
40”x40”x10mm half mirror (float-glass)
Angle between projection plane and
mirror can be changed by rotating the
mirror
Physical Arrangement

10. PBAR – Projection-based Augmented Reality
André Stork, Oliver Bimber,
R. de Amicis
Working principle
The Extended Virtual Table

11. PBAR – Projection-based Augmented Reality
André Stork, Oliver Bimber,
R. de Amicis
The Extended Virtual Table
VR Mode
Gaze-directed (non-
simultaneous) visual
extension of virtual
environments
Projection PlaneMirror

12. PBAR – Projection-based Augmented Reality
André Stork, Oliver Bimber,
R. de Amicis
The Extended Virtual Table
AR Mode
Virtual-real object
occlusion through
phantoms (Breen et al.,
1996)
Ratio of intensity of the
transmitted light and the
reflected light depends on
angle between mirror and
projection plane
AND
on the lighting situation
60°
80°
no clipping
plane

13. PBAR – Projection-based Augmented Reality
André Stork, Oliver Bimber,
R. de Amicis
Use of VR and AR within the PLC
VR as key factor for industry and key
technology to remain competitive.
VR replaces expensive physical
prototypes through its digital
counterpart within the product
development process
AR small brother of VR ( yet !)
First prototypes have been developed in
the construction, training and
maintenance area
Augmented Prototyping as new field of
application
Product
Definition
Design Analysis
Produc-
tion
Planning
Produc-
tion
Use
Mainte-
nance Recycling
Product life cycle
Engineering applications

14. PBAR – Projection-based Augmented Reality
André Stork, Oliver Bimber,
R. de Amicis
Engineering applications
New kinds of applications:
 Hybrid Assembling
 Augmented Design Review
 Hybrid Modeling and sketching
 Visual Inspections of modeled parts
 etc., e.g. hybrid ergonomic design PMU
VMU

15. PBAR – Projection-based Augmented Reality
André Stork, Oliver Bimber,
R. de Amicis
Hybrid assembling
Manipulating the distant
object ‚behind‘ the mirror
Occlusion of the virtual part
through the real part
Direct manipulation
above the VT
Move the virtual object
‚through‘ the mirror
Engineering applications

16. PBAR – Projection-based Augmented Reality
André Stork, Oliver Bimber,
R. de Amicis
Visual inspection of molded parts
Engineering applications

17. PBAR – Projection-based Augmented Reality
André Stork, Oliver Bimber,
R. de Amicis
Augmented Prototyping
VR :
• Visualisation and interaction
• BUT limited haptic feedback
Rapid Prototyping (RP) :
• Fast physical prototype
• good haptic feedback
• BUT only one finish at a time
 Augmented Prototyping
Engineering applications

18. PBAR – Projection-based Augmented Reality
André Stork, Oliver Bimber,
R. de Amicis
Sketching free-form surfaces in Mixed Realities
Engineering applications

19. PBAR – Projection-based Augmented Reality
André Stork, Oliver Bimber,
R. de Amicis
Engineering applications
PMU
VMU
Hybrid Mock-Up
(HMU)
Hybrid assembling
Direct interaction with physical mock-up through
reach-in set-up

20. PBAR – Projection-based Augmented Reality
André Stork, Oliver Bimber,
R. de Amicis
Ceiling projection
Two beamers
Project pre-filtered stereo images simultaneously
Passive shuttering using polarized light
Alternative set-ups
The Extended Virtual Table

21. PBAR – Projection-based Augmented Reality
André Stork, Oliver Bimber,
R. de Amicis
Augmented Design Review
The Virtual Showcase:
Conic half-silver mirror
‚walk-arounds‘
Engineering applications

22. PBAR – Projection-based Augmented Reality
André Stork, Oliver Bimber,
R. de Amicis
Multi-user AR-Setup
Half-silver mirror
• On a projection plane several views are presented
perspectively correct
• A real object inside the mirror prism can be augmented
Future steps

23. PBAR – Projection-based Augmented Reality
André Stork, Oliver Bimber,
R. de Amicis
Conclusions
VR
AR
Product
Definition
Design Analysis
Produc-
tion
Planning
Produc-
tion
Use
Mainte-
nance Recycling
Product life cycle

24. PBAR – Projection-based Augmented Reality
André Stork, Oliver Bimber,
R. de Amicis
IGD-Darmstadt
Univ. of Technology-Darmstadt
ZGDV-Darmstadt CAPCom business & solutions -
Darmstadt
IGD-Frankfurt Univ. of Frankfurt
IGD-Rostock
Univ. of Rostock
ZGDV-Rostock
Ctr. for Research in Computer Graphics-Providence
Brown Univ.- Providence
Rhode Island School of
Design- Providence
Univ. do Minho- Azurem Centro de Computacao Grafica Univ. of Technology-Nayang Ctr. for Advanced Media Technology- Nayang
Univ. of Technology-Vienna
Thank you !