VR/AR needs to be multi-modal and multi-format

Spatial Augmented Reality | Raskar | MIT Media Lab
Ramesh Raskar
MIT Media Lab
Cambridge, MA
Spatial Augmented Reality:
Overlay Inserting in the Real World

Multi-ModeMulti-Format

Augmented Reality (AR)
Augmented Human: Beyond X-ray vision
© Andrei State

World Input Process Output World
Sense beyond
human ability Abstract
Present
within human
comprehensibility

Camera,
Location
Sensors
Identify,
Track,
Abstract
Display,
Overlay
Interaction

Camera,
Location
Sensors
Identify,
Track,
Abstract
Display,
Overlay
Interaction
Henry FuchsCyborg, Bionic Eye, HMD

Bits
Photons
CV/Machine Learning
Optics
Sensors
Computational Imaging
Computational
Photography
AR – Imaging Centric View
Signal Processing
Light Transport
Displays

MR Laboratory’s COASTAR HMD
(Co-Optical Axis See-Through Augmented Reality)
Parallax-free video see-through HMD
Camera + Viewmaster LCD Display

MicroOptical display
Courtesy: Sam Ogden
Nomad display
Courtesy: Microvision
Courtesy: Azuma 1995
Near Eye Displays

Real world, Tracking free, Gear free,
Multi person, All 3D Cues

Head-mounted Display Projectors
© Andrei State
Multi-Mode Augmented Reality

hand-held
display
spatial
see-through
display
head-
mounted
display
retinal
display
Head-attached Body-attached spatial
projector
Spatial Augmented Reality
1998
Multi-Mode AR
Real
Object

Virtual Reflectance Virtual Illumination
InteractionVirtual Motion
ShaderLamps
1997 - 2000

Virtual Reflectance Virtual Illumination
Lamps for Colorfully Shading
the World
1997 - 2000

Shader Lamps on a Saturn

Guerilla Marketing
FHM Magazine

VienViewer (Luminetx)
Near-IR camera locates subcutaneous veins and project
their location onto surface of the skin.
Coaxial IR camera
+ Projector

Hong Hua, U of Arizona
Projective HMD
Jannick Rolland (U. Central Florida)
Frank Biocca (Michigan St. U.)

Camera,
Location
Sensors
Identify,
Track,
Abstract
Display,
Overlay
Interaction
Self-ID
(barcodes)

AR Issues
• Preprocessing:
– Authoring
• Runtime:
– Identification: Recognition of objects
• Using markers and visual tags
– Registration: Finding relative pose of display device
• Dynamic estimate of translation and rotation
• Render/Warp images
– Interaction:
• Widgets, Gesture recognition, Visual feedback

Siggraph 2004
Where are products
about to expire ?

ARToolKit
[Kato and Billinghurst 1999]
ARTag
[Fiala 2005]
Self-ID via Barcodes for
Camera
Phones

Bokode

Defocus
blur of
Bokode

Pieces of the Puzzle
Actuated Surfaces
Actuated Workbench [Pangaro, Maynes-Aminzade, Ishii UIST 2002]

Bits
Photons
CV/Machine Learning
Optics
Sensors
Computational Imaging
Computational
Photography
AR Centric View
Signal Processing
Light Transport
Displays

Camera,
Location
Sensors
Identify,
Track,
Abstract
Display,
Overlay
Interaction
Self-ID

Materials: Displays Contenders
Organic
LED
Light
Emitting
Polymers
E-Ink

Dream Material: Sticky Pixels
Project Once, pixels persist, day/night
Phosphorescent
Photochromic
+
Saakes, Chiu, Hutchinson, Naoya, Inami, Raskar 2010, slowdisplay.org

Oliver Bimber
Projecting on
Holograms
Hologram
Projected + Hologram

Camera,
Location
Sensors
Identify,
Track,
Abstract
Display,
Overlay
Interaction
Shape &
Appearance
Morphing
Self-Id

John Fan
Don’t overwhelm
Pattie Maes:
Monocle
Hiroshi Ishii
Tangible Bits
Mark Billinghurst
Fundamentals!
Steve Feiner
50 Years since Sutherland

Poor Man’s Palace

Poor Man’s Palace
Augment the world, projectors and proxy geometry, controlling light every millimeter at every millisecond.

Acknowledgements
• Camera Culture Group, MIT Media Lab
• MERL
• Amit Agrawal, Ashok Veeraraghavan, Ankit Mohan, Jack
Tumblin, Jeroen van Baar, Paul Beardsley, Remo Ziegler,
Thomas Willwacher, Srinivas Rao, Cliff Forlines, Paul Dietz,
Joe Marks, Darren Leigh
• Office of the Future group at UNC Chapel Hill
• Greg Welch, Kok-lim Low, Deepak B’padhyay, Aditi Majumder,
Michael Brown, Ruigang Yang
• Henry Fuchs, Herman Towles
• Wei-chao Chen
• Organizers
• + John Werner

• Special Effects in Real World
– Poor Man’s Palace
– Beyond Flat Displays
– Upgrade the world
• Tools
– Untethered solution for fusion
– Geometry, Photometry, Id
– Sense, Control, Compensate
– Projectors, RFID, Sensors
• Open Problems
– All senses: haptic, olfactory, proprioception
– Natural phenomenon, complex BRDF, other displays
– Signal Processing: Efficient overlays exploiting rank
– Materials: display, sticky pixels
• Next Challenge: Photorealistic AR around us

VR/AR needs to be multi-modal and multi-format

Recommandé

Recommandé

Contenu connexe

Plus de Camera Culture Group, MIT Media Lab

Plus de Camera Culture Group, MIT Media Lab (20)

Dernier

Dernier (20)

VR/AR needs to be multi-modal and multi-format