CHI 2013 DARE Course

Billinghurst and Duh 1
Designing Augmented Reality Experiences
Mark Billinghurst
University of Canterbury
Christchurch, New Zealand
Henry B.L. Duh
National University of Singapore
Singapore, Singapore
courses@chi2013.com
http://chi2013.acm.org/
Copyright is held by Billinghurst & Duh
CHI 2013, April 27–May 2, 2013, Paris, France.
ACM 13/04

Introduction

Instructors
 Mark Billinghurst
•  Director of HIT Lab NZ, University of Canterbury
•  Degrees in Electrical Engineering, Applied Mathematics
•  Research on collaborative AR, mobile AR, AR usability
•  More than 250 papers in AR, VR, interface design
 Henry Duh
•  Co-director Keio-NUS Joint International Research (CUTE) Center
•  Degrees in Psychology, Industrial design and Engineering
•  Research on interaction design and AR applications
•  More than 80 papers in HCI, AR and Design
Introduction

How Would You Design This?
 Put nice AR Picture here – and video

Or This?

  How to design effective AR experiences
  Understanding AR interaction design possibilities
  Hardware and software tools for rapid prototyping of AR applications
  Effective evaluation methods for AR applications
  Current areas of AR research that will contribute to future AR experiences
  Hands on experiences with AR applications
  Resources for your own research
What You Will Learn
Introduction

 Introduction [Mark]
 AR and the Interaction Design Process [Mark]
 Design Guidelines and Interaction Metaphors for AR [Mark]
 AR Development/Prototyping Tools [Mark]
 Afternoon Tea – Demos [Mark and Henry]
 AR Evaluation Methods [Henry]
 AR Design Case Studies [Henry]
 AR Research Directions [Mark]
Course Agenda
Introduction

Course Demos
 AR Authoring
BuildAR, Metaio Creator
 AR Browers
•  Junaio, Layar, Wikitude
 AR Gaming
•  Elite CommandAR, Transformers, etc..
 Marker Based Handheld AR
•  NASA and CCDU
 Outdoor AR
•  CityViewAR
 Displays
•  Vuzix, Google Glass

Course Motivation
 AR Needs Good Interaction Design
  AR increasingly popular but ergonomics, design and social
issues need to be addressed
  There is a need for deeper understanding of how to uncover,
design build and evaluate effective AR experiences
  AR authoring tools are making it easier than ever before to build
an AR experience, but there are few design guidelines
  Many AR applications are being developed, but there is little
formal evaluation being conducted
  AR experiences are being delivered without an understanding of
the interaction design/experience design process
Introduction

What is Augmented Reality?
 Defining Characteristics (Azuma 97)
•  Combines Real and Virtual Images
– Both can be seen at the same time
•  Interactive in real-time
– The virtual content can be interacted with
•  Registered in 3D
– Virtual objects appear fixed in space
Introduction
Azuma, R., A Survey of Augmented Reality, Presence, Vol. 6, No. 4, August 1997, pp. 355-385.

From Science Fiction to Fact
1977 – Star Wars
2008 – CNN
Introduction

AR Part of MR Continuum
Mixed Reality
Reality - Virtuality (RV) Continuum
Real
Environment
Augmented
Reality (AR)
Augmented
Virtuality (AV)
Virtual
Environment
"...anywhere between the extrema of the virtuality continuum."
P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays
IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.

AR History
 1960’s – 80’s: Early Experimentation
•  Military, Academic labs
 1980’s – 90’s: Basic Research
•  Tracking, Displays
 1995 – 2005: Tools/Applications
•  Interaction, Usability, Theory
 2005 - : Commercial Applications
•  Games, Medical, Industry, Mobile
Introduction

Core Technologies
 Combining Real and Virtual Images
•  Display technologies
 Interactive in Real-Time
•  Input and interactive technologies
 Registered in 3D
•  Viewpoint tracking technologies
Introduction
Display
Processing
Input Tracking

Display Technologies
 Types (Bimber/Raskar 2003)
 Head attached
•  Head mounted display/projector
 Body attached
•  Handheld display/projector
 Spatial
•  Spatially aligned projector/monitor
 HMD Optical vs. Video see-through
 Optical: Direct view of real world -> safer, simpler
 Video: Video overlay -> more image registration options
Introduction

Display Taxonomy

Input Technologies
 Tangible objects
•  Tracked items
 Touch (HHD)
•  Glove, touch
 Gesture
•  Glove, free-hand
 Speech/Multimodal
 Device motion
•  HHD + sensors
Introduction

Tracking Technologies
 Active
•  Mechanical, Magnetic, Ultrasonic
•  GPS, Wifi, cell location
 Passive
•  Inertial sensors (compass, accelerometer, gyro)
•  Computer Vision
•  Marker based
•  Natural feature tracking
 Hybrid Tracking
•  Combined sensors (eg Vision + Inertial)
Introduction

 Web Based AR
•  Flash, HTML 5 based AR
•  Marketing, education
 Outdoor Mobile AR
•  GPS, compass tracking
•  Viewing Points of Interest in real world
•  Eg: Junaio, Layar, Wikitude
 Handheld AR
•  Vision based tracking
•  Marketing, gaming
 Location Based Experiences
•  HMD, fixed screens
•  Museums, point of sale, advertising
Typical AR Experiences
Introduction

AR Becoming Big Business
 Marketing
•  Web-based, mobile
 Mobile AR
•  Geo-located information and service
•  Driving demand for high end phones
 Gaming
•  Mobile, Physical input (Kinect)
 Upcoming areas
•  Manufacturing, Medical, Military
 Rapid Growth
•  Market projected to grow 53% 2012 – 2016
•  Over $5 Billion USD in Mobile AR alone by 2017

Mobile AR Market Size

Commercial AR Companies
 ARToolworks (http://www.artoolworks.com/)
•  ARToolKit, FLARToolKit, SDKs
 Metaio (http://www.metaio.com/)
•  Marketing, Industry, SDKs
 Total Immersion (http://www.t-immersion.com/)
•  Marketing, Theme Parks, AR Experiences
 Qualcomm (http://developer.qualcomm.com/dev/
augmented-reality)
•  Mobile AR, Vuforia SDK
 Many small start-ups (String, Ogmento, etc)

The Interaction Design Process

“The product is no longer
the basis of value. The
experience is.”
Venkat Ramaswamy
The Future of Competition.
Interaction Design

experiences
services
products
components
Value
Gilmore + Pine:
Experience Economy
Function
Emotion
Interaction Design

experiences
applications
tools
components
Designing AR Experiences
Tracking, Display
Authoring
Interaction
Usability
Interaction Design

The Value of Good User
Experience
20c
50c
$3.50
Interaction Design

Good Experience Design
 Reactrix
•  Top down projection
•  Camera based input
•  Reactive Graphics
•  No instructions
•  No training
Interaction Design

Apple: The Value of Good Design
 Good Experience Design Dominates Markets
iPod Sales 2002-2007

Nokia N-Gage
 Great idea – bad experience design
 See - http://www.sidetalkin.com
Good: Handheld Gaming + Phone Bad: Look like a dork using it

Interaction Design
 Answering three questions:
•  What do you do? - How do you affect the world?
•  What do you feel? – What do you sense of the world?
•  What do you know? – What do you learn?
 The Design of User
Experience with Technology
“Designing interactive products to
support people in their everyday and
working lives”
Preece, J., (2002). Interaction Design
Interaction Design

Interaction Design is All About You
 Users should be involved
throughout the Design
Process
 Consider all the needs of
the user
•  Especially context of use
Interaction Design

Interaction Design Process
Interaction Design

Gabbard Model for AR Design
1. user task analysis
2. expert guidelines-based evaluation
3. formative user-centered evaluation
4. summative comparative evaluations
Gabbard, J.L.; Swan, J.E.; , "Usability Engineering
for Augmented Reality: Employing User-Based
Studies to Inform Design,”
Visualization and Computer Graphics, IEEE Transactions
on, vol.14, no.3, pp.513-525, May-June 2008

Gabbard Model in Context

Design Guidelines for AR
Design Guidelines

AR Interaction Design
 Designing AR System = Interface Design
•  Using different input and output technologies
 Objective is a high quality of user experience
•  Ease of use and learning
•  Performance and satisfaction

Design Considerations
 Combining Real and Virtual Images
•  Perceptual issues
 Interactive in Real-Time
•  Interaction issues
 Registered in 3D
•  Technology issues
Introduction

 Interface Components
•  Physical components
•  Display elements
– Visual/audio
•  Interaction metaphors
Physical
Elements
Display
ElementsInteraction
Metaphor
Input Output
AR Design Elements

AR UI Design
 Consider your user
 Follow good HCI principles
 Adapt HCI guidelines for AR
 Design to device constraints
 Using Design Patterns to Inform Design
 Design for you interface metaphor
 Design for evaluation

Consider Your User
 Consider context of user
•  Physical, social, emotional, cognitive, etc
 Mobile Phone AR User
•  Probably Mobile
•  One hand interaction
•  Short application use
•  Need to be able to multitask
•  Use in outdoor or indoor environment
•  Want to enhance interaction with real world

Good HCI Principles
 Affordance
 Reducing cognitive overload
 Low physical effort
 Learnability
 User satisfaction
 Flexibility in use
 Responsiveness and feedback
 Error tolerance

Norman’s Principles of Good Practice
•  Ensure a high degree of visibility
– allow the user to work out the current state of the system
and the range of actions possible.
•  Provide feedback
– continuous, clear information about the results of actions.
•  Present a good conceptual model
– allow the user to build up a picture of the way the system
holds together, the relationships between its different parts
and how to move from one state to the next.
•  Offer good mappings
– aim for clear, natural relationships between actions the
user performs and the results they achieve.

Adapting Existing Guidelines
 Mobile Phone AR
•  Phone HCI Guidelines
•  Mobile HCI Guidelines
 HMD Based AR
•  3D User Interface Guidelines
•  VR Interface Guidelines
 Desktop AR
•  Desktop UI Guidelines

iPhone Guidelines
 Make it obvious how to use your content.
 Avoid clutter, unused blank space, and busy
backgrounds.
 Minimize required user input.
 Express essential information succinctly.
 Provide a fingertip-sized target area for all links
and controls.
 Avoid unnecessary interactivity.
 Provide feedback when necessary

Applying Principles to Mobile AR
 Clean
 Large Video View
 Large Icons
 Text Overlay
 Feedback

AR vs. Non AR Design
 Design Guidelines
•  Design for 3D graphics + Interaction
•  Consider elements of physical world
•  Support implicit interaction
Characteristics Non-AR Interfaces AR Interfaces
Object Graphics Mainly 2D Mainly 3D
Object Types Mainly virtual objects Both virtual and physical objects
Object behaviors Mainly passive objects Both passive and active objects
Communication Mainly simple Mainly complex
HCI methods Mainly explicit Both explicit and implicit

Maps vs. Junaio
 Google Maps
•  2D, mouse driven, text/image heavy, exocentric
 Junaio
•  3D, location driven, simple graphics, egocentric

Design to Device Constraints
 Understand the platforms used and design for limitations
•  Hardware, software platforms
 Eg Handheld AR game with visual tracking
•  Use large screen icons
•  Consider screen reflectivity
•  Support one-hand interaction
•  Consider the natural viewing angle
•  Do not tire users out physically
•  Do not encourage fast actions
•  Keep at least one tracking surface in view
50
Art of Defense Game

Handheld AR Constraints/Affordances
  Camera and screen are linked
•  Fast motions a problem when looking at screen
•  Intuitive “navigation”
  Phone in hand
•  Two handed activities: awkward or intuitive
•  Extended periods of holding phone tiring
•  Awareness of surrounding environment
  Small screen
•  Extended periods of looking at screen tiring
•  In general, small awkward platform
  Vibration, sound
•  Can provide feedback when looking elsewhere
  Networking - Bluetooth, 802.11
•  Collaboration possible
  Guaranteed minimum collection of buttons
  Sensors often available
•  GPS, camera, accelerometer, compass, etc

Design Patterns
“Each pattern describes a problem which occurs
over and over again in our environment, and then
describes the core of the solution to that problem in
such a way that you can use this solution a million
times over, without ever doing it the same way twice.”
– Christopher Alexander et al.
Use Design Patterns to Address Reoccurring Problems
C.A. Alexander, A Pattern Language, Oxford Univ. Press, New York, 1977.

Handheld AR Design Patterns
Title Meaning Embodied Skills
Device Metaphors Using metaphor to suggest available player
actions
Body A&S Naïve physics
Control Mapping Intuitive mapping between physical and
digital objects
Body A&S Naïve physics
Seamful Design Making sense of and integrating the
technological seams through game design
Body A&S
World Consistency Whether the laws and rules in
physical world hold in digital world
Naïve physics
Environmental A&S
Landmarks Reinforcing the connection between digital-
physical space through landmarks
Environmental A&S
Personal Presence The way that a player is represented in the
game decides how much they feel like living
in the digital game world
Environmental A&S
Naïve physics
Living Creatures Game characters that are responsive to
physical, social events that mimic behaviours
of living beings
Social A&S Body A&S
Body constraints Movement of one’s body position
constrains another player’s action
Body A&S Social A&S
Hidden information The information that can be hidden and
revealed can foster emergent social play
Social A&S Body A&S

Example: Seamless Design
 Design to reduce seams in the user experience
•  Eg: AR tracking failure, change in interaction mode
 Paparazzi Game
•  Change between AR tracking to accelerometer input
Yan Xu , et.al. , Pre-patterns for designing embodied interactions in handheld augmented
reality games, Proceedings of the 2011 IEEE International Symposium on Mixed and
Augmented Reality--Arts, Media, and Humanities, p.19-28, October 26-29, 2011

Example: Living Creatures
 Virtual creatures should respond to real world events
•  eg. Player motion, wind, light, etc
•  Creates illusion creatures are alive in the real world
 Sony EyePet
•  Responds to player blowing on creature
55

Physical Elements
Design Guidelines

AR Design Space
Reality Virtual Reality
Augmented Reality
Physical Design Virtual Design

Design of Objects
 Objects
•  Purposely built – affordances
•  “Found” – repurposed
•  Existing – already at use in marketplace
 Affordance
•  The quality of an object allowing an action-
relationship with an actor
•  An attribute of an object that allows people to
know how to use it
– e.g. a door handle affords pulling

Norman on Affordances
"...the term affordance refers to the perceived
and actual properties of the thing, primarily
those fundamental properties that determine
just how the thing could possibly be used.
[...] Affordances provide strong clues to the
operations of things. Plates are for pushing.
Knobs are for turning. Slots are for inserting
things into. Balls are for throwing .. "
(Norman, The Psychology of Everyday
Things 1988, p.9)

Physical vs. Virtual Affordances
 Physical affordances
-  Physical and material aspects of real object
 Virtual affordance
-  Visual and perceived aspects of digital objects
 AR is mixture of physical and virtual
affordances
•  Physical
– Tangible controllers and objects
•  Virtual
– Virtual graphics and audio
-

Affordance Framework
William W. Gaver. 1991. Technology affordances. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems (CHI '91), Scott P. Robertson,
Gary M. Olson, and Judith S. Olson (Eds.). ACM, New York, NY, USA, 79-84.

Affordance Led Design
 Make affordances perceivable
•  Provide visual, haptic, tactile, auditory cues
 Affordance Led Usability
•  Give feedback
•  Provide constraints
•  Use natural mapping
•  Use good cognitive model

Example: AR Chemistry
 Tangible AR chemistry education (Fjeld)
Fjeld, M., Juchli, P., and Voegtli, B. M. 2003. Chemistry education: A tangible
interaction approach. Proceedings of INTERACT 2003, September 1st -5th
2003, Zurich, Switzerland.

Input Devices
 Form informs function and use

Picking up an Atom

AR Interaction Metaphors
Design Guidelines

 Interface Components
•  Physical components
•  Display elements
– Visual/audio
•  Interaction metaphors
Physical
Elements
Display
ElementsInteraction
Metaphor
Input Output
AR Design Principles

Interaction Tasks
 2D (from [Foley]):
•  Selection, Text Entry, Quantify, Position
 3D (from [Bowman]):
•  Navigation (Travel/Wayfinding)
•  Selection
•  Manipulation
•  System Control/Data Input
 AR: 2D + 3D Tasks and.. more specific tasks?
[Foley] The Human Factors of Computer Graphics InteractionTechniques Foley, J. D.,V.Wallace & P. Chan. IEEE
Computer Graphics and Applications (Nov.): 13-48. 1984.
[Bowman]: 3D User Interfaces:Theory and Practice D. Bowman, E. Kruijff, J. Laviola, I. Poupyrev Addison Wesley 2005

AR Interaction Metaphors
 Viewpoint Control
 Information Browsing
•  establish shared meaning
 3D AR Interfaces
 Augmented Surfaces
•  serve as cognitive artifacts
 Tangible AR

1. Viewpoint Control
 2D/3D virtual objects are
registered in 3D
•  “VR in Real World”
 Interaction
•  2D/3D virtual viewpoint control
 Applications
•  Visualization, training

2. Information Browsering
 Information is registered to
real-world context
•  Hand held AR displays
 Interaction
•  Manipulation of a window
into information space
 Applications
•  Context-aware information
displays
Rekimoto, et al. 1997

3. 3D AR Interfaces
 Virtual objects displayed in 3D
physical space and manipulated
•  HMDs and 6DOF head-tracking
•  6DOF hand trackers for input
 Interaction
•  Viewpoint control
•  Traditional 3D user interface
interaction: manipulation,
selection, etc.
Kiyokawa, et al. 2000

4. Augmented Surfaces
 Basic principles
•  Virtual objects are projected on a surface
•  Physical objects are used as controls for
virtual objects
•  Support for collaboration
 Rekimoto, et al. 1998
•  Front projection
•  Marker-based tracking
•  Multiple projection surfaces

5. Tangible User Interfaces
 Create digital shadows for
physical objects
 Foreground
•  graspable UI
 Background
•  ambient interfaces

Lessons from Tangible
Interfaces
 Physical objects make us smart
•  Norman’s “Things that Make Us Smart”
•  encode affordances, constraints
 Objects aid collaboration
 Objects increase understanding

TUI Limitations
 Difficult to change object properties
•  Can’t tell state of digital data
 Limited display capabilities
•  projection screen = 2D
•  dependent on physical display surface
 Separation between object and display
•  Augmented Surfaces

Tangible AR Metaphor
 AR overcomes limitation of TUIs
•  enhance display possibilities
•  merge task/display space
•  provide public and private views
 TUI + AR = Tangible AR
•  Apply TUI methods to AR interface design

 Space-multiplexed
•  Many devices each with one function
–  Quicker to use, more intuitive, clutter
–  Real Toolbox
 Time-multiplexed
•  One device with many functions
–  Space efficient
–  mouse

Tangible AR: Tiles (Space
Multiplexed)
 Tiles semantics
•  data tiles
•  operation tiles
 Operation on tiles
•  proximity
•  spatial arrangements
•  space-multiplexed

Tangible AR: Time-
multiplexed Interaction
 Use of natural physical object manipulations to control
virtual objects
 VOMAR Demo
•  Catalog book:
–  Turn over the page
•  Paddle operation:
–  Push, shake, incline, hit, scoop

Object Based Interaction:
MagicCup
 Intuitive Virtual Object Manipulation
on a Table-Top Workspace
•  Time multiplexed
•  Multiple Markers
–  Robust Tracking
•  Tangible User Interface
–  Intuitive Manipulation
•  Stereo Display
–  Good Presence

Tangible AR Design Principles
 Tangible AR Interfaces use TUI principles
•  Physical controllers for moving virtual content
•  Support for spatial 3D interaction techniques
•  Time and space multiplexed interaction
•  Support for multi-handed interaction
•  Match object affordances to task requirements
•  Support parallel activity with multiple objects
•  Allow collaboration between multiple users

Interaction with Handheld AR
 Embodied Interaction
•  Focuses on the device itself
•  Touch, gesture, orientation, etc
 Tangible Interaction
•  Direct manipulation of known objects
•  Tracking objects
 Egocentric vs. Exocentric Interaction
•  Egocentric – inside out (eg outdoor AR browsing)
•  Exocentric – outside in (eg marker based AR)

Handheld AR Metaphors
HandHeld AR
Wearable AR
Output:
Display
Input
Input &
Output

Handheld Interface Metaphors
 Tangible AR Lens Viewing
•  Look through screen into AR scene
•  Interact with screen to interact with
AR content
–  Eg Invisible Train
 Tangible AR Lens Manipulation
•  Select AR object and attach to device
•  Use the motion of the device as input
–  Eg AR Lego

Case Study 1: 3D AR Lens
Goal: Develop a lens based AR interface
 MagicLenses
•  Developed at Xerox PARC in 1993
•  View a region of the workspace differently to the rest
•  Overlap MagicLenses to create composite effects

3D MagicLenses
MagicLenses extended to 3D (Veiga et. al. 96)
  Volumetric and flat lenses

AR Lens Design Principles
 Physical Components
•  Lens handle
–  Virtual lens attached to real object
 Display Elements
•  Lens view
–  Reveal layers in dataset
 Interaction Metaphor
•  Physically holding lens

Case Study 2: LevelHead
 Physical Components
•  Real blocks
 Display Elements
•  Virtual person and rooms
 Interaction Metaphor
•  Blocks are rooms

AR Perceptual + Cognitive Issues
Design Guidelines

AR and Perception
 Creating the illusion that virtual images are
seamlessly part of the real world
•  Must match real and virtual cues
•  Depth, occlusion, lighting, shadows..

AR as Perception Problem
 Goal of AR to fool human senses – create illusion that
real and virtual are merged
 Depth
•  Size
•  Occlusion
•  Shadows
•  Relative motion
•  Etc..

Central goal of AR systems is to fool the
human perceptual system
 Display Modes
•  Direct View
•  Stereo Video
•  Stereo graphics
 Multi-modal display
•  Different objects with different display modes
•  Potential for depth cue conflict
Perceptual Issues
D. Drascic and P. Milgram. Perceptual issues in augmented reality. In M. T. Bolas, S. S. Fisher, and J.
O. Merritt, editors, SPIE Volume 2653: Stereoscopic Displays and Virtual Reality Systems III, pages
123-134, January/February 1996.

Perceptual Issues
 Combining multiple display modes
•  Direct View, Stereo Video View, Graphics View
 Conflict between display modes
•  Mismatch between depth cues

Perceptual Issues
 Static and Dynamic registration mismatch
 Restricted Field of View
 Mismatch of Resolution and Image clarity
 Luminance mismatch
 Contrast mismatch
 Size and distance mismatch
 Limited depth resolution
 Vertical alignment mismatches
 Viewpoint dependency mismatch

Types of Perceptual Issues
 Environment: Issues related to the environment itself.
 Capturing: Issues related to digitizing the environment
 Augmentation: Issues related to the design, layout, and
registration or AR content
 Display device: Technical issues associated with the
display device.
 User: Issues associated with user perceiving content.
E. Kruijff, J. E. Swan, and S. Feiner. Perceptual issues in augmented reality revisited.
9th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2010, pp. 3--12.

Depth Cues
 Pictorial: visual cues
•  Occlusion, texture, relative brightness
 Kinetic: motion cues
•  Relative motion parallax, motion perspective
 Physiological: motion cues
•  Convergence, accommodation
 Binocular disparity: two different eye images

Depth Perception

Occlusion Handling

Cognitive Issues in AR
 Three categories of issues
•  Information Presentation – displaying virtual
information on the real world
•  Physical Interaction – content creation,
manipulation and navigation in AR
•  Shared Experience – collaboration and
supporting common experiences in AR
Li, Nai, and Henry Been-Lirn Duh. "Cognitive Issues in Mobile Augmented Reality:
An Embodied Perspective." Human Factors in Augmented Reality Environments.
Springer New York, 2013. 109-135.

Information Presentation
 Information Presentation
•  Amount of information
•  Clutter, complexity
•  Representation of information
•  Navigation cues, POI representation
•  Placement of information
•  Head, body, world stabilized
•  View combination
•  Multiple views

Twitter 360
 www.twitter-360.com
 iPhone application
 See geo-located tweets in real world
 Twitter.com supports geo tagging

Wikitude – www.mobilizy.com
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah

Information Filtering

Physical Interaction
 Physical Interaction
•  Navigation
•  Direct Manipulation
•  Embodied vs. Tangible
•  Multimodal interaction
•  Content creation

Outdoor AR: Limited FOV

Possible solutions
 Overview + Detail
•  spatial separation; two views
 Focus + Context
•  merges both views into one view
 Zooming
•  temporal separation

 TU Graz – HIT Lab NZ - collaboration
•  Zooming panorama
•  Zooming Map
Zooming Views

Gesture Based Interaction
 HMD-based AR frees the users hands
•  Natural hand based interaction
•  Intuitive manipulation – low cognitive load
 Example
•  Tinmith-Hand Two hand manipulation of 3D models
112

Shared Experiences
 Shared Experience
•  Social context
•  Bodily configuration
•  Artifact manipulation
•  Display space

TAT Augmented ID

Designing for Children
 Development Psychology Factors
•  Motor Abilities
•  Spatial Abilities
•  Logic Abilities
•  Attention Abilities
Radu, Iulian, and Blair MacIntyre. "Using children's developmental psychology to
guide augmented-reality design and usability." Mixed and Augmented Reality
(ISMAR), 2012 IEEE International Symposium on. IEEE, 2012.

Motor Abilities
Skill Type Challenging AR Interaction
Multiple hand coordination Holding phone in one hand and
using another hand to move
marker
Hand-eye coordination Using a marker to intercept a
moving object
Fine motor skills Moving a marker on a specified
path
Gross motor skills and endurance Turning body around to look at a
panorama

Spatial Abilities
Spatial memory Remembering the configuration of a large
virtual space while handheld screen shows
a limited view
Spatial Perception Understanding when a virtual item is on top
of a physical item
Spatial Visualization Predict when virtual objects are visible by
other people or virtual characters

Attention and Logic
Divided attention Playing an AR game, and making sure to
keep marker in view so tracking is not lost
Selective and executive
attention
Playing an AR game while moving outdoors
Remembering and reversing Remembering how to recover from tracking
loss
Abstract over concrete
thinking
Understanding that virtual objects are
computer generated, and they do not need
to obey physical laws
Attention Abilities
Logic and Memory

AR Development Tools

AR Authoring Tools
 Low Level Software Libraries
•  osgART, Studierstube, MXRToolKit
 Plug-ins to existing software
•  DART (Macromedia Director), mARx, Unity,
 Stand Alone
•  AMIRE, BuildAR, Metaio Creator etc
 Rapid Prototyping Tools
•  Flash, OpenFrameworks, Processing, Arduino, etc
 Next Generation
•  iaTAR (Tangible AR)

ARToolKit (Kato 1998)
 Open source – computer vision based AR tracking
 http://artoolkit.sourceforge.net/

ARToolKit Structure
 Three key libraries:
•  AR32.lib – ARToolKit image processing functions
•  ARgsub32.lib – ARToolKit graphics functions
•  ARvideo.lib – DirectShow video capture class
DirectShow
ARvideo.lib

Software
 Cross platform
•  Windows, Mac, Linux, IRIX, Symbian, iPhone, etc
 Additional basic libraries
•  Video capture library (Video4Linux, VisionSDK)
•  OpenGL, GLUT
 Requires a rendering library
•  Open VRML, Open Inventor, osgART, etc

OSGART Programming Library
 Integration of ARToolKit with a High-Level Rendering Engine
(OpenSceneGraph)
OSGART= OpenSceneGraph + ARToolKit
 Supporting Geometric + Photometric Registration

osgART Approach: AR
Scene Graph
Video
Geode
Root
Transform
3D Object
Virtual
Camera
Projection matrix from
tracker calibration
Transformati
on matrix
updated
from marker
tracking in
realtime
Video
Layer
Full-screen
quad with
live
texture
updated
fromVideo
source
Orthographic
projection

osgART:Features
 C++ (but also Python, Lua, etc).
 Multiple Video Input supports:
•  Direct (Firewire/USB Camera), Files, Network by
ARvideo, PtGrey, CVCam, VideoWrapper, etc.
 Benefits of Open Scene Graph
•  Rendering Engine, Plug-ins, etc

ARToolKit Family
ARToolKit ARToolKit NFT
ARToolKit (Symbian)
NyToolKit
- Java, C#,
- Android, WM
JARToolKit (Java)
FLARToolKit (Flash)
FLARManager (Flash)

Why Browser Based AR?
 High impact
•  High marketing value
 Large potential install base
•  1.6 Billion web users
 Ease of development
•  Lots of developers, mature tools
 Low cost of entry
•  Browser, web camera

FLARToolkit
Papervision 3D
Adobe Flash
AR Application Components

FLARToolKit Example
 Boffswana Living Sasquatch
 In first month
•  100K unique visits
•  500K page views
•  6 minutes on page

Low Level Mobile AR Tools
 Vuforia Tracking Library (Qualcomm)
•  Vuforia.com
•  iOS, Android
•  Computer vision based tracking
•  Marker tracking, 3D objects, frame markers
 Integration with Unity
•  Interaction, model loading, game logic

Junaio - www.junaio.com

Junaio Key Features
 Content provided in information channels
•  Over 2,000 channels available
 Two types of AR channels
•  GLUE channels – visual tracking
•  Location based channels – GPS, compass tracking
 Simple to use interface with multiple views
•  List, map, AR (live) view
 Point of Interest (POI) based
•  POIs are geo-located content

AREL
 Augmented Reality Environment Language
•  Overcomes limitations of XML by itself
•  Based on web technologies; XML, HTML5, JavaScript
 Core Components
1.  AREL XML: Static file, specifies scene content
2.  AREL JavaScript: Handles all interactions and animation. Any
user interaction send an event to AREL JS
3.  AREL HTML5: GUI Elements. Buttons, icons, etc
 Advantages
•  Scripting on device, more functionality, GUI customization

Result

BirdsView
 Location Based CMS
•  Add content, publish to Layar or Junaio
•  http://www.birdsview.de/

BirdsView on Junaio

BuildAR
 http://www.buildar.co.nz/
 Stand alone application
 Visual interface for AR model viewing application
 Enables non-programmers to build AR scenes

Metaio Creator
 Drag and drop Junaio authoring

Total Immersion D’Fusion Studio
 Complete commercial authoring platform
•  http://www.t-immersion.com/
•  Multi-platform
•  Markerless tracking
•  Scripting
•  Face tracking
•  Finger tracking
•  Kinect support

Others
 AR-Media
•  http://www.inglobetechnologies.com/
•  Google sketch-up plug-in
 LinceoVR
•  http://linceovr.seac02.it/
•  AR/VR authoring package
 Libraries
•  JARToolKit, MXRToolKit, ARLib, Goblin XNA

Research in AR Authoring
 iaTAR (Lee 2004)
•  Immersive AR Authoring
•  Using real objects to create AR applications

Rapid Prototyping
 Speed development time by using quick
hardware mockups
•  handheld device connected to PC
•  LCD screen
•  USB phone keypad
•  Camera

Build Your Own Google Glass
 Rapid Prototype Glass-Like HMD
 Myvu HMD + headphone + iOS Device + basic glue skills
•  $300 + less than 3 hours construction
  http://www.instructables.com/id/DIY-Google-Glasses-AKA-the-Beady-i/

BUNRATTY FOLK PARK
 Irish visitor attraction run by Shannon Heritage
 19th century life is recreated
 Buildings from the mid-west have been relocated to the
26-land surrounding Bunratty Castle
 30 buildings are set in a rural or village setting there.

AUGMENTED REALITY
154
In Bunratty Folk Park:
 Allows the visitor to point a camera at an exhibit, the
device recognises its by it’s location and layers digital
information on to the display
 3- dimensional virtual objects can be positioned with real
ones on display
 Leads to dynamic combination of a live camera view and
information

ITERATIVE DESIGN PROCESS
Prototyping and User Testing
 Low Fidelity Prototyping
• Sketches
• Paper Prototyping
• Post-It Prototyping
• PowerPoint Prototyping
 High Fidelity Prototyping
• Wikitude

Storyboarding
156

INITIAL SKETCHES
Pros:

• 
Good
for
idea
genera/on

• 
Cheap

• 
Concepts
seem
feasible

Cons:

• 
Not
great
feedback
gained

• 
Photoshop
not
fast
enough

for
making
changes

Post-it Note Prototyping
Camera
View
with
3D
Annota/on

• 
Selec/on
highlighted
in
blue
• 
Home
buBon
added
for
easy

naviga/on
to
main
menu

POWERPOINT PROTOTYPING
Beneﬁts

• 
Used
for
User
Tes/ng

• 
Interac/ve

• 
Func/onali/es
work

• 
Quick

• 
Easy
arrangement
of
slides

User
Tes/ng

• 
Par/cipants
found

• 
15
minute
sessions
screen
captured

• 
‘Talk
Allowed’
technique
used

• 
Notes
taken

• 
Post-‐Interview

WIKITUDE PROTOTYPE
User Testing
 Application well received
 Understandable
 Participants playful with the
technology

FINAL VIDEO PROTOTYPE
 Flexible
tool
for
capturing
the
use

of
an
interface

 Elaborate
simula/on
of
how
the

naviga/onal
aid
will
work

 Does
not
need
to
be
realis/c
in

every
detail

 Gives
a
good
idea
of
how
the

ﬁnished
system
will
work

AR Evaluation Methods

Why Evaluate AR
Applications?
 To test and compare interfaces, new technologies,
interaction techniques
 To validate the efficiency and efficient the AR interface
and system
 Test Usability (learnability, efficiency, satisfaction,...)
 Get user feedback
 Refine interface design
 Better understand your end users
 ...

Survey of AR Papers
 Edward Swan (2005)
 Surveyed major conference/journals (1992-2004)
– Presence, ISMAR, ISWC, IEEE VR
 Summary
•  1104 total papers
•  266 AR papers
•  38 AR HCI papers (Interaction)
•  21 AR user studies
 Only 21 from 266 AR papers had a formal user study
•  Less than 8% of all AR papers

HIT Lab NZ Usability Survey
  A Survey of Evaluation Techniques Used in
Augmented Reality Studies
•  Andreas Dünser, Raphaël Grasset, Mark
Billinghurst
 reviewed publications from 1993 to 2007
•  Extracted 6071 papers which mentioned
“Augmented Reality”
•  Searched to find 165 AR papers with User Studies

Types of Experimental
Measures Used
 Types of Experimental Measures
•  Objective measures
•  Subjective measures
•  Qualitative analysis
•  Usability evaluation techniques
•  Informal evaluations

Types of Experimental
Measures Used

Types of Experiments and topics
  Sensation, Perception & Cognition
•  How is virtual content perceived ?
•  What perceptual cues are most important ?
•  How to visualize augmented/overlay information on real environment?
•  Visual search/attention/salience issues of human performance
  Interaction
•  How can users interact with virtual content ?
•  Which interaction techniques are most efficient in certain context ?
  Collaboration & Social issues
•  How is collaboration in AR interface different ?
•  Which collaborative cues can be conveyed best ?
•  Privacy and security issues of AR interface

Types of AR User Studies

Summary
 Over last 10 years
•  Most user studies focused on user performance
•  Fewest user studies on collaboration
– MobileAR was not popular before 2009
•  Objective performance measures most used
•  Qualitative and usability measures least used

Sample Size
 … the more the better
•  for quantitative analysis:
•  rule of thumb approx. 15-20 or more (for cognitive
and lab type of experiment)
•  absolute minimum of 8-10 per cell
 Ideal sample size can be calculated - power analysis
•  Power (1- beta) => the chance to reject the null hypothesis
when the null hypothesis is false
•  Power is the probability of observing a difference when it really
exists
•  Power increases with sample size
•  Power decreases with variance
 Large effects can be detected with smaller samples
•  e.g. to discriminate mean speed between turtles and a rabbits

Data Collection and Analysis
 The choice of a method is dependent on the type of data that
needs to be collected
 In order to test a hypothesis the data has to be analysed using
a statistical method
 The choice of a statistical method depends on
the type of collected data
 All the decisions about an experiment should be made before
it is carried out

Observe and Measure
 Observations are gathered…
•  manually (human observers)
•  automatically (computers, software, cameras, sensors, etc.)
 A measurement is a recorded observation
 Objective metrics
 Subjective metrics

Typical objective metrics
 task completion time
 errors (number, percent,…)
 percent of task completed
 ratio of successes to failures
 number of repetitions
 number of commands used
 number of failed commands
 physiological data (heart rate,…)
 …

Typical subjective metrics
 user satisfaction
 subjective performance
 ratings
 ease of use
 intuitiveness
 judgments
 …

Data Types
 Subjective
•  Subjective survey
–  Likert Scale, condition rankings
•  Observations
–  Think Aloud
•  Interview responses
 Objective
•  Performance measures
–  Time, accuracy, errors
•  Process measures
–  Video/audio analysis
How easy was the task
1 2 3 4 5
Not very easy Very easy

Experimental Measures
Measure What does it tell us? How is it
measured?
Timings Performance Via a stopwatch, or
automatically by the device.
Errors Performance, Particular sticking
points in a task
By success in completing the
task correctly. Through
experimenter observation,
examining the route walked.
Perceived Workload Effort invested. User satisfaction Through NASA TLX scales
and other questionnaires.
Distance traveled and
route taken
Depending on the application, these
can be used to pinpoint errors and to
indicate performance
Using a pedometer, GPS or
other location-sensing system.
By experimenter observation.
Percentage preferred
walking speed
Performance By finding average walking
speed, which is compared
with normal walking speed.
Comfort User satisfaction. Device
acceptability
Comfort Rating Scale and
other questionnaires.
User comments and
preferences
User satisfaction and preferences.
Particular sticking points in a task.
Through questionnaires,
interviews and think-alouds.
Experimenter
observations
Different aspects, depending on the
experimenter and on the observations
Through observation and
note-taking

Statistical Analysis
 Once data is collected statistics can be used for analysis
 Typical Statistical Techniques
•  Comparing between two results
–  Unpaired T-Test (for between subjects – assumes normal distribution)
–  Paired T-Test (for within subjects – assumes normal distribution)
–  Mann–Whitney U (independent samples)
•  Comparing between > two results
–  Followed by post-hoc analysis – Bonferroni Test
–  Analysis of Variance – ANOVA
–  Kruskal–Wallis (does not assume normal distribution)

Case Study: A Wearable
Information Space
Head Stabilized Body Stabilized
An AR interface provides spatial audio and visual cues
Does a spatial interface aid performance?
– Task time / accuracy
M. Billinghurst, J. Bowskill, Nick Dyer, Jason Morphett (1998). An Evaluation of Wearable
Information Spaces. Proc. Virtual Reality Annual International Symposium.

Task Performance
 Task
•  find target icons on 8 pages
•  remember information space
 Conditions
A - head-stabilized pages
B - cylindrical display with trackball
C - cylindrical display with head tracking
 Subjects
•  Within subjects (need fewer subjects)
•  12 subjects used

Experimental Measures
 Objective
•  spatial ability (pre-test)
•  time to perform task
•  information recall
•  workload (NASA TLX)
 Subjective
•  Post Experiment Survey
–  rank conditions (forced choice)
–  Likert Scale Questions
-  “How intuitive was the interface to use?”
Many
Different
Measures

Post Experiment Survey
For each of these conditions please answer:
1) How easy was it to find the target?
1 2 3 4 5 6 7
1=not very easy 7=very easy
For the head stabilised condition (A):
For the cylindrical condition with mouse input (B):
For the head tracked condition (C):
Rank all the conditions in order on a scale of one to three
1) Which condition was easiest to find target (1 = easiest, 3 = hardest)
A: B: C:

Results
 Body Stabilization Improved Performance
•  search times significantly faster (One factor ANOVA)
 Head Tracking Improved Information recall
•  no difference between trackball and stack case
 Head tracking involved more physical work

Subjective Impressions
  Subjects Felt Spatialized Conditions (ANOVA):
•  More enjoyable
•  Easier to find target

Subjective Impressions
  Subject Rankings (Kruskal-Wallis)
•  Spatialized easier to use than head stabilized
•  Body stabilized gave better understanding
•  Head tracking most intuitive

AR Evaluation
 Field, Field, Field –
•  Field studies vs. Lab studies
•  Contextual design and evaluation
 Combined methods (qualitative and quantitative
studies)
•  Weakness of each method should be considered
 New/modified evaluation methods may need to be
developed
 Seek for more new evaluation case studies in AR

AR Design Case Study

“The Jackson Plan”
An Educational Location-based Handheld AR Game
Learning while in travel
Mobile AR Entertainment for
Children

The Jackson Plan
 Overview

‘The Jackson Plan’ is an educational discovery Mobile
Augmented Reality game that is set on the historical
urban plan of the same name (also known as the “Plan
of the Town of Singapore”)

Using multi-modality features on an Apple iPad2, players
collaboratively experience this location-based Mobile
Augmented Reality game around the several important
historical sites and events that revolve around Sir
Thomas Stamford Rafﬂes and his founding of the island
of Singapore in 1819.

The Jackson Plan 1822, is on
display at the Singapore
History Gallery, National
Museum of Singapore

 Learning Goals/Objectives

Unit
Learning objectives

Jackson
Plan

【Knowledge】
1. To acquire a better understanding of the key developments of
the Raffles’s arrival, its early settlers and Raffle’s town plan.
【Skills】
1. To explains the reasons for the founding of Singapore (1819).
2. To explain the importance of trade to Singapore.
3. To describe the contributions of key personalities and
immigrants to the growth and development of Singapore.
【Values & Attitudes】
1. To develop an interest in the past.
2. To appreciate culture heritage as well as to instill a sense of
courage, diligence and perseverance to Singapore.

History Syllabus for Lower Secondary, Year of Implementation: 2006. ISBN 981-05-1669-X.
Source: Curriculum Planning and Development Division, Ministry of Education, Singapore
Learning Content

Consideration
 How can a new technology help new
learning experience in cultural heritage?
 Interdisciplinary research (Design,
Technology, Education and Learning)
 System building, a single application or
 Recognition in each field
 Real deployment in schools
194

Theoretical Framework
“Situated cognition via
scaffolding mechanisms
([Vygotsky, 1978])”
Distinct HAR technology
pairings available in a
game, (0=No, 1=Yes),
resulting in four possible
eHAR game types and
play styles, each with an
implementation process.
Y.-N. Chang, R. K. C. Koh, and H. B.-L. Duh, "Handheld AR games - A triarchic conceptual design framework," in Mixed and Augmented Reality -
Arts, Media, and Humanities (ISMAR-AMH), 2011 IEEE International Symposium On, Basel, Switzerland, 2011, pp. 29-36.
Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press.

 Triarchic conceptual design framework

•  GPS navigation: Location-based implementation
for Cultural & Historical (contextual) explorations
•  Overlaying options: ‘Binoculars’ metaphor
(i.e., Panoramic Map)
•  Virtual properties
(game inventory)
•  Geo-tagging / (diary)
•  Blended mini games
(i.e. puzzles)
•  Tasks may exploit
platform’s hardware
features
(GPS,
Accelerometer)
•  Visual identification
of past and present
imagery
•  History comes to life by
exploiting location-
dependent contexts
•  Backend confirmation with
server connectivity
(‘Wizard of Oz’ possibility)
for dynamic situational
exchanges, i.e. messages,
images, induce player-
behaviors, etc
•  Promote Contextual Inquiry
& Collaboration
(Learning Strategies)
Theoretical Framework

The Jackson Plan
Textbook:
SINGAPORE: FROM SETTLEMENT
TO NATION - PRE-1819 TO 1971
(Marshall Cavendish Education)
Theme: Chapter 3 - What Part Did the
Different Immigrant Communities Play in
Singapore’s Development?
TheJackson
Plan
Prior Knowledge:
The settlement of
Singapore
-Why Raffles chose
Singapore
Singapore’s
central location
Central location
Excellent port
Good supply of
drinking water
The Dutch had not
occupied the island
Immigrants
Why immigrants
came
Singapore’s
town plan
Lieutenant Philip
Jackson,1822
Improve the
haphazard
building plan
Segregated
population groups
Populations from
trading goods /
countries
Chinese
Coolies
Samsui women
Indians
Labourers
Coolies
Malays Shipbuilders
Europeans Merchants
Arabs Traders

• Ideation: Use of Historical Illustrations / Images in Situated Augmented Views
•  Panoramic / Still - Visual Imageries of the Past +
GPS

Game Features, Mechanisms &
Platform - Ideations
• ‘Civic District Trail’ - A tourist’s DIY
exploration experience promoted by the
Singapore Tourism Board

Virtual
& physical
interaction
Manipulate
knowledge-collect
trading materials
(i.e. spices)
Geo-Tagging
the “right”
locations
Taking pictures
(Wizard of Oz)Blended casual
mini-games with
physical interaction
and collaboration
Game Features, Mechanisms & Platform

Chinese:
Chinatow
n
Indian
s
Europeans
&
Rich Asians
Malays &
Muslims
“Plan of the town of Singapore” by Lieutenant Phillip Jackson,1822
Commerci
al Square

Phase
Learning
Objectives
Learning Task(s) Time
1
Understanding of
activities
To understand the
Gameplay and
manipulation of iPad2
devices
- Introduction to
Gameplay
- Game introduction /
Mission Briefing
15
min
2
Constructing
Knowledge
To understand the
background of Singapore
settlement
- Information Collection:
Know who these
immigrants are
15
min
3 Mastering
To analyze how did the
immigrants contribute to
Singapore as a trading
centre
- Experience the entrepot
trade
20
min
4
Knowledge
Application
To make comparisons and
organize information of
the different contributions
of immigrants
- Make an accusation by
evidence (Gain a
summative feedback)
15
min
The Jackson Plan: Planned
Gaming Activities

The Trail

Game Design

The Jackson Plan - Features

Evaluation
• 72 students (36 pairs) took
part in the evaluation
• Secondary One classes
(~12-13 years old)
• They were equally divided into
2 main groups, Location-based
and Digital Book versions
Digital Book
Location-based AR

Platform
Apple iPad2
Apple iPad2

Collaboration
Yes
Yes

Interaction Type
Non-AR
Location-based AR

Play Space
Indoors
Outdoors

 The structure of knowledge
Evaluation

The Jackson Plan

Theory into Practice: Domain-Centric Handheld Augmented Reality Game
Design
Study 3 - Co-creativity fusions in interdisciplinary AR game developments

AR Research Directions

Billinghurst and Duh 218Vision of AR

To Make the Vision Real..
 Hardware/software requirements
•  Contact lens displays
•  Free space hand/body tracking
•  Speech/gesture recognition
•  Etc..
 Most importantly
•  Usability

Natural Interaction
 Automatically detecting real environment
•  Environmental awareness
•  Physically based interaction
 Gesture Input
•  Free-hand interaction
 Multimodal Input
•  Speech and gesture interaction
•  Implicit rather than Explicit interaction

AR MicroMachines
 AR experience with environment awareness and
physically-based interaction
•  Based on MS Kinect RGB-D sensor
 Augmented environment supports
•  occlusion, shadows
•  physically-based interaction between real and virtual objects

Operating Environment

Architecture
 Our framework uses five libraries:
•  OpenNI
•  OpenCV
•  OPIRA
•  Bullet Physics
•  OpenSceneGraph

System Flow
 The system flow consists of three sections:
•  Image Processing and Marker Tracking
•  Physics Simulation
•  Rendering

Physics Simulation
 Create virtual mesh over real world
 Update at 10 fps – can move real objects
 Use by physics engine for collision detection (virtual/real)
 Use by OpenScenegraph for occlusion and shadows

Rendering
Occlusion Shadows

Architecture
5. Gesture
• Static Gestures
• Dynamic Gestures
• Context based Gestures
4. Modeling
• Hand recognition/modeling
• Rigid-body modeling
3. Classification/Tracking
2. Segmentation
1. Hardware Interface
HITLabNZ’s Gesture Library

Architecture
5. Gesture
•  Static Gestures
•  Dynamic Gestures
•  Context based Gestures
4. Modeling
•  Hand recognition/
modeling
•  Rigid-body modeling
2. Segmentation
o  Supports PCL, OpenNI, OpenCV,
and Kinect SDK.
o  Provides access to depth, RGB,
XYZRGB.
o  Usage: Capturing color image,
depth image and concatenated
point clouds from a single or
multiple cameras
o  For example:
Kinect for Xbox 360
Kinect for Windows
Asus Xtion Pro Live

Architecture
5. Gesture
4. Modeling
modeling
2. Segmentation
o  Segment images and point
clouds based on color, depth
and space.
o  Usage: Segmenting images or
point clouds using color models,
depth, or spatial properties such
as location, shape and size.
o  For example:
Skin color segmentation
Depth threshold

Architecture
5. Gesture
4. Modeling
modeling
2. Segmentation
o  Identify and track objects
between frames based on
XYZRGB.
o  Usage: Identifying current
position/orientation of the
tracked object in space.
o  For example:
Training set of hand
poses, colors represent
unique regions of the
hand.
Raw output (without-
cleaning) classified on
real hand input (depth
image).

Architecture
5. Gesture
4. Modeling
modeling
2. Segmentation
o  Hand Recognition/Modeling
  Skeleton based (for low resolution
approximation)
  Model based (for more accurate
representation)
o  Object Modeling (identification and tracking
rigid-body objects)
o  Physical Modeling (physical interaction)
  Sphere Proxy
  Model based
  Mesh based
o  Usage: For general spatial interaction in AR/VR
environment

Method

Represent
models
as
collec1ons
of
spheres
moving
with
the

models
in
the
Bullet
physics
engine

Method

Render
AR
scene
with
OpenSceneGraph,
using
depth
map

for
occlusion

Shadows
yet
to
be
implemented

Results

Architecture
5. Gesture
4. Modeling
modeling
2. Segmentation
o  Static (hand pose recognition)
o  Dynamic (meaningful movement
recognition)
o  Context-based gesture
recognition (gestures with
context, e.g. pointing)
o  Usage: Issuing commands/
anticipating user intention and
high level interaction.

Multimodal Interaction
 Combined speech input
 Gesture and Speech complimentary
•  Speech
–  modal commands, quantities
•  Gesture
–  selection, motion, qualities
 Previous work found multimodal interfaces intuitive for
2D/3D graphics interaction

Free Hand Multimodal Input
 Use free hand to interact with AR content
 Recognize simple gestures
 No marker tracking
Point Move Pick/Drop

Multimodal Architecture

Multimodal Fusion

Hand Occlusion

User Evaluation
 Change object shape, colour and position
 Conditions
•  Speech only, gesture only, multimodal
 Measure
•  performance time, error, subjective survey

Experimental Setup
Change object shape
and colour

Results
 Average performance time (MMI, speech fastest)
•  Gesture: 15.44s
•  Speech: 12.38s
•  Multimodal: 11.78s
 No difference in user errors
 User subjective survey
•  Q1: How natural was it to manipulate the object?
–  MMI, speech significantly better
•  70% preferred MMI, 25% speech only, 5% gesture only

Natural Gesture
Interaction on Mobile
 Use mobile camera for hand tracking
•  Fingertip detection

Evaluation
 Gesture input more than twice as slow as touch
 No difference in naturalness

Intelligent Interfaces
 Most AR systems are stupid
•  Don’t recognize user behaviour
•  Don’t provide feedback
•  Don’t adapt to user
 Especially important for training
•  Scaffolded learning
•  Moving beyond check-lists of actions

Intelligent Interfaces
 AR interface + intelligent tutoring system
•  ASPIRE constraint based system (from UC)
•  Constraints
–  relevance cond., satisfaction cond., feedback

Domain Ontology

Intelligent Feedback
 Actively monitors user behaviour
•  Implicit vs. explicit interaction
 Provides corrective feedback

Evaluation Results
 16 subjects, with and without ITS
 Improved task completion
 Improved learning

Intelligent Agents
 AR characters
•  Virtual embodiment of system
•  Multimodal input/output
 Examples
•  AR Lego, Welbo, etc
•  Mr Virtuoso
–  AR character more real, more fun
–  On-screen 3D and AR similar in usefulness

Contact Lens Display
 Babak Parviz
•  University Washington
 MEMS components
•  Transparent elements
•  Micro-sensors
 Challenges
•  Miniaturization
•  Assembly
•  Eye-safe

Contact Lens Prototype

Conclusion

Conclusion
 There is need for better designed AR experiences
 Through
•  use of Interaction Design principles
•  understanding of the technology
•  use of rapid prototyping tools
•  rigorous user evaluation
 There a number of important areas for future research
•  Natural interaction
•  Multimodal interfaces
•  Intelligent agents
•  Novel displays

More Information
•  Mark Billinghurst
– mark.billinghurst@hitlabnz.org
•  Websites
– www.hitlabnz.org
•  Henry Duh
– hblduh@gmail.com

Resources

Websites
 Meta List of AR SDKs
•  http://www.icg.tugraz.at/Members/gerhard/augmented-reality-sdks
 ARToolKit Software Download
•  http://artoolkit.sourceforge.net/
 ARToolKit Documentation
•  http://www.hitl.washington.edu/artoolkit/
 ARToolKit Forum
•  https://www.artoolworks.com/community/forum/
 ARToolworks Inc
•  http://www.artoolworks.com/

 ARToolKit Plus
•  http://studierstube.icg.tu-graz.ac.at/handheld_ar/artoolkitplus.php
 osgART
•  http://www.osgart.org/
 FLARToolKit
•  http://www.libspark.org/wiki/saqoosha/FLARToolKit/
 FLARManager
•  http://words.transmote.com/wp/flarmanager/

AR Labs
 Europe
•  TU Graz, Cambridge U, TU Munich, FraunhoferIGD
 USA
•  Columbia U, Georgia Tech, USC
 Asia
•  KIST, KAIST
•  AIST, Kyoto U, NAIST, U of Tsukuba
•  NUS, UniSA, HITLab NZ
 Companies
•  Qualcomm, Nokia, Layar, Wikitube, Metaio

Books
 Interactive Environments with Open-Source
Software: 3D Walkthroughs and Augmented Reality
for Architects with Blender 2.43, DART 3.0 and
ARToolKit 2.72 by Wolfgang Höhl
 A Hitchhikers Guide to Virtual Reality by Karen
McMenemy and Stuart Ferguson
 Bimber, Raskar. Spatial Augmented Reality (2005)

 Books
Mobile Interaction Design
Matt Jones and Gary Marsden
Designing for Small Screens
Studio 7.5
Handheld Usability
Scott Weiss
Designing the Mobile User Experience
Barbara Ballard

Publication venues
  Conference
•  IEEE/ACM International Symposium in Mixed and Augmented Reality (IEEE/
ACM ISMAR) (ismar.net)
•  IEEE Virtual Reality (IEEE VR)
•  Korean-Japan Mixed Reality Workshop (KJMR)
  Journal
•  IEEE Transaction on Visualization and Computer Graphics (IEEE)
•  Computer & Graphics (Elsevier)
•  PRESENCE (MIT Press)
  Papers
•  Zhou, F., Duh, H.B.L., and Billinghurst, M. (2008). Trends in Augmented Reality Tracking,
Interaction and Display: A Review of Ten Years of ISMAR. in IEEE International Symposium
on Mixed and Augmented Reality (IEEE/ACM ISMAR) 193-202
•  Azuma, R., Baillot, Behringer, R., Feiner, S., Julier, S., MacIntyre, B., (2001). Recent
Advances in Augmented Reality, IEEE Computer Graphics and Applications, 34-47

More Papers
  E. Kruijff, J. E. Swan, and S. Feiner. Perceptual issues in augmented reality
revisited. 9th IEEE International Symposium on Mixed and Augmented
Reality (ISMAR), 2010, pp. 3--12.
  D. Drascic and P. Milgram. Perceptual issues in augmented reality. In M. T.
Bolas, S. S. Fisher, and J. O. Merritt, editors, SPIE Volume 2653:
Stereoscopic Displays and Virtual Reality Systems III, pages 123-134,
January/February 1996.
270

Developer Guidelines
 Palm
http://www.access-company.com/developers/documents/docs/ui/
UI_Design.html
 Zen of Palm guidelines
http://www.access-company.com/developers/documents/docs/
zenofpalm.pdf
 Motorola
http://developer.motorola.com/docstools/developerguides/
 iPhone Human Interface Guidelines
http://developer.apple.com/documentation/iPhone/Conceptual/iPhoneHIG/

Handheld HCI Design
Websites
Do’s and Don’ts of PocketPC design
http://www.pocketpcmag.com/_archives/Nov04/Commandements.aspx
Usability special interest group – handheld usability
http://www.stcsig.org/usability/topics/handheld.html
Usable Mobile website
http://www.smartgroups.com/groups/usablemobile
Mobile Coders Website
http://www.mobilecoders.com/Articles/mc-01.asp
Univ of Waikato Handheld Group
http://www.cs.waikato.ac.nz/hci/pdas.html
Mobile Interaction Website
http://www.cs.waikato.ac.nz/~mattj/mwshop.html

CHI 2013 DARE Course

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à CHI 2013 DARE Course

Similaire à CHI 2013 DARE Course (20)

Plus de Mark Billinghurst

Plus de Mark Billinghurst (20)

Dernier

Dernier (20)

CHI 2013 DARE Course