Talk on Designing Augmented Reality Experiences given by Mark Billinghurst at the AWE 2013 conference on June 5th 2013

1. Designing Augmented
Reality Experiences
Mark Billinghurst
mark.billinghurst@hitlabnz.org
The HIT Lab NZ, University of Canterbury
June 5th 2013

2. How Would You Design This?
 Put nice AR Picture here – and video

3. Or This?

4. DARE 101
1. Know the Technology
2. Design for User Experience
 All aspects of user experience
3. Follow good Interaction Design principles
 Discover, Design, Evaluate
4. Consider all the Design Elements
 Physical, Virtual and Metaphorical
5. Know Future Research Directions

5. Know the Technology

6. What is Augmented Reality?
 Defining Characteristics (Azuma 97)
• Combines Real and Virtual Images
– Both can be seen at the same time
• Interactive in real-time
– The virtual content can be interacted with
• Registered in 3D
– Virtual objects appear fixed in space
Azuma, R., A Survey of Augmented Reality, Presence, Vol. 6, No. 4, August 1997, pp. 355-385.

7. AR From Science Fiction to Fact
1977 – Star Wars
2008 – CNN

8. AR Part of MR Continuum
Mixed Reality
Reality - Virtuality (RV) Continuum
Real
Environment
Augmented
Reality (AR)
Augmented
Virtuality (AV)
Virtual
Environment
"...anywhere between the extrema of the virtuality continuum."
P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays
IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.

9. Core Technologies
 Combining Real and Virtual Images
• Display technologies
 Interactive in Real-Time
• Input and interactive technologies
 Registered in 3D
• Viewpoint tracking technologies
Display
Processing
Input Tracking

10. Display Technologies
 Types (Bimber/Raskar 2003)
 Head attached
• Head mounted display/projector
 Body attached
• Handheld display/projector
 Spatial
• Spatially aligned projector/monitor
 HMD Optical vs. Video see-through
 Optical: Direct view of real world -> safer, simpler
 Video: Video overlay -> more image registration options

11. Display Taxonomy

12. AR Input Technologies
 Tangible objects
• Tracked items
 Touch (HHD)
• Glove, touch
 Gesture
• Glove, free-hand
 Speech/Multimodal
 Device motion
• HHD + sensors

13. Tracking Technologies
 Active
• Mechanical, Magnetic, Ultrasonic
• GPS, Wifi, cell location
 Passive
• Inertial sensors (compass, accelerometer, gyro)
• Computer Vision
• Marker based, Natural feature tracking, model based
 Hybrid Tracking
• Combined sensors (eg Vision + Inertial)

14. Design for User Experience

15. “The product is no longer
the basis of value. The
experience is.”
Venkat Ramaswamy
The Future of Competition.
Interaction Design

16. experiences
services
products
components
Value
Gilmore + Pine: Experience Economy
Function
Emotion

17. experiences
applications
tools
components
Designing AR Experiences
Tracking, Display, Input
Authoring
Interaction
Usability

18. The Value of Good User Experience
Kenya: 20c
My house: 50c
Starbucks: $3.50

19. Good Experience Design
 Reactrix
 Top down projection
 Camera based input
 Reactive Graphics
 No instructions
 No training

20. Would You Wear This?

21. User Experience is All About You
 Designing good user
experience involves
many aspects
 Consider all the
needs of the user
 Especially context of
use

22.  Web Based AR
 Flash, HTML 5 based AR
 Marketing, education
 Outdoor Mobile AR
 GPS, compass tracking
 Viewing Points of Interest in real world
 Handheld AR
 Vision based tracking
 Marketing, gaming
 Location Based Experiences
 HMD, fixed screens
 Museums, point of sale, advertising
Typical AR Experiences

23. What Makes a Good AR Experience?
 Compelling
 Engaging, ‘Magic’ moment
 Intuitive, ease of use
 Uses existing skills
 Anchored in physical world
 Seamless combination of real and digital

24. Demo: colAR
 Turn colouring books pages into AR scenes
 Markerless tracking, use your own colours..
 Try it yourself: http://www.colARapp.com/

25. Follow Good Interaction
Design Principles

26. Interaction Design
 Answering three questions:
 What do you do? - How do you affect the world?
 What do you feel? – What do you sense of the world?
 What do you know? – What do you learn?
 The Design of User
Experience with Technology
“Designing interactive products to
support people in their everyday
and working lives”
Preece, J., (2002). Interaction Design

27. Interaction Design Process
Interaction Design

28. AR UI Design
 Consider your user
 Follow good HCI principles
 Adapt HCI guidelines for AR
 Design to device constraints
 Using Design Patterns to Inform Design
 Design for you interface metaphor
 Design for evaluation

29. Consider Your User
 Consider context of user
 Physical, social, emotional, cognitive, etc
 Mobile Phone AR User
 Probably Mobile
 One hand interaction
 Short application use
 Need to be able to multitask
 Use in outdoor or indoor environment
 Want to enhance interaction with real world

30. AR vs. Non AR Design
 Design Guidelines
 Design for 3D graphics + Interaction
 Consider elements of physical world
 Support implicit interaction
Characteristics Non-AR Interfaces AR Interfaces
Object Graphics Mainly 2D Mainly 3D
Object Types Mainly virtual objects Both virtual and physical objects
Object behaviors Mainly passive objects Both passive and active objects
Communication Mainly simple Mainly complex
HCI methods Mainly explicit Both explicit and implicit

31. Maps vs. Junaio
 Google Maps
 2D, mouse driven, text/image heavy, exocentric
 Junaio
 3D, location driven, simple graphics, egocentric

32. Design to Device Constraints
 Understand the platform and design for limitations
 Hardware, software platforms
 Eg Handheld AR game with visual tracking
 Use large screen icons
 Consider screen reflectivity
 Support one-hand interaction
 Consider the natural viewing angle
 Do not tire users out physically
 Do not encourage fast actions
 Keep at least one tracking surface in view32
Art of Defense Game

33. Design Patterns
“Each pattern describes a problem which occurs
over and over again in our environment, and then
describes the core of the solution to that problem in
such a way that you can use this solution a million
times over, without ever doing it the same way twice.”
– Christopher Alexander et al.
Use Design Patterns to Address Reoccurring Problems
C.A. Alexander, A Pattern Language, Oxford Univ. Press, New York, 1977.

34. Handheld AR Patterns
Title Meaning Embodied Skills
Device Metaphors Using metaphor to suggest available player
actions
Body A&S Naïve physics
Control Mapping Intuitive mapping between physical and
digital objects
Body A&S Naïve physics
Seamful Design Making sense of and integrating the
technological seams through game design
Body A&S
World Consistency Whether the laws and rules in
physical world hold in digital world
Naïve physics
Environmental A&S
Landmarks Reinforcing the connection between digital-
physical space through landmarks
Environmental A&S
Personal Presence The way that a player is represented in the
game decides how much they feel like living
in the digital game world
Environmental A&S
Naïve physics
Living Creatures Game characters that are responsive to
physical, social events that mimic behaviours
of living beings
Social A&S Body A&S
Body constraints Movement of one’s body position
constrains another player’s action
Body A&S Social A&S
Hidden information The information that can be hidden and
revealed can foster emergent social play
Social A&S Body A&S

35. Example: Seamless Design
 Design to reduce seams in the user experience
 Eg: AR tracking failure, change in interaction mode
 Paparazzi Game
 Change between AR tracking to accelerometer input
Yan Xu , et.al. , Pre-patterns for designing embodied interactions in handheld augmented reality
games, Proceedings of the 2011 IEEE International Symposium on Mixed and Augmented Reality--
Arts, Media, and Humanities, p.19-28, October 26-29, 2011

36. Example: Living Creatures
 Virtual creatures respond to real world events
 eg. Player motion, wind, light, etc
 Creates illusion creatures are alive in the real world
 Sony EyePet
 Responds to player blowing on creature
36

37. Rapid Hardware Prototyping
 Speed development time by using quick hardware mockups
 Handheld connected to PC, LCD screen, USB phone keypad,
Camera
 Can use PC tools for rapid application development
 Flash, Visual Basic, etc

38. Build Your Own Google Glass
 Rapid Prototype Glass-Like HMD
 Myvu HMD + headphone + iOS Device + basic glue skills
 $300 + less than 3 hours construction
 http://www.instructables.com/id/DIY-Google-Glasses-AKA-the-Beady-i/

39. Why Evaluate AR Applications?
 To test and compare interfaces, new technologies,
interaction techniques
 To validate the efficiency and efficient the AR
interface and system
 Test Usability (learnability, efficiency, satisfaction,...)
 Get user feedback, Better understand your users
 Refine interface design
 Better understand your end users
 ...

40. HIT Lab NZ Usability Survey
 A Survey of Evaluation Techniques Used in
Augmented Reality Studies
 Andreas Dünser, Raphaël Grasset, Mark Billinghurst
 reviewed publications from 1993 to 2007
 Extracted 6071 papers which mentioned “Augmented
Reality”
 Searched to find 165 AR papers with User Studies

41. Types of Experiments and topics
 Sensation, Perception & Cognition
 How is virtual content perceived ?
 What perceptual cues are most important ?
 How to visualize augmented/overlay information on real environment?
 Visual search/attention/salience issues of human performance
 Interaction
 How can users interact with virtual content ?
 Which interaction techniques are most efficient in certain context ?
 Collaboration & Social issues
 How is collaboration in AR interface different ?
 Which collaborative cues can be conveyed best ?
 Privacy and security issues of AR interface

42. Gabbard Model for AR Design
1. user task analysis
2. expert guidelines-based evaluation
3. formative user-centered evaluation
4. summative comparative evaluations
Gabbard, J.L.; Swan, J.E.; , "Usability Engineering
for Augmented Reality: Employing User-Based
Studies to Inform Design,”
Visualization and Computer Graphics, IEEE Transactions
on, vol.14, no.3, pp.513-525, May-June 2008

43. Gabbard Model in Context

44. Consider All Design Elements

45.  Interface Components
 Physical components
 Display elements
- Visual/audio
 Interaction metaphors
Physical
Elements
Virtual
Elements
Interaction
Metaphor
Input Output
AR Design Elements

46. AR Design Space
Reality Virtual Reality
Augmented Reality
Physical Design Virtual Design

47. Design of Objects
 Objects
 Purposely built – affordances
 “Found” – repurposed
 Existing – already at use in marketplace
 Affordance
 The quality of an object allowing an action-
relationship with an actor
 An attribute of an object that allows people to
know how to use it
- e.g. a door handle affords pulling

48. Affordance Led Design
 Make affordances perceivable
 Provide visual, haptic, tactile, auditory cues
 Affordance Led Usability
 Give feedback
 Provide constraints
 Use natural mapping
 Use good cognitive model

49. Example: AR Chemistry
 Tangible AR chemistry education (Fjeld)
Fjeld, M., Juchli, P., and Voegtli, B. M. 2003. Chemistry education: A tangible interaction
approach. Proceedings of INTERACT 2003, September 1st -5th 2003, Zurich,
Switzerland.

50. Input Devices
 Form informs function and use

51. AR Interaction Metaphors
 Information Browsing
 View AR content
 3D AR Interfaces
 3D UI interaction techniques
 Augmented Surfaces
 Tangible UI techniques
 Tangible AR
 Tangible UI input + AR output

52. 1. Information Browsing
 Information is registered to
real-world context
 Hand held AR displays
 Interaction
 Manipulation of a window
into information space
 Applications
 Context-aware
information displays
Rekimoto, et al. 1997

53. 2. 3D AR Interfaces
 Virtual objects displayed in 3D
physical space and manipulated
 HMDs and 6DOF head-tracking
 6DOF hand trackers for input
 Interaction
 Viewpoint control
 Traditional 3D user interface
interaction: manipulation,
selection, etc.
Kiyokawa, et al. 2000

54. 3. Augmented Surfaces
 Basic principles
 Virtual objects are projected on a surface
 Physical objects are used as controls for
virtual objects
 Support for collaboration
 Rekimoto, et al. 1998
 Front projection
 Marker-based tracking
 Multiple projection surfaces

55. Lessons from Tangible Interfaces
 Physical objects make us smart
 Norman’s “Things that Make Us Smart”
 encode affordances, constraints
 Objects aid collaboration
 establish shared meaning
 Objects increase understanding
 serve as cognitive artifacts

56. TUI Limitations
 Difficult to change object properties
 Can’t tell state of digital data
 Limited display capabilities
 projection screen = 2D
 dependent on physical display surface
 Separation between object and display
 Augmented Surfaces

57. 4. Tangible AR Metaphor
 AR overcomes limitation of TUIs
 enhance display possibilities
 merge task/display space
 provide public and private views
 TUI + AR = Tangible AR
 Apply TUI methods to AR interface design

58. Tangible AR Demo
 Use of natural physical object
manipulations to control virtual objects
 VOMAR Demo
 Catalog book:
- Turn over the page
 Paddle operation:
- Push, shake, incline, hit, scoop

59. Object Based Interaction: MagicCup
 Intuitive Virtual Object Manipulation
on a Table-Top Workspace
 Time multiplexed
 Multiple Markers
- Robust Tracking
 Tangible User Interface
- Intuitive Manipulation
 Stereo Display
- Good Presence

61. Tangible AR Design Principles
 Tangible AR Interfaces use TUI principles
 Physical controllers for moving virtual content
 Support for spatial 3D interaction techniques
 Time and space multiplexed interaction
 Support for multi-handed interaction
 Match object affordances to task requirements
 Support parallel activity with multiple objects
 Allow collaboration between multiple users

62. Example 1: AR Lens
 Physical Components
 Lens handle
- Virtual lens attached to real object
 Display Elements
 Lens view
- Reveal layers in dataset
 Interaction Metaphor
 Physically holding lens

63. Example 2: LevelHead
 Physical Components
 Real blocks
 Display Elements
 Virtual person and rooms
 Interaction Metaphor
 Blocks are rooms

64. Know Future Research
Directions

65. The Vision of AR

66. To Make the Vision Real..
 Hardware/software requirements
 Contact lens displays
 Free space hand/body tracking
 Speech/gesture recognition
 Etc..
 Most importantly
 Usability/User Experience

67. Natural Interaction
 Automatically detecting real environment
 Environmental awareness
 Physically based interaction
 Gesture Input
 Free-hand interaction
 Multimodal Input
 Speech and gesture interaction
 Implicit rather than Explicit interaction

68. AR MicroMachines
 AR experience with environment awareness
and physically-based interaction
 Based on MS Kinect RGB-D sensor
 Augmented environment supports
 occlusion, shadows
 physically-based interaction between real and
virtual objects

69. Physics Simulation
 Create virtual mesh over real world
 Update at 10 fps – can move real objects
 Use by physics engine for collision detection (virtual/real)
 Use by OpenScenegraph for occlusion and shadows

70. Rendering
Occlusion Shadows

71. Gesture Input Architecture
5. Gesture
• Static Gestures
• Dynamic Gestures
• Context based Gestures
4. Modeling
• Hand recognition/modeling
• Rigid-body modeling
3. Classification/Tracking
2. Segmentation
1. Hardware Interface

72. Results

73. Free Hand Multimodal Input
 Use free hand to interact with AR content
 Recognize simple gestures
 No marker tracking
Point Move Pick/Drop

74. Multimodal Architecture

75. Multimodal Fusion

76. Hand Occlusion

77. Conclusion

78. Conclusion
 There is need for better designed AR experiences
 Through
 use of Interaction Design principles
 understanding of the technology
 use of rapid prototyping tools
 rigorous user evaluation
 There a number of important areas for future research
 Natural interaction, Multimodal interfaces, Intelligent agents, …

79. More Information
• Mark Billinghurst
– mark.billinghurst@hitlabnz.org
• Websites
– www.hitlabnz.org

80. 80
Resources