Lecture 11 of the COMP 4010 class on Augmented Reality and Virtual Reality. This lecture is about VR applications and was taught by Mark Billinghurst on October 19th 2021 at the University of South Australia
Mark BillinghurstDirector at HIT Lab NZ à University of South Australia
3. How can we Interact in VR?
• How can VR devices create a natural user experience?
4. Universal 3D Interaction Tasks in VR
• Object Interaction
• Selection: Picking object(s) from a set
• Manipulation: Modifying object properties
• Navigation
• Travel: motor component of viewpoint motion
• Wayfinding: cognitive component; decision-making
• System control
• Issuing a command to change system state or mode
5. Selection and Manipulation
• Selection:
• specifying one or more objects from a set
• Manipulation:
• modifying object properties
• position, orientation, scale, shape, color, texture, behavior, etc.
7. Ray-casting technique
• “Laser pointer” attached
to virtual hand
• First object intersected by
ray may be selected
• User only needs to
control 2 DOFs
• Proven to perform well
for remote selection
• Variants:
• Cone casting
• Snap-to-object rays
9. World-in-miniature (WIM) technique
• “Dollhouse” world held in
user’s hand
• Miniature objects can be
manipulated directly
• Moving miniature objects
affects full-scale objects
• Can also be used for
navigation
Stoakley, R., Conway, M., & Pausch, R. (1995). Virtual Reality on a WIM: Interactive Worlds in
Miniature. Proceedings of CHI: Human Factors in Computing Systems, 265-272, and
Pausch, R., Burnette, T., Brockway, D., & Weiblen, M. (1995). Navigation and Locomotion in
Virtual Worlds via Flight into Hand-Held Miniatures. Proceedings of ACM SIGGRAPH, 399-400.
10. Navigation
• How we move from place to place within an environment
• The combination of travel with wayfinding
• Wayfinding: cognitive component of navigation
• Travel: motor component of navigation
• Travel without wayfinding: "exploring", "wandering”
11. Types of Travel
• Exploration
• No explicit goal for the movement
• Search
• Moving to specific target location
• Naïve – target position not known
• Primed – position of target known
• Maneuvering
• Short, precise movements changing viewpoint
12. Gaze Directed Steering
• Move in direction that you are looking
• Very intuitive, natural navigation
• Can be used on simple HMDs (e.g. Google Cardboard)
• But: Can’t look in different direction while moving
13. TelePortation
• Use controller to select end point
• Usable with 3DOF contoller
• Jump to a fixed point in VR
• Discrete motion can be confusing/cause sickness
14. Redirected Walking
• Address problem of limited
walking space
• Warp VR graphics view of
space
• Create illusion of walking
straight, while walking in circles
Razzaque, S., Kohn, Z., & Whitton, M. C. (2001, September). Redirected walking.
In Proceedings of EUROGRAPHICS (Vol. 9, pp. 105-106).
15. Wayfinding – Making Cognitive Maps
• Goal of Wayfinding is to build Mental Model (Cognitive Map)
• Types of spatial knowledge in a mental model
• landmark knowledge
• procedural knowledge (sequence of actions required to follow a path)
• map-like (topological) knowledge
• Creating a mental model
• systematic study of a map
• exploration of the real space
• exploration of a copy of the real space
• Problem: Sometimes perceptual judgments are incorrect
within a virtual environment
• e.g. users wearing a HMD often underestimate dimensions of space,
possibly caused by limited field of view
16. Designing VE to Support Wayfinding
• Provide Landmarks
• Any obvious, distinct and non-mobile
object can serve as a landmark
• A good landmark can be seen from
several locations (e.g. tall)
• Audio beacons can also serve as
landmarks
• Use Maps
• Copy real world maps
• Ego-centric vs. Exocentric map cues
• World in Miniature
• Map based navigation
17. System Control
• Issuing a command to change system state or mode
• Examples
• Launching application
• Changing system settings
• Opening a file
• Etc.
• Key points
• Make commands visible to user
• Support easy selection
18. 2D Menus in VR
• Many examples of 2D GUI and floating menus in VR
Nested Pie Menu
2D Menu in VR CAVE
19. How Can we Design Useful VR?
• Designing VR experiences that meet real needs
20. The Interaction Design Process
Evaluate
(Re)Design
Identify needs/
establish
requirements
Build an
interactive
version
Final Product
Develop alternative prototypes/concepts and compare them
And iterate, iterate, iterate....
21. Methods for Identifying User Needs
Learn
from
people
Learn
from
analogous
settings
Learn
from
Experts
Immersive
yourself in
context
22. Is VR the Best Solution?
• Not every problem can be solved by VR..
• Problems Ideal for Virtual Reality, have:
• visual elements
• 3D spatial interaction
• physical manipulation
• procedural learning
• Problems Not ideal for Virtual Reality, have:
• heavy reading, text editing
• many non-visual elements
• need for connection with real world
• need for tactile, haptic, olfaction feedback
23. Typical VR Interface Metaphors
• Direct Manipulation
• Reach out and directly grab objects
• Ray Casting
• Select objects through ray from head/hand
• Vehicle Movement
• Move through VR environment through vehicle movement
24. Affordances in VR
• Design interface objects to show how they are used
• Use visual cues to show possible affordances
• Perceived affordances should match actual affordances
• Good cognitive model - map object behavior to expected
Familiar objects in Job Simulator Object shape shows how to pick up
25. UX Guidelines for VR
• The Four Cores of UX Design for VR
• Make interface Interactive and Reactive
• Design for Comfort and Ease
• Use usable Text and Image Scale
• Include position audio and 3D sound
https://www.dtelepathy.com/blog/philosophy/ux-guide-designing-virtual-reality-experiences
26. Cardboard Design Lab
• Mobile VR App providing examples of best practice VR
designs and user interaction (iOS, Play app stores)
27. What is Evaluation?
•Evaluation is concerned with
gathering data about the usability
of a design or product by a
specified group of users for a
particular activity within a specified
environment or work context
29. Quick and Dirty
• ‘quick & dirty’ evaluation: informal feedback from
users or consultants to confirm that their ideas are
in-line with users’ needs and are liked.
• Quick & dirty evaluations are done any time.
• Emphasis is on fast input to the design process
rather than carefully documented findings.
30. Usability Testing
• Recording typical users’ performance on typical
tasks in controlled settings.
• As the users perform tasks they are watched &
recorded on video & their inputs are logged.
• User data is used to calculate performance times,
errors & help determine system usability
• User satisfaction questionnaires & interviews are
used to elicit users’ opinions.
31. Field/Ethnographic Studies
• Field studies are done in natural settings
• The aim is to understand what users do naturally
and how technology impacts them.
• In product design field studies can be used to:
- identify opportunities for new technology
- determine design requirements
- decide how to introduce new technology
- evaluate technology in use.
32. Predictive Evaluation
• Experts apply their knowledge of typical
users, often guided by heuristics, to
predict usability problems.
• Can involve theoretically based models.
• A key feature of predictive evaluation is
that users need not be present
• Relatively quick and inexpensive
33. Pilot Studies
• A small trial run of the main study.
• Can identify majority of issues with interface design
• Pilot studies check:
- that the evaluation plan is viable
- you can conduct the procedure
- that interview scripts, questionnaires,
experiments, etc. work appropriately
• Iron out problems before doing the main study.
34. Controlled Experiments
• Designer of a controlled experiment should
carefully consider:
• proposed hypothesis
• selected subjects
• measured variables
• experimental methods
• data collection
• data analysis
35. Subjects
• The choice of subjects is critical to the validity of the
results of an experiment
• subjects group should represent expected user population
expected user population
• Consider subject factors such as:
• age group, education, skills, culture, technology background
• The sample size should be large enough (10+) to be
statistically representative of the user population
36. Hypothesis and Variables
• Hypothesis: prediction of the experiment outcome
• Experiments manipulate and measure variables
under controlled conditions
• There are two types of variables
• independent: variables that are manipulated to create
different experimental conditions
• e.g. number of items in menus, colour of the icons
• dependent: variables that are measured to find out the
effects of changing the independent variables
• e.g. speed of menu selection, speed of locating icons
37. Experimental Methods
Randomly
assigned
Statistical data analysis
Experimental
task
Condition
2
Condition
3
Condition
1
Subjects
data data data
Between-
groups
Randomly
assigned
Statistical data analysis
Subjects
data data data
Within-
groups
Experimental
tasks
Condition
2
Condition
3
Condition
1
Experimental
tasks
Condition
1
Condition
3
Condition
2
Experimental
tasks
Condition
1
Condition
2
Condition
3
38. Data Types
• Subjective (Qualitative)
• Subjective survey
• Likert Scale, condition rankings
• Observations
• Think Aloud
• Interview responses
• Objective (Quantitative)
• Performance measures
• Time, accuracy, errors
• Process measures
• Video/audio analysis
How easy was the task
1 2 3 4 5
Not very easy Very easy
41. Virtual Reality Applications
• Ideal applications for VR should:
• Be strongly visual, have 3D spatial elements
• Benefit from first person immersion
• Benefit from 3D manipulation/navigation
• Support Autonomy, Interaction and Presence (AIP Cube)
• Etc..
43. Many Possible Types of VR Applications
From https://www.slideshare.net/ampnewventures/virtual-reality-vr-continuum-amp-new-ventures
45. Potential Disruption for Existing Domains
https://www.slideshare.net/BDMIFund/the-emerging-virtual-reality-landscape-a-primer
46. Example VR Applications
• Education
• Google Expeditions
• Medicine
• Virtual Characters
• Entertainment
• The Void, Zero Latency
• Art + Design
• Tilt Brush
• Collaboration
• Facebook Spaces
49. Google Expeditions
• Goal: Provide low cost educational VR experience
• Based on Google Cardboard VR platform
• Different roles:
• Guide— person leading an expedition on a tablet
• Explorer— person following an expedition on a phone.
• Usage
• Used by millions of students
• Over 1000 educational experiences developed
• Royal Collection Trust, American Museum of Natural History, etc.
50. Teacher Led VR Experiences
• Teacher/Guide uses tablet to control the experience
• Selects the virtual tour experience
• Guide sees tour script, can select immersive scenes to view
• Guide sees focus point and where individual students are looking
• Students connect as followers, look at what guides highlight
Guide Interface
51. System
• Hardware
• Google Cardboard mobile viewer
• Smart phones + tablet (class set)
• Wireless router
• Software
• Viewer and Guide applications (iOS/Android)
• 360 image/video VR experiences
Class set for 30 students
52. Example Experiences
• Over 1000 locations/experiences
• Great barrier reef, Great Wall of China, Grand Canyon, etc.
54. Feedback
• Teacher/student survey (100 people)
• 65% experienced a “Wow” moment during Google expedition
• Noted the variety of educator styles and approaches possible
• People enjoyed “The feeling of ‘being’ there”
From https://www.slideshare.net/zoesujon/google-expeditions-virtual-reality-and-the-classroom
55. Limitations
• But 53% of participants identified some problems:
• Difficult for some people who wore glasses
• Some complained of eye strain, headaches or nausea
• Some staff were reluctant/resistant to use the leader tablet
• Issues of disabilities and inclusion
56. Key Findings
• Low-cost VR/mobile VR can provide a valuable
educational experience
• Visit different locations, different times, etc.
• Teach interaction key
• Acting as guide, providing educational context
• VR requires more work
• Address simulator sickness, ergonomic issues, etc.
• Immersion/Presence creates learning
• Immersion creates memorable educational experience
57. Challenges/Solutions
• Making VR accessible
• Designing for phones, tablets, low cost viewers
• Synchronizing content with all viewers
• Teacher controlled viewing
• Teacher can guide experiences
• Engaging interaction on simple viewers
• Head pointing based interaction, button input
• Supporting Educational goals
• Providing compelling educational content
59. Virtual Patients
• Problem
• Many doctors have poor doctor/patient skills
• Have limited opportunity during training to learn skills
• Solution
• Virtual patients that doctors can communicate with naturally
• Artificial agents with speech understanding
60. Typical System Setup
• Trainee in front of projection screen
• Speech and gesture recognition
• Intelligent agent on screen
Johnsen, K., Raij, A., Stevens, A., Lind, D. S., & Lok, B. (2007, April). The validity of a virtual human experience for interpersonal
skills education. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 1049-1058). ACM.
62. Key Findings
• Virtual Humans can replace actors in training
• interaction skills used with a virtual human translate to
the interaction skills used with a real human
• Students feel a strong sense of co-presence
• Having character respond to speech and gesture
increases immersion
• VR is capable of creating realistic characters
• Life size, intelligent backend, speech recognition
• Skills learnt transfer to real world
63. Challenges/Solutions
• Training in medical environment
• Design for training in medical exam room
• Use projected VR not HMDs
• Natural interaction
• Support speech and gesture interaction
• Tactile/haptic feedback
• Use prosthetics to add support for palpation and other
tactile interaction between doctor and virtual patient
• Supporting Educational goals
• Give virtual character domain knowledge
65. Large Scale VR Gaming
• Provide multi-player VR gaming in warehouse space
• Examples
• The Void - https://www.thevoid.com/
• Zero Latency - https://zerolatencyvr.com/
66. Typical System
• Wide Area Tracking
• Computer vision, lights/reflective balls
• > 120 cameras for 300 m2
space
• Backpack VR system
• Haptic feedback, wireless HMD
• Real Props
• Tracked objects, walls
Tracking cameras
Backpack system
68. Key Findings
• Wide area tracking possible
• vision based systems can create large scale wide areas
tracking, fast enough for game play
• Shared gameplay improves experience
• Focus on collaborative experiences, using avatar
representations and roll division
• Haptic feedback significantly increases presence
• Use of physical props (objects, walls)
• Content is king
• Systems need compelling content/game place
69. Challenges/Solutions
• Wide area tracking
• Computer vision tracking of people
• Over 100 cameras + multiple servers
• Freedom of movement
• Custom wireless VR backpacks
• Ruggedized HMDs, weapon props
• Natural interaction
• Redirected walking, tangible props
• Compelling content
• Multi-sensory feedback, custom game platform
71. Tilt Brush
• Intuitive 3D immersive drawing/sculpting program
• Developed by Patrick Hackett and Drew Skillman 2014
• Acquired by Google in 2015/Open sourced 2021
• https://www.tiltbrush.com/
72. Functionality
• Goal: Extremely natural 3D painting/sculpting
• User Interface
• Two handed interface designed for two controllers (Vive, Rift)
• Brush in dominant hand, tool palette in non-dominant
• Typical drawing functionality – color, brush width, undo/redo, etc..
• Content sharing
• Created content can be exported/shared in 2D/3D formats
76. Key Findings
• Use familiar tools
• Tilt brush interface has familiar sculpting/painting tools –
e.g. brush size, colour pallet, etc
• Use intuitive interface
• Two handed tools with natural metaphor – one hand for
pallet/menu, one hand for painting/sculpting
• Provide Magical experience
• Provide experience not possible in real world, e.g.
changing body scale, painting in 3D, etc.
• Create a community
• Provide ways for people to share content
77. Challenges/Solutions
• Intuitive Interface
• Very natural metaphor – painting in space
• Two handed interface – map to VR controllers
• Familiar menu objects from paint programs
• Need for limited training
• Provide in app training, tool tips
• Content sharing
• Enable content to be exported in variety of formats
• Video, animated GIFs, 2D images, 3D files
• Engaging Experience
• Provides novel immersive artistic experience
81. A wide variety of communication cues used.
Speech
Paralinguistic
Para-verbals
Prosodics
Intonation
Audio
Gaze
Gesture
Face Expression
Body Position
Visual
Object Manipulation
Writing/Drawing
Spatial Relationship
Object Presence
Environmental
Face to Face Communication
82. Face to Face Communication
Audio Cues
Visual Cues
Environmental Cues
88. Why is Video Conferencing so Tiring?
• Physical factors
• Poor posture
• Staring at fixed screen
• Longer hours/work anytime
• Cognitive factors
• Loss of non-verbal cues, eye gaze
• Need to focus more intently
• Multi-tasking/split attention
• Social factors
• Aware of being watched
• Seeing yourself
• Constant staring
“It's almost like you're
emoting more because
you're just a little box on a
screen”. “I’m just so tired.”
“You have to fill in the
gaps. And that takes
cognitive energy. You get
tired more quickly.”
“For somebody who’s
really dependent on those
non-verbal cues, it can be
a big drain not to have
them.”
92. Limitations with Current Technology
•Lack of spatial cues
• Person blends with background
•Poor communication cues
• Limited gaze, gesture, non-verbal communication
•Introduction of artificial seams
• Separation of task/communication space
93. Facebook Spaces
• Collaborative VR environment
• VR meeting and interaction space (up to 4 people)
• Focus on communication
• Speech and gesture based
• https://www.facebook.com/spaces
94. System Interaction
• Designed for Oculus Rift/HTC Vive
• Upper body tracking, touch controllers
• Simple interaction
• Loading scenes, direct object manipulation
• Content creation
• Selfie pictures, simple sketching
96. Facebook Workrooms
• Designed to support meetings
• Meeting seating
• Shared blackboard
• Limited movement
• Support for real devices
• Keyboard/mouse input
• Calibrated computer screen
• Private space/public space
• Rich communication cues
• Lip sync
• Natural gestures
103. Other Examples
• Many other examples of collaborative VR
• Rec Room, High Fidelity, AltspaceVR
• Sansar, VR chat, etc..
104. Mozilla Hubs
• Web based social VR
• Customizable avatars, spatial audio, 3D environments
• Multi-device support
• HMDs, Desktop, Mobile
• Open source
• https://hubs.mozilla.com
106. Mozilla Hubs Spoke Creator
• Drag and drop scene editing
• Upload assets
• 3D objects, textures, animaton
• Customize content – Sketchup, Google Poly, Sketchfab, etc
109. Key Findings
• Minimal social cues okay
• Even simple avatars can provide rich social experience
• Create shared social context
• Important to place users in same shared Virtual Reality
environment/shared social context
• Audio is key
• Provide low latency audio, spatial audio cues
• Create a reason for communicating
• Why should people want to connect? Create shared
activity/reason for people to conference
110. Challenges/Solutions
• Create shared sense of Presence
• Use common background, shared objects
• Natural communication
• Support non-verbal behaviour, speech/gesture input
• Intuitive interaction
• Map real body motion onto Avatars
• Limited ability to navigate/move through environment
• Engaging Experience
• Shared content creation, experience capture
114. Best VR Apps of 2019 (Digital Trends)
• ALLUMETTE – VR Stop motion film
• Google Earth – Travel/geography
• Kingspray Graffiti – Art/content creation
• The FOO Show - VR Talk show
• Virtual Desktop – Use desktop in VR
• www.digitaltrends.com/virtual-reality/best-virtual-reality-apps/
123. The ultimate display would, of course, be a room within
which the computer can control the existence of matter. A
chair displayed in such a room would be good enough to sit
in. Handcuffs displayed in such a room would be confining,
and a bullet displayed in such a room would be fatal..
Sutherland, Ivan. "The ultimate display." (1965).
139. Future Visions of VR: Ready Player One
• https://www.youtube.com/watch?v=LiK2fhOY0nE
140. Today vs. Tomorrow
VR in 2021 VR in 2045
Graphics High quality Photo-realistic
Display 110-150 degrees Total immersion
Interaction Handheld controller/some gesture Full gesture/body/gaze
Navigation Limited movement Natural
Multiuser Few users Millions of users
143. “.. the technologies that will significantly
affect our lives over the next 10 years
have been around for a decade. The
future is with us ... The trick is learning
how to spot it”
October 2004
Bill Buxton
145. Conclusions
• AR/VR/MR is becoming commonly available
• Significant advances over 50+ years
• In order to achieve Sutherland’s vision, need research in
• Display, Tracking, Input
• New MR technologies will enable this to happen
• Display devices, Interaction, Tracking technologies
• There are still significant areas for research
• Social Acceptance, Perception, Collaboration, Etc.