1. NOVEL INTERFACES FOR AR SYSTEMS
Mark Billinghurst
mark.billinghurst@unisa.edu.au
September 2022

2. THE EMPATHIC COMPUTING LABORATORY

3. Two Amazing Locations
• University of South Australia, Adelaide
• Highest number of AR research outputs in world
• University of Auckland, Auckland
• Top ranked university in New Zealand

4. One Amazing Team
• Staff (6)
• 2 faculty, 3 post docs, 1 engineer
• Students (39)
• 17 PhD (+2), 2 Masters, 6 undergraduate, 14 interns

6. “Only through
communication
can Human Life
hold meaning.”
Paulo Freire

7. Modern Technology Trends
1. Improved Content Capture
• Move from sharing faces to sharing places
2. Increased Network Bandwidth
• Sharing natural communication cues
3. Implicit Understanding
• Recognizing behaviour and emotion

8. Natural
Collaboration
Implicit
Understanding
Experience
Capture

9. Natural
Collaboration
Implicit
Understanding
Experience
Capture
Empathic
Computing

10. “Empathy is Seeing with the
Eyes of another, Listening with
the Ears of another, and Feeling
with the Heart of another..”
Alfred Adler

11. Empathic Computing Research Focus
Can we develop systems that allow
us to share what we are seeing,
hearing and feeling with others?

12. Key Elements of Empathic Systems
•Understanding
• Emotion Recognition, physiological sensors
•Experiencing
• Content/Environment capture, VR
•Sharing
• Communication cues, AR

13. Example Projects
• Remote collaboration in Wearable AR
• Sharing of non-verbal cues (gaze, pointing, face expression, emotion)
• Shared Empathic VR experiences
• Use VR to put a viewer inside the players view
• Measuring emotion
• Detecting emotion from heart rate, GSR, eye gaze, etc.

14. Empathy Glasses (2016)
• Combine together eye-tracking, display, face expression
• Implicit cues – eye gaze, face expression
+
+
Pupil Labs Epson BT-200 AffectiveWear
Masai, K., Sugimoto, M., Kunze, K., & Billinghurst, M. (2016, May). Empathy Glasses. In Proceedings of the
34th Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. ACM.

15. Remote Collaboration
• Eye gaze pointer and remote pointing
• Face expression display
• Implicit cues for remote collaboration

17. Shared Sphere – 360 Video Sharing
Shared
Live 360 Video
Host User Guest User
Lee, G. A., Teo, T., Kim, S., & Billinghurst, M. (2017). Mixed reality collaboration through sharing a
live panorama. In SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications (pp. 1-4).

19. 3D Live Scene Capture
• Use cluster of RGBD sensors
• Fuse together 3D point cloud

20. Scene Capture and Sharing
Scene Reconstruction Remote Expert Local Worker

21. AR View Remote Expert View

22. View Sharing Evolution
• Increased immersion
• Improved scene understanding
• Better collaboration
2D 360 3D

23. NOVELAR INTERFACES

24. Current Generation of AR Glasses
• Trend towards lightweight thin displays
• Offload processing onto second device
• High bandwidth connectivity to cloud services

26. The Challenge of AR Interaction
AR HMD INTERACTION
Several Techniques
• Controllers
• Gestures
• Touch input
Limitations
• Imprecise input
• Simple graphics

27. The Challenge of AR Interaction
HANDHELD AR INTERACTION
Advantages
• Precise touch input
• High resolution display
Limitations
• On viewed on phone screen
• Narrow field of view

28. The Opportunity
AR HMDs tethered to handheld devices/phones
○New opportunity for interaction/display
○Wide field of view of HMD and Precise input of HHD

29. Motivation
Complementary nature of
HMDs/HHDs

30. Previous Work
Our previous work
• Use touch input on phone to interact with AR HMD content
• Use tablet to provide 2D view of 3D AR conferencing space
Bleeker 2013
Budhiraja 2013

32. Secondsight
A prototyping platform for rapidly testing cross-device interfaces
• Enables an AR HMD to "extend" the screen of a smartphone
Key Features
• Can simulate a range of HMD Field of View
• Enables World-fixed or Device-fixed content placement
• Supports touch screen input, free-hand gestures, head-pose selection
Reichherzer, C., Fraser, J., Rompapas, D. C., & Billinghurst, M. (2021, May). Secondsight: A framework for cross-device augmented
reality interfaces. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-6).

33. Implementation
Hardware
• Meta2 AR Glasses (82o FOV)
• Samsung Galaxy S8 phone
• OptiTrack motion capture
system
Software
• Unity game engine
• Mirror networking library

34. Input

35. Content Placement
© 2021 SIGGRAPH. All Rights Reserved.

36. Field of View Simulation

37. Map application
© 2021 SIGGRAPH. All Rights Reserved.

38. Puzzle application
© 2021 SIGGRAPH. All Rights Reserved.

39. Pilot user study (4 users)
© 2021 SIGGRAPH. All Rights Reserved.
Goal: Explore effect of AR HMD FOV
Task: Data visualisation task
● Find correspondances on health data
Conditions: Simulated Field of View
● 15, 30 , 55, 82 degrees
Measures
● Subjective Feedback on 5-point Likert Scale
○ How difficult was it to use the system (5 = very easy)
● Observing virtual window placement
● Condition ranking

40. Pilot study results
© 2021 SIGGRAPH. All Rights Reserved.
• Users felt wider FOV was easiest to use
• Most users had similar window placement
• Useful feedback on how to improve user interface
Typical Window Placement
FOV Ease of Use
15 deg 1.00 +/- 0.0
30 deg 1.75 +/- 0.5
55 deg 2.75 +/- 1.25
82 deg 3.75 +/- 1.25
Average Ease of Use Rating

41. COLLABORATIVE SYSTEMS

42. Social Panoramas
• Capture and share social spaces in real time
• Supports independent views into Panorama
Reichherzer, C., Nassani, A., & Billinghurst, M. (2014, September). [Poster] Social panoramas using wearable
computers. In 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) (pp. 303-304). IEEE.

43. Implementation
• Google Glass
• Capture live image panorama (compass +
camera)
• Remote device (tablet)
• Immersive viewing, live annotation

44. Interface
Glass View
Tablet View

45. Social Panorama
https://www.youtube.com/watch?v=vdC0-UV3hmY

46. Lessons Learned
• Good
• Communication easy and natural
• Users enjoy have view independence
• Very natural capturing panorama on Glass
• Sharing panorama enhances the shared experience
• Bad
• Difficult to support equal input
• Need to provide awareness cues

47. • Using AR/VR to share communication cues
• Gaze, gesture, head pose, body position
• Sharing same environment
• Virtual copy of real world
• Collaboration between AR/VR
• VR user appears in AR user’s space
Piumsomboon, T., Dey, A., Ens, B., Lee, G., & Billinghurst, M. (2019). The effects of sharing awareness cues
in collaborative mixed reality. Frontiers in Robotics and AI, 6, 5.
Sharing Virtual Communication Cues (2019)

48. Sharing Virtual Communication Cues
• AR/VR displays
• Gesture input (Leap Motion)
• Room scale tracking
• Conditions
• Baseline, FoV, Head-gaze, Eye-gaze

50. Sharing Gaze Cues
How sharing gaze behavioural cues can improve remote collaboration in Mixed Reality environment.
➔ Developed eyemR-Vis, a 360 panoramic Mixed Reality remote collaboration system
➔ Showed gaze behavioural cues as bi-directional spatial virtual visualisations shared
between a local host (AR) and a remote collaborator (VR).
Jing, A., May, K. W., Naeem, M., Lee, G., & Billinghurst, M. (2021). eyemR-Vis: Using Bi-Directional Gaze Behavioural Cues to Improve Mixed
Reality Remote Collaboration. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-7).

51. System Design
➔ 360 Panaramic Camera + Mixed Reality View
➔ Combination of HoloLens2 + Vive Pro Eye
➔ 4 gaze behavioural visualisations:
browse, focus, mutual, fixated circle

52. System Design
Browse Focus
Mutual Fixed
Circle-map

53. Multi-Scale Collaboration
• Changing the user’s virtual body scale

55. Sharing: Communication Cues (2018)
• What happens when you can’t see your colleague/agent?
Piumsomboon, T., Lee, G. A., Hart, J. D., Ens, B., Lindeman, R. W., Thomas, B. H., & Billinghurst, M.
(2018, April). Mini-me: An adaptive avatar for mixed reality remote collaboration. In Proceedings of the
2018 CHI conference on human factors in computing systems (pp. 1-13).
Collaborating Collaborator out of View

56. Mini-Me Communication Cues in MR
• When lose sight of collaborator a Mini-Me avatar appears
• Miniature avatar in real world
• Mini-Me points to shared objects, show communication cues
• Redirected gaze, gestures

58. Results from User Evaluation
• Collaboration between user in AR, expert in VR
• Hololens, HTC Vive
• Two tasks
• Asymmetric, symmetric collaboration
• Significant performance improvement
• 20% faster with Mini-Me
• Social Presence
• Higher sense of Presence
• Users preferred
• People felt the task was easier to complete
• 60-75% preference
“I feel like I am talking
to my partner”

59. Technology Trends
• Advanced displays
• Wide FOV, high resolution
• Real time space capture
• 3D scanning, stitching, segmentation
• Natural gesture interaction
• Hand tracking, pose recognition
• Robust eye-tracking
• Gaze points, focus depth
• Emotion sensing/sharing
• Physiological sensing, emotion mapping

60. Sensor Enhanced HMDs
Eye tracking, heart rate,
pupillometry, and face camera
HP Omnicept Project Galea
EEG, EMG, EDA, PPG,
EOG, eye gaze, etc.

61. Multiple Physiological Sensors into HMD
• Incorporate range of sensors on HMD faceplate and over head
• EMG – muscle movement
• EOG – Eye movement
• EEG – Brain activity
• EDA, PPG – Heart rate

63. Brain Synchronization

64. Pre-training (Finger Pointing) Session Start

65. Post-Training (Finger Pointing) Session End

66. Brain Synchronization in VR
Gumilar, I., Sareen, E., Bell, R., Stone, A., Hayati, A., Mao, J., ... & Billinghurst, M. (2021). A comparative study on inter-
brain synchrony in real and virtual environments using hyperscanning. Computers & Graphics, 94, 62-75.

69. asfd

71. NeuralDrum
• Using brain synchronicity to increase connection
• Collaborative VR drumming experience
• Measure brain activity using 3 EEG electrodes
• Use PLV to calculate synchronization
• More synchronization increases graphics effects/immersion
Pai, Y. S., Hajika, R., Gupta, K., Sasikumar, P., & Billinghurst, M. (2020). NeuralDrum: Perceiving Brain
Synchronicity in XR Drumming. In SIGGRAPH Asia 2020 Technical Communications (pp. 1-4).

72. Set Up
• HTC Vive HMD
• OpenBCI
• 3 EEG electrodes

74. Results
"It’s quite interesting, I actually felt like my
body was exchanged with my partner."
Poor Player Good Player

75. Fabric Electrodes
• Can use conductive threads to create soft fabric EEG/GSR electrodes

77. • Advanced displays
• Real time space capture
• Natural gesture interaction
• Robust eye-tracking
• Emotion sensing/sharing
Empathic
Tele-Existence

78. Empathic Tele-Existence
• Move from Observer to Participant
• Explicit to Implicit communication
• Experiential collaboration – doing together

79. CONCLUSIONS

80. Conclusions
• Empathic Computing Laboratory
• 50 people split between Auckland/Adelaide
• Empathic Computing Research Focus
• Systems that enhance understanding
• Natural collaboration + Experience capture + Implicit understanding
• Combining AR, VR, Physiological sensing
• Research directions
• Hybrid AR, Novel collaboration, brain synchronization, etc.

81. IVE

82. ARIVE
• Australasian Researchers in Interactive and Virtual Environments
• Collecting together all the best AR/VR researchers in Australia and NZ
• 8 institutions, 260 researchers, > 30 million USD in funding
• Looking for Industry Partners for research consortium

84. www.empathiccomputing.org
@marknb00
mark.billinghurst@auckland.ac.nz