1. EMPATHIC COMPUTING AND
COLLABORATIVE IMMERSIVE ANALYTICS
Mark Billinghurst
mark.billinghurst@unisa.edu.au
July 2022

3. Collaborative Visualization

4. Need for More Research
•Collaborative Visualisation related to:
• Scientific Visualisation
• Information Visualisation
• Visual Analytics
•All well established fields over last 20 years
• But little existing research in Collaborative Visualisation
• E.g. from 1990 – 2010, over 1600 papers in main Viz conferences
• Only 34 papers published in Collaborative Visualisation, ~2%

5. Collaborative Immersive Analytics
• Intersection between Mixed Reality, Visual Analytics, CSCW

6. CIA Example: EVL CAVE2

7. Modern Communication Technology Trends
1. Improved Content Capture
• Move from sharing faces to sharing places
2. Increased Network Bandwidth
• Sharing natural communication cues
3. Implicit Understanding
• Recognizing behaviour and emotion

8. Natural
Collaboration
Implicit
Understanding
Experience
Capture

9. Natural
Collaboration
Implicit
Understanding
Experience
Capture
Empathic
Computing

10. Empathic Computing Research Focus
Can we develop systems that allow
us to share what we are seeing,
hearing and feeling with others?

11. Key Elements of Empathic Systems
•Understanding
• Emotion Recognition, physiological sensors
•Experiencing
• Content/Environment capture, VR
•Sharing
• Communication cues, AR

12. Sharing Virtual Communication Cues
• AR/VR displays
• Gesture input (Leap Motion)
• Room scale tracking
• Conditions
• Baseline, FoV, Head-gaze, Eye-gaze

13. Piumsomboon, T., Dey, A., Ens, B., Lee, G., & Billinghurst, M. (2019). The effects of sharing
awareness cues in collaborative mixed reality. Frontiers in Robotics and AI, 6, 5.

14. Sensor Enhanced HMDs
Eye tracking, heart rate,
pupillometry, and face camera
HP Omnicept Project Galea
EEG, EMG, EDA, PPG,
EOG, eye gaze, etc.

15. GALEA: Multiple Physiological Sensors into HMD
• Incorporate range of sensors on HMD faceplate and over head
• EMG – muscle movement
• EOG – Eye movement
• EEG – Brain activity
• EDA, PPG – Heart rate

17. Brain Synchronization

18. Pre-training (Finger Pointing) Session Start

19. Post-Training (Finger Pointing) Session End

20. Brain Synchronization in VR
Gumilar, I., Sareen, E., Bell, R., Stone, A., Hayati, A., Mao, J., ... & Billinghurst, M. (2021). A comparative study on
inter-brain synchrony in real and virtual environments using hyperscanning. Computers & Graphics, 94, 62-75.

23. asfd

25. Applying Empathic Computing to CIA
• Providing communication cues in immersive visualization
• Gaze, gesture, body motion, avatar representation
• Measuring cognitive load and stress
• Adapting interface and visualization
• Exploring communication measures
• Brain synchronization, non-verbal cues, etc.
• Sharing perspectives
• Same virtual body, joint interaction
• Hybrid interfaces
• Combining multiple devices AR/VR/MR view

26. Conclusions
• Need for more research in Collaborative Visualization
• Especially Collaborative Immersive Analytics
• Empathic Computing
• Systems that enhance understanding
• Natural collaboration + Experience capture + Implicit understanding
• Combining AR, VR, Physiological sensing
• Applying Empathic Computing to CIA
• Sharing communication cues, brain synchronization, etc.

27. More Information..
Billinghurst, M., Cordeil, M., Bezerianos, A.,
& Margolis, T. (2018). Collaborative
immersive analytics. In Immersive
Analytics (pp. 221-257). Springer, Cham.

28. www.empathiccomputing.org
@marknb00
mark.billinghurst@unisa.edu.au