keynote speech by Mark Billinghurst at the Workshop on Transitional Interfaces in Mixed and Cross-Reality, at the ACM ISS 2021 Conference. Given on November 14th 2021

1. RESEARCH DIRECTIONS IN
TRANSITIONAL INTERFACES
Mark Billinghurst
mark.billinghurst@unisa.edu.au
ACM ISS 2021
Workshop on Transitional Interfaces in
Mixed and Cross-Reality: A new frontier?
November 14th 2021

5. Computer Interfaces
• Separation between real and digital worlds
• WIMP (Windows, Icons, Menus, Pointer) metaphor

6. Rekimoto, J. and Nagao, K. 1995. The world through the computer: computer augmented interaction with real world environments.
Making Interfaces Invisible
(c) Internet of Things

7. Internet of Things (IoT)..
• Embed computing and sensing in real world
• Smart objects, sensors, etc..
(c) Internet of Things

8. Virtual Reality (VR)
• Users immersed in Computer Generated environment
• HMD, gloves, 3D graphics, body tracking

9. Augmented Reality (AR)
• Virtual Images blended with the real world
• See-through HMD, handheld display, viewpoint tracking, etc..

10. From Reality to Virtual Reality
Internet of Things Augmented Reality Virtual Reality
Real World Virtual World

11. Milgram’s Mixed Reality (MR) Continuum
Augmented Reality Virtual Reality
Real World Virtual World
Mixed Reality
"...anywhere between the extrema of the virtuality continuum."
P. Milgram and A. F. Kishino, (1994) A Taxonomy of Mixed Reality Visual Displays
Internet of Things

12. Milgram’s Mixed Reality (MR) Continuum
Augmented Reality Virtual Reality
Real World Virtual World
Mixed Reality
Internet of Things

13. Transitions (2001)
• Interfaces need to support transitions along the RV continuum
• Augmented Reality is preferred for:
• augmenting real objects
• co-located collaboration
• Immersive Virtual Reality is preferred for:
• experiencing world immersively
• sharing views, remote collaboration

14. The MagicBook (2000)
Reality Virtuality
Augmented
Reality (AR)
Augmented
Virtuality (AV)

15. MagicBook Demo
https://www.youtube.com/watch?v=tNMljw0F-aw

16. Technology
• Reality
• No technology
• Augmented Reality
• Camera – tracking
• Switch – fly in
• Virtual Reality
• Compass – tracking
• Pressure pad – move
• Switch – fly out

17. Features
• Seamless transition from Reality to Virtuality
• Reliance on real decreases as virtual increases
• Supports egocentric and exocentric views
• User can pick appropriate view
• Independent Views
• Privacy, role division, scalability
• Collaboration on multiple levels:
• Physical Object, AR Object, Immersive Virtual Space
• Egocentric + exocentric collaboration
• multiple multi-scale users

18. The MagicBook
• Billinghurst, M., Kato, H., & Poupyrev, I.
(2001). The MagicBook: a transitional AR
interface. Computers & Graphics, 25(5),
745-753.

19. Mobile AR Transitional Interfaces
• Using transition from Ego-centric to Exo-centric to increase awareness
Mulloni, A., Dünser, A., & Schmalstieg, D. (2010, September). Zooming interfaces for augmented reality
browsers. In Proceedings of the 12th international conference on Human computer interaction with
mobile devices and services (pp. 161-170).

20. https://www.youtube.com/watch?v=JLxLH9Cya20

21. A Conceptual Model
• Theoretical framework for
understanding transitional interfaces
Grasset, R., Mulloni, A., Billinghurst, M., & Schmalstieg, D.
(2011). Navigation techniques in augmented and mixed reality:
Crossing the virtuality continuum. In Handbook of Augmented
Reality (pp. 379-407). Springer, New York, NY.

22. Applying Conceptual Model to MagicBook

23. Important Research Questions
• Presence
• How does user’s sense of Presence change?
• Awareness
• How does the user perceive the switch between contexts?
• Interaction Techniques
• What are the best interaction techniques for transitional interfaces?
• Usability
• How can we measure the usability of transitional interfaces?
• Collaboration
• How can transitional interfaces be used to improve collaboration?

24. Example: AR Lens for Transitional Interfaces
Looser, J. C. A. (2007). AR magic lenses: Addressing the challenge of focus and
context in augmented reality.
• AR Lens
• Create illusion that user is holding real lens
• Support lens-based viewing/selection

25. AR Lens Demo

26. Example: Multi-Scale Collaboration
Changing the user’s virtual body scale
Piumsomboon, T., Lee, G. A., Irlitti, A., Ens, B., Thomas, B. H., & Billinghurst, M. (2019, May). On the shoulder of the giant: A
multi-scale mixed reality collaboration with 360 video sharing and tangible interaction. In Proceedings of the 2019 CHI
conference on human factors in computing systems (pp. 1-17).

28. Sharing: Separating Cues from Body
• What happens when you can’t see your colleague/agent?
Piumsomboon, T., Lee, G. A., Hart, J. D., Ens, B., Lindeman, R. W., Thomas, B. H., & Billinghurst, M. (2018, April). Mini-me: An adaptive
avatar for mixed reality remote collaboration. In Proceedings of the 2018 CHI conference on human factors in computing systems (pp. 1-13).
Collaborating Collaborator out of View

29. Mini-Me Communication Cues in MR
• When lose sight of collaborator a Mini-Me avatar appears
• Miniature avatar in real world
• Mini-Me points to shared objects, show communication cues
• Redirected gaze, gestures

30. https://www.youtube.com/watch?v=YrdCg8zz57E

31. User Study (16 participants)
• Collaboration between user in AR, expert in VR
• Hololens, HTC Vive
• Two tasks:
• (1) asymmetric, (2) symmetric
• Key findings
• Mini-Me significantly improved performance time (task1) (20% faster)
• Mini-Me significantly improved Social Presence scores
• 63% (task 2) – 75% (task 1) of users preferred Mini-Me
“I feel like I am
talking to my
partner”

32. Jun Rekimoto, Sony CSL
Invisible Interfaces

33. Invisible Interfaces
Jun Rekimoto, Sony CSL

34. Invisible Interfaces
Jun Rekimoto, Sony CSL

35. Example: Visualizing Sensor Networks
• Network of Humidity Sensors
• ZigBee wireless communication
• Use Mobile AR to Visualize Humidity
Rauhala, M., Gunnarsson, A. S., & Henrysson, A. (2006, September). A novel interface to sensor
networks using handheld augmented reality. In Proceedings of the 8th conference on Human-
computer interaction with mobile devices and services (pp. 145-148).

36. Invisible Interfaces
Jun Rekimoto, Sony CSL

37. UbiVR – CAMAR
CAMAR Companion
CAMAR Viewer
CAMAR Controller
GIST - Korea

38. The Metaverse
• Neal Stephenson’s “SnowCrash”
• The Metaverse is the convergence of:
• 1) virtually enhanced physical reality
• 2) physically persistent virtual space
• Metaverse Roadmap
• http://metaverseroadmap.org/

39. Metaverse Dimensions
• Augmentation technologies that layer information onto our
perception of the physical environment.
• Simulation refers to technologies that model reality
• Intimate technologies are focused inwardly, on the identity
and actions of the individual or object;
• External technologies are focused outwardly, towards the
world at large;

40. Metaverse Components
• Four Key Components
• Virtual Worlds
• Augmented Reality
• Mirror Worlds
• Lifelogging

41. M
ixed
R
eality

42. Metaverse Transitions

43. Possible Research Directions
• Lifelogging to VR
• Bringing real world actions into VR, VR to experience lifelogging data
• AR to Lifelogging
• Using AR to view lifelogging data in everyday life, Sharing physiological data
• Mirror Worlds to VR
• VR copy of the real world, Mirroring real world collaboration in VR
• AR to Mirror Worlds
• Visualizing the past in place, Asymmetric collaboration
• And more..

44. Conclusions
• Transitional interfaces
• Research crossing boundaries
• AR/VR Transitions along the MR continuum
• Seamless transitions
• Ego-centric and exo-centric views
• Collaboration on multiple levels
• Many directions for future research
• Crossing boundaries, Metaverse

46. www.empathiccomputing.org
@marknb00
mark.billinghurst@unisa.edu.au