Keynote address by Mark Billinghurst at the IUI 2014 conference in Haifa Israel, February 27th, 2014.

1. Using Augmented Reality to Create
Empathic Experiences
Mark Billinghurst
mark.billinghurst@hitlabnz.org
The HIT Lab NZ, University of Canterbury
February 27th 2014

5. Courtesy Matt Rettig, CMU

18. Processing Power
Adapt
Experience
Operate

19. Beyond the Desktop

20. Intelligent User Interfaces
 AI + HCI: User Interface involving some
elements of Artificial Intelligence
 Computer having model of user/domain
 First IUI Conference in 1997
 Readings in IUI (Wahster 1998)

21.  Microsoft Clippy (1997)
 MS Office Intelligent User Interface

23. Intelligent User Interfaces

24. Multiple Intelligences
 Frames of Mind: The
Theory of Multiple
Intelligences
 Howard Gardner (1983)

25. Multiple Intelligences

26. Emotional Intelligence
 Emotional Intelligence - Why it
can matter more than IQ
 Goldman (1995)
 Identify, assess, and control
the emotions of oneself, of
others, and of groups

27. Foundations of Emotional Intelligence

28. Empathy
Empathy

30. Empathy vs. Intelligence
 Intelligence: the power of one’s brain, divided
into many categories and used in numerous ways.
 Empathy: the power of one’s heart, expressing
one’s true emotions with themselves, those
around them, and their own world.

31. IQ vs EQ?

32. Mirror Neurons
 Neuron that fires both when an animal acts and it
observes the same action performed by another
 Giacomo Rizzolatti, Univ. of Palma (1980s/90s)

34. Empathic Computing
1. Computing systems that can
understand your feelings and emotions
2. Computing systems that help you
better understand the feelings of others

35. Affective Computing
 Ros Picard – MIT Media Lab
 http://affect.media.mit.edu

36. Appliances That Make You Happy
 Jun Rekimoto – Univ. Tokyo
 Smile detection + smart devices

37. Can we develop interfaces
that allow us to be more
empathetic to others?

38.  adsf

39. Empathy Computing Requirements
 Basic Requirements
 Making the technology transparent
 Empathy Definition
 Seeing with the eyes of another
 Hearing with the ears of another
 Feeling with the heart of another

40. Augmented Reality
1977
2008

41. Using AR for Empathy
 Augmented Reality can:
 Remove technology barriers
 Enhance communication
 Change perspective
 Share experiences
 Enhance interaction in real world

42. Communication Seams
Communication Space
Task Space
 Technology introduces artificial seams in the
communication (eg separate real and virtual space)

43. Removing Barriers: Shared Space
 Face to Face interaction, Tangible AR metaphor
- ~3,000 users (Siggraph 1999)
 Easy collaboration with strangers
 Users acted same as if handling real objects
Billinghurst, M., Poupyrev, I., Kato, H., & May, R. (2000). Mixing realities in shared space: An augmented
reality interface for collaborative computing. In Multimedia and Expo, 2000. ICME 2000. 2000 IEEE
International Conference on (Vol. 3, pp. 1641-1644).

44. Enhancing Face to Face Communication
 AR Pad
 Handheld AR device
 AR shows viewpoints
 Users collaborate easier
 Show communication cues
Virtual Viewpoint Visualization
Mogilev, D., Kiyokawa, K., Billinghurst, M., & Pair, J. (2002, April). AR Pad: An interface for face-to-face AR
collaboration. In CHI'02 extended abstracts on Human factors in computing systems (pp. 654-655).

45. Changing Perspective
 CamNet (1992)
 British Telecom
 Wearable Teleconferencing
 audio, video
 Remote collaboration
 Sends task space video
 Similar CMU study (1996)
 cut performance time in half

46. WACL: Remote Expert Collaboration
 Wearable Camera/Laser Pointer
 Independent pointer control
 Remote panorama view

47. WACL: Remote Expert Collaboration
 Remote Expert View
 Panorama viewing, annotation, image capture
Kurata, T., Sakata, N., Kourogi, M., Kuzuoka, H., & Billinghurst, M. (2004, October). Remote collaboration
using a shoulder-worn active camera/laser. In Wearable Computers, 2004. ISWC 2004. Eighth
International Symposium on (Vol. 1, pp. 62-69).

49. View Through Google Glass
Always available peripheral information display
Combining computing, communications and content capture

50. Ego-Vision Collaboration
 Google Glass
 camera + processing + display + connectivity

51. Current Collaboration on Glass
 First person remote conferencing/hangouts
 Limitations
 Single POV, no spatial cues, no annotations, etc

52. Sharing Space: Social Panoramas
 Capture and share social spaces in real time
 Enable remote people to feel like they’re with you

53. Key Technology
 Google Glass
 Capture live image panorama (compass + camera)
 Capture spatial audio, live video
 Remote device (desktop, tablet)
 Immersive viewing, live annotation

54. Capturing Space: Real World Capture
 Hands free AR
 Portable scene capture (color + depth)
 Projector/Kinect combo, Remote controlled pan/tilt
 Remote expert annotation interface

55. Remote Expert View

56. Capturing Behaviours
 3 Gear Systems
 Kinect/Primesense Sensor
 Two hand tracking
 http://www.threegear.com

57. Skeleton Interaction + AR
 HMD AR View
 Viewpoint tracking
 Two hand input
 Skeleton interaction, occlusion

58. Ghostman
 Use AR to capture and overlay your actions
into remote persons space
 Eg remote therapy

59. Looking to the Future
What’s Next?

60. Brain to Brain Control
 Rajesh Rao, University of Washington
 First Brain to Brain control

61. System Architecture

62. Scaling Up
 Seeing actions of millions of users in the world
 Augmentation on city/country level

63. AR + Smart Sensors + Social Networks
 Track population at city scale (mobile networks)
 Match population data to external sensor data
 medical, environmental, etc
 Mine data to improve social services

66. Research Challenges
 How to convey emotion?
 How to measure empathy?
 Interface/interaction models?
 How to communicate emotion?
 Scaling up to city/country scale?

67. Conclusion

68. Harvard Grant Study
 $20 million, 75 years study
 268 Harvard graduates
 456 disadvantaged people
 Led by George Valliant
 What makes us happy?
 warmth of relationships throughout
life have the greatest positive impact
on "life satisfaction".

69. “The seventy-five years and twenty million dollars
expended on the Grant Study points to a
straightforward five-word conclusion: Happiness is
love. Full stop.”
George Valliant

70. Conclusions
 Empathic Computing
 Sharing what you see, hear and feel
 AR Enables Empathic Experiences
 Removing technology
 Changing perspective
 Sharing space/experience
 Many directions for future research

71. More Information
• Mark Billinghurst
– Email: mark.billinghurst@hitlabnz.org
– Twitter: @marknb00
• Website
– http://www.hitlabnz.org/