This document discusses challenges for the next generation of interactive media using computer vision sensors. It presents three prototypes developed by the author: 1) AquaTop Display, an interactive water surface that uses water-specific gestures for input; 2) BallCam, a camera inside a ball that stabilizes spinning video to provide ball's point of view footage of sports; 3) Unaware Gaze Navigation, which guides users' gaze with subtle blurring without noticing. The author concludes that rapid prototyping and advanced implementation are needed to develop novel ideas into unique products for vision-based human-computer interaction.

1. USING VISION SENSORS FOR
INNOVATIVE HCI
- CHALLENGES FOR THE NEXT INTERACTIVE MEDIA -
Hideki Koike
Tokyo Institute of Technology

2. OUTLINE
➤ Vision-based HCI
➤ Challenges for the next interactive media
➤ Environment: Interactive Surface on the Water
➤ Contents: Spectating Ball Sports from Ball’s POV
➤ Advertisement: Unaware Gaze Navigation
➤ Lessons learned from our prototype development

3. VISION-BASED HCI

4. NEW I/O DEVICES HAVE BEEN CHANGING HCI
IBM PC Macintosh DataLand
CUI GUI PUI
I: keyboard
O: character display
I: keyboard, mouse
O: bitmap display
I: gesture recognition/
speech recognition
O: large display/wearable

5. ADVANCES IN COMPUTER VISION
➤ Hardware:
➤ Camera: smaller, higher resolution, and less expensive.
➤ GPU: faster and less expensive.
➤ Software:
➤ Open software (e.g. OpenCV, AR Toolkit)
wired unwired

6. SPECIAL USE CAMERAS
➤ Example:
➤ far/near infrared, polarization, high-speed, depth, mocap,
action cam, eye tracking, light field
➤ Can see what human cannot
➤ Enable natural and intuitive UI
far infrared camera polarization camera depth camera

7. CHALLENGES FOR
THE NEXT INTERACTIVE MEDIA

8. AQUATOP DISPLAY:
INTERACTIVE SURFACE ON THE WATER

11. PROTOTYPE

12. WATER SPECIFIC GESTURES
Poking Stroking
Scooping Hitting

13. WATER SPECIFIC GESTURES
poking as selecting
stroking as swiping
scooping as dragging

14. COMPARING TO TRADITIONAL UI
selection page scroll transition exit app.
PC’s GUI
mouse
click
slider click&drag
win. close
button
Touch UI touch swipe touch&drag
home
button
AquaTop poke stroke scoop hit

15. IMMERSION BY FLUID SIMULATION

16. AQUATOP DISPLAY [2012]

17. EVALUATION: VISIBILITY
0.26 g/l
1.10 g/l

18. EVALUATION: POINTING ACCURACY
poking touching

19. AWARD
➤ Laval Virtual 2013: Grand Prix
➤ METI Innovative Technologies 2013: Special Award (Culture)
➤ TV Tokyo World Business Satellite 2013: Toretama Award &
Kickstarter Award
➤ and more.

20. BALLCAM!:
SPECTATING BALL SPORTS FROM
BALL’S POV

21. BALL’S POV
➤ 360 degree omnidirectional video streaming
➤ Ball’s Point of View
time

22. PROTOTYPE
➤ What is difficult?
➤ The ball is spinning, sometimes very
fast (> 600rpm)!
➤ How to stabilize the video from
spinning ball?
first prototype: The iBall [2011]
second prototype

23. BALLCAM! [2013]

24. OVERVIEW
Image Subsampling
View Expansion
triplet image sixlet image
Motion Interpolation
Distortion Correction

25. STEP 1: CORRECTING LENS DISTORTION AND ROLLING SHUTTER
P : back-projection
K: calibration matrix
R(t): camera rotation
Gauss-Newton method

26. STEP 2: SELECTING IMAGES USING INTENSITY

27. STEP 3: EXTENDING VIEWS USING MULTIPLE FRAMES
frame t+1
frame t+2
triplet image N
sixlet image
triplet image N+1
frame t

28. STEP 4: IMAGE INTERPOLATION
image
interpolation
Mf: forward mapping
Mb: backward mapping
MI: Identity mapping

29. BALLCAM! [2013]

30. OUR GOAL
➤ Rugby World Cup in Japan [2019]
➤ Tokyo Olympic [2020]
➤ World Cup Soccer [2022]
20162011 2012 2013 2014 2015

31. UNAWARE GAZE NAVIGATION

33. EXPERIMENTS

34. EXPERIMENT 1
no blur(σ=0) σ=5
experi
ment
image
heat
map

35. EXPERIMENT 1
no blur(σ=0) σ=5
experi
ment
image
heat
map

36. EXPERIMENT 1
blur strength and the time to fixation

37. EXPERIMENT 2

38. EXPERIMENT 2

39. USAGE SCENARIO

40. LESSONS LEARNED FROM OUR
PROTOTYPE DEVELOPMENT

41. HOW IDEAS BECOME TO A UNIQUE PRODUCT
new(?) idea
rapid prototyping
advanced
implementation
unique
product

42. NEW IDEA(?)
➤ In most cases, it is not.
➤ Survey other work carefully.
➤ OK. It seems to be a new idea!
➤ However, 10,000 people in the world are inspiring the same
idea at this moment.

43. RAPID PROTOTYPING
➤ Let’s make a prototype ASAP.
➤ The idea may be not so interesting as you thought.
➤ The idea may be more interesting than you thought.
➤ OK. It seems to be interesting!
➤ However, 100 people in the world are making the same
prototype at this moment.

44. ADVANCED IMPLEMENTATION
➤ Let’s do serious implementation.
➤ Work harder.
➤ However, you may face very difficult technical issues.
➤ Use the wisdom of predecessors.
➤ You may be only one who finally develop the system!

45. CONCLUSION
➤ Vision-based HCI is a key to the next HCI.
➤ Effective use of special use cameras enables natural and
intuitive interaction.
➤ Challenges for the next interactive media:
➤ Environment, Contents, and Advertisement.
➤ Rapid prototyping and advanced implementation make your
ideas unique.

46. REFERENCES
➤ AquaTop Display
➤ Y. Matoba et.al.: “AquaTop Display: A True Immersive Water Display System”, Proc. on
ACM SIGGRAPH 2013, Emerging Technologies, Article 4, 2013.
➤ H. Koike et.al.: “AquaTop Display: Interactive Water Surface for Viewing and Manipulating
Information in a Bathroom”, ACM Int’l Conf. on Interactive Tabletops and Surfaces
(ITS’13), pp.155-164, 2013.
➤ BallCam
➤ R.Funakoshi et.al.: “Activity-Aware Video Stabilization for BallCam”, Proc. on ACM User
Interface Software and Technology (UIST’16 Adjunct), pp.197-198, 2016.
➤ R. Funakoshi et.al.: “Synthesizing Pseudo Straight View form A Spinning Camera Ball”,
7th Augmented Human Int’l Conf. (AH’16), pp.30:1-30:4, 2016.
➤ K. Horita et.al.: “Experiencing the Ball’s POV for Ballistic Sports”, Proc. of the 4th
Augmented Human Int’l Conf. (AH’13), pp.128-133, 2013.
➤ Unaware Blur
➤ H. Hata et.al.: “Visual Guidance with Unnoticed Blur Effect”, Int’l Working Conf. on
Advanced Visual Interfaces (AVI2016), pp.28-35, 2016.