This slide describes our proposition of concept regarding an OS wide user interface for free gestures input devices such as Leap Motion. This was presented in GNOME Asia 2015 on 9th May 2015. We discusses how we tackle the design issue and our prototype of the concept implemented in GNOME and browser.

1. PROTOTYPING GNOME UI
FOR GESTURAL INPUT
Presented by
Adityo Pratomo ( ) / Surya University
Hidayat Febiansyah / Surya University
GNOME.Asia 2015
@kotakmakan
(@havban)

2. ABOUT US
We are lecturers at the Informatics Department at Surya
University
Adityo Pratomo got a Master Degree in Interaction Design
from University of Sydney, previously worked at an
advertising agency
Hidayat Febiansyah an IT wizard... jk went with yellow
Jaket, obtained PhD from Sun Moon University, Republic
of Korea

3. USER INPUT DEVICES
Devices act as a medium for user to give input to computer

4. EVOLUTION OF USER INPUT DEVICE
Punch Card

5. Keyboard

6. Mouse

7. Scanner

8. Graphic Tablet

9. Touchscreen

11. Voice Input

12. Tangible User Interface

13. Air Gesture Input

14. TODAY CONDITION
Huge variety of input device

15. (Still) Follows the WIMP (Windows, Icon, Menu, Pointer)
style
Term coined in 1980 by Merzouga Wilberts
Popularized by Apple Macintosh in 1984

16. Windows
Runs a self contained program, isolated from other program,
run at the same time in other windows

17. Icon
Shortcut for an action that the computer performs

18. Menu
Text or icon-based selection system that selects and executes
programs or tasks

19. Pointer
An onscreen symbol that represents movement of a physical
device that the user controls to select icons, objects, data
elements, etc

20. GESTURAL INPUT
Provides gestures (mostly fingers) input for users

21. Touchpad

23. Implemented in many OS

24. NEW GESTURAL INPUT DEVICES
Introduces free form gestures as means to interact

25. Microsoft Kinect
Full body input, including voice
No finger detection

26. Leap Motion
Fingers input only
Have built-in finger gestures detection

27. Intel RealSense
Essentially a much smaller Kinect
Windows only SDK

28. Generally speaking, hardware wise, these devices do their job
well (can be improved though)

29. PROBLEMS
Come from the software implementation of it Especially, for
productive, non-entertainment use

30. Metro UI
Supposed to be designed for touch
Falls flat for these kind of interactions

31. Kinect Guidelines
Still involves choosing and confirming action
Chooses a big icon
Confirms by hanging there for a period of time

32. Direct Mouse to Leap Motion
Barely usable since it involves high accuracy in picking
object
Lost in translation between 2D (where the feedback
happens) and 3D (where the action happens)

34. OS Wise
Information are structured inside tree, explored by moving
down branches, presented one node at a time (nothing
wrong with this)
Interaction wise, requires moving between directories,
picking up objects, confirming and working on that (this
requires specific interaction technique)

36. GOALS AND MOTIVATIONS
To suggest an interaction model for an OS-wide application
of free gestures input
We view this interaction model as something that's
human and natural, appreciating human's inherent
motoric ability
To push further research of this area, without waiting for
big corp to do it (hooray Open Source)

37. INFLUENCES AND PREVIOUS WORKS
This project takes cues from other projects that people
already worked on

38. GNOME Shell
Minimal use of icon, users are forced to memorize keyboard
shortcuts

39. Controlling GNOME with Leap
http://www.joaquimrocha.com/2013/08/09/controlling-
gnome-with-leap/

40. Leap Motion DBus
https://github.com/jamespcole/leapmotion-dbus

41. 9 to 5
a Leap Motion puzzle game
https://www.youtube.com/watch?v=rz_wBzWCAXw

42. Kids Note Training System
A tool for kids to learn about musical notes and test their
hearing

43. DESIGN CONSIDERATION
Applied in Leap Motion
Complementary to mouse and keyboard
Can be used with only one hand
Focus on gesture use with minimal icon involved
Main use case for productivity, non-entertainment
functionality

44. ISOLATING THE PROBLEM
Bigest problem is telling people how to choose an object
using gestures

45. This is NOT a replacement for mouse
Mouse is precise, gestures are not.

46. Giving a bigger icon doesn't necessarily means it'll be
easier to pick
It still requires a dexterity to accurately choose the object
Plus, how to confirm that choice? Inaccurately pressing the
air?

47. OUR SOLUTION
To translate imprecise gestures into an act of choosing an
object and confirming it

48. OUR PROTOTYPE
Leap Motion-controlled windowing system
UI-model prototype for desktop
Both will demonstrate how choosing object is done with
gestures

49. ARCHITECTURE
Application
OS WIndowing System
System Event
Leap Motion API
OS

50. FEATURES
Alternative live tile windowing system prototype
All contents are live and directly editable
Activated using gestures
Mapping of various gestures into keyboard shortcut
Translation of keyboard shortcuts into various events
Usage of GNOME windowing capability
Keyboard shortcuts receiver inside application

51. GESTURES
CircleGesture() – A circular movement by a finger.
SwipeGesture() – A straight line movement by the hand
with fingers extended.
ScreenTapGesture() – A forward tapping movement by a
finger.
KeyTapGesture() – A downward tapping movement by a
finger.

52. MAPPING SHORTCUTS
Connecting the element using keyboard shortcuts
e.g. ctrl+alt+=
No need to access lower level component
Portable deployment
Leap motion control is only complementary input, not
primary
Other input can be proceded with existing keyboard or
mouse

53. COMPONENTS
Leapjs Controller
NWJS
RobotJs
With added functionality to accept multiple flags shortcut
Packery.js
Other: node-open, jquery, scrollTo, Knobjs, Togglejs, node-gyp
https://developer.leapmotion.com/documentation/javascrip
http://nwjs.io/
https://github.com/octalmage/robotjs
http://packery.metafizzy.co

54. CHARACTERISTICS
Open Sourceness
Platform Agnostic
as long as the device supports HTML5, specifically node-
webkit
Easy to configure
Manage shortcuts
Easy to develop
Common technology platforms

55. SYSTEM MODE
Running with existing gnome shortcut mode
Move among workspaces
ctrl+alt+- => Swipe down
ctrl+alt+= => Swipe up
Move among windows
alt+tab => Swipe left
alt+shift+tab => Swipe right
Inner applications windows movement
alt+` => keytap

56. DEMO MODE
Move among widgets one step
ctrl+alt+i => Swipe left
ctrl+alt+y => Swipe right
Move among widgets with rolling
ctrl+alt+i * => Rotate finger right
ctrl+alt+y * => Rotate finger left
Focused widget → Full size widget
ctrl+alt+space => keytap

59. FUTURE GOALS
User test
Integrate with other technologies (VR, haptic sensors)
Possibility into integration with file system that behaves
like a graph database
Formalize the system

60. CONCLUSION
Air gesture-based interaction will come along in the next
few years (or even decades)
Software and hardware capabilities are there, they just
need to be unified by a proper design system
Requires further development to integrate with other
technologies and to user test it rigorously

61. COME JOIN THE PARTY
adityo.pratomo@surya.ac.id hidayat.febiansyah@surya.ac.id
http://www.github.com/lunchboxav/leap-gnome