In science fiction and action films, gestural interfaces are everywhere, and new gestural input technologies generate a lot of anticipation with their kickstarter videos. And yet in the real world, gestural input (with the exception of multitouch) has gained little traction. Why is that? We've been working with a variety of gestural technologies, trying to incorporate them into professional products, and we have learned where the problems are, and what we need to do to cross the divide between hype and practice. This presentation looks at the current state of gestural input technologies, analyses the strengths and failings of each, and charts a course to using them successfully. We present a case study showing how and why gestures need to be curated across different form-factors, and give some tips on how to test.

1. © 2014 Autodesk
Hollywood Has a Lot to Answer For:
Making Gestural Input Work for the Real World
Ian Hooper
Principal UX Designer
UXPA 2015
John Schrag
UX Architect, Media & Entertainment Division

2. © 2014 Autodesk

3. © 2014 Autodesk
Part One: A Brief History of Gestural Interaction

4. © 2014 Autodesk
What do we mean by gestures?
A gesture is a motion of the body that
contains information.
Waving goodbye is a gesture. Pressing a key on a keyboard is not a
gesture because the motion of a finger on its way to hitting a key is neither
observed nor significant. All that matters is which key was pressed.
“
”
Kurtenbach and Hulteen (1990):

5. © 2014 Autodesk
§ Human to Human Interaction:
§ Music Conducting
§ Dance, Theatre & Magic
§ Sign Language & informal salutations and signaling
§ Early Electronic Systems
§ Musical instruments (Theremin, Sackbut)
§ Sensors (The Clapper, Photodetectors)
Historical Context

6. © 2014 Autodesk
A long road from theory to practice
Direct Manipulation of Graphical Objects
1990197019601950 1980
The Mouse
Windows
Research
Commercialization
2000 2010
Gesture Recognition
derived from: Brad A. Myers (1998). A brief history of human-computer interaction technology. Interactions, vol 5(2), pp. 44-54

7. © 2014 Autodesk
1981: Xerox Star 8010
§ Original Price: $16,595
§ Professional workstation
was the inspiration for the
Macintosh and all
Windows / Icons / Mouse /
Pointer (WIMP) interfaces
Image: digibarn.com

8. © 2014 Autodesk
1983: HP-150
§ Original Price: $2795
§ Early personal computer
had infrared touch-screen
capability
Image: columbia.edu

9. © 2014 Autodesk
1984: Apple Macintosh
§ Original Price: $2,495
§ Popularized WIMP
§ Introduced the ‘drag’
gesture in place of a
dedicated mouse button
Image: oldcomputers.net/

10. © 2014 Autodesk
§ Games
Outside the mainstream
Dataglove (Fifth Dimension Technologies)
§ Niche
Phantom Omni (SensAble Technologies)
Xbox Kinect (Microsoft)
Playstation Move (Sony) Wii Remote (Nintendo)

11. © 2014 Autodesk
1998: Dance Dance Revolution
(Konami)
2001: Black & White
(Lionhead Studios)
2003: Eye Toy
(Sony)
§ Games
Outside the mainstream

12. © 2014 Autodesk
§ Art
Outside the mainstream
1985: Video Place (Myron Krueger)
Projector
Camera
Projector Screen
Back-lit screen
2000: Body Language (Nathaniel Stern)

13. © 2014 Autodesk
The Renaissance
Image: Apple
§ 2007: iPhone
§ 2010: iPad
§ 2012: Cintiq
Image: Wacom

14. © 2014 Autodesk
§ The four best-known players in mid-air gestural input
devices right now are:
§ Built-in cameras
§ Kinect camera
§ LEAP Motion device
§ Thalmic Labs’ Myo device
A Look at the Hardware

15. © 2014 Autodesk
Built-in Camera
§ Cheap and ubiquitous
§ Open-source libraries for
extracting gesture
§ Slower (image processing
in software)
§ Bad depth sense
§ Error prone
Image: Dell

16. © 2014 Autodesk
Kinect Camera
§ Reads full body or upper
body and creates a 3d
skeleton
§ Close up: includes gaze
direction, head position,
fingers
§ Further back: no fingers
Image: Microsoft

17. © 2014 Autodesk
LEAP Motion Device
§ Provides full hand skeletal
model
§ Has a limited volume –
hard to see its limits
§ Occlusion is a problem
Image: LEAP

18. © 2014 Autodesk
Thalmic Labs Myo
§ Senses hand pose using
myoelectrics
§ Tracks rotation &
accelleration
§ No occlusion problem
§ Usable anywhere
§ Only handles 5 poses
§ Requires training
Image: Thalmic Labs

19. © 2014 Autodesk
Case Study: Curating Multi-touch Interaction Across
Form-Factors

20. © 2014 Autodesk
§ Started in January 2010.
§ Wanted to create a common, good standard for using
multi-touch for 3d Navigation.
§ Wanted to avoid everyone doing it differently
The History

21. © 2014 Autodesk
§ The interactions should be easy to learn and retain.
§ They should fit in naturally with existing standard 2d
interactions.
§ They should be suitable for common and key multitouch
hardware setups.
§ Blah blah other stuff
Our research goals

22. © 2014 Autodesk
First Test (July 2010)
§ Mac Laptop running
Mudbox
§ Wacom Intuos 3 tablet
§ iPhone providing a
multitouch surface
§ Also some “soft buttons”
on the phone

23. © 2014 Autodesk
The Gestures

24. © 2014 Autodesk
§ Navigation Gestures intuitable? - almost
§ Learnable? – Yes!
§ Comfortable? – Hard to gesture on that tiny surface
§ Soft buttons? – Bad idea; users are looking at the
screen, and must shift focus to use buttons. This breaks
their flow.
First Test Results

25. © 2014 Autodesk
Second Test (Nov 2010)
§ Introducing the iPad!
§ Tested gesture retention
§ Compared navigation
speed to mouse+hotkey

26. © 2014 Autodesk
§ Gestures learnable? – Yes!
§ Retained after 7-14 days? – Yes!
§ Comfortable? – Yes, on the iPad
§ Preferred? – Yes… except for fixable technical
problems (lag, etc)
§ Faster? – Yes
Second Test Results

27. © 2014 Autodesk
Third Test (April 2012)
§ Goal was to try the new
engine with many different
kinds of people
§ Try it on new form factor –
27” PPI multi-touch
monitor with stylus
§ First test with on-screen
touch

28. © 2014 Autodesk
§ Experience was surprisingly consistent for people of all
sizes/shapes/genders/etc.
§ Users kept trying to touch interface buttons, even when
they knew it didn’t work.
§ Camera tilt and zoom interactions felt wrong, unlike in
earlier tests – Huh?
Third test results

29. © 2014 Autodesk
This feels right

30. © 2014 Autodesk
Same interaction on a touch screen feels wrong

31. © 2014 Autodesk
Obvious finger/screen mapping

32. © 2014 Autodesk
§ Try separate behaviours for touchscreens and touch-
tablets.
§ Used 27” PPI touch/stylus monitor, vs large monitor
with Wacom Intuos 5 tablet.
§ Test new 3, 4, and 5-finger gestures (Mudbox specific).
Fourth Test (July 2012)

33. © 2014 Autodesk
§ Users perceived the different zoom/roll behaviours on
different devices to be the same
§ 4 and 5 finger gestures were awkward on the small
tablet, but fine on the big screen.
Fourth Test Results

34. © 2014 Autodesk
§ How do you map tablet touches to multiple 3d views?
Related Considerations

35. © 2014 Autodesk
§ Just another non-display multitouch tablet like the Intuos,
so we used the same behaviour.
§ WRONG.
Adding Magic Trackpad Support
Image: Apple

36. © 2014 Autodesk
§ Even though this is not a screen device, it controls a
persistent pointer on the screen.
§ The entire surface is mounted over a big button, so that
you can tap it (lightly) or click it (harder) like a mouse.
How is the Magic Trackpad different?

37. © 2014 Autodesk
Magic Mouse
§ All the problems of the
magic trackpad, and more!
§ Tiny touch area
§ Cursor moved by touch
OR by moving mouse
§ Frequent false touch
detection

38. © 2014 Autodesk
§ Testing with users is critical.
§ Obvious device classification is not enough; dig for the
salient differences.
§ Sometimes you have to do things differently to make
things feel the same across form-factors.
Conclusions

39. © 2014 Autodesk
Part Three: What’s Wrong with Mid-Air Gestures?

40. © 2014 Autodesk
Mid-Air Gesture Control is Really Cool
§ It’s been around for ages
§ It’s completely natural
§ The demos look
awesome
§ Why don’t we use it
everywhere?
Image: Minority Report (2002) Twentieth Centurey Fox

41. © 2014 Autodesk
§ Cost of the equipment?
§ Availability of hardware?
§ To hard to install?
§ Too computationally intensive?
§ No killer application?
§ The user experience sucks?
Why we don’t use mid-air gestures everywhere

42. © 2014 Autodesk
§ Cost of the equipment?
§ Availability of hardware?
§ To hard to install?
§ Too computationally intensive?
§ No killer application?
§ The user experience sucks?
Why we don’t use mid-air gestures everywhere

43. © 2014 Autodesk
§ It looks awesome in demos.
§ BUT in practice, it’s often:
§ Disappointing
§ Hard to control
§ No better (and often worse) than existing interactions
Primary Reason

44. © 2014 Autodesk
§ Awesome
YouTube
Demo!
Google Earth + Leap Motion

45. © 2014 Autodesk
“A lot of potential, but it needs much more development”
“It’s much more intuitive than clicking with a mouse…
but it’s not actually doing what I want.”
Maya + Leap Motion
Image: Digital Arts Online

46. © 2014 Autodesk
§ Gorilla arm
§ Gesture delineation & intentionality
§ No tactile feedback
§ Context and mapping
§ Internationalization
Why Is Mid-Air Gestural Interaction Hard?

47. © 2014 Autodesk
Gorilla Arm
Photo by Richard Ruggiero, public domain
§ Holding your arms up to
interact is quickly tiring.
Diagram from Austin, Gilbert (1806): Chiromania; or a Treatise on Rhetorical Delivery
Seriously?

48. © 2014 Autodesk
Gesture Delineation & Intentionality
Photo by: Biswarup Ganguly, used under Creative Commons license (CC)

49. © 2014 Autodesk
Context and Mapping
Image: Bolt, Richard A. “Put That There”: Voice and Gesture at the graphics interface. SIGGRAPH 1980.

50. © 2014 Autodesk
No tactile feedback
Image: Syntact ultrasonic mid-air haptic feedback device by Ultrasonic-Audio.com

51. © 2014 Autodesk
Internationalization
Frames from the short film Hand Gestures, by Jeff Werner, Vancouver, Canada

52. © 2014 Autodesk
Doomed?
No.

53. © 2014 Autodesk
How do we
know?

54. © 2014 Autodesk
1. Because Kinect Games are pretty Awesome
Frame from Kinect Star Wars

55. © 2014 Autodesk
2. Because music conductors don’t
suffer from Gorilla Arm
Photo courtesy of John Schrag

56. © 2014 Autodesk
“Consumed Endurance” – metric for Gorilla arm
Diagram from Consumed Endurance: A Metric to Quantify Arm Fatigue of Mid-Air Interactions. Juan David Hincapié-Ramos, Xiang Guo, Paymahn Moghadasian,
Pourang Irani,University of Manitoba

57. © 2014 Autodesk
§ We do it with voice, eye contact, facial expression,
repetition.
§ This is an area of active research, both in HCI and
linguistics (ASL)
3. Because we can all delineate gestures

58. © 2014 Autodesk
§ The question isn’t:
“is mid-air gestural interaction good”
§ The question is:
“What is it good for?”
4. Because we understand this:

59. © 2014 Autodesk
§ Is there a killer app for mid-air gestural input?
(aside from Games)
The Killer App?

60. © 2014 Autodesk
The Killer App: VR and AR
Image: technologyreview.com
Video: ign.com

61. © 2014 Autodesk
Case Study: Mid-Air Gestures with the Myo Device

62. © 2014 Autodesk
§ Test the suitability of the Myo device for VR/AR
navigation applications.
§ Come up with useful design guidelines and mental
models to help us design better gestural interactions with
this class of device
§ Improve our own shared gesture-interpretation library for
other Autodesk developers.
Our research goals

63. © 2014 Autodesk
§ “Gorilla Arm”… no use of upper arm
§ To delineate… use hand poses
§ Occlusion… not a problem with this device
§ Context… not absolute mapping needed
§ Tactile feedback… not an issue for this task
Designing the Gestural Interaction

64. © 2014 Autodesk
First Test
§ PC running Stingray
game engine
§ Thalmic Labs Myo
arm band
§ Calibrated Myo
§ Users received
training

65. © 2014 Autodesk
§ Continuous pose detection is problematic on the Myo
§ Holding poses continuously strains the hand
§ Non-mnemonic gestures were poorly recalled
§ Instinctive movement often overrode training
§ Gesture success depends on personal physiology
First Test Results

66. © 2014 Autodesk
Second Test (informal)
§ Changed steering to
more natural mapping
§ Removed need to
hold hand poses
§ Added speed control
§ Various small fixes

67. © 2014 Autodesk
§ Gestures are deep-rooted: adapt to your user’s
movement patterns where possible.
§ User testing is still crucial – test with different
physiologies.
§ Remember to measure physical comfort when testing –
people may not mention it unless asked.
§ Simplify your problem domain if necessary
§ Don’t give up right away
Conclusions

68. © 2014 Autodesk
The Real World is Not Magic
Video: https://www.youtube.com/channel/UC0bM0PxqixYjYY7ypt5mqGw
§ Gestural control can
seem like magic…
IF it is designed well
§ When you want to get
something done…
cool is irrelevant

69. © 2014 Autodesk
John Schrag @jvschrag
Ian Hooper +IanHooperUX

70. Autodesk is a registered trademark of Autodesk, Inc., and/or its subsidiaries and/or affiliates in the USA and/or other countries. All other brand names, product names, or trademarks belong to their
respective holders. Autodesk reserves the right to alter product and services offerings, and specifications and pricing at any time without notice, and is not responsible for typographical or
graphical errors that may appear in this document.
© 2014 Autodesk. All rights reserved.