Gesture-Aware Remote Controls: Guidelines & Interaction Techniques

GESTURE-AWARE REMOTE CONTROLS:
GUIDELINES & INTERACTION TECHNIQUES
Gilles Bailly, Dong-Bach Vo, Eric Lecolinet and Yves Guiard
Telecom ParisTech, LTCI - CNRS

mardi 15 novembre 2011

Interactive television

More and more services and
contents:
• Electronic program guide

• Video on demand

• Web based widgets

• Web applications

• Personal content

2
Dong-Bach Vo ‒ ICMI 2011


contents:

• Video on demand




3


contents:

• Video on demand




4


contents:

• Video on demand




5


contents:

• Video on demand




6

Interacting with interactive television

Inspired from computer
interaction:
• Deep hierarchical menus

• Long lists

7


interaction:

• Long lists

8


interaction:

• Long lists

9


interaction:

• Long lists

10

Remote control

Problems:
• Browsing

• Commands

11

Remote control

Problems:
• Browsing

• Commands

12

Remote control

Problems:
• Browsing

• Commands

13

Remote control

Problems:
• Browsing

• Commands

Limited
expressiveness

14

Improving expressiveness

Solution?
• Add a new button for each new
command

15


Solution?
command Pultius TV remote control
[Artlebedev 07]

16


Solution?
command

Can we beneﬁt from
moving the remote control
in 3D space to improve
expressiveness?

17

Approach

Advantages of 3D gestures:
• Allow large vocabulary of gestures

• Can be easy to remember

• Free visual focus from the remote
control

• Comfortable to perform gestures at
home [Rico et al. 10]

• Remote controls can remain small and
easy to hold

18

1. Interactive television
2. Related work
3. Observing spontaneous gestures
4. Exploring new gesture dimension
5. Designing new interaction techniques
6. Conclusion

19

Related work

3D gestures oﬀer 6 degrees of
freedom:
• 3 translations Y
• 3 rotations Z

X

20

Related work

3D gestures oﬀer 6 degrees of
freedom:
• 3 translations Y
• 3 rotations Z
Yaw
Roll

Pitch
X

21

Related work

Rotations:
• 1 dimension

• 2 dimensions

• Discretization of tilt angle
with visual feedback
[Rahman et al. 09]

Motion marking menu system [Oakley 03]

Pitch
Dong-Bach Vo ‒ ICMI 2011 X 22

Related work

Rotations:
• 1 dimension

• 2 dimensions

• Discretization of tilt angle
with visual feedback
[Rahman et al. 09]
GesText [Jones et al. 10]

Roll Z

Pitch
X 23

Related work

Rotations:
• 1 dimension
Y
• 2 dimensions
Z
• Discretization of tilt angle Yaw
with visual feedback 12 levels
[Rahman et al. 09] Roll
16 levels

24

2. Related work
4. Exploring new gesture dimension
6. Conclusion

25

Experiment 1: spontaneous gestures

What gestures users
perform spontaneously
for moving targets? Y
• Which degree(s) of freedom? Z
Yaw
• Is it context dependant
(smartphone vs. remote
controls)? Roll

Pitch
X
26


Task:
• Perform the gesture that would
move the target to this position

Procedure:
• Smartphone vs. Remote control

• Observation from video recordings

27


Task:
• Perform a gesture which has
produced this eﬀect

Procedure:
• Smartphone vs. Remote control

• Observation from video recordings

28

9

Number of participants
8
7
6
5
Results: 4
3
• Either rotations or translations 2
1
• 14 of 18 users perform rotations 0
(S) Condition (RC) Condition
• Pitch and Roll only in smartphone Pitch & Roll rotations
condition
Pitch & Yaw rotations

• Pitch and Yaw in remote control X & Y translations
condition

• Roll can be used for something else

29

9

8
7
6
5
Results: 4
3
1
condition

condition

Z
Roll

Pitch
X 30

9

8
7
6
5
Results: 4
3
1
condition

condition Y

Yaw
ntrol condition

Pitch
X 31

9

8
7
6
5
Results: 4
3
1
condition

condition Y

• Roll can be used for something else Z
Yaw
Roll

Pitch
X 32

2. Related work
4. Exploring roll gestures
6. Conclusion

33

Experiment 2: exploring roll gestures

What we know:
• 16 items selection with visual feedback
[Rahman 09]

• 3 items selection in eyes free selection
[Oakley 07]

34


Question:
How many levels users can control
without visual feedback?

Task:
Use any roll amplitude you need to
reach the target
• Roll - Press - Roll - Release strategy

• No feedback provided

Procedure:
• 9 participants

• Items set size (5, 7, 9, 11 items)

35


Question:

Task:
reach the target


Procedure:
• 9 participants


36


Question:

Task:
reach the target


Procedure: Step 1: Roll to prepare the gesture
• 9 participants


37


Question:

Task:
reach the target


Procedure: Step 2: Press the trigger
• 9 participants


38


Question:

Task:
reach the target


Procedure: Step 3: Roll gesture
• 9 participants


39


Question:

Task:
reach the target


Procedure: Step 4: Release the trigger
• 9 participants


40


Question:

Task:
reach the target


Procedure:
• 9 participants


41


Question:

Task:
reach the target


Procedure:
• 9 participants


42

Experiment 2: results

Angular range:
• Diﬀers largely among 162° 326°
participants

• Depends on item-set size

Minimum and maximum
Angular variation: angular range for 7 items

• Constant variation between two
contiguous items

43


Angular range: 400

• Diﬀers largely among
participants

∆ angle (in degrees)
300

• Depends on item-set size
200

100
Angular variation:
• Constant variation between two 0
0 1 2 3 4 6 7 8 9 10 11
contiguous items
number of items in set

44


Angular range: Mean angular varation for 7 items

• Diﬀers largely among 200
participants 160
120
• Depends on item-set size 80
40
0

Angular variation: -40
-80
• Constant variation between two -120
contiguous items -160
-200
-3 -2 -1 0 1 2 3
Left Center Right

45


Recognition:
• KNN algorithm
• euclidian distance on angular
variation
• cross-validation technique

Results:
• Very good for 5 items (96.3%)
• Insuﬃcient for 9 and 11 items
• 7 items:
• 87.7% for cross-validation

• 96% with user speciﬁc training
46


100
Recognition: 96.3

• KNN algorithm 75

accuracy (in %)
variation 50
25
Results:
• Very good for 5 items (96.3%) 0
5 items 7 items 9 items 11 items
mean accuracy rate
• 7 items:

47


100
Recognition:
• KNN algorithm 75 77.1

accuracy (in %)
• euclidian distance on angular 63.1
variation 50
25
Results:
mean accuracy rate
• 7 items:

48


100
Recognition:
87.7
• KNN algorithm 75

accuracy (in %)
variation 50
25
Results:
mean accuracy rate
• 7 items:

49

2. Related work
6. Conclusion

50

Designing interaction techniques

Goals:
• two-level circular menus

• that combine gestural modalities
• Pitch & Yaw

• Roll

• Directional pad buttons

51


Design space:
• 1 modality per menu
level

• D-pad used as a
baseline

• Unpromising
techniques dismissed
at pretest

52


Design space: 2nd Level
Pitch & Yaw Roll Directional Pad
• 1 modality per menu 1st Level
level

• D-pad used as a Pitch & Yaw
baseline

• Unpromising
techniques dismissed Roll
at pretest

Directional Pad

53


level

baseline

• Unpromising
at pretest

Directional Pad Baseline

54


level

Gesture trigger

penalty
baseline

• Unpromising Biomechanical Poor results in Gesture trigger
constraint pretests penalty
at pretest

Directional Pad Baseline

55


level

3D multi-strokes

menu
3D gestures

parallelization
Gesture trigger

penalty
baseline

• Unpromising Biomechanical Poor results in Gesture trigger
constraint pretests penalty
at pretest

• 5 techniques to
Directional Pad
2D/3D gestures 2D/3D gestures
Baseline
evaluate parallelization parallelization

56

Experiment 3: interaction techniques

Procedure:
• Menu with 4 * 4 items

• Combination of modalities

• User feedback

• 13 participants

Task:
• Perform the gesture
corresponding to the symbolic
stimulus

57


Best modality combinations Mean accuracy rate by technique

for accuracy: 100,0
97.4 98.4 94.2
80,0
• Pad / Pad (baseline)

Accuracy in %
60,0
• Pad / Pitch & Yaw (better than
baseline) 40,0

• Pad / Roll 20,0

0
Pad/Pad Pad/P&Y Pad/Roll

58


Best modality combinations Mean time selection by technique
for speed: 1700
1650

• Similar results 1275 1400 1390

Time in ms
• Pad / Roll is a bit slower 850

425

0
Pad/Pad Pad/P&Y Pad/Roll

59


User satisfaction: Subjective preferences
7
• Pad / Pad 6
5
• then Pad / Pitch & Yaw
4

• then P&Y / P&Y 3
2
• then Pad / Roll 1
0
Speed Accuracy Pleasant Tiring Easy to learn

Pad/Pad Pad/P&Y Pad/Roll P&Y/P&Y

60

Multimedia system menu

Novice mode:
• 2-level circular menu

• 2 modalities

• Novice mode:
• 1st level : Pitch and Yaw

• 2nd level : Roll

61


Novice mode:
• 2-levels circular menu

• 1st level selection

• 2nd level selection

62


Novice mode:
• 2-levels circular menu

• 1st level selection

• 2nd level selection

63


Expert mode:
• Same gestures

• But the menu does not
appear

• Implicit learning of
gestures

64

2. Related work
6. Conclusion

65

Conclusion

What we have learned:
• Experiment 1:
• Users rather use rotations when interacting with
handheld devices
• With a remote control, they use pitch and yaw to interact
with a distant screen
• Experiment 2: Roll gestures without feedback work fine:
• for 5 levels
• for 7 levels with user specific training
• Using button and 3D gestures can efficiently augment
remote control expressiveness

66

Conclusion

What we have learned:
• Experiment 3:

• 3D gestures provide good performance

• especially when combined with d-pad buttons

• Conclusion:
• Moving the remote control into space is indeed an eﬀective
way to improve expressiveness

67

Thank you for your attention!

GESTURE-AWARE REMOTE CONTROLS:
GUIDELINES & INTERACTION TECHNIQUES

http://perso.telecom-paristech.fr/ via/VIA_Project.html
dong-bach.vo@telecom-paristech.fr


Gesture-Aware Remote Controls: Guidelines & Interaction Techniques

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (19)

Similaire à Gesture-Aware Remote Controls: Guidelines & Interaction Techniques

Similaire à Gesture-Aware Remote Controls: Guidelines & Interaction Techniques (11)

Gesture-Aware Remote Controls: Guidelines & Interaction Techniques