New forms of interactions such as the gesture-based control of interfaces could enable interaction in situations where hardware controls are missing and support impaired people where other controls fails. The rich spectrum of combining hand postures with movements offers great interaction possibilities but requires extensive user testing to figure out a user interface navigation control with a sufficient performance and a low error rate. In this paper we describe a model-based interface design based on assembling interactors and multimodal mappings to design multimodal interfaces. We propose to use state charts for the rapid generation of different user interface controls to accelerate user testing. First we describe how our approach can be applied to design direct manipulation interfaces that rely on a mouse-based interface control. Then we present how we applied the interactor based design to quickly generate several variants of a gesture-based interface navigation control to demonstrate our approach. We comparethe three most promising variants in a user test and report about the test results.

1. Sebastian Feuerstack I Sebastian@Feuerstack.org29. Januar 2015
Modelling of User Interfaces based on State-
Charts to Ease the Test and Evaluation of
Multimodal Interactions
Sebastian Feuerstack, Mauro Dos Santos Anjo, Jessica Colnago und
Ednaldo Pizzolato
Dr.-Ing. Sebastian Feuerstack
Universidade Federal de São Carlos
Departamento de Computacão

2. 229. Januar 2015
Basic Question
How to model Interactions that
span Modes and Media ?

3. 329. Januar 2015
Why modeling?
• One (visual) language that
– is declarative and precise to discuss &
execute
– Is located between Tool and Code
– Supports different forms of interaction and
flexible multimodal prototyping
– Supports paradigm design

4. 429. Januar 2015
What is Out of
scrope ?
• No transformations, no process !
– Anticipation, long processes, complex
transformations, foreign process
– Instead: Interactor assembly like GUI building
– System Design vs. Application Design
• Tools
– Visual language -> Tool support

5. The Augmented “Drag-and-Drop 529. Januar 2015
Use Case: Test and Evaluation
Gesture-based Interface Navigation
• Rapid model-based Design and Comparison
of three variants
:IN:hand_gestures
NoHands
[one_hand]
wait_one
previous
select
OneHand
[no_hands]
selected
confirmed
[confirm]
select
next
next
Navigation
CommandPredecessor
Ticker
started
[1s]/tick
previous
Successor
previous
next
tick
tick
previous /
closer / farer
next /
closer /farer
select confirm
start_p
tick
start_n
tick

6. 629. Januar 2015
How to model?
• Models
– Interactors (Abstract & Concrete Media, Mode)
• Static: class diagram, Behavior: state-chart
– Mappings Mode-To-Media Synchronization
• Custom Notation

7. 729. Januar 2015
Abstract Media Model

8. 829. Januar 2015
Abstract Media Model
Single Choice
Aggregates a set of
Entities from that only
one can be chosen at
a time
Examples: Direction (left or right), Shopping Cart
Further Properties:
Container -> Aggregation, Discrete, Output to the User
Contains Single Choice Elements that are Inputs

9. 929. Januar 2015
Abstract Behavior Model
AUI:AIO:AIChoiceElement:
AISingleChoiceElement
initialized
listed
focus
suspended
organize
defocus
organized
suspend
focused
organize
Presenting
chosen
unchosen
drag
dragging
unchoose
drop
H
[in(focused)] choose
/ aios=find(parent.childs.chosen);
aios.all.unchoose
next||prev||parent
/aio=find(act);
aio.focus
Selectionpresent

10. 1029. Januar 2015
IR:IN:HandGesture:FlexibleTicker
NoHands
right_hand_appeared
wait_one
select
Right Hand
right_hand_disappeared
selected
confirmed
confirm
select
Navigation
Command
SpeedAdjustment
fastest
entry/
t = 800ms;
restart_ticker
faster
entry/
t = 1000ms;
restart_ticker
closer
farer
previous
nextnormal
entry/
t = 1200ms;
restart_ticker
closer
farer
previous
next
next
Predecessor
previous
Successor
tick
tick
start_p
tick
start_n
tick
Speed
started
entry/start_ticker
exit/stop_ticker
[1s]/tick
IR:IN:HandGesture:FlexibleTicker
NoHands
right_hand_appeared
Right Hand
right_hand_disappeared
IR:IN:HandGesture:FlexibleTicker
NoHands
right_hand_appeared
Right Hand
right_hand_disappeared
Navigation
Command
IR:IN:HandGesture:FlexibleTicker
NoHands
right_hand_appeared
wait_one
select
Right Hand
right_hand_disappeared
selected
confirmed
confirm
Navigation
Command
IR:IN:HandGesture:FlexibleTicker
NoHands
right_hand_appeared
wait_one
select
Right Hand
right_hand_disappeared
selected
confirmed
confirm
select
Navigation
Command
previous
next
previous
next
next
Predecessor
previous
Successor
tick
tick
start_p
tick
start_n
tick
IR:IN:HandGesture:FlexibleTicker
NoHands
right_hand_appeared
wait_one
select
Right Hand
right_hand_disappeared
selected
confirmed
confirm
select
Navigation
Command
previous
next
previous
next
next
Predecessor
previous
Successor
tick
tick
start_p
tick
start_n
tick
Speed
started
entry/start_ticker
exit/stop_ticker
[t]/tick
IR:IN:HandGesture:FlexibleTicker
NoHands
right_hand_appeared
wait_one
select
Right Hand
right_hand_disappeared
selected
confirmed
confirm
select
Navigation
Command
SpeedAdjustment
fastest
entry/
t = 800ms;
restart_ticker
faster
entry/
t = 1000ms;
restart_ticker
closer
farer
previous
nextnormal
entry/
t = 1200ms;
restart_ticker
closer
farer
previous
next
next
Predecessor
previous
Successor
tick
tick
start_p
tick
start_n
tick
Speed
started
entry/start_ticker
exit/stop_ticker
[t]/tick
Mode Model
(Example: Gesture Interactor)

11. 1129. Januar 2015
Multimodal Mapping
(Combining Mode with concrete Media)

12. 1229. Januar 2015
Multimodal Mapping
The Drag-and-Drop (abstract)
C
dst=SingleChoice.focused
C
Tw<0,3s
sc.drag
dst.drop(aios)
sc=SingleChoiceElement.focused
Tw<0,3sLeftButton.pressed
LeftButton.released
sc.parent.drop
fail

13. 1329. Januar 2015
Conclusions & Future Work
What’s the advantage?
• Detailed Modeling of Multimodal
Interactions (Media, Mode) including
Behavior
• Paradigm Design
• No gap between Design- and Runtime
• No foreign process

14. Whats next?
Design of Multimodal Interaction 1429. Januar 2015
Visit our website
– All papers
– Videos
– Open Source
Software
– MINT Framework
http://www.multi-access.de
Focus on:
– Fusion
– Paradigm Design
– Formalization
– Tools
– MINT Framework

15. 1529. Januar 2015