This tutorial aims to help user interface designers and developers to understand the issues involved in multi-device interactive applications, which can be accessed through both mobile and stationary devices even exploiting different interaction modalities (graphical, vocal, gesture, ...). It will provide a discussion of the possible solutions in terms of concepts, techniques, languages, and tools, with particular attention to Web environments. The tutorial will deal with the various strategies in order to adapt the user interface according to the interaction resources available, also discussing what results can be obtained through model-based approaches when multi-device interfaces are considered. It will consider how to address such issues both when authoring multi-device interfaces and when user interfaces for different devices are dynamically adapted and can even migrate seamlessly across them to follow the mobile user. Thus, it will discuss how to support task continuity across multiple devices in examples of distributed and migratory interfaces and related usability issues.

1. Adaptation and Continuity
in Multi-Device Environments
i Multi-D i E i
M lti t
Fabio Paternò
HIIS Laboratory
ISTI-C.N.R.
Pisa, Italy
http://giove.isti.cnr.it/
Tutorial Goals
 How to obtain interfaces able to adapt to multiple
device types while preserving usability?
 State of art in terms of approaches design criteria
approaches, criteria,
tools
 Particular attentiont to adaptation in Web applications
and use of model-based techniques
 Understanding the space of the possible solutions in
order to better apply them and think about new
pp y
solutions
 Consider how to address the device adaptation issue
both at design time and at run-time
 Discussion on how adaptation and continuity can be
supported in migratory user interfaces 2
1

2. Structure
 Introduction, Basic Concepts, Issues
 Usability and Task in Multi-device environments
Multi device
 Authoring Multi-Device Interfaces
 Model-based Support for Multi-device Interfaces
 Tools for Desktop-to-Mobile Adaptation
 Adaptation Customization
 Migratory Interfaces
 Partial / Trans-modal Migration
 Conclusions and Discussion
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 –
http://hiis.isti.cnr.it 3
Why Adaptation to the
Context of Use
User:
• Preferences
Environment: • Knowledge
•Position • Goals
• Light • Background
• Noise, … •…
• Social aspects Device:
•Privacy • Screen resolution
•Collaboration • Connectivity
•……. • Browser
•…
4
2

3. Display Size and Resolution
 Personal Computer (PC) usually varies between
800x600 and 1920x1200 pixels,
 Mobile devices usually between 240x240 and
960x640 pixels (Iphone 4)
 Simple phones have even lower resolutions
 It varies more with mobile devices than desktop
ones
 The Moore Law continuously changes these
numbers!
b !
5
Mobile Interaction Techniques Variation
http://www.w3.org/TR/mwabp/#bp-presentation-
http://www.w3.org/TR/mwabp/#bp-presentation-interaction
 Focus Based: The browser focus moves through elements
 The current focus of the page is easily determined because the focus element will be
highlighted;
 Focus area can move from one selectable element to another (e.g. from link to link) even when
(e g
widely spaced
 Pointer Based: Key-based navigation controls a pointer that
can cover any part of the screen
 Selectable elements that are associated with each other need to be close as moving the pointer
can be slow;
 Selectable elements need to be large enough to be easily selected -- since the pointer often
moves in steps of between 5 - 10 pixels;
 Selectable elements should have rollovers to make it clear when the pointer has entered their
active area.
 Touch Based: Events are related directly to a finger or stylus
touch position on the screen.
 Selectable elements may be widely spaced since the user can select them directly;
 Selectable elements must be large enough to be easily selected (e.g. list items should have a
height of at least 30px);
 No elements are in focus until they are selected so extra information cannot be passed to the
user (e.g. rollovers will not work).
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 –
http://hiis.isti.cnr.it 6
3

4. Usability in Mobile Interaction
 Minimize text input
 Exploit the elements of the mobile device, such
y
as accesskeys
 Consistency between platforms
 Prevent user error
 The purpose of the interface elements should be
clear
 Avoid overloading the user interface with many
elements
 Limit the need for scrolling
 Short time access
 Access to small pieces of information
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 –
http://hiis.isti.cnr.it 7
Supporting Tasks in Multi-
Multi-
platform Environments
 Platform definition
 Same task on multiple platforms in the same
p p
manner
 Same task on multiple platforms but with
different user interface elements
 Same main task, with different levels of
subtasks
 Dependencies among tasks performed on
different platforms
 Tasks meaningful only on some platform types
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 –
http://hiis.isti.cnr.it 8
4

5. Same task on multiple platforms
with different user interfaces
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 –
http://hiis.isti.cnr.it 9
Task Meaningful only on
Some Platforms
Search for flights
Promotions
Reservations
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – 10
http://hiis.isti.cnr.it
5

6. Authoring Multi-device
Multi-
Interfaces
 Platform-specific
Platform specific authoring (e.g.
(e g
Amazon)
 Multiple-device authoring (e.g. with
CSS)
 Single authoring
 Automatic re-authoring
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 –
http://hiis.isti.cnr.it 11
Multiple Device Authoring
Damask
(James Lin PhD – CHI’08)
•Sketches
•Layers
•Patterns
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 –
http://hiis.isti.cnr.it 12
6

7. Automatic Re-Authoring
Re-
 Scaling, such as Safari on IPhone
 Transducing, t
T d i translates elements i t
l t l t into
other formats, and compresses and
converts images to match device
characteristics, such as Mowser or
Skweezer,
 Transforming goes f th t modify b th
T f i further to dif both
contents and structures originally designed
for desktop systems to make them suitable
to display on small screens
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – 13
http://hiis.isti.cnr.it
Transforming (examples)
 Single column, (for example Opera SSR)
eliminates scrolling in one dimension, it
g
greatly increases the amount of scrolling in
y g
the other dimension.
 Fisheye (for example Fishnet) is a fisheye
Web browser that shows a focus region at a
readable scale while spatially compressing
page content outside the focus region
 Overview + detail splits a Web page into
multiple sections and provides an overview
li l i d id i
page with links to these sections.
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 –
http://hiis.isti.cnr.it 14
7

8. Narrow Solution (e.g. Opera SSR)
15
………. A lot of vertical scrolling !!!!
Narrow Solution
 The order for the content follows that of the
markup file starting with the top
 The images are scaled to the size of the
screen
 The text is always visible and the content is
compacted without blank spaces
 Content that requires space such as maps
and tables can become unreadable
 Sometimes it is difficult to understand that
the page has changed because the initial
part is the same
16
8

9. Requirements for the Nokia
Solution (CHI 2006, Roto et al.)
 Remove the need of horizontal scrolling to read
text
 Provide enough contextual information to give an
idea of the page structure and the current
location in it
 Don’t destroy the original page layout
 Don’t introduce modal interaction
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 –
http://hiis.isti.cnr.it 17
MiniMap Method
(CHI 2006, Roto et al.)
Nokia S60 phones Overview + Detail
Original Compacted Mini-Map
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 –
http://hiis.isti.cnr.it 18
9

10. Application of Information
Visualization Techniques
Fisheye calendar
allows complex
tasks to be
completed more
quickly
Focus + Context
In DateLens
[Bederson et al 02, 04]
19
Model-
Model-based approaches
 Allow designers and developers to concentrate
on main semantic aspects
 Languages th t represent such aspects
L that t h t
 Avoid need to learn and manage many
implementation languages
 Linking semantic information and
implemementation elements
p
 Interoperability through many possible
implementation languages
 Facilitate support of assistive technology
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 –
http://hiis.isti.cnr.it 20
10

11. Abstraction Levels
in Interactive Systems
 Task and object – Activity oriented
 – I want to select a work of art
 Abstract Interface – Platform Independent
 – Single selection object with high cardinality
 Concrete Interface –Platform Dependent
 – List Interaction object with X elements
 Implementation
 – List object in Java or XHTML or ....
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 –
http://hiis.isti.cnr.it 21
The ConcurTaskTrees
Notation for Task Models
Hierarchical Temporal relations
structure
Task Allocation
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – 22
http://hiis.isti.cnr.it
11

12. Support for Applications
based on Web Services
(http://giove.isti.cnr.it/tools/Mariae)
Task Model
Web TM
Services Language
Multitouch AUI
MARIA
Desktop AUI PDA AUI Language
Phone AUI
Specification Specification
Specification
Annotations
M‐touch
Multitouch Phone
Desktop CUI PDA CUI PDA
Phone CUI CUI
Specification Specification CUI Desktopp
Language
Specification Language CUI
Language
Specifications Languages
Reverse Engineering
Automatic Model Creation !
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 –
http://hiis.isti.cnr.it
24
12

13. Adaptation in Authoring Environments
(SUPPLE, Gajos and Weld)
Weld)
 Takes a functional specification of the interface, the device-
specific constraints, a typical usage trace, and a cost function.
 The cost function is based on user preferences and expected
speed of operation
 SUPPLE’s optimization algorithm, finds the user interface which
minimizes the cost function while also satisfying all device
constraints.
Adaptation and Continuity in Multi-Device Environments 25
Architectural Solutions for
Automatic Web Adaptation
Application server-side adaptation
Request/Device capability
Server Client
Adapted Content
Proxy-side adaptation
Request/
Request Device capability
Server Proxy Client
Content Adapted content
Client-side adaptation
Request
Server Client
Content Adapted content
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 –
http://hiis.isti.cnr.it 26
13

14. Dynamic User Interface
Adaptation
 Use of Logical Description Languages
 Existing Web desktop Applications
 Automatic user interface generation
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 –
http://hiis.isti.cnr.it
Parametric Bidimensional
Cost-
Cost-based Semantic Redesign
Font adjustments according
user preferences
Images
Interactor replacement
Long texts
Grouping
Relation
Data tables
Layout tables
Paternò, Zichitella, HCSE 2010
14

15. Adaptation Solutions
http://mowser.com http://www.skweezer.com
•Limited reduction of
image dimensions
•Uses predefined style sheets •Aims to reduce
•No support for tables or long texts horizontal scrolling
Customizing Adaptation
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI
2010 – http://hiis.isti.cnr.it 30
15

16. Customizing Adaptation
31
Browsing Large Table in
Small Screens
Ohnishi and Tajima,
UIST’08
Ease comparison of cells far from each other
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile
HCI 2010 – http://hiis.isti.cnr.it 32
16

17. Page Summarization
 The Abstraction-based approach uses sentence
manipulation techniques like reduction
reduction,
compression and reformulation.
 The Extraction-based approach assigns scores
to sentences in order to select those which
better represents the whole text
 Feature based (e.g. term frequency, sentence
p
position, attributes…); machine learning, graph
, ); g, g p
based techniques
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 –
http://hiis.isti.cnr.it 33
Micro-
Micro-level Summarization
(PowerBrowser, Buyukkokten et al.)
PowerBrowser,
 The importance of a keyword depends on the
frequency it occurs in a text and in a larger
collection
 A word within a given text is considered most
important if it occurs frequently within the text, but
infrequently in the larger collection
 The significance factor of a sentence is derived
from an analysis of its constituent words
 The sentences in which the greatest number of
e se te ces c t e g eatest u be o
frequently occurring distinct words are found in
closest proximity to each other are probably
important
 MEAD is a public multi-document summarization
system, which provides more flexible support in this
area (see http://www.summarization.com/mead/)
17

18. Adaptation in Multi-User and
Multi-
Multi-
Multi-Device environments
 WebSplitter (Han et al., CSCW’00)- collaborative Web browsing by
creating personalized partial views of the same Web page depending
on the user and the device. Developers have to specify the Web
content in XML and define a policy file indicating what content tags
should be shown for each device and user.
Concepts and Tools for Multi-Device User Interfaces 35
How People Use
Multiple Devices (Dearman & Pierce ‘08)
 A recent study in US of 27 people from academic and
industrial research revealed that on average they
employ more than five computing devices
 Associating a user’s activities with a particular device is
problematic for multiple device users because many
activities span multiple devices
 Device use varies by user and circumstance; users assign
different roles to devices both by choice and by constraint.
 Users employ a variety of techniques for accessing
information across devices but participants reported
managing information across their devices as the
most challenging aspect of using multiple devices.
36
18

19. Flexible Access in Multi-
Multi-
Device environments
 Moving objects: across interactive
devices th
d i through pick-and-drop
h i k dd
 Distributed user interfaces: application
logic receiving input from multiple
devices
 Migratory user interfaces: change
g y g
device, interface migration with state
preservation
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 –
http://hiis.isti.cnr.it 37
Pick-and-
Pick-and-drop
Jun Rekimoto, UIST’97 / CHI’98
A Direct-manipulation Technique for
Multiple-Computer Environments
38
19

20. PUC
(Personal Universal Controller) [CMU, Nichols, Myers]
Personal Mobile
Automatically Device
Generated
G t d
Interface
Control
Abstract
Specification State Feedback
39
Why Migratory Interfaces
 Our life is becoming a multi-device experience
 One of the main source of frustration is that we
need to restart for each device change
 Need for continuous access to interactive
services across various devices
 Migratory user interfaces can transfer among
different devices (from ‘source’ devices to
(
‘target’ devices), so as to allow the users to
continue their tasks
 Application domains such as shopping, bids for
auction on line, games, making reservations
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – 40
http://hiis.isti.cnr.it
20

21. Migration Phases
 Device Discovery: Devices notify their presence
 When to Migrate: Migration Trigger (User or
System or Mixed initiative)
 Where to Migrate: Identification of Target and
its resources
 What to Migrate: User interface and/or
Application logic
 How to Migrate: Adaptation (depending on the
type of target)
 State Persistence: Source state extraction and
association to target version
 Activation in the target device: Upload the
adapted version at the point in which the user left
off on the source device 41
Usability in Migration
 Relevant aspects to continuity
 Time
 Time required by the migration process
db h
 Time from the last interaction in the source device
 Adaptation process
 When adaptation makes unclear how to continue the task
 Predictability:
 The target device
 Which part of the user interface migrate
 Where the result of an interaction will be presented
 Learning: to get familiar with the migration
process 42
21

22. What is the state to preserve
 User Input
 Focus
 Cookies
 Session
 History
 Bookmarks
 JavaScripts
J S i t
Task: SelectMenu
value: Vegetable
Your Selected 43
Menu is Vegetable
Web Session Migration Using
Dynamic 2D Barcodes
A. Alapetite, PUC 14(1): 45-52 (2010)
Append the ID of the current session
(token) to the current URL query string
When the Web server receives a request
from a new device (the mobile phone)
with a known existing session ID, the
server will provide the same content as
the one served to the previous device
(the desktop).
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 –
http://hiis.isti.cnr.it 44
22

23. Approaches to Migration
 Bharat and Cardelli (UIST’95) - the migration of entire
applications
 Chung and D
Ch d Dewan (UIST’96) - when migration is
h i ti i
triggered the environment starts a fresh copy of the
application process in the target system, and replays
the saved sequence of input events to the copy. This
solution does not support interface adaptation.
 Kozuch and Satyanarayanan (2002) - solution for
migration based on the encapsulation of all volatile
execution state of a virtual machine (migration of
applications among desktop or laptop systems)
 Melchior, Grolaux, Vanderdonckt (EICS’09) - solution for
distributed user interfaces with extension of Tcl/Tk
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – 45
http://hiis.isti.cnr.it
Architectural Framework
Device 1
Migration Platform
Application
• User Interface Migration Framework
Mi ti F k
• Application Logic •Server
• Connection to OPEN Migration Services
Client Orchestration
OPEN Client
Migration Services
g
•R
Runs i b k
in background
d User Interface migration
• Connects to OPEN Application logic migration
platform framework State persistence
Context management
Trigger management
46
23

24. Device discovery and selection
Target Device Selection
•same environment
•device availability
•same user (or shared)
•interaction features
Example representation of the migration environment
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 –
http://hiis.isti.cnr.it 47
Device Selection
Ghiani,
PhD
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 –
http://hiis.isti.cnr.it 48
24

25. Example Architecture Migration
Device Discovery (1)
Migration Trigger (6)
Migration
Client
Request Page (2)
Desktop
Application
Cookies
Reverse
Semantic Redesign
Provide Annotated Page (5) Proxy State Mapper
Server Generator
Upload (8)
Trasmission of DOM + current state through callback PDA
AJAX (7)
Provide Page (4)
Request Page (3)
Application Server 49
Migrating between mobile
and Digital Tv
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI
2010 – http://hiis.isti.cnr.it 50
25

26. Example Migration (Video)
(shopping scenario)
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 –
http://hiis.isti.cnr.it 51
Trans-
Trans-Modal Migration
Vocabularyy
Sender Subject Date Size
&Time

Luois Last minute offer May 25 1k
lestat 9:23
 Information organization George
Clow
Re: Cameleon
meeting
June 3
11:02
3k
Different concrete objects
Luois Learn about Java June 5 2k
Lestat 12:20

Mary Rex Re: Cameleon June 7 1k
meeting 10:00
 Not a simple one to one mapping
 Task not Supported
 Task implemented with different concrete
objects
 Adaptation of runtime data
52
26

27. Differences in task support
implementation
(grouping sound) Please say your name.
Please say your surname.
yy
Please say the date of reservation.
What type of menu do you prefer: fish;
meat; vegetable (grouping sound)
Your name is …. Your surname is ….
You have booked a table on … and You
prefer…
(grouping sound) If you want to cancel the
reservation , say delete; If you want to
confirm the reservation, say ok (grouping
sound).
Grouping Operator Object that Support the task
“Provide date”
53
Adaptation of runtime data
Your name is Louis You have booked
Louis.
a table on 3 July
July.
(grouping sound) Please say your
name.
Please say your surname.
Please say the date of reservation.
Name: Louis
Date: 3 July
What type of menu do you prefer:
fish; meat; vegetable (grouping
sound)
Your name is …. Your surname is ….
You have booked a table on … and You
prefer…
(grouping sound) If you want to
cancel the reservation , say delete; If
you want to confirm the reservation,
say ok (grouping sound).
54
27

28. Application Logic Reconfiguration
Clausthal
University in
OPEN Project
Partial Migration
(Mobile HCI 2010 paper)
56
28

29. Partial Migration
Partial Migration with Interface
Structure Representation
29

30. Partial Migration with
Direct Selection
59
Partial Migration with
Direct Selection (video)
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – 60
http://hiis.isti.cnr.it
30

31. Research Agenda
 Automatic Desktop-to-Vocal Adaptation
 Migration involving multiple users
g g p
 Integration of wider set of interaction modalities
in migratory environments
 Migration from multiple devices to multiple
devices
 Authoring Environments for Distributed User
Interfaces
I t f
 Privacy and Security of Migratory User Interfaces
 Applying Machine Learning Techniques to User
Interface Adaptation
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – 61
http://hiis.isti.cnr.it
More Info at
 EU OPEN Project http://www.ict-open.eu
 EU SERENOA Project htt //
http://www.serenoa-fp7.eu/
f 7 /
 W3C group on model-based interfaces
http://www.w3.org/2005/Incubator/model-based-ui/charter/
 EU Artemis SMARCOS Project http://www.smarcos-
project.eu/
 Forthcoming S i
F th i Spinger Book on Mi t
B k Migratory
Interactive Applications in Ubiquitous
Environments
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – 62
http://hiis.isti.cnr.it
31

32. Don’t Forget EICS 2011 !
Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – 63
http://hiis.isti.cnr.it
32