This document describes a blended prototyping system that allows designers to create paper prototypes of mobile applications and test them digitally on a tabletop surface. The system uses either color detection or barcode recognition to digitize paper sketches and define interactive components. It aims to enhance paper prototyping without disrupting the design process through unnecessary shifts in fidelity between paper and digital tools. The document outlines the hardware, software, interaction techniques, alternatives considered, and an evaluation method to measure the success of the new techniques based on parameters like quickness, ease-of-use, distraction, and user experience.
[2024]Digital Global Overview Report 2024 Meltwater.pdf
Enhancing Tabletop Design of Mobile Apps
1. Enhancing the interaction space
of a tabletop computing system
to design paper prototypes
for mobile applications
Master Thesis
Francesco Bonadiman
2016
4. Blended Prototyping
● tabletop system based on hand-drawn paper sketches
○ converted into digital versions
→ projected on the table
→ expanded further
2
5. ● tabletop system based on hand-drawn paper sketches
○ converted into digital versions
→ projected on the table
→ expanded further
○ transformed into testable applications
→ on a mobile device
Blended Prototyping
2
6. ● tabletop system based on hand-drawn paper sketches
○ converted into digital versions
→ projected on the table
→ expanded further
○ transformed into testable applications
→ on a mobile device
● enhances development of mobile applications
○ accelerates early design phases
Blended Prototyping
2
9. Hardware
● video projector
○ projects mobile frames & prototypes
● webcam
○ recognizes barcodes of screens
● DSLR camera
○ shoots HQ pics of the sketches
3
10. Hardware
● video projector
○ projects mobile frames & prototypes
● webcam
○ recognizes barcodes of screens
● DSLR camera
○ shoots HQ pics of the sketches
● tablet
○ allows to perform actions on the prototypes
3
16. Advantages of Paper Prototyping
● potential of Paper-Prototyping & mobile devices = combined
6
17. Advantages of Paper Prototyping
● potential of Paper-Prototyping & mobile devices = combined
● first impression of not-yet-developed product
6
18. Advantages of Paper Prototyping
● potential of Paper-Prototyping & mobile devices = combined
● first impression of not-yet-developed product
● paper → cheap, fast & intuitive (Snyder)
○ no unimportant details → iterative refinement (Nielsen)
6
19. Advantages of Paper Prototyping
● potential of Paper-Prototyping & mobile devices = combined
● first impression of not-yet-developed product
● paper → cheap, fast & intuitive (Snyder)
○ no unimportant details → iterative refinement (Nielsen)
○ quickly create multiple design alternatives (Landay)
6
20. Advantages of Paper Prototyping
● potential of Paper-Prototyping & mobile devices = combined
● first impression of not-yet-developed product
● paper → cheap, fast & intuitive (Snyder)
○ no unimportant details → iterative refinement (Nielsen)
○ quickly create multiple design alternatives (Landay)
● same usability issues as Hi-Fi discovered
○ benefits of early usability data = 10+ times bigger (Snyder)
6
21. ● encourages collaboration & critiques →
interdisciplinary & creative (no “gaps”)
Advantages of the System
7
22. ● encourages collaboration & critiques →
interdisciplinary & creative (no “gaps”)
● testing within real-life scenarios (de Sá / Carriço)
→ in thousand of different usage conditions
Advantages of the System
7
23. ● encourages collaboration & critiques →
interdisciplinary & creative (no “gaps”)
● testing within real-life scenarios (de Sá / Carriço)
→ in thousand of different usage conditions
● add code & functionalities
○ define dynamic interface behavior
○ smooth transition to development
Advantages of the System
7
25. Core features
Via TABLET → Determine widgets & semantics
● manually define “hotspots” on prototype
● turn these into different design patterns
8
26. Core features
Via TABLET → Determine widgets & semantics
● manually define “hotspots” on prototype
● turn these into different design patterns
● create links between the prototypes
● remove components & connections
8
27. Core features
Via TABLET → Determine widgets & semantics
● manually define “hotspots” on prototype
● turn these into different design patterns
● create links between the prototypes
● remove components & connections
● convert into working code
8
28. Problem
● several devices + media → different degrees of fidelity
○ paper & office supplies → computers & smart objects
9
29. Problem
● several devices + media → different degrees of fidelity
○ paper & office supplies → computers & smart objects
BUT
9
30. Problem
● several devices + media → different degrees of fidelity
○ paper & office supplies → computers & smart objects
BUT
Changing fidelity of tool & modality of interaction
→ disrupts creative design process
9
31. Problem
● several devices + media → different degrees of fidelity
○ paper & office supplies → computers & smart objects
BUT
Changing fidelity of tool & modality of interaction
→ disrupts creative design process
→ shifting continuously confuses users
9
33. Fidelity Clash
● combination of low- & high-tech solutions = puzzling
○ one single user stops ideation process
→ to perform any action
10
34. Fidelity Clash
● combination of low- & high-tech solutions = puzzling
○ one single user stops ideation process
→ to perform any action
○ breaking of collaborative & creative moment (Snyder)
10
35. Fidelity Clash
● combination of low- & high-tech solutions = puzzling
○ one single user stops ideation process
→ to perform any action
○ breaking of collaborative & creative moment (Snyder)
● users distracted → do not interact anymore
○ get unfocused → lose the “flow”
10
36. Fidelity Clash
● combination of low- & high-tech solutions = puzzling
○ one single user stops ideation process
→ to perform any action
○ breaking of collaborative & creative moment (Snyder)
● users distracted → do not interact anymore
○ get unfocused → lose the “flow”
→ perceived as too technical, isolating & distracting
10
38. Objectives
● replace high-tech interactions with low-tech approaches
● new interaction techniques → not to shift fidelity of media
11
39. Objectives
● replace high-tech interactions with low-tech approaches
● new interaction techniques → not to shift fidelity of media
THEN
11
40. Objectives
● replace high-tech interactions with low-tech approaches
● new interaction techniques → not to shift fidelity of media
THEN
● find & implement alternative solutions to tablet
11
41. Objectives
● replace high-tech interactions with low-tech approaches
● new interaction techniques → not to shift fidelity of media
THEN
● find & implement alternative solutions to tablet
● low-tech approaches to define “hotspots” & perform actions
○ keep interaction techniques learnable & usable
11
44. Tasks
1. digitize a screen by taking a picture
2. duplicate a whole screen
13
45. Tasks
1. digitize a screen by taking a picture
2. duplicate a whole screen
3. detect a component (button, image or textbox)
13
46. Tasks
1. digitize a screen by taking a picture
2. duplicate a whole screen
3. detect a component (button, image or textbox)
4. connect two screens (link a button to next screen)
13
47. Tasks
1. digitize a screen by taking a picture
2. duplicate a whole screen
3. detect a component (button, image or textbox)
4. connect two screens (link a button to next screen)
5. remove a component from a screen
13
48. Tasks
1. digitize a screen by taking a picture
2. duplicate a whole screen
3. detect a component (button, image or textbox)
4. connect two screens (link a button to next screen)
5. remove a component from a screen
6. remove a connection between two screens
13
49. Tasks
1. digitize a screen by taking a picture
2. duplicate a whole screen
3. detect a component (button, image or textbox)
4. connect two screens (link a button to next screen)
5. remove a component from a screen
6. remove a connection between two screens
7. remove a whole screen
13
50. Alternatives considered
● Voice recognition
● Object recognition
● Gesture recognition
● Special pen
● Special button
● Colored objects
● Barcode recognition
● Transparent layers
14
51. Alternatives considered
● Voice recognition
● Object recognition
● Gesture recognition
● Special pen
● Special button
● Colored objects
● Barcode recognition
● Transparent layers
● most approaches = unfeasible
○ noisy / confusing environment
○ need of not-yet-existing hardware
○ time constraints
14
54. A. Color Detection
● no specific hardware
● dynamic & “playful” interaction
Concept: different color = different component
16
55. A. Color Detection
● no specific hardware
● dynamic & “playful” interaction
Concept: different color = different component
→ colored markers + semi-transparent BBPapier
→ users can see through without “ruining” sketches
16
56. A. Color Detection
● no specific hardware
● dynamic & “playful” interaction
Concept: different color = different component
→ colored markers + semi-transparent BBPapier
→ users can see through without “ruining” sketches
→ JavaCV & OpenCV
16
57. → Procedure
● control card = “toolbox” → paint with markers
● same color → fill up the chosen component
17
58. → Procedure
● control card = “toolbox” → paint with markers
● same color → fill up the chosen component
● digitize the sketch → creating threshold
○ average color in toolbox → range (+30/-30)
○ exterior pixels discarded → central values
→ avoid mistakes & colors overlapping
17
59. → Procedure
● control card = “toolbox” → paint with markers
● same color → fill up the chosen component
● digitize the sketch → creating threshold
○ average color in toolbox → range (+30/-30)
○ exterior pixels discarded → central values
→ avoid mistakes & colors overlapping
● threshold = calculate component’s RGB value
● any pixel within → associated to specific UI component
17
60. → Approximation
Once pixels & colors detected
● algorithm recognizes contours of shape
● approximates it to a rectangle
● component created & projected
18
61. → Approximation
Once pixels & colors detected
● algorithm recognizes contours of shape
● approximates it to a rectangle
● component created & projected
Need for filling component
● algorithm not efficient only for outline
● rough drawings typical of sketching
18
63. B. Barcode Recognition
Barcode marker = tool for specific operations
● take pictures of device
● connect screens
● recognize & delete components
● copy screens
19
64. B. Barcode Recognition
Barcode marker = tool for specific operations
● take pictures of device
● connect screens
● recognize & delete components
● copy screens
Algorithm continuously running
→ checks if new barcodes are detected
19
67. Further improvements
Sidebar → alternative approach for some functionalities
● rectangular transparent area + at top of table
● if barcode placed inside → operation performed
21
68. Further improvements
Sidebar → alternative approach for some functionalities
● rectangular transparent area + at top of table
● if barcode placed inside → operation performed
Clock (progress indicator) → added onto center-top area
21
69. Further improvements
Sidebar → alternative approach for some functionalities
● rectangular transparent area + at top of table
● if barcode placed inside → operation performed
Clock (progress indicator) → added onto center-top area
● immediate visual feedback to users
● if appears → users realize they are performing an action
21
70. Further improvements
Sidebar → alternative approach for some functionalities
● rectangular transparent area + at top of table
● if barcode placed inside → operation performed
Clock (progress indicator) → added onto center-top area
● immediate visual feedback to users
● if appears → users realize they are performing an action
● 3 seconds = time-frame users might change their mind
→ effective to avoid accidental mistakes
21
72. Modality A: Sidebar
● digitize a screen → taking a picture
○ by placing single barcode inside
23
73. Modality A: Sidebar
● digitize a screen → taking a picture
○ by placing single barcode inside
● duplicate a screen
○ by placing two barcodes inside
○ one is already digitized (source)
○ one is empty (destination)
23
75. Modality A: Colors
● toolbox projected
→ boxes for button, image, textbox…
● BBPapier over toolbox → paint with color
● same color → BBPapier over component
24
76. Modality A: Colors
● toolbox projected
→ boxes for button, image, textbox…
● BBPapier over toolbox → paint with color
● same color → BBPapier over component
● digitize screen with toolbox visible on table
→ colored square displayed to represent component
24
77. Modality A: Colors
To connect two screens
● same + use FROM → TO box
To delete component → digitize again
● physically remove colored BBPapier
● or cover it with white paper
To remove connection
● same with button-connector
25
78. Modality B
● camera tool to digitize
● copy tool to duplicate
● handles to detect component
● arrow tool to connect screens
● rubber tool to delete
○ component
○ connection
○ screen (only way)
26
79. Dependent Variables
● measure overall success of new interaction techniques
○ if quick & easy-to-use without distracting user
○ which of the two modalities = most effective
27
80. Dependent Variables
● measure overall success of new interaction techniques
○ if quick & easy-to-use without distracting user
○ which of the two modalities = most effective
● prove if efficient & offers satisfactory user-experience
27
81. Dependent Variables
● measure overall success of new interaction techniques
○ if quick & easy-to-use without distracting user
○ which of the two modalities = most effective
● prove if efficient & offers satisfactory user-experience
4 parameters:
→ Quickness → Ease-of-use → Distraction → User-Experience
27
83. Quickness & Ease-of-Use
Quickness
● time for task completion
● evaluation is videotaped
● Effective Time
○ duration of task
→ without system errors
→ avoids malfunctionings
28
84. Quickness & Ease-of-Use
Quickness
● time for task completion
● evaluation is videotaped
● Effective Time
○ duration of task
→ without system errors
→ avoids malfunctionings
Ease-of-use
● % successfully completed tasks
● ratio calculated by checking
interactions executed perfectly
28
85. Quickness & Ease-of-Use
Quickness
● time for task completion
● evaluation is videotaped
● Effective Time
○ duration of task
→ without system errors
→ avoids malfunctionings
Ease-of-use
● % successfully completed tasks
● ratio calculated by checking
interactions executed perfectly
○ users make no mistakes
○ no significant problems
○ not need any help / hint
28
86. Distraction
Two factors: quickness & workload index (RTLX)
● unweighted (Raw) version of NASA Task Load Index (TLX)
● 6 subscales:
○ Mental, Physical, Temporal Demand; Performance; Effort; Frustration
29
87. Distraction
Two factors: quickness & workload index (RTLX)
● unweighted (Raw) version of NASA Task Load Index (TLX)
● 6 subscales:
○ Mental, Physical, Temporal Demand; Performance; Effort; Frustration
● own survey → filled in by users after every task
● 7-points linear scales → then averaged = RTLX
● the lower → the less demanding & distracting task is
29
88. User-Experience
● dimensions of AttrakDiff Survey → studies by Hassenzahl
○ “how users rate the usability and design of your interactive product”
30
89. User-Experience
● dimensions of AttrakDiff Survey → studies by Hassenzahl
○ “how users rate the usability and design of your interactive product”
● own custom version → filled in after finishing whole study
● 28 7-points semantic differential scales
○ opposite adjectives at both poles ("good - bad" / "human - technical")
○ implicitly divided into 4 dimensions (Pragmatic Quality, Hedonic
Quality - Identity, Hedonic Quality - Stimulation, Attractiveness)
30
90. Independent Variables
● Task: 1 to 7
● Modality: A | B
● Group: AB | BA
● Area of expertise: HCI | IT | Other
● Sketching familiarity: 1 to 5
● Mobile familiarity: 1 to 5
● Tabletop use: Yes | No
● Lighting conditions: Day | Night
31
91. Experiment Design
● two different design solutions → A/B Testing
● “within-subject” design → every user tests both versions
○ no interactions in common → no influence
32
92. Experiment Design
● two different design solutions → A/B Testing
● “within-subject” design → every user tests both versions
○ no interactions in common → no influence
● to avoid possible bias = AB & BA groups
32
93. User Demographics
● one user at a time → focus on interactions
○ easier to observe & more difficult to get biased
33
94. User Demographics
● one user at a time → focus on interactions
○ easier to observe & more difficult to get biased
● 24 participants → 11 females & 13 males
● half aged 18-24, 10 aged 25-34, two older
● 7 background in HCI, 8 IT, rest other expertise
● via Facebook groups & personal connections
● reward = tasty gift or house utensils
33
96. Results
● reviewed videos from GoPro
● investigate how long took users to complete tasks
● detect when they had troubles & needed help
35
97. Results
● reviewed videos from GoPro
● investigate how long took users to complete tasks
● detect when they had troubles & needed help
● SPSS → ONE-WAY ANOVA test
○ ideal for type of Dependent Variables (continuous)
○ valid for any group of users of study
○ robust to violations in underlying assumptions
35
98. Quickness
● Modality B faster than Modality A
○ time for A = 40.1s, B only 23.25s
○ statistically significant (p < .05)
36
99. Quickness
● Modality B faster than Modality A
○ time for A = 40.1s, B only 23.25s
○ statistically significant (p < .05)
● Effective Time (without system errors)
○ time for A = 29.24s, B only 18.38s
○ statistically significant (p < .05)
36
100. Quickness
● Modality B faster than Modality A
○ time for A = 40.1s, B only 23.25s
○ statistically significant (p < .05)
● Effective Time (without system errors)
○ time for A = 29.24s, B only 18.38s
○ statistically significant (p < .05)
→ color detection = long process
36
102. Ease-of-use
● Modality A easier to use than B
● perfectly accomplished tasks:
○ ratio: A = 84%, B = 60%
○ statistically significant (p < .05)
37
103. Distraction
● A almost as demanding as B
○ A RTLX index = 1.90
○ B RTLX index = 1.84
38
104. Distraction
● A almost as demanding as B
○ A RTLX index = 1.90
○ B RTLX index = 1.84
● On a 7-points scale →
not statistically significant
(F1,286 = 0.289, p = 0.591)
38
106. User-Experience
● users feel “assisted by the product”
○ usability can be improved
● users are “stimulated by the product”
○ users can identify with it
○ room for hedonic improvement
39
107. User-Experience
● users feel “assisted by the product”
○ usability can be improved
● users are “stimulated by the product”
○ users can identify with it
○ room for hedonic improvement
● system considered as “rather desired”
○ attractiveness above average
39
123. Task 6 Analysis
● rubber (B)
○ faster
○ less demanding
● color detection (A)
○ easier to use
○ experience?
46
124. Interesting facts
● people with background in HCI
→ performed more easily & faster
● good sketching abilities
→ same + found system more attractive
47
125. Interesting facts
● people with background in HCI
→ performed more easily & faster
● good sketching abilities
→ same + found system more attractive
● people familiar with mobile devices
→ faster + system less distracting
47
126. Interesting facts
● people with background in HCI
→ performed more easily & faster
● good sketching abilities
→ same + found system more attractive
● people familiar with mobile devices
→ faster + system less distracting
● during day-time → better performance & lower RTLX
47
127. Considerations
● accuracy = ratio of task’s precision
○ global accuracy = 86%
○ color detection = 67%
→ more delicate + error-prone
48
128. Considerations
● accuracy = ratio of task’s precision
○ global accuracy = 86%
○ color detection = 67%
→ more delicate + error-prone
● barcodes + hands into digitized image → distraction + inexperience
48
129. Considerations
● accuracy = ratio of task’s precision
○ global accuracy = 86%
○ color detection = 67%
→ more delicate + error-prone
● barcodes + hands into digitized image → distraction + inexperience
● if BBPapier wavy → grey area projected
○ projector’s brightness → shadow (“ghost shape”)
48
130. Projector’s beam
● when toolbox at sides of projection → outer areas appear darker
○ either no color is detected
○ “fake” grey color recognized → multiple components created
49
131. Projector’s beam
● when toolbox at sides of projection → outer areas appear darker
○ either no color is detected
○ “fake” grey color recognized → multiple components created
● most system errors
○ excessively intense light causing reflections on table
49
132. Projector’s beam
● when toolbox at sides of projection → outer areas appear darker
○ either no color is detected
○ “fake” grey color recognized → multiple components created
● most system errors
○ excessively intense light causing reflections on table
→ closest barcodes could not be detected
→ devices continuously refreshing
49
133. Projector’s beam
● when toolbox at sides of projection → outer areas appear darker
○ either no color is detected
○ “fake” grey color recognized → multiple components created
● most system errors
○ excessively intense light causing reflections on table
→ closest barcodes could not be detected
→ devices continuously refreshing
○ “effective time” → task completed if interaction is correct
49
137. Which better?
● color detection (A)
○ easier to use
○ takes longer time
● barcode recognition (B)
○ definitely faster
○ harder to understand
51
138. Which better?
● color detection (A)
○ easier to use
○ takes longer time
● barcode recognition (B)
○ definitely faster
○ harder to understand
● both have similar workload
○ around 1.9 out of 7
○ positively low →
not too distracting
51
141. Ideal implementation
1-2. sidebar → digitize & duplicate screens
2. copy tool too → duplicate without moving papers
3-4. both approaches to detect components & create connections
a. color detection = from scratch + multiple tasks at same time
52
142. Ideal implementation
1-2. sidebar → digitize & duplicate screens
2. copy tool too → duplicate without moving papers
3-4. both approaches to detect components & create connections
a. color detection = from scratch + multiple tasks at same time
b. handles & arrow = fast & easy-to-use
52
143. Ideal implementation
1-2. sidebar → digitize & duplicate screens
2. copy tool too → duplicate without moving papers
3-4. both approaches to detect components & create connections
a. color detection = from scratch + multiple tasks at same time
b. handles & arrow = fast & easy-to-use
5-6-7. rubber tool to remove components, connections & screens
→ implement separate ones to avoid possible mistakes
52
145. Future Work
● visual feedback when projections disappear from table
● icon near clock → tells which action is being performed
53
146. Future Work
● visual feedback when projections disappear from table
● icon near clock → tells which action is being performed
● exploring color detection further
○ better algorithms to calculate the threshold
○ better ways to detect shapes through contours
53
147. Future Work
● visual feedback when projections disappear from table
● icon near clock → tells which action is being performed
● exploring color detection further
○ better algorithms to calculate the threshold
○ better ways to detect shapes through contours
● barcodes = built with wood / opaque plastic → reduce reflections
53
148. Future Work
● visual feedback when projections disappear from table
● icon near clock → tells which action is being performed
● exploring color detection further
○ better algorithms to calculate the threshold
○ better ways to detect shapes through contours
● barcodes = built with wood / opaque plastic → reduce reflections
● compare interaction techniques with original system
53
149. Future Work
● visual feedback when projections disappear from table
● icon near clock → tells which action is being performed
● exploring color detection further
○ better algorithms to calculate the threshold
○ better ways to detect shapes through contours
● barcodes = built with wood / opaque plastic → reduce reflections
● compare interaction techniques with original system
● test system in real-life scenario → inside company or startup
53