1. Investigating
Perceptible
Affordances
of
Touch-­‐Based
NUIs:
New
Designs
Paradigms
for
Novel
Interactions
Jacques Chueke
London, UK, May 2011
George Buchanan
(1st Supervisor)
Lecturer, Centre for HCI Design
Stephanie Wilson
(2nd Supervisor)
Lecturer, Centre for HCI Design
Master in Design, PUC-Rio, RJ, Brazil
PhD Researcher at the Centre for HCI Design
School of Informatics, City University London

4. Extreme Reality Ltd. XTR3D (Android)
Phones with hover state (SONY XPERIA) and eye gaze interaction (SAMSUNG S4)

5. It’s not supposed to be like this…
SIRI

6. Or like this...
Gmail
Motion,
April
2011

7. Is there a problem?

8. (…) How is anyone to know, first, that this
magical gesture exists, and second, in which
settings it operates?
Norman, D. Nielsen, J. (2010)
One of the powers of modern computers is
discoverability, you can explore, but with gesture
systems it’s a pain. It’s amazing how many things
people don’t know about the computers they use
and there’s no way to find out.
Norman, D. (2012)

9. •Shift from mouse and keyboard towards touch-based devices:
•No hover state, no close button, no right click, no keyboard shortcuts...

10. •Strange hybrids emerged...

11. •Unfamiliarity with novel visual metaphors
within NUIs.
•The ‘hidden menus and toolbars’ issue.
•The ‘hidden gestures’ issue.

14. Windows 8

15. Josh
Clark,
'Buttons
are
a
Hack',
2011:
http://vimeo.com/21796111

18. BlackBerry Playbook, 2012

19. Some theory...

20. "Perceived or Perceptible Affordances" are
actions you understand just by looking
at the object, before you start using it (or
feeling it, if it's a physical device rather than
an on-screen UI element).
Gaver (1991), Norman (1999 & 2008),
Amant (1999), Nielsen (2008)

21. + WHAT IS IT?
+ WHAT AM I SUPPOSED TO DO?
+ WHAT’S GOING TO HAPPEN IF I DID?
= GOOD PERCEPTIBLE AFFORDANCE
WHAT"S GOING TO HAPPEN IF I DID IS BEING REGARDED NOWADAYS AS 'FEEDFORWARD', CHECK
THE WORK OF Wensveen & Djajadiningrat and Vermeulen

22. PERCEPTIBLE AFFORDANCE

23. INPUT
NUI [MODE OF INTERACTION LAYER]
PERCEPTIBLE AFFORDANCE
Post-WIMP GUI [INTERFACE LAYER]
OUTPUT
Execution Cycle
Evaluation Cycle
InterpretationPerception Evaluation
Formulation of
Intention
Specification of
Actions Sequence
Execution
*
Based
on
Norman’s
Theory
of
Action,
1986

24. NUI [MODE OF INTERACTION LAYER]
Post-WIMP GUI [INTERFACE LAYER]
Execution Cycle
Evaluation Cycle
PERCEPTIBLE AFFORDANCE
Amantʼs Conceptual Framework on Perceptible Affordances (1999).
Wigdor & Wixton (2011) review of Wuʼs theory of RCT.

25. 12th point system
Observation Phase
• Perception
1. Perception (Yes/No)
• Identification
2. What is it?
3. What should you do?
• Comprehension
4. How many fingers?
5. Touch/hold or swipe?
6. In which direction?
7. What will change?
SCORE
“0” for wrong assessment/inability to respond
“0.5” for lack of confidence or right assessment/interaction on the 2nd and 3rd times
“1” for correct response/interaction
Interaction Phase
• Registration
8. Touch to confirm
• Continuation
9. Perform touch/hold or swipe
10. Perform direction
• Termination
11. System status
12. Restore status (restarting
the observation phase)

26. Post-WIMP GUI [INTERFACE LAYER]Evaluation Cycle

27. NUI [MODE OF INTERACTION LAYER]Execution Cycle

28. The Eye Tracking study

30. Eye Tracking study:
order + focus + planning + expectation + bias = MENTAL MODEL

31. Issues to be investigated/Interactions to tested (08)
• Hidden Menus and toolbars
Unveiling Menu
Bringing the picture to the page
• Object Manipulation
Press and hold for options
• Workspace Manipulation
Flip pages
• Hidden gestures for iOs Manipulation
Zoom In/Out
Alternate between running apps
Unveil task switcher
Minimize app

32. DESIGN 01: “SMUDGY”

45. DESIGN 02: “DOTS”

58. 1. Unveiling Menu
Series 02” “SMUDGY”
Series 01: “DOTS”
Observation Phase
• Perception: 1. Perception (Yes/No)
• Identification: 2. What is it? 3. What should you do?
• Comprehension: 4. How many fingers? 5. Touch/hold or swipe? 6. In which direction? 7. What will change?
Interaction Phase
• Registration 8. Touch to confirm
• Continuation 9. Perform touch/hold or swipe 10. Perform direction
• Termination 11. System status 12. Restore status (restarting the observation phase)

59. 2. Bringing the picture to the page
Series 02” “SMUDGY”
Series 01: “DOTS”
Observation Phase
• Perception: 1. Perception (Yes/No)
• Identification: 2. What is it? 3. What should you do?
• Comprehension: 4. How many fingers? 5. Touch/hold or swipe? 6. In which direction? 7. What will change?
Interaction Phase
• Registration 8. Touch to confirm
• Continuation 9. Perform touch/hold or swipe 10. Perform direction
• Termination 11. System status 12. Restore status (restarting the observation phase)

60. 3. Press and hold for options
Series 02” “SMUDGY”
Series 01: “DOTS”
Observation Phase
• Perception: 1. Perception (Yes/No)
• Identification: 2. What is it? 3. What should you do?
• Comprehension: 4. How many fingers? 5. Touch/hold or swipe? 6. In which direction? 7. What will change?
Interaction Phase
• Registration 8. Touch to confirm
• Continuation 9. Perform touch/hold or swipe 10. Perform direction
• Termination 11. System status 12. Restore status (restarting the observation phase)

61. 4. Zoom In/Out
Series 02” “SMUDGY”
Series 01: “DOTS”
Observation Phase
• Perception: 1. Perception (Yes/No)
• Identification: 2. What is it? 3. What should you do?
• Comprehension: 4. How many fingers? 5. Touch/hold or swipe? 6. In which direction? 7. What will change?
Interaction Phase
• Registration 8. Touch to confirm
• Continuation 9. Perform touch/hold or swipe 10. Perform direction
• Termination 11. System status 12. Restore status (restarting the observation phase)

62. 5. Flip pages
Series 02” “SMUDGY”
Series 01: “DOTS”
Observation Phase
• Perception: 1. Perception (Yes/No)
• Identification: 2. What is it? 3. What should you do?
• Comprehension: 4. How many fingers? 5. Touch/hold or swipe? 6. In which direction? 7. What will change?
Interaction Phase
• Registration 8. Touch to confirm
• Continuation 9. Perform touch/hold or swipe 10. Perform direction
• Termination 11. System status 12. Restore status (restarting the observation phase)

63. 6. Alternate applications
Series 02” “SMUDGY”
Series 01: “DOTS”
Observation Phase
• Perception: 1. Perception (Yes/No)
• Identification: 2. What is it? 3. What should you do?
• Comprehension: 4. How many fingers? 5. Touch/hold or swipe? 6. In which direction? 7. What will change?
Interaction Phase
• Registration 8. Touch to confirm
• Continuation 9. Perform touch/hold or swipe 10. Perform direction
• Termination 11. System status 12. Restore status (restarting the observation phase)

64. 7. Unveil task switcher
Series 02” “SMUDGY”
Series 01: “DOTS”
Observation Phase
• Perception: 1. Perception (Yes/No)
• Identification: 2. What is it? 3. What should you do?
• Comprehension: 4. How many fingers? 5. Touch/hold or swipe? 6. In which direction? 7. What will change?
Interaction Phase
• Registration 8. Touch to confirm
• Continuation 9. Perform touch/hold or swipe 10. Perform direction
• Termination 11. System status 12. Restore status (restarting the observation phase)

65. 8. Minimize application
Series 02” “SMUDGY”
Series 01: “DOTS”
Observation Phase
• Perception: 1. Perception (Yes/No)
• Identification: 2. What is it? 3. What should you do?
• Comprehension: 4. How many fingers? 5. Touch/hold or swipe? 6. In which direction? 7. What will change?
Interaction Phase
• Registration 8. Touch to confirm
• Continuation 9. Perform touch/hold or swipe 10. Perform direction
• Termination 11. System status 12. Restore status (restarting the observation phase)

66. More questions than
answers...

67. How people are making sense of new technologies?
How to create better designs adapted to NUI?
When is the best moment to present the visual cue? In
which context?
Automatic at first runs? How many times? If unrequired
would imply in loss of control? Will be obtrusive?
Perhaps only if summoned/triggered by user action?
Interactive tutorials only? Mandatory tutorials?
Conflict of specific application mode x OS mode. How to
design something which is CLEARLY related to either?

68. “The most profound technologies are those that disappear.
They wave themselves into the fabric of everyday life until
they are indistinguishable from it.” Weiser (1991)
The equipment fades into the background. This unspoken
background against which our actions are played out is at
the heart of Heideggerʼs view of being-in-the-world.”
Dourish (2005: 109)

69. Thank you for your attention.
Questions?
Jacques Chueke
jacques.chueke.1@city.ac.uk
@jchueke