When interacting with physical objects, we receive sensory feedback about the characteristics of these objects. Specifically, the sounds resulting from touching a surface provide information about the material of the surface and about one's own touching behaviour. Current developments in interactive systems are opening up new avenues in the use and design of both physical and virtual objects. These developments have the potential to change people’s behaviours, perceptions and emotions – elements of user experience that can be measured. This talk presents the results of a study on auditory feedback as example of how research methods from cognitive psychology and human computer interaction can be used to evaluate these potential changes. This research may inform the design of physical and virtual objects and help enhance everyday user experiences.

1. Sonification of virtual and real surface tapping:
evaluation of behaviour changes, surface perception
and emotional indices
Enrico Furfaro
UX Architect @ We Love Digital
24th October 2015
Ana Tajadura-Jiménez, Research Fellow, UCL
Nadia Berthouze, Professor in Affective Computing, UCL
Frédéric Bevilacqua, Head Researcher, IRCAM
UCD 2015 - Humanity in digital landscapes: Mind, Cognition and Psychology

2. 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 2
Dr. Nadia Berthouze,
Professor in Affective
Computing, UCL
Frédéric Bevilacqua,
Head Researcher, IRCAM
Enrico Furfaro
UX Architect
Dr. A. Tajadura-Jiménez,
Research Fellow and
Principal Investigator

3. 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 3
Dr. A. Tajadura-Jiménez,
Research Fellow and
Principal Investigator
Dr. Nadia Berthouze,
Professor in Affective
Computing, UCL
Frédéric Bevilacqua,
Head Researcher, IRCAM
Enrico Furfaro
UX Architect

4. What does Sonification mean?
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 4

5. Three aspects
Auditory displays
• Systems that systematically transform data into
sounds that can be manipulated by human users
through an interface.
Sonification
• Sonification is this transformation: data into sounds
to convey information.
Sonic interaction design
• It is the study and exploitation of sound as one of
the principal channels conveying information.
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 5

6. Why is it important?
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 6

7. Why is it important?
• Interaction with objects is increasingly mediated
through their digital representation.
• Interaction with object is multisensory.
• Touch, vision and hearing contributing and
interacting with each other
To what extent do we make use of the information
conveyed by auditory feedback? How do we measure it?
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 7
Google’s Project Soli

8. The research
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 8

9. Our aim
To explore how the sounds resulting from tapping on a
surface inform :
1. The applied strength when tapping.
2. The user’s ability to tap & changes in emotional
states.
3. The physical features of the surface material
(hardness).
And to test a multidimensional measurement approach
to evaluate user experience related to these three
dimensions.
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 9
Me tapping on a table

10. 1. Tapping behaviour
Changes might occur because of the auditory action loop.
• The effect for which users try to adjust their actions
based on the auditory feedback they receive.
Example: the audio feedback in real time helped elite
rowers to adjust their strokes and improved their
performance. (Schaffert et al.’10).
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 10
Listening to the sound of the boat motion increases boat velocity
(Schaffert et al.’10)
Audio
Feedback
Motor Behaviour

11. 2. User’s ability to tap
Users’ ability to tap might be affected because of the
sense of body can be altered by sound feedback .
• Sense of body = the mental representation of body.
• Crucial for our interaction with our environment.
• Obtained and continuously updated through sensing
and acting.
Example - the perceived length of participants' arm can
be altered by presenting real-time sound feedback
suggesting an altered distance. (Tajadura-Jiménez et
al.’12).
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 11
Tajadura-Jiménez et al.’12

12. 2. User’s emotional states
Body movement is a medium to express and to regulate
one’s own emotions.
• We expected changes in behaviour and emotions.
For example, the strokes in ‘Fruit Ninja’ were
successfully used to detect players’ emotional states
(frustrated, excited, relaxed, bored) during the
gameplay.
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 12
Gao et al.’12

13. 3. Physical features of the material
Changing audio feedback can result in changes in an
object perceived material properties.
• Both in case of natural surfaces and virtual haptic
surfaces.
Example:
• Reduced high frequencies led to biased perception
of sand paper’s texture (smoother). (Guest, et al.
2002)
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 13
Sandpaper

14. 3. Physical features of the material
Changing audio feedback can result in changes in an
object perceived material properties.
• Both in case of natural surfaces and virtual haptic
surfaces.
Examples:
• Attenuating high frequencies led to biased
perception of sand paper (smoother). (Guest, et al.
2002)
• higher frequencies or louder sound while biting
crisps led to biased perception of crispness and
freshness (Zampini & Spence, 2003)
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 14
Kermit the frog eating crisps

15. The prototype
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 15

16. tapping on “real” surface
tapping on “virtual” surface
The tapping action triggers, in real-time, the presentation of pre-recorded tapping
sounds
Surfaces & Components
24/10/2015 Enrico Furfaro, WLD | Sonification of surface tapping | UCD 2015 16
Two modes: “real” and “virtual”.

17. The measures
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 17

18. 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 18
Design & procedure
Virtual
Real
Six conditions for
each participant
“Strong” sound
feedback
“Medium” sound
feedback
“Weak” sound
feedback
Design
2 surface type x 3 sound level strength
Timeline for each condition
0 s 10 s
baseline1
70 s 80 s
baseline2Audio-feedback phase
time
Questionnaire
x

19. 1. Measuring tapping behaviour
The accelerometer was used to measure the tapping
behaviour.
• Acceleration = tap’s strength
• Interval between taps = frequency of taps
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 19

20. Galvanic skin response (GSR)
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 20
2. Measuring emotional experience and ability to tap
Self Assessment Manikin (SAM)
Valence
Arousal
Dominance

21. 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 21
2. Measuring emotional experience and ability to tap
7-point Likert scales: physical strength, ability to tap,
aggressiveness.
Now please, draw an horizontal line through the vertical scale to indicate how
much mental effort you feel you had to invest to complete the tapping task.
Subjective Mental Effort Questionnaire (SMEQ)

22. To measure perceived surface hardness we used a 7-point Likert scale.
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 22
3. Measuring surface hardness

23. The results
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 23

24. 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 24
Results
Real vs. Virtual surface
• Harder surface, larger perceived strength and ability to tap, less stressed & more in control.
Surface hardness
• Virtual surface perceived softer when ‘weak’ feedback.
Emotions & ability to tap
• Audio motor incongruences led to unpleasant arousing experiences.
• Real surface. More negative valence when weak sound feedback and less able to tap.
• Virtual surface. Weak sound led to higher GSR values.
Behaviour
• Real & virtual surface. introducing the sound feedback sped up movements and decreased acceleration.
• Real surface. Weak sounds led to higher maxima acceleration

25. 24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 25
Takeaways
When measuring and designing digital representation
of objects, consider that audio feedback plays a
fundamental role in complementing or substituting
haptic feedback.
1. it informs about physical properties of objects.
2. it can be used to manipulate and/or lead motor
behaviour.
3. if congruent to the action, it leads to a more
pleasant user experience.
4. it might affect the sense of body which relates to
perceived body characteristics and self-esteem
Example of measures:
• Self Assessment Manikin (SAM)
• 7-point Likert scales
• Subjective Mental Effort Questionnaire (SMEQ)
• Bandura’s self-efficacy test
• GSR
• Sensors (for example accelerometer and piezo) to
capture data and for sonification of objects.

26. Implications for design
1. To complement when interacting with digitally
represented objects.
Digital representation of objects characterized by a
limited amount of haptic & visual feedback:
• when extreme precision in applied strength is critical
e.g. touch-less surgery or dismantling bombs.
• Shopping on-line (tactile-sensory substitution)
perceived material properties & emotion responses
are important .
2. To guide motor behaviour.
Motor behaviour changes in a self-controlled way:
• Physical rehabilitation (VR/augmented reality).
• Games promoting health- or fun-movement
(rehabilitation, education, entertainment).
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 26
Nicole Stenger wearing one of the first VR prototypes in the early 90ies

27. THANKS!
QUESTIONS?
24/10/2015 Enrico Furfaro, We Love Digital | Sonification of surface tapping | UCD 2015 27
enrico@welovedigital.com
@fasteedio

28. Sonification of Surface tapping original paper
• Tajadura-Jimenez, A; Bianchi-Berthouze, N; Furfaro, E; Bevilacqua, F; (2015) Sonification of surface tapping
changes behavior, surface perception, and emotion. IEEE Multimedia , 22 (1) 48 - 57.
• Furfaro, E., Berthouze, N., Bevilacqua, F., Tajadura-Jiménez, A. (2013) Sonification of surface tapping:
Influences on behaviour, emotion and surface perception. Interactive Sonification Workshop (ISon 2013),
Bielefeld, 9th-10th December, 2013.
Sonification and Sonic Interaction design
• T. Hermann, A. Hunt, and J.G. Neuhoff, The sonification handbook. Berlin:Logos Publishing House, 2011.
• Franinović, Karmen, and Stefania Serafin. Sonic interaction design. Mit Press, 2013.
Sonification of boat motion improves athletes’ movements
• N. Schaffert, K. Mattes, and A.O. Effenberg, “Listen to the boat motion: acoustic information for elite rowers”,
in Proc. Interaction Sonification workshop (ISon), Stockholm, Sweden, 2010
Sound feedback affects arm’s length perception
• A. Tajadura-Jiménez, A. Väljamäe, I. Toshima, T. Kimura, M. Tsakiris and N. Kitagawa, “Action sounds recalibrate
perceived tactile distance”. Current Biology, 22(13), 2012, pp. R516-R517.
24/10/2015 Enrico Furfaro, WLD | Sonification of surface tapping | UCD 2015 28
References

29. Fruit Ninja to detect players’ emotional states
• Y. Gao, N. Bianchi-Berthouze, and H. Meng, “What does touch tell us about emotions in touchscreen-based
gameplay?”, in ACM Transactions on Computer-Human Interaction 19(4), pp. 31, 2012.
Body movements and emotions
• N. Bianchi-Berthouze, “Understanding the role of body movement in player engagement”. Human Computer
Interaction 28(1), 2012, pp. 42-75.
• A. Kleinsmith, R. De Silva, and N. Berthouze, “Grounding affective dimensions into postures description”,
Lecture Notes in Computer Science, Berlin / Heidelberg: Springer, 3784, 2005, pp.263-270.
Sound and perception of objects’ characteristics
• Guest, S., Catmur, C., Lloyd, D. & Spence, C. (2002). Audiotactile interactions in roughness perception.
Experimental Brain Research, 146(2), 161-171
• D. Merrill, and H. Raffle, “The sound of touch”, in Proc. ACM Conference on Human Factors in Computing
Systems 2007, ACM Press, 2007, pp. 2807-2812.
• Zampini, M., & Spence, C. (2004). The role of auditory cues in modulating the perceived crispness and
staleness of potato chips. Journal of sensory studies,19(5), 347-363
24/10/2015 Enrico Furfaro, WLD | Sonification of surface tapping | UCD 2015 29
References