Short talk presented at the CanUX 2015 conference (Ottawa). "Embrace UX and adapt your evaluation methods accordingly!" Carine Lallemand, University of Luxembourg

1. Embrace UX… and adapt
your evaluation methods accordingly!
Carine Lallemand
University of Luxembourg
@carilall

2. -Prof. Marc Hassenzahl (2013)
« Strangely, while I find the proposition to consider the
experience before the thing quite a radical change,
many practitioners and academics of HCI happily
embrace experience – however, without changing
much in their approach. »

3. To think about experience first, we have to know
what experience is about
This knowledge has an impact on design practice

4. The nature and complexity of UX involves a deep
change in the methods we use
UX is highly dynamic
The memory of an experience matters more than
the experience itself
UX is highly contextual
UX is more than usability1
2
3
4

5. UX is more than usability
Thüring & Mahlke, 2007
A system’s perceived attractiveness is based on the perception
of its pragmatic and hedonic qualities
System
User
Context
Interaction
characteristics
Perception of non-instrumental qualities
Emotions
Perception of instrumental qualities
Components of User Experience
Consequences
overall evaluation,
acceptance,
intention to use,
choice of
alternatives
1

6. System
User
Context
Interaction
characteristics
Perception of non-instrumental qualities
Emotions
Perception of instrumental qualities
Components of User Experience
Consequences
overall evaluation,
acceptance,
intention to use,
choice of
alternatives
Thüring & Mahlke, 2007
Usability scales
(SUS, QUIS, SUMI, WAMMI, etc)
traditional usability questionnaires focus on
pragmatic aspects only… this is not enough!
1
UX is more than usability

7. We need to assess both pragmatic and hedonic
perceived qualities of a system
AttrakDiff scale
(Hassenzahl et al., 2003)
User Experience
Questionnaire
(Laugwitz et al., 2008)
meCUE scale
(Minge & Riedel, 2013)
1

8. UX is highly contextual
Context
User System
Social context
Technical context
Temporal context
Task context
Physical context
Time
2

9. user testing in a
controlled
environment
expert evaluation
traditional evaluation methods assess UX in an
artificial environment
Context
User System
Time
2
UX is highly contextual

10. We need to evaluate UX in a natural or realistic setting
Field testing and
observation
"In-sitro" user testing
(Kjeldskov et al., 2004)
Experience sampling
(Csikszentmihalyi , 1990)
2

11. Before usage
Anticipated UX
Imagining
experience
During usage
Momentary UX
Experiencing
After usage
Episodic UX
Reflecting on an
experience
Over time
Cumulative UX
Recollecting multiple
periods of use
When:
What:
How:
UX White Paper, 2010
There are several time spans of UX
UX starts before the interaction and doesn’t end immediately
after the interaction
UX is highly dynamic
3

12. traditional evaluation methods focus on
momentary UX… this is not enough!
UX White Paper, 2010
user testing psychophysiological
measurements
Before usage
Anticipated UX
Imagining
experience
During usage
Momentary UX
Experiencing
After usage
Episodic UX
Reflecting on an
experience
Over time
Cumulative UX
Recollecting multiple
periods of use
When:
What:
How:
UX is highly dynamic
3

13. The memory of an experience matters more than
the experience itself
Episodic UX is a reconstruction,
a remembered experience biased by cognitive
processes
The momentary experience is not as important
as the way it is remembered.
It’s the memory of an experience that
influences user’s behavior and the way he
talks or recommends the product to someone
4

14. We need to assess UX across time and to focus
on the memory of experiences
UX Curve
(Kujala et al., 2011)
Diary methods
Retrospective UX assessment
Analytic scale
(Karapanos et al., 2010)
Longitudinal study
3 4

15. Established evaluation methods only explore a
limited part of UX
single user testing
sessions
psychophysiological
measurements
expert evaluationusability scales
As we gain a deeper understanding of UX, we have to adapt
the methods we use to design or evaluate it.

16. You already embraced UX…
will you now consider changing something
in your approach?
-Prof. Marc Hassenzahl (2013)
« …many practitioners and academics of HCI happily embrace experience –
however, without changing much in their approach. »

17. References
Csikszentmihalyi, M. (1990). Flow. The psychology of optimal experience, Harper and Row.
Hassenzahl, M., Burmester, M., Koller, F. (2003) AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität.
In: Ziegler, J., Szwillus, G. (eds.) Mensch & Computer 2003. Interaktion in Bewegung, pp. 187–196. B.G. Teubner, Stuttgart. / AttrakDiff. Internet
Resource http://www.attrakdiff.de.
Karapanos, E., Zimmerman, J., Forlizzi, J., & Martens, J. (2010). Measuring the Dynamics of Remembered Experience Over Time. Interacting with
Computers, 22 (5), doi: 10.1016/j.intcom.2010.04.003
Karapanos, E., Martens, J.-B., Hassenzahl, M. (2010) On the Retrospective Assessment of Users’ Experiences Over Time: Memory or Actuality?.
CHI’10 extended abstracts on Human factors in computing systems. Atlanta, ACM Press.
Kjeldskov, J., & Skov, M. B. (2007). Studying Usability In Sitro: Simulating Real World Phenomena in Controlled Environments, International Journal
of Human-Computer Interaction, 22 (1-2).
Kujala,S., Roto,V., Väänänen-Vainio-Mattila,K., Karapanos,E., & Sinnelä, A. (2011). UX Curve: A method for evaluating long-term user experience.
Interacting with Computers, 23, 473-483.
Lallemand, C., Gronier, G., & Koenig, V. (2015). User experience: A concept without consensus? Exploring practitioners’ perspectives through an
international survey. Computers in Human Behavior, 43, 35-48.
Lallemand, C. (2015). Towards Consolidated Methods for the Design and Evaluation of User Experience. (Doctoral dissertation). University of
Luxembourg. https://publications.uni.lu/handle/10993/21463
Laugwitz, B, Held, T., & Schrepp, M. (2008). Construction and evaluation of a user expe- rience questionnaire. In A. Holzinger (Ed.) USAB 2008,
LNCS 5298. Berlin: Springer-Verlag.
Minge, M., & Riedel, L. (2013). meCUE – Ein modularer Fragebogen zur Erfassung des Nutzungserlebens. Presented at Mensch & Computer 2013.
Roto, V., Law, E., Vermeeren, A., & Hoonhout, J. (2011). User Experience White Paper: Bringing clarity to the concept of user experience. Result
from Dagstuhl Seminar on Demarcating User Experience, Finland.
Thüring, M., & Mahlke, S. (2007). Usability, aesthetics and emotions in human-technology interaction. International Journal of Psychology, 42(4),
253-264.