Final Keynote by Regina Bernhaupt at IHC 2008, Porto Alegre, Brasil, 24. October, 2008

1. Evaluating Usability and User Experience
in Non-Traditional Environments
IHC 2008 Porto Alegre, Brasil
21 to 24 October, 2008
Regina Bernhaupt

2. Think about it ….
* How to evaluate the usability of this presentation …
* And how to evalute the User Experience during this
presentation …

3. Non-Traditional Environments
* Stressed
* Not being well
* Critical Situations
* Relax
* Fun
* Operational Situations

4. Evaluating Usability & UX in Non-Traditional Environments
Overview
* Usability and User Experience as Key Concepts
* Current Evaluation Approaches
* Three Case Studies: Adoption and Development of
Evaluation Methods
* How to evaluate in non-traditional environments

5. Usability and UX as Key Concepts

6. Definition of Usability
* Usability is concerned with making
systems easy to learn and use.
* A usable system is one which
enables users to perform their job
effectively and efficiently.
* Usability is the „effectiveness, efficiency
and satisfaction with which a specified
set of users can achieve a specified
set of tasks in a particular environment“.
ISO 9241-11

7. Usability Decomposed
* Flexibility: Dialog Initiative, Multi-threading,
Task migratability, Substitutivity, Customizability, …
* Learnability: Predictability, Synthesizability, Familiarity,
Generalizability, Consistency, ….
* Robustness: Observability, Recoverability,
Responsiveness, Task Conformance, …..

8. Fun as Factor of Usability (Caroll and Thomas, 1988)
„We realize that many people will read this article as a
joke. To this extend, we are the victims of our own
analysis: there are risks in being serious about fun.
Still though, we continue to see, without humor, the
prospect of a decade of research analysis possibly failing
to provide the leverage it could on designing systems
people will really want to use by ignoring what could be a
very potent determinant of subjective judgements of
usability – FUN“ (p. 23)

9. Is User Experience different from Usability?
For Playing the Game:
Congratulationskey
press any – you won!!

11. Defining UX
* Experiences, Feelings and Needs of the user before,
during and after interacting with the product in various
contexts.
* How do we evaluate?

12. Current Evaluation Approaches

13. Evaluation: Process Model

15. Limitations of Evaluation Methods
* Real usage and user experiences differ from lab usage
* Field studies: adoption of methods needed
* UX is difficult to evalute: lots of factors
* UX: when do we evaluate what and how?

16. Categories of Evaluation: Usability
Heuristic
Evaluation
Model-based
Evaluation
Design
Rationale Experiments
User Studies
User Testing

17. Adoption and Development of New Evaluation Methods

18. Case Study 1: Satellite Control Room
* Usage of new interaction technique
· Multimodal Interfaces
· Multi-touch including voice
· Usability Problems lead to enormous damage/loss

19. Case Study 1: Satellite Control Room
* Problems:
· You are not allowed to access the users
· You can not test in real situations
· Usability Problems lead to considerable
damage/loss
* Solution:
· Full modeling of the interaction
· Combine formal modeling state description of
the model to investigate usability problems (*)
· (*) Bernhaupt, Palanque, Winckler, Navarre (2007); Bernhaupt, Palanque (in preparation)

20. Adapting Usability Evaluation Methods
* Combination of Methods:
· Usability Studies with model-based evaluation (*)
* Adapting Methods:
· Heuristic Evaluation with usage scenarios (videos) (**)
· Cognitive Walkthrough with usage scenarios (*)
· Performing Heuristic Evaluation in the field (**)
* Bringing the „Field“ into the „Lab“
· Sports equipment to simulate running (***)
* In general: Combination and embedding of methods
* (*) Bernhaupt et al. (2007) (**) Po, Howard, Vetere, Skov (2004) (***) Kjeldskov and Stage (2003)

21. Adopting Usability Evaluation Methods
for Non-Traditional Environments
Heuristic
Evaluation
Model-based
Evaluation Experiments
Design
Rationale
User Studies
User Testing

22. User Experience Evaluation

23. Categories of Evaluation: User Experience
Probing
Questionnaire
Focus Groups
UX Sampling
Interviews
Experiments User Studies
Play Testing

24. Case Study 2: Playful Probing

25. Cultural Probing and Variations
* Probing Approach (Gaver et al., 1999)
* Several Variations: Mobile Probes, Technology Probes, …
* Insights into daily life e.g. to understand user experience
factors

26. Playful Probing
* Problem:
· Privacy
· Real usage situations: long-term usage
· Adressing all users during study
* Solution (*) :
· Make Probing „Playful“
· Develop Games to extend the methods
· Use creative/playful material/questions
* (*) Bernhaupt, Weiss, Obrist, Tscheligi (2007),

27. Playful Probing: Material
* Games, especially developed to address the research
questions, to increase participation

28. Playful Probing: Material

29. Playful Probing: Results (*)
* more involvement
* more insights
* more „material“
* helps investigating UX
(*) e.g. Bernhaupt et al. 2007
·

30. Case Study 3: Evaluating UX in Games
* Developing/Using Games to evaluate aspects of UX
* Effects of Emotion

32. Social and Ambient Factors related to UX
* Emotional Flower can be played by a group of people
* We use an ambient display to enable the players to see
the progress of others
* All flowers including the chosen nickname are displayed,
the screen is in a public area

33. Investigation aspects of UX: Manipulating Emotions
* Need for attentive user interfaces
* Emotions are a major
influence on peoples attitude
* Emotional Interfaces are
dealing with human emotions
* Goal: make UX positive

34. Results (*)
* Emotion can be influenced (but not on a longer term)
* Social influences are important for UX in games
* UX depends on usage situations, time is an important
factor
* UX design must take into account different characters of
users (e.g. more/less play-oriented users)

35. Categories of Evaluation: User Experience
Playful Probing
Questionnaire
Focus Groups
Emotional
Interviews Flowers
Experiments User Studies

36. How to Evaluate in Non-Traditional Environments
* Choose the right set of evaluation methods based on the
context the final product will be used in
* Evaluation should take place in real usage context with
real users – but this is not realistic for several domains
(security, airtraffic, cockpits, satellite control rooms,
homes, cars, public places, …)
* Modify and develop methods to allow the evaluation in
non-traditional environments or adapt methods to reflect
the real usage environment

37. Evaluation in Non-Traditional Environments
* Real usage context vs. Lab/Field Study
* Investigate the real usage problems and the user
experiences
* Add valuable information for the development of products
that are usable and that are perceived with a positive user
experience

38. Problems for Evaluation in
Non-Traditional Environments
* Users not accessible
* Non-standardized technologies (e.g. Set-top box in living
room)
* Evaluation is typcially expensive
* Sometimes not reproducable insights – especially for UX.

39. Solutions - Examples
* Adaption of Methods:
· Playful Probing
· Creative cultural probing
· User Experience Sampling
· Field usability studies
* Development of new methods and learning from „related“
disciplines:
· Evaluating User Experience in Games [Bernhaupt, R., Springer
Verlag, Mid 2009]

40. Future Work
* Identification of factors influencing the evaluation
in non-traditional environments
* Development of new or adapted methodologies and their
validation
* Framework for comparing evaluation methods with
respect to usage in non-traditional environments

41. Contacts
* Regina Bernhaupt
* IRIT ruwido
* Groupe IHCS
* 118, Route de Narbonne Köstendorferstr. 8
* 31062 Toulouse 5321 Neumarkt a. W.
* Regina.Bernhaupt@irit.fr
* Regina.Bernhaupt@ruwido.com

42. Literature
* IF you have DIFFICULTIES in finding any of the
publications I authored, please send me an e-mail to
request the article!
* Regina.Bernhaupt@sbg.ac.at

43. Literature: Playful Probing and Playful Evaluation Methods
(Bernhaupt et al.)
* PLAYFUL PROBING
* Bernhaupt, R., Weiss, A., Obrist, M., Tscheligi, M.(2007). Playful Probing: Making
Probing more Fun, INTERACT 2007, 10 -14 Sep 2007, Springer LNCS, 606-619.
* Beck, E., Obrist, M., Bernhaupt, R., Tscheligi, M., (2008). Instant Card Technique: How
and Why to apply in User-Centered Design. Conference on Participatory Design (PDC)
2008, Bloomington, Indiana, USA, to appear.
* Weiss, A., Wurhofer. D., Bernhaupt, R., Beck, E., and Tscheligi, M. (2008) “This is a
Flying Shopping Trolley”- A Case Study of Participatory Design with Children in a
Shopping Context, Conference on Participatory Design (PDC) 2008, Bloomington,
Indiana, USA, to appear.
* STUDIES USING PLAYFUL PROBING
* Bernhaupt, R., Obrist. M., Weiss, A., Beck, E. & Tscheligi, M. (2008). Trends in the Living
Room and Beyond. Computers in Entertainment, Special Issue, Vol. 6, 1, online.
* Obrist, M., Bernhaupt, R. and Tscheligi, M. (2008). Interactive TV for the Home: An
ethnographic study on users’ requirements and experiences. International Journal of
Human-Computer Interaction, Volume 24, Issue 2, 174-196.
* Bernhaupt, R., Wilfinger, D., Weiss, A., and Tscheligi, M. (2008) An Ethnographic Study
on Recommendations in the Living Room: Implications for Design of iTV Recommender
Systems, Proc. of EuroiTV 2008, Springer LNCS 5066, 92-101.

44. Literature: Evaluation Methods – Introduction/Overviews
* Bernhaupt, R., Mihalic, K. and Obrist M. (2008). Methods for Usability
Evaluation of Mobile Applications. to appear in: Handbook of
Research on User Interface Design and Evaluation for Mobile
Technology, Lumsden, J. et al. (eds.), IGI Global, p. 742 - 755.
* Bernhaupt, R, Palanque, P., Winckler, M. and Navarre, D. ( 2007)
Usability Study of Multi-Modal Interfaces using Eye-Tracking, Interact
2007, 10 – 14 Sep 2007, Springer LNCS, p 412-424.
* Bernhaupt, R., Palanque, P., Winkler, M. and Navarre, D. (2007).
Supporting Usability Evaluation of Multimodal Safety Critical
Interactive Applications using Dialogue and Interaction Models. In
book: Maturing Usability: Quality in Software, Interaction and Value.
Law, E., et al. (eds.) Springer 95-127.

45. Literature: User Experience
* Hassenzahl, M. and Tractinsky, N. (2006) User Experience – a
research agenda, Behaviour and Infomration Technology, Vol. 25, No.
2, March-April 2006, 91-97.
* Blythe, M., Hassenzahl, M. and Wright, P. C. (2004), More Funology,
Interactions, 11, 36-77.
* Carroll, J.M. and Thomas, J. C. (1988) Fun. SIGCHI Bulletin, 19. 21-24.
* Norman, D. (2004) Emotional Design: Why we love (or hate) everyday
things (New York: Basic Books).
* Blythe, M., Overbeeke, C., Monk, A.f., and Wright, P. C. (Eds), (2003)
Funology: From Usability to Enjoyment (Dordrecht: Kluwer).
* Tractinsky, N., Katz, A.S. and Ikar, D. (2000). What is beautiful is
usable. Interacting with Computers, 13, 127-145.

46. Literature: User Experience - Emotion/Affect
* Desmet, P.M.A., Overbeeke, C.j. and Tax, S.J.E.T (2001) Designing
Products with added emotional value: development and application of
an approach for research through design. The Design Journal, 4. 32-
47.
* Bernhaupt, R., Boldt, A., Mirlacher, T., Wilfinger, D., Tscheligi, M.
(2007). Emotional Flower: How Social Community Building is
Influenced by an Emotional Interface Game; to AoiR 2007, online
version at:
http://conferences.aoir.org/papers.php?first_letter=B&cf=6.
* Bernhaupt, R., Boldt, A., Mirlacher, T., Wilfinger, D., and Tscheligi, M.
2007. Using emotion in games: emotional flowers. In Proceedings of
the international Conference on Advances in Computer Entertainment
Technology (Salzburg, Austria, June 13 - 15, 2007). ACE '07, vol. 203.
ACM Press, New York, NY, 41-48.

47. Literature: Evaluating Games
* R. Bernhaupt (to appear) Evaluating User Experience in
Games: Methods and Approaches, Springer Verlag, HCI-
Series.
* Bernhaupt, R., Ijsselsteijn, W., Mueller, F., Tscheligi, M.
and Wixon, D. (2008). Evaluating User Experiences in
Games, In CHI '08 Extended Abstracts on Human Factors
in Computing Systems (Florence, Italy, April 05 - 10,
2008). CHI '08. ACM, New York, NY, 3905-3908.
* Position Papers at:
http://workshops.icts.sbg.ac.at/chi2008uxgames/

48. Literature: Evaluation in Non-Traditional Environments
* Bernhaupt, R. (accepted) Application of Usability and User Experience
Evaluation Methods for the Mass-Customization of Products and
Services. (Eds).Mourlas, C., & Germanakos, P. Mass Customization for
Personalized Communication Environments: Integrating Human
Factors, IGI Global Publication.
* Bernhaupt, R., Obrist, M., Tscheligi, M. (accepted) Methods for
involving users in the development of social interactive TV: Enhancing
usability and user experience in non-traditional environments, 15.
pages, Ed. Caesar, P., Geerts, D., and Chorianopoulous, K. Social
Interactive Television: Immersive Shared Experiences and
Perspectives. IGI Global Publication.