Presents an introduction to some basic metrics for usability and some current trends in UX evaluation methods. Includes some indicative examples from UX evaluation studies conducted by the author
5. The Ancestor: Human Computer Interaction
Human – Computer Interaction, is “a discipline concerned
with the design, evaluation and implementation of interactive
computing systems for human use and the study of major
phenomena surrounding them’’.
6. User friendly? Usable? …
Once upon a time we were talking about “user friendly”
systems…
… so general and vague
Then usability emerged and was originally derived from the
term ‘‘user friendly’’
… but usability is still a general and vague term
Many definitions and different approaches to how usability
should be measured
7. What is usability anyway?
The ISO definition (9241-11)
Usability is defined as: ‘‘the extent to which a product can be
used by specified users to achieve specified goals with
effectiveness; the extent to which the intended goals of use are
achieved,
efficiency; the resources that have to be expended to achieve the
intended goals and
satisfaction; the extent to which the user finds the use of the
product acceptable,
in a specified context of use’’.
8. Examples of usability measures (1)
Measures of effectiveness:
Task completion (number of correct tasks, number of tasks
where users failed to finish within a set time, number of
tasks where users gave up)
Accuracy measures (quantify the number of errors users
make during the process of completing tasks)
Recall measures (how much information users can recall
after the use of an interface)
Quality of outcome (attempts to measure the outcome of
the tasks, measures of understanding e.g. tests of what has
been learned from an e-learning system)
9. Examples of usability measures (2)
Measures of efficiency
Time (refers to measures of how long users take to
complete tasks)
Input rate (e.g. words per minute)
Mental effort (mental resources spend on interaction, e.g.
heart rate variability, subjective time estimation)
Usage patterns (number of times a certain action has been
performed, how much information users access when
solving tasks, deviation from the optimal solution etc.)
10. Examples of usability measures (3)
Measures of satisfaction
Preference measures capture which interface users prefer
using
Specific attitudes towards the interface (liking, fun,
annoyance etc.), toward the content of interface (quality of
information, organization of information etc.)
Perception of outcomes (users’ assessment of their
performance, users’ perception of learning, users’ confidence
in the solution to tasks)
11. From Usability to User Experience (1)
UX - A relatively new field - Lack of unique definition…
But let’s take for example the ISO definition on UX:
“A person's perceptions and responses that result from the use
or anticipated use of a product, system or service”
12. From Usability to User Experience (2)
Usability
• Focused on
optimizing task
efficiency
• Removing
problems in the
user interface
• Studying
interactive
products mainly
used for work
UX
• Dynamic
context-
dependent and
subjective
• Focuses also on
hedonic
qualities of use
(identification,
stimulation,
attractiveness
etc.)
14. As we shift to User Experience…
Challenge#1: Need for new approaches and methods
Existing methods are somewhat unable -it is claimed- to
capture experience
Challenge#2:The choice of UX evaluation method
Qualitative or quantitative methods?
Only few studies combine qualitative and quantitative methods
15. What Dimensions of Experience Are
Studied?
Usability
• Effectiveness
• Efficiency
• Satisfaction
UX
• Emotions and affect
• Enjoyment
• Aesthetics
• And many more…
16. What Dimensions of Experience Are
Studied?
Source: Bargas- Avila and Hornbaek, 2012
17. How do we measure UX?
What are the methods applied in UX
Qualitative (50 %), quantitative (33%), combination (17%)
How UX data are collected?
Questionnaires are still the dominant UX evaluation method
(53%).
Source: Bargas- Avila and Hornbaek, 2012
19. When to use which UX evaluation methods
Decision
Qualitative
vs.
Quantitative
Attitudinal
vs.
Behavioral
Context of
system/
product use
Source: http://www.nngroup.com/articles/which-ux-research-methods/
20. The Attitudinal vs. Behavioral Dimension
to understand,
measure, or
inform change of
people's stated
beliefs
e.g. self-reported
methods such as
surveys,
interviews, focus
groups, etc.
Attitudinal to understand
"what people do"
with minimal
interference from
the method itself
e.g.A/B testing,
observation and
video recording,
first click testing,
eyetracking, etc.
Behavioral
Source: http://www.nngroup.com/articles/which-ux-research-methods/
21. The Qualitative vs. Quantitative Dimension
- the data is usually
being gathered directly
- they are much
better suited for
answering question
about why or how
to fix a problem
Qualitative - the data is
gathered indirectly,
through an
instrument, such as a
survey or a web
server log etc.
- they do a much
better job
answering how
many and how
much type of
questions.
Quantitative
Source: http://www.nngroup.com/articles/which-ux-research-methods/
22.
23. The Context of Use Dimension
This dimension can be described by the following options:
Natural or near-natural use of the system/product
Scripted use of the system/product
Not using the system/product during the study
A hybrid of the above
Source: http://www.nngroup.com/articles/which-ux-research-methods/
24.
25. What about the use of UX evaluation
methods across the development phases ?
26. UX evaluation methods across development
phases
Product Development Phase
Strategize Optimize Assess
Goal: Inspire, explore and
choose new
directions and
opportunities
Inform and optimize
designs in order to
reduce risk and
improve usability
Measure product
performance against
itself or its
competition
Approach: Qualitative and
Quantitative
Mainly Qualitative
(formative)
Mainly Quantitative
(summative)
Typical
methods:
Ethnographic field
studies, focus groups,
diary studies,
surveys, data mining
or analytics etc.
e.g. cardsorting, field
studies, observation,
paper prototype and
usability studies, etc.
e.g. Usability
benchmarking,
surveys, A/B testing,
etc.
Source: http://www.nngroup.com/articles/which-ux-research-methods/
28. Walls of Nicosia
The context:
"TheWalls of Nicosia” is an interactive application that runs on a multi-
touch table at the Leventeio Museum, Nicosia
Students from elementary schools as museum visitors
Goal:To conduct a comparative
summative evaluation
Methods used:
Multiple choice knowledge tests
UX questionnaires (FunToolkit)
Observation
Indicative results:
Mainly quantitative analysis (pre and post test analysis, t-tests)
29. UX questionnaire for children: the
FunToolkit (Read, 2007)
The Again-Again table
The Smileyometer
Source: http://hrast.pef.uni-lj.si/docs/research/Serious%20games/evaluation_paper_Janet.pdf
30. Long term UX evaluation of League of
Legends (1)
The context:
The well known MMOG “League of Legends”
Players were students at TEI of Kavala
Goal: to identify how UX of players
changed over time
Methods used:
An adaptation of UX Curve
Self reported method by drawing curves that depict changes in
UX over time
31. Long term UX evaluation of League of
Legends (2)
Indicative results:
Reason
Categories Positive Negative
General UX 43 14
Ease of use 41 26
Immersion 43 10
Social
interaction 38 12
Engagement 32 10
Total 197 72
General UX Curve for each player
Number of reasons for UX dimensions
32. Heuristic Evaluation of LMS
The context:
A well known cloud-based Learning Management System
platform
Goal: to identify usability problems and possible areas for
UX improvement
Methods used:
Heuristic evaluation
Heuristic evaluation protocol adapted by Mehlenbacher et al.
(2005)
Indicative results:
An evaluation report that contains the identified usability and
UX problems, their severity, and relevant redesign suggestions
33. Typical structure of the report of heuristic
evaluation
A numeric identifier of the problem
A short description of the problem (and likely difficulties
for the user)
Specific context (defined as location of the problem in the
interface): we identified this generically in terms of
affecting the:
Structure of the system, including layout, navigation, and basic help
functions such as print, help, etc.
Content, including actual content for study, exploration, examples,
exercises, and so on
Global issues, i.e., affecting both structure and learning content
The heuristic(s) used
The severity rate: severe, moderate, and minor
34. First Click testing with mockups
The context:
A project for the design of new MOOC interfaces
Goal: to create, inform and optimize MOOC interface
design
Methods used:
Interviews and first click testing with users
37. Some UX resources:
World’s premier magazine for both academics and industry:
http://interactions.acm.org/
A comprehensive list of UX methods:
http://www.allaboutux.org/
An indicative list of UX tools:
http://www.smashingmagazine.com/2011/10/20/comprehensive
-review-usability-user-experience-testing-tools/
Other sources on the web:
http://uxmag.com/
http://www.usability.gov/
http://www.uxmatters.com/index.php
38. Key takeaways:
You can’t afford NOT to evaluate UX
Always use a combination of methods
Don’t forget Usability
Do it properly…