The document discusses evaluating and improving software usability. It begins with an introduction that outlines the importance of usability, what usability is, and what user experience (UX) is. The agenda then lists topics that will be covered, including usability modeling and measurements, case studies on measuring and improving usability, and a summary. Key points are made about how usability relates to quality and user satisfaction. Frameworks for understanding usability and UX are presented, distinguishing between pragmatic and hedonic aspects of the user experience. Potential attributes for measuring usability in actual use are also listed.

1. Evaluating and
Improving
Software Usability
902 : Thursday, 9:30am - 10:45am
Philip Lew
www.xbosoft.com

2. Understand,
Evaluate
and
Improve
2

3. Agenda
• Introduc7on
– Importance
of
usability
– What
is
usability?
– What
is
UX?
• Usability
Modeling
and
Measurements
• Case
studies
on
measuring
and
improving
• Summary

4. What
is
Quality?
Pertaining
to
soEware
J
Requirements…
4

5. Importance
of
Usability

6. Why
is
Usability
Important?
• Usability
is
important
especially
for
web
applica7ons
– And
now
especially
mobile
applica7ons
• Saas,
mobile
are
in
alignment
with
and
soon
becoming
equivalent
to
cloud
• Most
prevalent
implementa7on
method
in
the
next
5
years
• Without
good
usability:
– Users
will
leave
the
applica7ons
– For
mobile,
if
they
can’t
learn
in
30
seconds,
they
won’t
come
back

7. Web
and
Mobile
have
Changed
the
Terrain
• Business
models
have
changed
– Instead
of
paying
upfront
and
‘owning’
the
soEware
– Pay
as
you
go,
pay
by
subscrip7on
• Behavior
and
expecta7ons
have
changed

8. What
is
Usability
In
rela7on
to
quality
In
rela7on
to
user
experience

9. When
you
think
Usability…
• Naviga7on
• Efficiency
• Responsiveness-­‐performance
• Learnability
• AZrac7veness
Anything
else
come
to
• Understandability
mind?

10. Current
Research
–
Usability
User
Quality
Experience
Satisfaction Usability
ISO 9241-11
ISO
9126-1
Usability
Design ISO
Experts Product 25010
Owners

11. Usability
-­‐
Key
Characteris7c
of
Product
Quality
Source: ISO 25010

12. Quality
in
Use
ISO 25010

13. Notes
on
Sa7sfac7on
and
Usability
don’t have
what I want I’m unsatisfied • Sa7sfac7on
is
a
subjec7ve
feeling
Highly usable dependent
on
many
software things
other
than
usability:
My password
doesn’t work – A
user
can
be
highly
sa7sfied
but
the
applica7on
with
low
Nice weather I’m usability.
today satisfied! – An
applica7on
can
be
! highly
usable
(high
low usability
software usability)
but
the
user
is
not
sa7sfied!
Finished
my work today

14. Usability-­‐Major
Component
of
Quality
Quality
and
Usability
in
the
so1ware
development
lifecycle
Usability
Usability
• Can
be
measured
from
the
design
point
of
view
or
of
the
product
• Can
be
measured
‘in-­‐use’
with
real
users

15. Evolu7on
of
SoEware
quality
ISO 9000 ISO 9126 ISO 9241 ISO 25010
Software Software Software
Type of Process
Quality Quality Quality
quality quality
(internal) (external) (in use)
What is
measured
Software
processes
Code Product ?
CMMI
How
measured?
assessment
model
white box
testing
black box
testing ?
CMMI
Who Assessment Programmer Tester End User
measures?
Company

16. What
is
Usability-­‐Effect
of
the
SoEware
Product
Degree to which specified
users can achieve specified
goals with effectiveness,
efficiency and satisfaction in a
specified context of use.
Source: ISO 25010

17. Usability
in
Actual
Usage
• User
role
specified
users
• Objec7ve
specified
goals
• Task
• Environment
specified
context
of
use
• Domain
• …
What
else
can
you
think
of?

18. Usability and
User
Experience
UX, Usability and Quality in Use
• Difficult to understand their relationships
• Lack of consensus in meaning in literature
• Not totally clear how related to quality
• Standard definition for user experience is still not available

19. Some
UX
Defini7ons
UX DESCRIPTION
D1 a continuous process of user engagement with the product [10]
D2 entire set of affects that results in user-product interaction [11]
D3 the evolution of usability [12]
D4 elaboration of the satisfaction component of usability [13]
D5 a categorization of “do-goals” (pragmatics) and “be-goals”(hedonics) [3], [14]
D6 infinite small experiences relating to people, products and contexts [6]
D7 consequence of user’s and product’s characteristics when interacted in a
specific environment [1]
D8 degree to which specified users can achieve actual usability, safety, and
satisfaction in use in a specified context of use [15]
D9 A person's perceptions and responses that result from the use or anticipated use
of a product, system or service [16]
D10 the degree of positive or negative emotions that can be experienced by a
specific user in a specific context during and after product use and that
motivates for further usage [17]

20. User Experience (UX)
• Pragmatic (do-goals) : refers to the product's
perceived ability to support the task achievement
and focuses on the product’s utility and usability in
completing tasks, the ‘do-goals’ of the user.
• Hedonic (be-goals) : product's ability to support
the user’s achievement of ‘be-goals’, such as
being satisfied, pleasure.
• Real goal of the end user is fulfilling be-goals such
as being autonomous, competent, related to
others, stimulated, etc.”
• Be-goals are the driver of UX
Hassenzahl

21. Rela7ng
Usability
and
User
Experience
Quality
508
User
Interface
Func7onality
Learnability
characteris7c
compliancy
Usability
Product
Text/Sound
Appropriate
Easy
to
use
Easy
to
learn
aZributes
aZributes
Func7ons
UX
Pragma7c
Effec7ve
and
Efficient
Do
Goals
Usability
Effec7veness
in
use
Efficiency
in
use
Learnability
in
use
Measurement
UX
Hedonic
Sa7sfac7on
(pleasure,
sense
of
community…)
Be
Goals
Sa7sfac7on
Sense
of
Pleasure
Comfort
Trust
Measurement
Community
Context
USER
EXPERIENCE
21

22. Now
that
we
understand
it
ALL…
We
Need
to
Define
Usability
For
Our
Own
Purposes
22

23. Let’s
Define
Usability/UX
From
the
Product
Viewpoint
Usability
Characteris7c
1
Naviga7on
Understandable
Characteris7c
2 Characteris7c
n
Subcharacteris7c
1
Control
Stability
Subcharacteris7c
2
Previous-­‐Next
Subcharacteris7c
n
AZribute
1
Home
Loca7on
AZribute
2
Save
Loca7on
Ease
of
finding
AZribute
n

24. Defining
Usability
from
an
Effect-­‐Real
usage
Point
of
View
Usability
Quality
Effec7veness
Efficiency
Sa7sfac7on
Characteris7c
n
Accuracy
Completeness
Subcharacteris7c
n
Errors
AZribute
2
AZribute
3
AZribute
1

25. Some
Poten7al
AZributes
Measurements
for
‘in
use’
Usability
• Effec9veness
– Comple7on
rates
Can
you
think
– Error
rate
of
any
others,
– Help
usage
par9cular
to
• Efficiency
your
applica9on?
– Task
7me
– Backtracking
• Learnability
– Learning
rate
– Task
7me
devia7on

26. Adding
in
UX
Sa9sfac9on
in
use
• Degree
to
which
users
are
sa7sfied
in
a
specified
context
of
use.
Sa7sfac7on
is
the
user’s
response
to
interac7on
with
the
product,
including
amtudes
towards
use
of
the
product.
Subdivided
into
sub-­‐
characteris7cs:
– Likability
(cogni7ve
sa7sfac7on)
UX Component
– Pleasure
(emo7onal
sa7sfac7on)
– Comfort
(physical
sa7sfac7on)
– Trust
26

27. Example
of
Product
Quality
Model
with
AZributes
(Operability in ISO 25010)
External
Quality
Requirements
(for
Shopping
Cart
Entity)
1 Usability
1.1 Understandability
1.1.1 Icon/label
ease
to
be
recognized
1.1.2 Information
grouping
cohesiveness
1.2 Learnability
1.2.1 ………………………………………………………..
1.3 Ease
of
Use
1.3.1 Control
permanence
1.3.2 Helpfulness

28. Learnability
• Degree
to
which
the
soEware
product
enables
users
to
learn
its
applica7on
-­‐
7
principles
(ISO
9241-­‐10)
– Suitability
for
the
task
-­‐
should
be
suitable
for
the
user’s
task
and
skill
level
– Self-­‐descrip7veness-­‐
should
be
clear
what
the
user
should
do
next
– Controllability
-­‐
the
user
should
be
able
to
control
the
pace
and
sequence
of
the
interac7on
– Conformity
with
user
expecta7ons
-­‐
should
be
consistent
– Error
tolerant
-­‐
forgiving
– Suitability
for
individualiza7on
-­‐
should
be
able
to
be
customized
to
suit
the
user
– Suitability
for
learning
-­‐
should
support
learning
28

29. Ease
of
use
• Degree
to
which
the
soEware
product
makes
it
easy
for
users
to
operate
and
control
it.
– controllability
– error
tolerance
(by
operator)
– conformity
with
user
expecta7ons
29

30. Ease
of
Use
-­‐
Helpfulness
• Degree
to
which
the
soEware
product
provides
help
when
users
need
assistance
including
help
that
is:
– easy
to
find
– comprehensive
– effec7ve
30

31. Example:
Usability
Measurement
ACribute
Scale How Calcula9on Goal Current
Help
Percent
of
Compare
% 90% 40%
completeness Menu
items
menus
and
with
help
items
help
Ease
of
Keystrokes
Sample
50
average 3.5 12
access to
items
find/use
a
feature/
func7on/
informa7on
Consistency Number
Examine
integer 1 5
loca7ons
for
menus
and
same
buZon doc.
What types of usability measures are these?
31

32. Using
a
measurement
model
Evalua7on
Usability
composed of Indicator
Metrics
Evalua7on
Evalua7on
Characteris7c
Measurement
composed of Func7on
Evalua7on
Evalua7on
Measurements
Sub-­‐characteris7cs
32

33. Usability-­‐Review
usability
usability
in
use
UX
• Can
be
measured
from
the
design
point
of
view
or
of
the
product
• Can
be
measured
‘in-­‐use’
with
real
users

34. Measuring
Usability
Methods
and
models
Now
that
we
know
what
it
is,
and
what
to
measure,
then
HOW?
34

35. Measurable
AZributes
• Usability
characteris7c
• Descrip7on
and
purpose
Once you have a
model (what you
• How
to
measure
are going to
• What
is
measured
measure), then
• Measurement/Calcula7on
you start doing IT!
• Range
(min,
max)
• Metric
• Objec7ve
(goal)
versus
Current
(indicator)

36. UX/Usability
Measurement
Methods
Focus
groups
Ques9on
naires
Observa9on
Heuris9c
Labs
Evalua9on
Logging
Let’s
get
started

37. Measurement
methods
Expert
Evalua9on
(Heuris9c
evalua9on)
• Define
a
set
of
rules
or
criteria
with
measurements
and
evaluate
against
them
Web-­‐based
logs
• Collect
user
ac:vity
data
– Mistakes
and
errors
– How
long
it
takes
to
complete
tasks
– Comple:on
rates
Ques9onnaires
• Quan7ta7ve
subjec7ve
measurement
of
UX
characteris7cs
37

38. Heuris7c
Evalua7on
Using
a
checklist
But
a
liZle
more
complicated
than
yes/no
38

39. Heuristic Evaluation of a Pharmacy
application
Each usability attribute has a quantifying metric.
• For
example,
for
Predic9ve
textual
aCribute,
users
should
be
able
to
understand
a
buCon’s
results
prior
to
pressing
it.
• Direct
metrics
need
to
be
designed,
i.e.:
– (0)
No
support
at
all
– (1)
Par9al
– (2)
Complete
• Need
a
mapping
from
0,
1,
2
to
something
more
understandable,
i.e.
2
=
100,
and
1
=
60
• Need
an
indicator
to
interpret
the
level
of
sa9sfac9on
met
with
decision
criteria
with
acceptability
ranges
in
a
percentage
scale:
– 0-­‐40
(unsa9sfactory
–red)
means
changes
must
take
place
with
high
priority;
– 40-­‐70
(marginal
–yellow)
indicates
a
need
for
improvement
ac9ons;
– 70-­‐100
indicates
a
sa9sfactory
level
–green-­‐
for
the
analyzed
aCribute.
39

40. 2.2.2 Error Recovery Support
" In the current state, users filling a new prescription are
supported well in error recovery (2.2.2) in automatic cursor
positioning
40

41. Heuristic Evaluation – Pharmacy Software
The purpose of the evaluation
• Understand the external quality level of the learnability sub-characteristic for
filling a new prescription
Table 2 - excerpt of the whole current evaluation 41

42. Heuris7c
Evalua7on
of
Usability-­‐
Shopping
Cart
External Quality Requirements Measure EI value P/GI value
Global Quality Indicator 61.97%
1 Usability 60.88%
1.1 Understandability 83%
1.1.1 Icon/label ease to be recognized 100%
1.1.2 Information grouping cohesiveness 66%
1.2 Learnability 51.97%
1.2.1 ……………………………………………… …
1.3 Operability 49.50%
1.3.1 Control permanence 100%
1.3.2 Expected behaviour 50%
2 Content Quality 63.05%
2.1 Content Suitability 63.05%
2.1.1 Basic Information Coverage 50%
2.1.1.1 Line item information completeness 2 50%
2.1.1.2 Product description appropriateness 50%
2.1.2 Coverage of other Contextual Information 76.89%
2.1.2.1 ……………………………………………….. …
2.1.2.2 Return policy information completeness 33%

43. JIRA Heuristic Usability Evaluation

44. Usability
Logging
Collec7ng
user
ac7vity
and
behavior
Tradi7onally
used
for
other
things
like…
44

45. Usability
Logging
• Iden7fy
users
by
using
session
ID
to
iden7fy
a
unique
user.
• Itera7vely
insert
code
into
the
applica7on
• Collect
data
• Analyze
the
data
for
each
aZribute
in
different
dimensions
and
aggrega7ons
• Determine
the
need
for
further
calcula7ons
and
what
aZributes
to
measure
further
• Revise
the
data
we
are
collec7ng,
adding
or
decreasing
granularity

46. Some
easy
and
some
hard
46

47. Jira Evaluation Summary
47

48. Using
Both
Product
Perspec7ve
and
the
User
Perspec7ve
• If
we
find
some
aZribute
with
low
performance
from
the
user
perspec7ve
• Then
we
can
go
back
to
the
product
perspec7ve,
figure
out
why,
and
improve
48

49. JIRA Evaluation by Attribute
Excerpt
49

50. For
example:Login
BuZon
Visibility
(product
usability
aZribute)
50

51. Using
Ques7onnaires
The
process:
1. Use
your
model
of
what
you
want
to
measure
and
improve
2. Design
the
ques7onnaire
according
to
the
model
3. Execute
the
ques7onnaire
4. Analysis

52. One
UX
Model

53. Ques7onnaires
Used
SUMI
(So1ware
Usability
Measurement
Inventory
)
QUIS
(Ques:onnaire
for
User
Interface
Sa:sfac:on
)
PSSUQ
(Post-­‐Study
System
Usability
Ques:onnaire
)
ASQ
(A1er-­‐Scenario
Ques:onnaire
)
PUTQ
(Purdue
Usability
Tes:ng
Ques:onnaire
)
PHUE
(Prac:cal
Heuris:cs
for
Usability
Evalua:on
)
SUS
(System
Usability
Scale
)
IUI
(Isometrics
Usability
Inventory)

54. 54

55. 55

56. What
do
you
do
with
all
this?
56

57. Itera7ve
Improvement
• Measure
external
quality
–
usability
with
heuris7cs
• Measure
in-­‐use
usability
with
logging
or
ques7onnaires

58. Goal
is
to
Understand,
Evaluate
and
Improve
58

59. Next
Steps
• Produce
an
ac7on
plan
– What
usability
aZributes
are
important
to
your
organiza7on?
• Develop
a
model
– What
data
can
you
collect/Which
technique
can
you
use
• Maybe
some
elements
of
the
model
drop
out-­‐can’t
be
measured
that
easily
• Start
collec7ng
and
developing
benchmarks

60. Conclusion
• Usability
and
UX
are
both
abstract
concepts
– Cri7cal
component
of
quality
• Defining
is
different
for
each
organiza7on
• Need
a
model
for
your
organiza7on
• The
model
is
the
founda7on
of
what
to
measure
• Once
you
can
measure,
then
you
can
evaluate
and
improve

61. Resources
These are all journal publications that can
be accessed via these links… 61

62. Thanks
Ques7ons
and
Answers
Please
fill
out
an
evalua:on
form
and
drop
it
in
the
collec:on
basket
located
at
the
back
of
the
room.
Philip
Lew
@xbosoE
philip.lew@xbosoE.com
phone:
408-­‐350-­‐0508
www.xbosoE.com