Although usability and user experience may seem synonymous, they are separate and much different concepts. While usability is well defined in standards, UX has no agreed upon definition because it relates to a more nebulous attribute-user satisfaction. Both are, however, key ingredients for successful system deployment. Because they don’t know how to measure and evaluate UX, many teams ignore this important attribute until the end of development. Philip Lew discusses how to model both usability and UX by breaking each attribute down into measurable characteristics-learnability, user effectiveness, user efficiency, content quality, user errors, and more. Phil shows you how to derive measurements and metrics that your development and team can employ to benchmark, analyze, and improve both usability and UX. Beyond the measurements, Phil discusses case studies in which measurements have driven significant usability and user experience improvements.
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
The Why and How of Usability and User Experience (UX) Testing
1.
BT12
Concurrent Session
11/8/2012 3:45 PM
"The Why and How of Usability and
User Experience (UX) Testing"
Presented by:
Philip Lew
XBOSoft
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888‐268‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
2. Philip Lew
XBOSoft, Inc.
After working for twenty years in various management and technical positions in product
development and management, CEOPhilip Lew now leads the direction and strategy of
XBOSoft (xbosoft.com). He has helped hundreds of organizations assess the quality of their
software, examine software quality processes, and set forth measurement plans so they can
consistently improve software quality using systematic methods. Phil has authored articles in
IEEE and ACM journal publications and trade journals; presented at several conferences on
web application usability, user experience, and quality evaluation; and is conducting postdoctorate research focused on user experience measurement frameworks and software
learnability.
3. The Why and How of Usability and
User Experience (UX) Testing
Philip Lew
Agenda
• What is usability and UX and why
important?
i
t t?
• Usability and UX – What are they
• How to set up models and measurements
• Case study and examples in testing and
evaluating applications
1
4. What is Usability-UX and Why
is it Important
The Year of Mobile?
• Last year was the year of mobile in terms
of th growth i b th mobile usage and th
f the
th in both
bil
d the
availability of mobile sites and mobileapps.
• Time to redesign your mobile site,
because your existing version is probably
far below users growing expectations for
users'
user experience quality.
• What do you think users expect?
4
2
5. History Repeats Itself
• In the beginning, the Web was experimental —it was
acceptable to have a somewhat shaky, experimental
website.
• Many sites were crippled by misguided design advice
• Many didn't know any better (because they didn't do
usability studies). Now, people simply expect websites
to work.
• For mobile, last year, it might have been cool simply to
mobile
year
have an app.
• Now, that app better be good. Expectations have gone
up.
5
Web and Mobile User
Expectations
• Business models have
changed
– Instead of paying
upfront and ‘owning’
the software
– Pay as you go, pay by
p
subscription
• Cloud and mobile
converge
• Behavior and
expectations have
changed
3
7. Usability - Design Perspective
•
•
•
•
•
•
Understandability
Learn-ability
Operability
Attractiveness
Navigation
Responsiveness-performance
What else can you think
of?
9
Usability-Effect
Degree t which specified
D
to hi h
ifi d
users can achieve specified
goals with effectiveness,
efficiency and satisfaction in a
specified context of use.
Source: ISO 25010
10
5
8. Usability-Effect
“Context” and “Specified”
specified users
• User role
• Objective
specified goals
• Task
• Environment
specified context of use
• Domain
What else can you think
•…
of?
11
Usability-Effect
User Experience
• Satisfaction
– The degree to which users are
g
satisfied in a specified context of
use. Satisfaction is further
subdivided into subcharacteristics:
• Likability (cognitive satisfaction)
• Pleasure (emotional satisfaction)
• Comfort (physical satisfaction)
• Trust
• Including many other factors experienced
over time and other channels
Source: ISO 25010
12
6
9. Task Ease
(effectiveness and efficiency)
• Buttons are a key tool in the user experience designer’s
box of tricks.
• Prioritize tasks - Some tasks more important than others.
• Understand the objectives of the application
• Understand which tasks are really important.
• Majority of your app’s value is provided by a small
number of tasks.
• Remove any friction that inhibits the user’s progress.
• Choose your words carefully to make it clear what the
buttons or functions do
What could
add ‘friction’ ?
13
Design and Test For the Task
• Highly specific tasks.
– You are in an electronics store looking at the Minolta
GS234 camera. It costs $120 25 in the store Check
camera
$120.25
store.
costco.com to see if you can get a better price online.
• Directed, but less specific.
– Find a moisturizer with SPF 45 or above that is
suitable for your skin." (Use Walmart app.)
• Open-ended, but restricted to a predetermined site-app.
– "See if you can find any interesting pictures related to
today's news." (While using the USA Today app.)
• Web-wide tasks that let users go anywhere.
– "Find out which is the tallest building in the world."
14
7
10. Usability
Basic Design Issues and
Concepts
Design
Testing and Evaluation
15
Design Concepts Test and
Evaluation Criteria
•
•
•
•
Context
Positioning
Color usage
Size usage
Size usage
8
11. User Context is King
• Mis-Targeted user
background can
b k
d
cause loss of
effectiveness
• Know your users.
Who are your users?
What are their special
needs/characteristics?
17
Positioning
• Effectiveness of
application buttons
pp
is affected by
position.
• Emphasis is
gained by
– removing clutter
– placing the button
where the user’s
eyes will mostly
likely be.
18
9
12. Using Color
• Color creates emphasis.
• Dependent on the importance of the
application’s function.
• Make easy to for user to see the most
important stuff.
• Color can affect usefulness.
usefulness
19
LinkedIn - Color
Create an
account!!!
• The ‘View Full Profile’ button
– LinkedIn wants you to click that.
– You’ll be prompted to create an account!
• Button has a unique color that isn’t shared with any other part of
the design.
• Page has tons of information, this button still stands out.
20
10
13. Don’t Overuse a Color
21
Size Matters
• Time taken to point at an object is directly influenced
by the size of that object.
• Big is beautiful depending on function-what you want
the user to do.
• Size can dictate the button’s importance over
everything else on the page.
• Take the Firefox page for example, Mozilla don’t mess
around with subtlety here.
y
• Good design communicates priority.
• With one massive button on the page, that priority is
obvious.
22
11
14. Using Size AND Color
23
Multivariate Testing
• Small changes can make a big
difference.
difference
• Laura Ashley arrived at this design
following multi-variate testing of 5
different options
• Differences in the testing included:
g
– Link colors and locations
• The "Go to checkout" button was
dark gray instead of green
• Achieved 11% increase in checkouts
24
12
15. Let’s Go Mobile
Usability and UX For Mobile
25
Usability - Desktop to Mobile
• Just loading websites on the phone is not
enough
• What matters for users
– Time to load the website
– User interface
– Accessibility of various functions available
• Usability design needs to change,
otherwise usability effect and UX will suffer
26
13
16. Different Platforms Need
Different Design
• Horizontal swiping now used to "flip" through deck-ofcards or carousel features.
– Include a visible cue when people can swipe
• Avoid swipe ambiguity: don't use the same swipe
gesture to mean different things on different areas of the
same screen.
– Use the same meanings for mobile phones
and tablets
– Mobile-device users typically expect to horizontally
swipe while desktop websites avoid horizontal
scrolling
• Mobile sites should perform better than full sites when
used on a mobile device.
Nov 2012
Better Software East 2012
27
Let’s Step Through a Few
Examples
Usability and UX For Mobile
28
14
17. Laura Ashley
Mobile Site
• Same as web version
• G
Green sticks out
ti k
t
29
Laura Ashley
• What is wrong here?
30
15
18. Laura Ashley – 9/2012
31
LA MS
• Consistent green
• Ob i
Obviously want you t
l
t
to
click and add to the
bag
32
16
19. LA
Shopping bag
• Clear green signals…
BUY
33
LA MobileApp
• Main Screen
• D
Decent size b tt
t i buttons
• No priority or
importance weighting
34
17
20. LA Mobile App
• Big buttons
• G d for big fi
Good f bi fingers
• Semi-meaningful
icons on the bottom
– Note contextual
location
35
Buy or Share?
• No placement of
importance or priority
36
18
21. Select Size
• Good navigation
• G d size b tt
Good i buttons
• No priority
37
In my Basket
• What should I do
now?
• Buy or share?
38
19
22. Shopping
Basket
• Empty or Checkout?
39
Bloomberg
• Simple Information
App
• Simple colors
• Meaningful Icons
• Big enough to see
40
20
23. Fidelity
• Easy to use scroll
– Thumb Friendly
• Simple colors
• Very limited task
based functionality
41
Fidelity
• Size and Importance
• P tt easy to see
Pretty
t
what they want you to
do
• Consistent icons on
bottom
– Si l
Simple
– Not everything
42
21
24. Yelp
• Easy on the eyes
• S i
Semi-meaningful
i f l
icons
43
Yelp
• Consistent
• Fi
Finger friendly
f i dl
• Consistent location of
icons on bottom
Similar to Fidelity
What does this mean?
44
22
25. Making Assumptions
• Web sites automatically
sense the platform and
switch t a ‘ bil site’
it h to ‘mobile it ’
(for instance, United
Airlines, Google, and
others)
– Provide mobile users with
more efficient web
e pe e ce
experience.
• Has the potential of
making too many
assumptions regarding
the users’ expectations.
45
Web-Mobile Integration
• Overall integration of web-apps needs to be thought out
carefully with mobile scenarios in mind.
• Friend wanted to show me a product at
www.costco.com.
• He sent me a link from his iPhone. I was at my desktop,
and opened the link and got the mobile version.
Nov 2012
Better Software East 2012
46
23
26. Give a Choice to Avoid Making
the Wrong Assumption
• Managing user expectations
of how the application
should behave needs to be
thought out carefully.
• Context of the user and their
expectations is key
• LinkedIn gives you a choice
when accessing their site;
an optimized mobile
experience via an mobileapp, or just their ordinary
website.
Nov 2012
Better Software East 2012
47
Determine the Needs at Hand
• What need is the mobile application attempting to
mobilize?
• How can the workflow on a mobile device be
designed more efficiently to accomplish the task?
• How can the specific features and characteristics
of a mobile device improve and complement the
experience in contrast to the normal web-based
p
application or other mobile platforms?
Nov 2012
Better Software East 2012
48
24
27. Determine the Needs at Hand
• Fidelity implements top
scenarios their users want
and optimizing efficiency for
p
g
y
those user scenarios ONLY.
• Filling an order was a user
scenario optimized just for
the mobile platform.
• Note that it only has 4 data
items to fill in with 2 being
in,
scroll buttons, while the
biggest button indicates
precisely what the purpose
of the task is.
Nov 2012
Better Software East 2012
49
Mobile Usability Design
Best Practices Summary
• Quick - Can you read the text without zooming or
scrolling?
• Simple Navigation-Task Oriented
• Thumb Friendly Design for the small screen.
Don’t make users struggle to click tiny areas that are
• Visibility
much smaller than their fingers.
• Easy to Convert/Complete the task
• Contextual
• Seamless with main webapp
• When you have a smaller screen, you must limit the
number of features to those that matter the most for the
mobile use case.
50
25
28. What Is Designed, We Test
Understanding Design Leads to
Better U d t di
B tt Understanding, E l ti
Evaluation,
and Testing
What does
want to
achieve in the Usability UX
Standpoint?
52
26
29. Introduction to
• iMedTablet - patient care coordination for
all types of long term care providers
long-term
providers.
• Allows the user to securely access and
document patient care information in realtime utilizing Cloud technology, GPS
technology and intra-facility
gy
y
communications.
• Designed to seamlessly integrate with
many pharmacy and long-term care
software systems.
53
- Login Screen
54
27
30. -Med Admin Record
55
• Users are _________.
• Heavily context oriented based on user
expectations and habits.
– Best design is not equal to the best
usability.
28
32. Design-Test and Evaluate
Design
Test and
Evaluate
Test and
Evaluate
Release
59
Let’s Define Usability
Product (Design) Viewpoint
Usability
Characteristic 1
Navigation
Subcharacteristic 1
Control Stability
Position
Attribute 1
•
•
•
Characteristic 2
Simpleness
Subcharacteristic 2
Button Visability
# Buttons
Attribute 2
Characteristic n
Subcharacteristic n
Color Usage
Attributes expressed hierarchically
Any number of sub-levels is OK
Achieving the sub-attributes=achieving the high level attribute -measurable
30
33. Defining Usability from an
Effect-Real usage Point of View
Quality y
y
Usability
Effectiveness
Efficiency
Accuracy
Errors
Satisfaction
Completeness
Attribute 2
Attribute 3
Characteristic n
Subcharacteristic n
Attribute 1
Potential Attributes Measurements
for ‘Effect’ Usability
• Effectiveness
– Completion rates
– Error rate
– Help usage
• Efficiency
– Task time/Speed
• Learnability
– Learning rate
– Task time deviation
– Change in task times over time period
31
34. Measurable Attributes
•
•
•
•
•
•
•
•
Attribute name
Description and purpose Need a model
before you can
How to measure
start measuring…
What is measured
Measurement/Calculation
Range (min, max)
Objective
Current
Usability Measurement Methods
Focus
groups
Walk
Throughs
Satisfaction
S if i
Surveys
Labs
Heuristic
Evaluation
Logging
Let’s get started
32
35. Usability Measurement
Attribute
Scale
How
Measure or Objective Current
Calculation
Help Access
Percent of Server Log
Users
Users
files
Accessing
Help
Task
Simplicity
%
30%
40%
Keystrokes Heuristic
%
to
measurement
find/use a
feature/fu
nction/inf
ormation
90% < 3
50%
Task Times
Time
Server log
finish‐time files
start
%
90% < 3
50%
Accuracy
Number
errors
Integer or
%
<5
10
Server Log
files
65
Sample Heuristic Evaluation
Usability
1. Understandability
1.1 Label/icon recognizable
1.2 Function‐task recognizable
2. Navigation
2.1 Back‐Forth
2.2 Full‐mobile
3. Swiping
3.1 Consistency
3.2 User Awareness
y
4 Learnability
4.1 Error prevention
4.2 Error detection
Total
Measure
Elementary
2
4
50%
100%
3
2
75%
50%
3
2
75%
50%
1
1
Partial
75%
33%
33%
63%
63%
33%
58%
Through defining the model, then benchmarking with heuristic
evaluation, we were able to determine weak areas and make
recommendations for significant improvements
Nov 2012
66
33
36. Usability Logging
Measurement and Data Collection
• Identify users by using session ID
to identify a unique user.
• Iteratively insert code into the
application
• Collect data
• Analyze the data for each attribute
in different dimensions and
aggregations
• Determine the need for further
calculations and what attributes to
measure further
• Revise the data we are collecting,
adding or decreasing granularity
67
Satisfaction Surveys
The process:
1. Calculate the usability score (satisfaction) of
1 Calculate the usability score (satisfaction) of
version X
2. Do the survey
3. Change to version X.1 – make changes to the
software directly correlated to the usability
factors to either increase or decrease the
factors to either increase or decrease the
usability score
4. Do the survey again
5. See if differences made change impact the
survey results
68
34
37. Putting It All Together
Usability Design
User
Satisfaction
Usability in
Use
User
Experience
69
Getting Started with
Measurement
• Produce an action plan
– What usability attributes are important to your
organization?
• Develop a model
– What data can you collect/Which technique
can you use
• Maybe some elements of the model drop out-can’t
be measured that easily
– Start collecting and developing benchmarks
70
35
38. Conclusion-Usability and UX
• Paramount for today’s applications with users’ short
attention spans.
• Designs should be as simple as possible for the user
user.
• UX is heavily influenced by expectations.
• Make navigation easy on a small screen, thumb friendly,
and as intuitive as possible.
• If the user cannot figure it out in 30 seconds, they are
gone.
• Mobile-app versus your web-app - differentiate and
discern access from each platform.
• Determine the features you really need and optimize
screens for certain workflows rather than trying to do it all.
71
Conclusion-Measurement
• Defining UX is different for each organization
y
g
• Need a model for your organization
– What is most important to you depends on
your users and their behavior and expectations
• The model is the foundation of what to measure,
evaluate and improve
72
36
