2. measure with users
USABILITY
EVALUATION
Usability Definition
Usability Evaluation Overview
• Formative Vs Summative
• History
Usability Inspection Methods
• Heuristic Evaluation
- Overview
- Heuristics Principles
- What is a Usability Review?
- When & How of a Usability review
- What to include in a review?
- Computing severity ratings
- Advantages of a Usability Review
- Things to Remember
• Cognitive Walkthrough
- Overview
- How To Do It
• Pluralistic Walkthrough
- Overview
Usability Testing Overview
• Techniques to Gather Data
• User Testing- How to Do It
• Quick Tips
User- Centered Design Process
Usability Evaluation Benefits
• Technology
• Business
Index
3. measure with users
USABILITY
EVALUATION
“The extent to which the product can be used by specified users to
achieve specified goals with effectiveness, efficiency, and
satisfaction in a specified context of use.”
Usability Definition
International Organization for Standardization (ISO)
Usability 5 Quality Components/Usability Measure
4. measure with users
USABILITY
EVALUATION
Usability Evaluation is aimed at finding usability problems
in the user interface/ System, providing qualitative and
quantitative data about people and their behavior or
attitudes.
Usability evolution are accomplished by identifying
representative users, representative tasks and developing
a procedure for capturing usability problems with the
interface/system. During the design/testing/development
cycle of software development.
There are two types of user evaluations are carried out
• Formative evaluations
• Summative evaluations
“A usability defect is a
product characteristic that
makes it difficult or
unpleasant for users to
accomplish tasks supported
by the product.”
“Kahn and Prail”
Overview
5. measure with users
USABILITY
EVALUATION
Conducted during design as part of an iterative process,
allowing the designers to see how their target
user/audience responds to the interface. These methods
provide qualitative data.
Heuristic Evaluation, User Interface Inspections, Thinking-aloud Testing,
Pluralistic Usability walkthroughs, and Cognitive Walkthroughs are some
methods that can be used for formative evaluation.
Conducted after completion of products, used in product
comparisons, to identify problems prior to a redesign.
Summative studies are designed to yield quantitative data
that can be tested for statistical significance.
Formative
Summative
"When the cook tastes
the soup, that’s
formative;
when the guests taste the
soup, that’s summative.“
- Robert Stakes
Formative Vs Summative
6. measure with users
USABILITY
EVALUATION
Jean Scholtz, National Institute of Standards and Technology
Timeline for usability evaluations in the last 30 years.
Users were first used as the source of usability feedback
but models have been also used for over 20 years. Expert
feedback was developed in heuristic reviews and cognitive
walkthroughs and has been used since the early 90s.
History
7. measure with users
USABILITY
EVALUATION
Heuristic Evaluation/Usability Review: is the most informal
method and involves having usability specialists/experts to
judge whether interface design established usability principles.
(the “heuristics” )
Cognitive Walkthrough: is a step-by-step method
exploring/inspecting user interface with a set of questions and
creating success/failure stories on the users’ behavior to
simulate users problem, understandability and ease of learning.
Pluralistic Walkthroughs: uses group meetings where users,
developers, and usability expert step through a scenario,
discussing each dialogue element.
Formal Usability Inspections: Evaluators walkthrough tasks with
the user's goals and purpose in mind, similar to cognitive
walkthroughs, although the emphasis is less on cognitive theory
and more on encountering usability problems.
Usability Inspection Methods
There’s no such thing as
a “user error”!
Cognitive: thought process toward awareness or knowledge
8. measure with users
USABILITY
EVALUATION
Feature Inspection: lists sequence of features used to
accomplish typical tasks, checks for long sequences,
cumbersome steps, steps that would not be natural for users
to try, and steps that require extensive knowledge/
experience in order to assess a proposed feature set.
Consistency Inspection: has designers representing multiple
projects inspect an interface to see whether it does things in
the same way as their own designs.
Standards Inspection: has an expert on some interface
standard ‘inspect the interface for compliance.
Usability Inspection Methods
There’s no such thing as
a “user error”!
9. measure with users
USABILITY
EVALUATION
Heuristic evaluation is a method for finding the usability
problems in a user interface design so that they can attend
to as part of an iterative design process.
Heuristic evaluation involves having a set of evaluators (3 -
5) examine the interface and judge its compliance with
recognized usability principles (“the heuristics")
Heuristic evaluation
Evaluators VS Problem found – report Nielsen’s
10. measure with users
USABILITY
EVALUATION
Visibility of system status
Match between system and the real world
User control and freedom
Consistency and standards
Error prevention
Recognition rather than recall
Flexibility and efficiency of use
Aesthetic and minimalist design
Help users recognize, diagnose, and recover from errors
Help and documentation
1
2
3
4
5
6
7
8
9
10
The Heuristics Principles
Nielsen’s Heuristics
Ref. Link: http://www.useit.com/papers/heuristic/heuristic_list.html
11. measure with users
USABILITY
EVALUATION
A Usability Review (more formally known as ‘Heuristic
Evaluation’ is a technique for identifying usability
issues.
A Usability Review:
• Is cheaper to conduct than formal usability testing
• Can be completed in a very short period
• Can be conducted at any stage of the design.
• Usability Reviews are somewhat subjective,
because no real users are involved.
Objectives of a review:
• Identify likely trouble-spots
• Recommend directions and concepts for improved
design (both tactical and strategic)
• Finds ‘obvious’ problems that are NOT really
obvious.
What is a Usability
Review?
Usability Review
12. measure with users
USABILITY
EVALUATION Usability Review
When ?
A Usability Review is particularly appropriate if an application
has a large number of serious usability problems, or if an
application is not sufficiently mature for usability testing with
real users. In a hostile environment, a Usability Review is open
to the accusation of being only one opinion against the opinion
of others. In such circumstances, consider usability testing
instead.
How?
At least two people should independently review the
application, to maximize the number of issues identified. In
general, avoid being given a walk-through of the application
before beginning, as it is best to approach the task with a fresh
and un-biased viewpoint. However, you should know:
• The purpose of the application
• The characteristics of the intended users and their levels of
domain knowledge
• The context of use.
Usability Review
13. measure with users
USABILITY
EVALUATION
Before you start, create a working document for your notes. This
allows you to capture first impressions, which you may not
readily recall once you have begun working with the application.
Once you begin your review, use a standard notation for
capturing issues. For each issue, note:
• An exact description of the issue. Write this in the voice that
you will use in your report, to facilitate subsequent 'cutting-
and-pasting' with a minimum of re-writing.
• Occurrence point of the issue
• The usability principle (heuristic) violated/ standard in
interface
• An estimate of severity.
• A recommendation
What to include in a
Review
Usability Review
14. measure with users
USABILITY
EVALUATION
• Frequency with which the problem occurs – is it common or
rare?
• Impact of the problem if it occurs – will it be easy or difficult
for users to overcome?
• Persistence of the problem – is it a one-time problem that
users can overcome once they know about it or will users
repeatedly be bothered by the problem?
Computing Severity
Ratings
Usability Review
15. measure with users
USABILITY
EVALUATION
Navigation & ease of information retrieval
Maximize the ease with which site visitors can find both the
information they require and the content you want them to
access
Page layout & calls-to-action
Make sure there are enough calls-to-actions and that they're
appropriate and noticeable
Persuasiveness & cross-selling
Ensure the site and individual pages persuade site visitors to do
what you want them to do (make a purchase, contact you etc.)
Removing obstacles
Make it as easy as possible for people to do what they - and
more importantly you - want them to do
Content, marketing & brand
Ensure that the site's content engages and persuades users and
that the site's USPs are communicated clearly and effectively
Search
Enhance the design and performance of the site's search engine
so users can find what they want with ease
Advantages of a
Usability Review
Usability Review
16. measure with users
USABILITY
EVALUATION
• Precisely why you consider each issue to be relevant from a usability
perspective - avoid personal opinions that may be grounded in
prejudice.
• Acknowledge the positive aspects of the system under review -
remember that some readers may be hostile to or upset by your
findings.
• Give careful, meaningful and reasoned comments.
• Provide clear recommendations. E.g.: Decide whether the application
can be 'patched', or whether more fundamental changes are required.
• Summarize your findings in the beginning of the report. Do not expect
all readers to have time to go through the issues in detail.
• Combine and print the issues from each reviewer, and group like
issues together. This should give you the basic structure for the body of
your report.
• Review your working notes to ensure that all the issues you noted
have been included as issues and not as recommendations.
• Peer review the checklists to see if all reviewers are in agreement, and
examine any discrepancies.
• Do NOT make a mockery of the design of the application.
Things to Remember
Usability Review
17. measure with users
USABILITY
EVALUATION
Its a more explicitly detailed procedure where evaluators
asks himself a set of questions about the expectations of
users’ behavior to assesses interface design and come up
with a success/failure stories to simulate user’s problem
and solving process at each step through the dialogues.
This technique is best used in the design stage of
development. The evaluators may include usability
experts, designers, software developers or people from
marketing, etc.
Cognitive Walkthrough
Cognitive: thought process toward awareness or knowledge
18. measure with users
USABILITY
EVALUATION
Prototype
Prepare Low / High fidelity prototype
Choose a user ( representative user)
Explore their skill levels with both technology and the
problem domain
Evaluation Tasks
Define the Goal & Tasks to be achieved in the prototype
Action Sequence
Determine the step by step actions the user ideally should
take to complete the task. Every step should be explicitly
listed
1
Cognitive Walkthrough
How To Do It
19. measure with users
USABILITY
EVALUATION
Task Walkthrough
The modulator walk through (discuss) the tasks with
respect to the interface/mockups and action sequences/
scenarios with the evaluators/ participants and create a
credible success /failure story for each individual step.
Ask the following questions, for each step, while looking at
the interface and create a credible story based around
these questions
• Will the user try to achieve the right effect?
• Will the user notice that the correct action is available?
• Will the user associate the correct action with the effect
that user is trying to achieve?
• If the correct action is performed, will the user see that
progress is being made towards their goal?
2
Cognitive Walkthrough
How To Do It
20. measure with users
USABILITY
EVALUATION
Reports (Actionable Data )
Create report with actionable data include steps where the
user might not know what to do, steps where they don't
know if they were successful, and gaps in the design,
including success/failure stories
Cognitive Walkthrough
How To Do It 3
21. measure with users
USABILITY
EVALUATION
In the design stage, when paper prototype is available, a
group of users, developers, and usability expert meet
together to step through a set of tasks, discussing and
evaluating the usability of a system. The more people
looking for problems, the higher the probability of finding
problems.
Five Characteristics
• Inclusion of representative users, product developers,
and Usability professionals
• The application’s screens are presented (as hard copy)
in the same order as they would appear to the user
• All participants are asked to assume the role of the user
• Participants write down what actions they, as users,
would take for each screen before the group discusses
the screens
• When discussing each screen, the representative users
speak first
Pluralistic Walkthrough
22. measure with users
USABILITY
EVALUATION
In a usability evaluation, you are predicting the problems or
successes that users will have with the system/interface. A
usability test tells you whether your predictions are valid and
also helps to find new usability problems.
The aim of the Usability testing is to gather qualitative user data;
combines interviewing and observation of a user in a real-time.
Usability testing is especially effective in determining:
• Naming (section/button labels/Words)
• Organization (information grouped into meaningful
categories/ better placed)
• First-time use and discoverability
• Effectiveness
• Emotional Response
Techniques popularly used to gather data during a usability test
include think aloud protocol, Co-discovery Learning, eye
tracking, Remote Testing, Question-asking Protocol, Clickstream
analysis, Facial Recognition and Psychometric Data Gathering
and Analysis
Overview
23. measure with users
USABILITY
EVALUATION
Think Aloud Protocol
During the course of a usability test, the test users are asked
to verbalize/speak out their thoughts, feelings, and opinions
while interacting with the system. It is very useful in
capturing a wide range of cognitive/ mental processes of
activities.
Co-discovery Learning
During a usability test, two test users attempt to perform
tasks together while being observed. They are to help each
other in the same manner as they would if they were
working together to accomplish a common goal using the
product. They are encouraged to explain what they are
thinking about while working on the tasks.
Techniques to Gather Data
24. measure with users
USABILITY
EVALUATION
Eye Tracking
Eye tracking is the process of measuring either the point of
gaze ("where we are looking") or the motion of an eye relative
to the head. Watching what people are looking at adds a level
of interest to usability studies. Following the eye at work
makes you feel as though you are in the user’s head, thinking
with him and able to understand how user looks the interface.
Eye tracking solutions
Eye Tracking is a combined software and hardware package
designed to capture the conscious and unconscious eye gaze
movement of a respondent.
Eye Tracking is a Unique and Reliable Method for Assessing
Design Impact or Usability of websites, print media, package
design, virtual store shelving, etc.
Hardware Solutions & Support
noldus.com, smi.de , faceLAB, Tobii TX300 Eye Tracker,
EyeTech, etc.
Techniques to Gather Data
26. measure with users
USABILITY
EVALUATION
Remote Testing
Remote usability testing is used when testers are separated in
space and/or time from the participants. This means that the
testers cannot observe the testing process directly and that
the participants are usually not in a formal usability
laboratory.
Another is different-time different-place testing, where the
user's test session is guided and logged through a special
piece of software.
Tools : Morae, usertesting.com, intuitionhq.com, CrazyEgg, etc.
Question-Asking Protocol
During a usability test, besides letting the test users to
verbalize their thoughts as in the thinking aloud protocol, the
testers prompt them by asking direct questions about the
product, in order to understand their mental model of the
system and the tasks, and where they have trouble in
understanding and using the system.
Techniques to Gather Data
27. measure with users
USABILITY
EVALUATION
Clickstream
Clickstream analysis/clickstream analytics is the process of
collecting, analyzing, and reporting aggregate data about
which pages visitors visit in what order There are two levels
of clickstream analysis, traffic analysis and e-commerce
analysis.
Facial Recognition
facial recognition technique are used to capture information
about the shape of a face and this information is then used
for identification and the verification of identity.
Psychometric Data Gathering and Analysis
Psychometric(Psychology + Metrics) is the study of behavior
and mental processes. Psychometric testing or psychometric
analysis is to measure or assess the psychology of a person.
Techniques to Gather Data
Clicksteam
Facial
Recognition
28. measure with users
USABILITY
EVALUATION
Plan and Strategize on:
• The Goals for the test
• The Screens & System for the test
• Tasks for the user to be tested
• Data to be collected and analyzed
• Methodology to collect the data
• Location of the test
• Recording Setup
• Role of the participants
Identify Test Users
User should be as representative as possible of the
intended users of the system. One should take additional
care to involve average users. If a larger group of testers
are to be engaged, one should select users from several
different subpopulations to cover the main different
categories of expected users. (Novice to Expert users)
2
“You’re testing the system
not the User Knowledge “
1How To Do It
User Testing
29. measure with users
USABILITY
EVALUATION
Test Location
Usability testing is preferably conducted in User’s
environment. Else, out of fear of seeming dumb,
incompetent, many users may avoid talking about software
behaviors that they find problematic.
Design Usability Test
• Pre test user briefing points, questions & confidential
agreements
• Participant Tasks
• Goals & Tasks
• Test User Questions
• Post Test Questions & Survey
Pilot Test
Pilot testing can also be used to refine the experimental
procedure and to clarify the definitions, Test tasks, Test
plan, Questionnaires.
How To Do It
User Testing
3
4
30. measure with users
USABILITY
EVALUATION
Perform User Testing (UT)
• Explain the User about the aim of the test
• Explain User if any recording, keystroke logging, or other
monitoring that is used
• Explain and get the confidential agreements if required
• Brief the goal and tasks which the tester needs to perform
• Choose appropriate techniques/method to gather data (
Think aloud method/Eye Tracking/Question-Asking
Protocol)
• Define the roles for each participant:
• User: Performs the task as planned
• Observer: Takes note of every action and reactions of the
user
• Moderator: Probes with questions and helps the user to
overcome any system limitations
• Upon completion of the test, analyze the data captured and
identify Usability problems
How To Do It
User Testing
5
31. measure with users
USABILITY
EVALUATION
• Observe, be quiet, watch, understand
• Don't explain, suggest, defend the design
• Don't ask the test user's opinion
• Help user to overcome the limitations of the prototype
• Hand out the test tasks one at a time
• Keep a relaxed atmosphere in the test room
• Avoid disruptions
• Never indicate in any way that the user is making
mistakes or is too slow
• If necessary, stop the test if it becomes too unpleasant
• End by stating that the user has helped you find areas of
improvement
• Never report results in such a way that individual users
can be identified
• Cash or an appropriate 'Thank You' gift
User Testing – Quick Tips
While UT
After UT
33. measure with users
USABILITY
EVALUATION System / Interface Design
• Usability evaluation will help you in identifying system’s Learnability,
Efficiency, Memorability, Errors & Users Satisfaction.
• Risks are managed and reduced by introducing usability evaluation
in early stage of the software development cycle.
• Evaluation provides qualitative feedback and helps to improve your
system / interface design.
• Helps you identify issues and potential problems before the
development starts.
• Help to discover real needs and tasks of the user’s in the early stage
of the design process.
• Decrease user acclimation (accustomed to a new environment) time,
errors and increased user’s productivity.
• Provide a competitive advantage
• Help to analyze user’s goals, which results in the business meeting
its targets.
• Will help in reducing the rework and help in meeting customer
requirements (80 percent of software rewrites are due to the fact
that important functionality was missed the first time)
• Help to build customer and users confidence.
34. measure with users
USABILITY
EVALUATION Business
• Gain a competitive advantage.
• Generate rapid ROI and gain quicker time to market by avoiding big
changes late in development.
• An easy-to-use system means fewer calls to customer service
• Early prediction of potential problems will reduce the project
lifecycle cost
• Provide more follow-on business due to satisfied customers & users
• Look better, demo better and sell better, as they are more intuitive
to use, faster and more effective results in profitability and revenue
• Loyal customers provide free word-of-mouth exposure results in the
business revenue
35. measure with users
USABILITY
EVALUATION
Usability Inspection Methods- Jakob Nielsen, Robert L Mack
Usability Engineering- Jakob Nielsen
Measuring The User Experience- Tom Tullis, Bill Albert
About Face 3.0- Alan Cooper, Robert Reimann, and Dave Cronin
Beyond The Usability Lab - Bill Albert, Tom Tullis, Donna Tedesco
usability.gov
useit.com
usabilityfirst.com
stcsig.org
wikipedia.org
usabilityhome.com
webcredible.co.uk
Enhancing the Explanatory Power of Usability Heuristics – Jakob Nielsen
Usability inspection methods - research and practice- David G. Novick
Usability Inspection Methods- Jakob Nielsen
The Pluralistic Usability Walk-Through Method- S. Riihiaho
Eyetracking Methodology- Kara Pernice and Jakob Nielsen
The cognitive walkthrough method: A practitioner’s guide. In Nielsen, J., and Mack,
Usability inspection methods. New York, NY: John Wiley & Sons, Inc.
Wharton, C., Rieman, J., Lewis, C., and Polson, P. (1994).
Courtesy : istockphoto.com
Books
Online
Whitepapers
USABILITY
EVALUATION References
