7. WHY ARE USABILITY STUDIES
IMPORTANT?
http://www.flickr.com/photos/0olong/479270001/
8. WHAT DOES YOUR
INTERFACE SAY?
We hope you have excellent eyesight.
We hope you have excellent eyesight.
We hope you have excellent eyesight.
9. Yeah, hmm, we don’t know what that error message
is either.
http://www.flickr.com/photos/kk/206835799/
10. Sorry we couldn't be bothered to test our site
actually makes sense to anyone outside our
organization.
http://www.flickr.com/photos/ballookey/222118221/
17. DO NOT BE ALARMED
Preparation will take longer than
steps 2 & 3 combined
18. PREP: GOALS
Talk to stakeholders:
“What are the business goals of the study?”
“What do we need to learn?”
“When do we need it?”
“What’s our budget?”
19. PREP: WHAT TO TEST
Idea
Information architecture
Prototype
Staging site
Production site You can test at
any point during
the project
lifecycle!
20. PREP: METHODOLOGY
Moderated vs unmoderated
Online tool vs own setup
Remote with screenshare vs in person
One-on-one vs many-to-one
21. PREP: METHODOLOGY
Moderated vs unmoderated
Online tool vs own setup
Remote with screenshare vs in person
One-on-one vs many-to-one
23. PREP: WHO TO TEST
Type of skillset
Expertise
New users vs. current users
5 people per study is ideal
24. PREP: TESTING
ENVIRONMENT
How will they access the site?
Content and user account setup is needed?
Do you need to set up separate sites for each? Or re-set
between sessions? (hint: use Drush)
26. PREP: WRITING TASKS
Do not use words that are in the UI
Phrase things in the way people think and in plain language
Write in cues for moderator (e.g. “Begin recording” or
“Observe how they ____”)
Dependencies between the tasks & task order
Write moderated assists
Ask them to rate their experience
28. PREP: THE SCREENER
Develop a screener to weed
out the ‘wrong’ participant
Find out when they’re
available (use Doodle?)
Set expectations with
participants (when, how
long, compensation)
29. PREP: RECRUITING
When: 0-2 days before you want to talk to them, if
moderated
How: social networks, online free ads, groups or
organizations, meetups, even family and friends
“General” participant profile: can use a bar or cafe
Send them to your screener, e.g. bit.ly/example
31. PREP: OBSERVATION FORM
Priority Participants who had the
issue (frequency)
Issue summary (quotes!)
Other things: totals,
Link to screenshot estimated fix date,
comments
Task name
34. MODERATED STUDY
She’s working on a site that is highly
focused on a specific type of user: the
Drupal developer. She needs
feedback from Drupal developers...
anyone else won’t provide the
knowledge that is needed to improve
the site.
15-60 minute session | 5 participants | 3-5 tasks |
3-5 pre- and post-session questions
35. CONDUCTING A MODERATED STUDY
Use your study outline prepared script
Participant needs come above everything else
Get good information from them
Gonna talk
about this in
detail at 13:45!
38. TREEJACK STUDY
He’s working on a Drupal distribution
that is being redeveloped for Drupal
7. The previous distribution has some
issues, so will users understand the
new information architecture? He
needs feedback from both new and
current users.
10 minute session | 40-400 (120 optimum) participants |
3-10 tasks | 3-5 pre- and post-session questions
39.
40. TREEJACK
When? Anytime...tree is isolated
• Where would they go to do ___?
• Can they find a specific piece of content?
• Do navigation labels make sense?
• Are my subsections well-organized?
• How does Tree A compare to Tree B?
41. BUILDING THE TREE
Build the structure in a spreadsheet, one column for each
level
Include sample content
Include controls within the site, if that’s what you want to
test (e.g. “add comment”)
Only the leaves of the tree are selectable as answers,
branches are not.
43. TREEJACK SETUP
Think about what you want to validate, then write the tasks
Decide the right answers; used to calculate the success of
your tasks
Customize pre-session and post-session question
message options
44.
45. TREEJACK PROTIPS
Run a few moderated trials with people unfamiliar with your
site, to catch the WTFs ....
Set your ‘right’ answers at the very end, as you will blow
them away if you rebuild the tree
Consider hacky chunking of options
Don’t allow skip tasks, but include [None of these]
46. THE RESULTS
What you’ll get is several graphs & complex visuals to
represent the paths people took, the success of each
task, etc
Analyze and add to your observation spreadsheet
47. USERTESTING.COM STUDY
He works at a university library. He
isn’t a usability expert but wants to
make sure that the students can
perform their primary tasks easily and
quickly (like finding a book, making
the reservations for a meeting
room). Unfortunately, he has little
time and little budget.
15 minute session | 5-8 participants |
4-5 tasks | 3-4 post-session questions
48. USERTESTING.COM SETUP
Really flexible: show them a URL to live site or prototype
Give them detailed instructions at each step
Write hints, just in case they missed it
Pre-session and post-session question and message
options
52. THE RESULTS
What you’ll get is recordings of the tester’s browser and
voice, plus the answers they fill in at the end.
Analyze and add to your observation spreadsheet
53. THE ANALYSIS & DEBRIEF
Positive comments Have a scribe, if no other
stakeholders are present
Issues/Problems/Bugs and the session isn’t
recorded
Observations
If recording, revisit the
Quotes tapes and make notes
Annotated screenshots
Take notes during the
session
54. ANALYSIS
Team (or just you) should review observations and
condense, if possible
Categorize the positive negative/observations
Severity: Low/Medium/ High or 1-5
Frequency: Rare/ Common / Very Common
Scope: Local/ Global
55. EXECUTIVE SUMMARY
There were no Urgent problems uncovered. Overall, the response was positive and
participants could find their way towards achieving the tasks quickly.
Task scores
Tasks are rated from 1 (low) to 5 (high) by the moderator for efficiency and
effectiveness, based on what the participant does and says. Anything 3.5 or high is
acceptable, but 4 or higher is the goal!
Average overall score by task:
[Task 1] = 3.85
[Task 2] = 3.01
[Task 3] = 3.55
[Task 4] = 3.66
When asked what their overall experience was, given everything they’d seen in the
session, the average score provided by participants was 3.6.
56. GOING OVER RESULTS
Get all stakeholders together
Within 2 days of final study
Informal discussion
Make recommendations
57. NEEDS MOAR RAPID
• Test, debrief
and present • Inventtasks and questions
findings within 3 days based on participants
thoughts = less prep, less
• Recruit and test at an event analysis
e.g. from the booth or
nabbing people in the • Have feedback incorporated
hallway into current or next sprint
• Getdecision-makers to
observe sessions
58. TOOLS AND ARTICLES
ClickHeat, ClickTale Usertesting.com
CrazyEgg UserVue
Ethnio Verifyapp
Five Second Test
Feedback Army Article: Extremely Rapid Usability Testing
Invision App http://www.upassoc.org/
upa_publications/jus/2009may/
Loop11 pawson1.html
Mechanical Turk
Morae Presentation: Rapid turnaround usability
testing: not just a pipe dream http://
Open Hallway www.slideshare.net/usableinterface/rapid-
OptimalWorkshop (OptimalSort, turnaround-usability-testing-not-just-a-
Treejack, ChalkMark) pipe-dream
Usabilla
59. DRUPAL USABILITY
DIRECTORY
http://groups.drupal.org/usability
Usability Group for Drupal
http://groups.drupal.org/node/223959
Content Creation UX Study Plan
http://drupal.org/community-initiatives/drupal-core/usability
Drupal 8 User Experience
http://www.d7ux.org/
Drupal 7 User Experience Project
60. Wanna volunteer in the workshop?
Needed:
3 moderators
3 participants
several observers
for
3 short studies on
groups.drupal.org
drupal.org
drupalshowcase.com
65. MODERATING
Use a study guide
Explain the purpose of the study and what to expect
Get consent to record
Encourage thinking out loud
Assure them it’s a test of the software, not their skills or
abilities
Explain your role as an unbiased observer
66. MODERATING
Establish rapport, but be professional
Ensure their physical and emotional comfort
Be observant
Be greedy for information
Watch the time
Give assists, but only if needed
Thank them
