Read more at confrere.com/usertest Everyone wants to make intuitive products. But you can’t really know they’re intuitive, if you haven’t tested them with actual users. So why don’t we user test more often? It might be because we think we’re lacking the time, budget, equipment or simply the knowledge to do it. But that doesn’t have to be true. CONTENTS Tools and methods: Get to know 7 easy and affordable techniques to do user testing Planning and prioritising: When and what should you test? Tasks and moderating: How do you make sure you’re not leading users during the test? Recruiting: How many people do you need to test with, and where can you find them? Observation: How should you interpret what you see and take notes during a user test? Analysis: How can make sure your usability findings lead to actual improvements?

1. Easy & affordable
user testing
Ida Aalen / @idaaa
CPO, Confrere
ida@confrere.com
Workshop

2. confrere.com/usertest

3. Task 1
Read the title and the intro.
Write down 1-5 questions you
expect will be answered on this
page.

4. Task 2
Read the text.
• Were your questions answered?
• Is there content that could be cut
or should be expanded upon?

5. #1
Micro-testing content.
• Time: 30 min+
• Price: Mostly free

6. Question 1
Read the title and description. Write down 3-4 questions you expect
will be answered on this page.
Don’t peek!

8. Feedback on
needs, not
grammar and
typos

9. Feedback on
needs, not
grammar and
typos

11. • What equipment is supported?
• Can this service be used on
mobile devices?
• How can I know that it is safe?
• What is the flow like, step-by-
step, when setting up a call?
• How much does it cost?
• How do you embed/link this on
your own site?

12. User testing can be
easy & affordable.

15. Why do
user testing?

16. User testing is
real users solving
real tasks.

18. Do you recognise this icon?
Survey with 4273 respondents, 2014
Mobile:
53%
said no
Desktop:
73%
said no

19. …but I can’t see the
football…

22. Colleagues
are not your users.

23. Designers
are not your users.

24. Developers
are not your users.

25. 1 in 3
problems found
by experts are
false alarms
1 in 2
user problems
are overlooked
by experts

26. Only users can let you know
whether it’s intuitive.
But how many users?

27. 5

28. N (1-(1-L)n)

29. 0%
25%
50%
75%
100%
Number of test subjects
0 3 6 9 12 15
Shareobservedproblems
Small and frequent
tests are better than
bigger and
infrequent tests

32. User testing
• …it’s the only way to know if the design truly is “intuitive”
• …you’re taking big risks when you’re not testing
• …it improves communication with stakeholders
And it’s cheaper than you think :)

33. #2
Information architecture
• Time: 30 min+
• Price: Free / $99-199 per month

39. #3
First click.
• Time: 30 min+
• Price: Free / $99-199 per month

44. Tasks.

45. Finding possible issues
Customer service and receptionists
What are people calling and asking about?
Website analytics
Where do users drop off? Where do they spend more time than expected?
Search logs
Are there things that people aren't able to find? Which words do they use?

46. You should test things that…
• are done frequently or by many users
• has serious consequences if users fail with their task
• are hangups within the organization, but need to be
determined if they're actual issues

47. What we want to find out:
Is it clear to physicians
that they can use Confrere
for video appointments?

48. Our question
≠
User task

49. Open & explorative
What would you do to
find a video calling
service that’s suitable
for physicians?
Specific & measurable
Could you find the
page which explains
how Confrere is
HIPAA compliant?
Partially open
Go to confrere.com.
What would you do
to figure out whether
this is suitable for
physicians?

50. Open & explorative
• If you want to find
more problems
• It’s ok to give
different tasks to
different people
• More realistic, but
also more time-
consuming
Specific & measurable
• When you know
there’s a problem,
but you want to
figure out why
• Easier to measure
and compare
• Can be leading
Partially open

51. Phrasing the tasks
• Don’t use the exact same phrases as on the website
• Give the users a problem to solve, rather than asking them to
“find” something
• Adapt the tasks to each person, to make them more realistic

52. What do we want to know?
“Is it clear to physicians that they can use Confrere for video
appointments?”
How do we phrase it?
“Go to confrere.com. What would you do to figure out whether
this is suitable for physicians?”

53. Intro
• User’s rights and informed consent
• Introductory questions / warm-up
Tasks
• The user tries to solve this, while the moderator sits next to them
Closing
• General feedback
• Thanking the user
Typical
duration
15-60 min

54. Time schedule
• You’ll always underestimate how long the tasks take
• Begin with 2-5 tasks, and do a trial test
• Prioritize the tasks, in case you don't have time to get to all of
them
• Make sure you have time to test the equipment
• Breaks between each test: At least 15 minutes of break time
per hour!

55. Moderating.

56. Help the
person to
relax
Make it
easier for
observers to
understand
Make sure we
follow the test
plan

57. Help the person to relax.

58. “Would you like something
to drink?”

60. Checklist before beginning
• Welcome the person, help them feel comfortable
• Go through rights and consent form, make it clear that we’re
testing the system, not you
• Explain the “think out loud”-method
• Ask if the person has any questions before you start
• Remember to turn on the recording!

61. Help the person
think out loud

62. What are you thinking about?

63. I’m looking for the menu…
Help the user
think out loud
and ask open
questions

64. What happens if I click here?
What do you think might
happen if you click there?
Make
sure you’re
not helping
the user

65. How to help someone think
out loud
• What are you looking for now?
• What are you thinking about now?
• What did you expect would happen just now?
• …but don’t interrupt

66. How do you lead a test,
without leading the user?

67. I’ve made the design for the
new website and I’d love to
hear what you think!
The team that designed the
site needs help figuring out
if it's really working.

68. You’re really good at this!
You’re solving all the tasks!
Thank you! Let’s move on
to the next task.

69. It would have gone so much
faster if you had just clicked
the menu-button.
There's really big room for
improvement here. Thank you
for helping us identify this!

70. Sigh…

71. The user
is there to help
you

72. #3
Get out.
• Time: 1-4 hours
• Price: Snacks

73. It’s this easy:

76. (Translated
by Google)

80. What did you find?

83. Analysis in 1-2-3
Is there anything you observed…
…across several users on the same task??
…across several tasks?
You start doing the analysis between each individual test.

84. Observation
and notes.

85. Don’t listen.
Observe.

86. Rationalization
Michael S. Gazzaniga:
“The Interpreter”
http://youtu.be/mJKloz2vwlc

87. Differentiate between…
Observation
What the person did and said during the task
Interpretation
Why you believe the person did what they did
Improvements
What you believe could be a solution to the problem

88. Example
Observation
• The user did not click the “Buy”-button
Interpretations
• The button should be placed elsewhere
• The button has the wrong color or shape
• The microcopy “buy” sounds too committing
• The user didn’t think they really were supposed to “buy” since it’s
just a test

89. Quantitative
Qualitative
Self-reported Observed
Surveys
A/B-testing
User testing
Interviews
Focus groups Field studies
Google Analytics
Key word analysis
“Was this
helpful?”
Crawling
social media

90. Read this!

91. How to take notes?
• Remember the purpose of the task - don’t write a novel on
what each person does
• Note which words the user is using
• Remember, it's mostly about what they're actually doing and
saying while they're doing it

92. Paper sketch
Paper prototype
Wireframes Sketch / Photoshop
InvisionApp
HTML prototype
without styling
HTML&CSS&JS
prototype
Interactive
Flat
Rough Detailed
Functional
Implemented
Text

93. #5
Scrolling, clicking and
hovering.
• Time: Done in 30 min
• Price: Free (e.g. Hotjar + Google Analytics)

98. #5
Was this helpful?
• Time: 15 min+
• Price: Often free

108. #4
Lab test on a budget.
• Time: 1 day for planning and recruiting, 1 day for the actual
test
• Price: Rewarding users

109. (Translated
by Google)

113. “Hugging”

115. If you need observers:
Test machine Observation
+ Tool for
screen sharing+

116. Informed
consent
• They can opt-out any time
• What data are you gathering and
why
• For what and when will you use it
• When will you delete it
• Check local rules/laws
KONTAKT OSS
+47 22 42 46 42
kontakt@netliferesearch.com
BESØK OSS
Stenersgaten 8, Oslo
http://netliferesearch.com/
Dine rettigheter som testperson
1. Det er ikke du som testes, men løsningen. Det er derfor ingen
krav til prestasjoner eller ferdigheter.
2. Det er frivillig å delta og du kan avbryte testen når som helst
3. Det blir sendt bilde av skjermen og testsituasjonen til et annet
rom
4. Din anonymitet sikres ved at det ikke brukes navn eller annen
informasjon som kan koble personen til notatene
5. Opptak av skjermbildet vil kunne bli vist i møter eller lignende
etter testen er ferdig, men det vil ikke være fokus på hvem du
er. Videoene vil kun være til internt bruk i prosjektet og vil
slettes etter tre måneder
6. Du skal behandles høflig og med respekt
Jeg bekrefter å ha fått den informasjon som er angitt ovenfor og at
jeg gir tillatelse til at opptak av skjermbildet kan bli vist til andre.
_________________________ _________________________
UNDERSKRIFT DATO
KUNDE:
PROSJEKT:

117. 20-30 min break
between each test

118. When do you need observers?
• If you need to align stakeholders
• In general, the more observers, the better, because different people
notice different things
• Bring your teammates from different disciplines (e.g. developers)
• Observers should stick to observing, and not introduce themselves to
the users

123. Analysis and
prioritisation.

124. The analysis must be done as
soon as possible, by the
people who did the user test.

125. Which of our findings or
observations should be
turned into to-dos?
What’s relevant for our
further work?

126. Documenting and sharing
your findings

127. As a tool for your team

128. Put screenshots
in InvisionApp in
advance
Add
observations
between (or
during) each
test

130. Connect
InvisionApp to
Slack, and your
teammates can
follow along

133. Vise alvoretWake-up calls

134. Innhold

136. A highlights video usually
does the trick.

137. Prioritizing
number of users consequences other sources cost

138. Make sure something
happens with your findings
• Each (prioritised) usability issue needs a to-do
• Each to-do must be assigned to a specific person

139. Do it together!

141. Summarizing
and prioritizing
in Google Sheets
or AirTable

143. Recruiting.

144. Selection criteria
• Diversity
• Language skills
• ICT skills
• Target group criteria
• Background knowledge or
experience with your service
Exclusion criteria
• People with too much
knowledge about your service
• People working with
development, design, web etc
Stricter
criteria
=
Harder to
recruit

145. Recruiting tips
• Friends of friends, social media, newsletters
• Go to where they are
• Run it outside of regular work hours
• Be creative about rewards
• Lower your standards

146. Avoiding drop-outs
• Send e-mails or texts with very clear instructions for how to
get there, sign them personally and add your phone number
• Send a text the day before or the same day to remind them
• Recruit one more person than you think you actually need

147. 7 methods:
1. Micro testing content
2. Information architecture
3. First click
4. Get out
5. Lab test on a budget
6. Scrolling, clicking and hovering
7. “Was this helpful?”
confrere.com/
usertest

148. 7 methods:
1.Micro testing content
2.Information architecture
3. First click
4. Get out
5.Lab test on a budget
6. Scrolling, clicking and hovering
7. “Was this helpful?”
confrere.com/
usertest

149. 7 methods:
1. Micro testing content
2. Information architecture
3. First click
4.Get out
5. Lab test on a budget
6.Scrolling, clicking and hovering
7.“Was this helpful?”
confrere.com/
usertest

150. 7 methods:
1. Micro testing content
2. Information architecture
3. First click
4.Get out
5.Lab test on a budget
6. Scrolling, clicking and hovering
7. “Was this helpful?”
confrere.com/
usertest

151. You learn so
much more by
meeting people
face-to-face!

152. Thank you!
confrere.com/usertest
Ida Aalen
Chief Product Officer
confrere.com