Have you ever wondered why your horoscope can predict your future, or what motivates people to report false information on a survey? These phenomena can be explained by everyday biases that can drastically impact user research studies. This introduction to bias in user research describes a number of biases we encounter in everyday life, and explains their impact in a user experience study. Senior Experience Researcher, Leah Samuelson, will help identify common biases, explain where they arise in user testing, and note best practices for avoiding them to get the most accurate data.

1. Powered by
What’s Next:
Understanding
User Research Bias
(and Why Your Horoscope is Right)

2. Leah Samuelson
Senior Experience Researcher
Ogilvy
Dayoán Daumont
Consulting Partner
Ogilvy Consulting
Welcome

3. What’s the weather
like in your city?
Tell us
where you
are dialing
in from!

4. August 20, 2019 4
1 3 4
General bias vs. user
experience bias
Agenda
Understanding
bias
2
Overview & best
practices
Researcher
bias
Overview & best
practices
Respondent
bias
Anything I can
answer for you
Questions

5. Understanding bias

6. August 20, 2019 6
A shortcut our brains take to
navigate constant stimulus and
make decisions quickly.
What is bias?
A cognitive bias refers to a
systematic illogical thinking
pattern that affects judgments
and decisions.
Biases can impact our actions
in positive and negative ways.

7. August 20, 2019 7
User experience design is the
process of creating products
that provide meaningful and
relevant experiences to users.
What is user
experience bias?
This involves research
throughout the entire process,
both in gathering information
and testing designs.
Biases in UX can cause a
respondent to subconsciously
respond untruthfully. They can
also cause a researcher to
evoke a certain response or
report inaccurately.

8. Researcher bias

9. Confirmation bias
Our natural inclination to seek out or pay greater
attention to information that supports our beliefs,
and discount that which does not.

10. 10
“Today you will
receive good advice
from a friend”
DAILY HOROSCOPE

11. A human tendency to develop personal opinions, giving
more weight to desired outcomes1
May cause the researcher to “seek out” the data that supports
their preferences
2
May lead to the observer-expectancy effect where the
researcher unknowingly steers the participant
3
August 20, 2019
Confirmation
bias
Our natural inclination to seek out or pay
greater attention to information that supports
our beliefs, and discount that which does not.

12. August 20, 2019
1 2 3 4Identify any
opinions or
assumptions you
have going into
testing
12
Ask yourself which
pieces of information
you readily accepted,
and those you
skimmed
Make sure your
sample size is large
enough to provide a
greater pool of
evidence
Write a discussion
guide beforehand
and stick to it
closely
Best practices – Confirmation Bias

13. Recency bias
An order of presentation effect that occurs when
more recent information is better remembered and
receives greater weight in forming a judgment than
earlier-presented information.

14. 14
Investors often base market
expectations on how it has
been performing recently
instead of assessing long-
term averages
INVESTING

15. Because it’s easier, we’re inclined to use our recent
experience as the baseline for what will happen in the future.1
Studied frequently in social psychology
2
Causes researchers to give more credence to what they heard
most recently instead of what they heard most often
3
Can greatly skew data toward the last interview that was
performed
4
August 20, 2019
An order of presentation effect that occurs
when more recent information is better
remembered and receives greater weight in
forming a judgment than earlier-presented
information.
Recency
bias

16. August 20, 2019 16
2 31
Analyze data in a
different order
than that of your
interview
sequence
Devise a process
for data analysis
before testing that
gives equal weight
to all responses
Create a debrief
document to fill
out with
colleagues after
each interview
Best practices – Recency Bias

17. Task-selection bias
The effect that takes place when a user knows that
a task can be completed, due to the mere prospect
of being asked to do it.

18. 18
How would you
access the virtual
dressing room?
USABILITY TESTING

19. This is a recently defined bias that is rooted more specifically
in user experience research1
Researchers in the UX field would not ask a user to do a task
unless it was possible to complete, thus clueing the user that
the feature is available and findable on the page2
When a user typically interacts with a website they do not
assume functionality that is not readily apparent
3
August 20, 2019
Task-
selection
Bias
The effect that takes place when a user
knows that a task can be completed, due to
the mere prospect of being asked to do it.

20. August 20, 2019 20
2
Get general feedback
Asking what features a user would
hope to have and where they would
expect to find them gives helpful
feedback without indicating what’s
already available.
3
Engage in out-of-
the-box activities
You can include creativity-prompting
exercises such as having respondents
draw out or describe their ideal
webpage design and functionality.
1
Ask open-ended
questions
When a user lands on a new page,
consider asking, “what stands out to
you here?” or “what would you expect
to find in this section?”
Best practices – Task-selection bias

21. Respondent Bias

22. Social desirability bias
The tendency of respondents to answer questions
in a manner that will be viewed favorably by others.

23. 23
“I love this design!
Delta always has
such a creative
approach”
DELTA AIRLINES

24. Social
Desirability
Bias
It can take the form of over-reporting "good behavior" or
under-reporting "bad," or undesirable behavior.1
We are evolutionarily predisposed toward this behavior
2
May be caused by social desirability toward others, toward
the researcher, or the company doing the testing
3
August 20, 2019
The tendency of respondents to answer
questions in a manner that will be viewed
favorably by others.

25. August 20, 2019
1 2 3 4
Frame
questions in
an objective
manner
Exhibit
neutral
responses and
body language
Leave the client
anonymous
until it’s
necessary to
reveal
Administer a
survey or
unmoderated
test
25
Best practices – Social desirability bias

26. Anchoring bias
The tendency to rely too heavily on the first piece
of information offered (the “anchor”) when making
decisions

27. 27
the initial price offered
sets a precedent for all
subsequent offers,
regardless of market
value
BUYING A USED CAR

28. During decision-making an individual depends on an initial
piece of information to make subsequent judgments1
In UX a respondent may often use the first design as an
anchor for the following designs
2
A researcher may show multiple designs for a company logo,
but the respondent uses the first design as their “gold
standard” in feedback for the rest3
August 20, 2019
The tendency to rely too heavily on the first
piece of information offered (the “anchor”)
when making decisions
Anchoring
bias

29. August 20, 2019 29
2
A/B test designs
when possible
Showing two or more variants
of a page at random is the best means
for negating this bias.
3
Deep-dive on customer
responses to encourage
more thoughtful
responses
Ask users subsequent questions
encouraging them to consider their
reasoning. This can also generate
feedback on elements that are not
anchored.
1
Alternate the order of
certain questions from
interview to interview
Think carefully about the sequence of
questions, especially ones that might
trigger the anchoring effect.
Best practices – Anchoring bias

30. Observer effect
The observer effect (also known as the Hawthorne
effect) is a tendency of some people to work
harder and perform better when they are
participants in an experiment.

31. 31
Productivity improved
when the lighting
improved… but also
when the lighting
diminished
HAWTHORNE WORKS

32. When people are aware that they are being observed, they
change their normal behavior unintentionally1
A user my want to complete a task quickly to demonstrate
their online savviness or to avoid the discomfort of silence
2
If you ask a participant to add items to an online shopping
cart, they may do so faster than normal or triple check the
quantity so as not to make a mistake3
The observer effect (also referred to as
the Hawthorne effect) is a tendency of some
people to work harder and perform better
when they are participants in an experiment.
August 20, 2019
Observer
effect

33. August 20, 2019 33
2
Ask warm-up
questions
Ask the user some light, easy questions
to get them comfortable and talking.
The more comfortable they are, the less
pressure they’ll feel to perform.
3
Encourage users to
think out loud
In setup and throughout testing,
encourage users to engage the think out
loud method, providing a stream of
consciousness as they work makes
responses more natural.
1
Remind the user
that this is not a
test
In the setup of any interview or
usability test it’s important to remind
users that there are no correct answers.
Best practices – Observer effect

34. Biases are nearly
impossible to avoid.
By keeping just a few
simple practices in
mind, you can improve
your skills as a
researcher, and
improve your data
accuracy.
BIAS WRAP UP

35. Questions?

36. Thank you.