Colleen Roller's presentation from the UXPA Boston 2014 conference.

1. Research Design
Gaining deeper insight
Colleen Roller
Author & Practitioner

2. Areas of research
 Behavioral decision theory
 The study of how people judge, decide and behave
 Cognitive psychology
 The study of how people perceive, think, remember
and learn
 Social psychology
 The study of how people are influenced and affected
by other people

3. No neutral design
 Preferences, judgments, and
decisions are remarkably sensitive to
the context in which they are made.

4. No neutral design

5. No neutral design

6. No neutral design

7. This presentation
 Uses research findings to inform
research design

8. Clever Hans

9. The power of the subconscious
Rosenthal/Jacobsen 1968

10. ‘When we expect certain behaviors of others,
we are likely to act in ways that make the
expected behavior more likely to occur.’
- Rosenthal and Babad

11. Confirmation bias
 The tendency to seek out and assign more
weight to evidence that confirms one’s
hypothesis, and ignore or underweight
evidence that could disconfirm it.

12. So what?
 We may be influencing participants –
and research findings – in ways
we’re not even aware of.

13. FRAMING

14. What is framing?

15. Framing
‘I’ve got great news. You managed
to avoid a salary decrease.’

16. Frame the study
 The purpose of this study is to test
the design, not you.

17. Child custody
Parent A
Average income
Average health
Average working hours
Reasonable rapport with the child
Relatively stable social life
Shafir, Simonson, Tversky, 1993
 Which parent should get sole custody?
Award – 64%
Parent B
Above-average income
Minor health problems
Lots of work-related travel
Very close relationship with the child
Extremely active social life

18. Child custody
Parent A
Average income
Average health
Average working hours
Reasonable rapport with the child
Relatively stable social life
Shafir, Simonson, Tversky, 1993
 Which parent should not get sole custody?
Award – 64%
Deny – 55%
Parent B
Above-average income
Minor health problems
Lots of work-related travel
Very close relationship with the child
Extremely active social life

19. Positive/negative
 What did you like/dislike…
 How useful, or not,…
 How confident, or not…
 Why would you continue/stop using…

20. How we ask the question
 Positive/negative
 Breadth/depth – broad/narrow

21.  Did you notice anything different
about this process…
 How does this process compare to
how you currently do this?

22.  What information do you need in
order to feel confident about
doing x…
 What would make you feel
confident about doing x…

23.  How often do you use the x
website…
 Tell me about how and when you
use the x website…

24. Counterintuitive frame
 What do you think of the brand?
 What does the brand think of you?

25. ‘A small question can never
hold a large answer.’
- Gerald Zaltman (How Customers Think)

26. Assumptions
 Be careful that your assumptions
aren’t built into the question
 Turn your assumptions into questions

27. Rating scales
 How successful have you been in life,
so far?
0 1 2 3 4 5 6 7 8 9 10
Not satisfied Very satisfied
-5 -4 -3 -2 -1 0 1 2 3 4 5
Not satisfied Very satisfied
34%
13%
Schwartz et al, 1991

28. Scholarships
 Two scholarships are available.
 You prefer a scholarship with less waiting
time and more money.
 Which one would you pick?
Sun, Li, Bonini 2010

29. Graphical scale
 Figure 1: wait time salient – preferred A
 Figure 2: $ amount salient – preferred B
Sun, Li, Bonini 2010

30. Graphical comparison
0 10 20 30 40
Toothpaste B
Toothpaste A
People (per 5000) with serious gum disease
E.R. Stone et al. 2003

31. Graphical comparison
Toothpaste A
30 out of 5000
Toothpaste B
15 out of 5000
E.R. Stone et al. 2003

32. So what?
 How does the frame itself influence the
meaning that people take away?

33. ORDERING OF QUESTIONS

34. Survey design
 Please rate your overall satisfaction.
 How many times have you had
repairs?

35. German students
 How happy are you these days?
 How many dates did you have last month?
Strack et al, 1988

36. Reverse order
 How happy are you these days?
 How many dates did you have last month?
 How many dates did you have last month?
 How happy are you these days?
Strack et al, 1988

37. ‘The sequential arrangement of information often
creates the very meaning of that information.
The order of presentation determines the reaction.’
- Dr. Frank Luntz (Words That Work)

38. So what?
 In what ways are you creating
(unintended) meaning?
 What are you measuring?

39. THE POWER OF WORDS

40. Different meanings
 Gambling v. gaming
 Liberal v. progressive
 Liquor v. spirits
 Used v. pre-owned
 Etc.

41. An auto accident
 At what speed did car A contact car B?
 At what speed did car A smash into car B?
Loftus/Palmer 1974
31.8 mph
40.5 mph
Actual: 12 mph

42. Power of words
 Make contact
 Smash into
 Crash into
 Bump into
 Run into
 Collide with

43. So what?
 How are your words impacting
 user perception – and consequently, research
findings?
 how your business partners perceive the
findings?

44. Delivering findings
 Users were…
 Aggravated…
 Annoyed…
 Confused…
 Disappointed…
 Frustrated…
 Irritated…
 Surprised…

45. Neutral
 Users…
 Said…
 Expected…
 Thought…
 Anticipated…

46. IN SUMMARY

47. Research design
‘The function of a research design is to ensure
that the evidence obtained enables us to answer
the initial question as unambiguously as possible.’

48. The design matters
There is no neutral design

49. ‘All research methods involve
compromises with reality.’
- Gerald Zaltman (How Customers Think)

50. The solution
 Be aware
 Control what we can
 How we ask questions
 Ordering of questions
 Choice of words
 Assumptions and their effect
 Use of ‘scale’ – charts, graphs, rating scales
 Comparison – graphs, ratings

51.  UXmatters.com
 UX magazine (uxmag.com)
Colleen Roller