Choosing The Right Research
Methods For Your Project
Susan Mercer – Senior Experience Researcher
smercer@madpow.com
@susanamercer
October 29, 2013

Hello, I’m Susan Mercer

BA and MSc in Geophysics

19 years in software and web UI and UX design

Developer

Designer

Web Producer

Product Manager

Researcher

MS Human Factors, Bentley University

Twitter: @susanamercer
2

Introduction
“It is tempting, if the only tool you have is a hammer, to
treat everything as if it were a nail.”
– Abraham Harold Maslow
Agenda:

Market Research & User Research

Study Goals

User Research Methods

Choosing the Right Methods
3

Marketing Research & User Research
4

STUDY GOALS
5

Defining Goals
“The most serious mistakes are not being made as a
result of wrong answers. The truly dangerous thing is
asking the wrong questions.”
– Peter Drucker
6

Defining Goals
“The most serious mistakes are not being made as a
result of wrong answers. The truly dangerous thing is
asking the wrong questions.”
– Peter Drucker
The first key step of a good study is to determine your goals
Good Study Goals:

Provide focus to the study

Become more specific as the project progresses

Focus on user behavior, not on interface specifics

Help drive actionable results to impact business or design decisions
7

Defining Goals
Consider:

Where are you in the project?

Identifying problems with current behaviors to identify opportunities?

Evaluating the appetite for a product feature?

Gathering feedback on an early design?

Testing an existing product?

What are you trying to learn?

Is there an existing artifact to test?

How much time do you have?

How much of a budget do you have?
8

USER RESEARCH METHODS
9

User Research Methods
Generative
Evaluative

Interviews

In-Person Usability

Ethnography

Remote Usability

Collaging

Unmoderated Usability

Surveys

Desirability Testing

Card Sorting

Usefulness Testing

Diary Studies

Surveys

Focus Groups

Information Architecture Testing

Eye-Tracking

Biometrics
10

USER INTERVIEWS
11

User Interviews
What is a User Interview:

A focused one-on-one conversation

In-person or via telephone

Familiar format for both participant and interviewer
12

User Interviews
How to ask good interview questions:

Open-Ended Questions


Who, What, When, Where, Why, How
Start with broad questions, then use prompts to fill in gaps

Let the participant guide the discussion at first to what they think is important
Example:
 Tell me about sharing the printer and copier with others.
 What works well?
 What doesn’t work?
 Do you need an ID, card, or code to use the shared printer/copier?
 Does your printer keep track of how many copies/dollars you have left on your “budget”?
 Do you know how much it costs per page to print?
 Is your company actively trying to manage printing costs? If so, how?
13

User Interviews
Pros
Cons
Moderator can ask follow on questions
to focus on topics of interest
Qualitative data takes longer to analyze
than quantitative
Gather rich, detailed data
Questions may not completely consistent
from participant to participant
Good for exploratory, formative
research
Need to manage talkative and quiet
participants
Small sample size yields effective data
Not as efficient when there are many
different demographics with truly
different needs
Can be done remotely via telephone
Relatively inexpensive to conduct
14

User Interviews
Additional Resources:
Understanding Your Users: A practical guide to user requirements. Catherine Courage & Kathy
Baxter, Morgan Kaufmann, 2005. Chapter 7
Interviewing Users: How to uncover compelling insights. Steve Portigal, Rosenfeld Media, 2013.
Mental Models: Aligning Design Strategy with Human Behavior. Indi Young, Rosenfeld Media, 2008.
Chapter 7
15

ETHNOGRAPHY
16

Ethnography
What is ethnography?

Observing users in their natural environment while they perform the tasks of
interest

Big “E” Ethnography

“A research strategy that allows researchers to explore and examine the cultures and
societies that are a fundamental part of the human experience” (Murchison)

Generally involves immersion of the researcher within the culture to be studied

Longitudinal and can take many years

Little “e” ethnography

UX Researchers perform little “e” ethnography

Typically involves a period of observation followed by an interview

Focuses on how users perform the tasks of interest and their overall environment
17

Ethnography
Good to examine:

What information users look for and where they find it

Who and what they interact with to get the task done

The order in which things happen, and whether that is important

Distractions, barriers, and interruptions that the participants encounter
18

Ethnography
Pros
Cons
See interactions with other people and
other products/systems
Logistics of setting up
observations/interviews in someone’s
home/office
Users will interact with their own items,
including cheat sheets or other “coping
mechanisms”
May be difficult to gain access to
individuals or certain environments, or for
extended periods of time
Good to understand how things are
really done vs. how they are supposed
to be done
Observing/interviewing many people
requires more time
Can yield insightful “aha” moments and
innovative ideas
On-site recording logistics can be
challenging
Observe natural interruptions and other
events which participants forget to
report in interviews
19

Ethnography
Additional Resources:
Ethnography in UX, Nathanael Boehm, uxmatters.com,
http://www.uxmatters.com/mt/archives/2010/06/ethnography-in-ux.php
Practical Ethnography: A guide to doing ethnography in the private sector, Sam
Ladner, Expected soon. http://www.practicalethnography.com/
Ethnography Essentials: Designing, conducting, and presenting your research,
Julian M. Murchison. Josey-Bass, 2010.
20

COLLAGING
21

Collaging
What is Collaging:
A creative activity to help uncover thoughts and attitudes.
Participants are given photographs, large paper and art supplies and asked to
create a collage about a central theme. They then explain their collage.

Can be done one-on-one or as a focus group activity

Can make some participants conscience of their lack of creative skills

Their storytelling about the college is the key

Followed by a short interview
22

Collaging
How to do it:
1.
Prepare materials
2.
Define focus
3.
Create the collage
4.
Have them tell their story
What does
“Saving Energy”
mean to you?
23

Collaging
Pros
Cons
Gather rich, detailed data
Qualitative data takes longer to analyze
Small sample size
Questions are not consistent from
participant to participant
Good for exploratory research – when
you don’t know the questions to ask
Some participants are not as open to
creative exercises as others
Good for uncovering values,
motivations, attitudes that are not topof-mind – participants may not even be
consciously aware of them
Preparation can be time-consuming
Good seeding activity for focus groups
Can help some participants verbalize
thoughts
24

Collaging
Additional Resources:
Collaging: Getting answers to the questions you don’t know to ask. Kyle Soucy,
Smashing Magazine, February 6, 2012.
http://uxdesign.smashingmagazine.com/2012/02/06/collaging-getting-answersquestions-you-dont-know-ask/
McKay, D., Cunningham, S. J., Thomson, K. Exploring the user experience through
collage. CHINZ ’06 Proceedings of the 7th ACM SIGCHI New Zealand’s chapter’s
international conference on Computer-human interaction: design centered HCI,
p. 109-115.
25

DIARY STUDIES
26

Diary Studies
What is a Diary Study:

A semi-longitudinal study used gather user behavior over time

Users are asked to keep a diary for a certain topic

When they interact with a device/website/etc.

Health, food, study habits, banking habits, etc.
When to use a diary study:

When a single research session will not truly capture users’ interactions

When the users’ environment plays a role in how they use the interface

If there is plenty of time in the project timeline
27

Diary Studies
Pros
Cons
Captures users’ data in their natural
setting
Reliant on participants keeping up
with their diaries
Users will be interacting with their
own items
Participants self-select content
Can capture data over time
Management of the data collection
mechanism
Must have time in the project timeline
28

Diary Studies
Additional Resources:
Observing the User Experience: A Practitioner’s Guide to User Research, Elizabeth Goodman,
Mike Kuniavsky, & Andrea Moed, Morgan Kaufman, 2012.
Diary Studies in HCI & Psychology, Demetrios Karis, UPA Boston 2011 Conference.
http://www.slideshare.net/UPABoston/diary-studies-in-hci-psychology
29

User Research Methods
Generative
Evaluative

Interviews

In-Person Usability

Ethnography

Remote Usability

Collaging

Unmoderated Usability

Surveys

Desirability Testing

Card Sorting

Usefulness Testing

Diary Studies

Surveys

Focus Groups

Information Architecture Testing

Eye-Tracking

Biometrics
30

EVALUATIVE METHODS
31

Evaluative Testing
Usable
Engaging
Useful
32

DESIRABILITY TESTING
33

Desirability Testing
What is Desirability Testing?
A set of tools for exploring participants’ emotional reactions to different visual
designs.
Use this method to:

Select a visual design which most closely aligns with desired traits

Evaluate a product’s visual design vs. competition to uncover differences in
brand perceptions or user experience

Evaluate a single design for emotional response
34

Desirability Testing
Open-Ended Questions:

Ask for initial reactions

Show a design to a participant and ask for their immediate reactions

Let them tell you whatever they want to tell you
Sentence Completions:

Start 5-10 sentences and ask them to finish them based on their reactions to the
design
Product Reaction Words:

Show them ~ 60 words – both positive and negative connotations

Ask them to select a small number which best represent the design
35

Sentence Completion
 Have participants complete sentences
 Gather focused feedback without being leading
Examples:
 This page makes me feel…
 Reading information on this page is…
 If I used this website all day, I would…
 I want to use this website because…
 I do not want to use this website because…
36

Product Reaction Words
Accessible
Dated
Hard to Use
Simplistic
Appealing
Dull
Helpful
Sophisticated
Approachable
Easy to Use
Impersonal
Sterile
Boring
Effective
Innovative
Stimulating
Busy
Efficient
Inspiring
Straight Forward
Clean
Energetic
Intimidating
Stressful
Clear
Engaging
Intuitive
Time-consuming
Comfortable
Enthusiastic
Inviting
Time-saving
Compelling
Exciting
Motivating
Too Technical
Complex
Familiar
Organized
Trustworthy
Confident
Fast
Overwhelming
Understandable
Confusing
Flexible
Patronizing
Usable
Convenient
Fresh
Predictable
Useful
Creative
Friendly
Professional
Valuable
Cutting Edge
Frustrating
Reliable
37

Product Reaction Words
Word Cloud: http://www.wordle.net
38

Product Reaction Words
Existing Design
Customers
Customers & Non-Customers
Non-Customers
Reliable
Accessible
Appealing
Confusing
Comfortable
Organized
Professional
Busy
Confident
Straight Forward
Approachable
Complex
Effective
Understandable
Efficient
Dull
Friendly
Easy to Use
Flexible
Hard to Use
Inviting
Clear
Helpful
Impersonal
Time-saving
Convenient
Overwhelming
Trustworthy
Clean
Boring
Useful
Familiar
Frustrating
Simplistic
Intimidating
Valuable
Stressful
Usable
Time-consuming
39

Product Reaction Words
Number of positive vs. negative words
Number of Words
70
60
50
40
Positive
30
Negative
20
10
0
Existing
Design A
Design B
Design C
40

Product Reaction Words
Words that the client desires the look and feel to portray
Existing
Design A
Design B
Design C
Appealing
Appealing
Appealing
Appealing
Clean
Clean
Clean
Clean
Efficient
Efficient
Efficient
Efficient
Engaging
Engaging
Engaging
Intuitive
Professional
Professional
Useful
Total
Professional
Useful
Useful
9
20
Useful
16
9
41

Desirability Testing
Pros
Cons
Gather qualitative feedback on
emotional response to designs
Choice of adjectives could introduce
subtle bias
Gather focused feedback in a neutral,
open-ended fashion
Generally, relatively small sample size
Product reaction words provides some
level of quantitative measurement, but
should be used carefully
Qualitative feedback can take longer to
analyze
42

Desirability Testing
Additional Resources:
Benedeck, J. & Miner, T. (2002) Measuring Desirability: New methods for evaluating
desirability in a usability lab setting. Proceedings of Usability Professionals Association,
2003, 8-12.
http://www.pagepipe.com/pdf/microsoft-desirability.pdf
Rapid Desirability Testing: A Case Study, Michael Hawley, uxmatters.com,
http://www.uxmatters.com/mt/archives/2010/02/rapid-desirability-testing-a-casestudy.php
Users play cards We keep score Magic Results, presentation by Carol Barnum and Laura
Palmer
http://www.slideshare.net/cbarnum/barnum-palmer-stc-2011-presentation
Kuhala, S. & Nurkka, P. (2012) Sentence Completion for Evaluating Symbolic Meaning.
International Journal of Design, 6(3), 1525.http://www.ijdesign.org/ojs/index.php/IJDesign/article/view/1166/523
43

USEFULNESS TESTING
44

Usefulness Testing
What is Usefulness Testing?
Usability testing identifies issues that prevent users from performing tasks
Usefulness testing evaluates the match between user needs and system features
1. Investigate how users do tasks today
2. Have them use the system or prototype
3. Ask questions about usefulness
45

Usefulness Testing
1. Background Questions:
 Investigate how users do things today – what works well and what doesn’t
Sample Questions:
 Tell me about your process for ordering equipment and supplies from XYZ Co.
 What do you typically need to order?
 How do you determine whom to contact?
 How often do you order supplies?
 What happened last time you felt your resolve to eat well sagged?
 What sources of support did you rely on?
 How did they help you?
 How did you get “back on track”?
46

Usefulness Testing
2. Show them the product / prototype
Watch for:
 Comments about usefulness
 “Wow. This would really make it easier to reorder the same things.”
 “I don’t like this. It would take me too long to do this part.”
 Neutral reactions
 “Eh. This is OK.”
 Probe: “Might this help you get your work done? Or not?”
 Be wary of “This isn’t how we do it today” comments
 Business processes may need to change in the future, so try to find out why something
would not work.
47

Usefulness Testing
3. Follow up Questions:
 After they have seen the system/prototype, how do they think it might work
for them?
Sample Questions:
 What are your overall impressions of the website?
 Based on what you saw today, how helpful would the system be to you? Why?
 What could we do to make this more helpful to you?
48

Usefulness Testing
Additional Resources:
Hawley, M. & Berlin, D. Beyond Usability Testing: Assessing the Usefulness of Your Design.
UPA Boston Conference, 2011.
49

CHOOSING THE RIGHT METHODS
50

How to Choose the Right Method(s)
1.
Make sure your research goals are clear

Know what you want to learn

Know what you will do with the information you gather

Know what decisions the business needs to make
2.
Know your constraints

Timeline?

Resources?

Budget?

Access to Users?
3.
Create a Methods Chart

The answers will become clear
51

Methods Chart
Method
Method 1
Pros
• List the advantages for
this method for this
specific project
Cons
• List the disadvantages
for this method for this
specific project
Candidate
Yes / No –
Would this method be a
good candidate for this
specific project
Method 2
Method 3
Etc…
52

Methods Chart Example
Method
Pros
Cons
Candidate
Interviews
• Good for capturing
motivations for behavior
change
• Can get good qualitative
details on why users
prefer each concept or
not
• Good to explore issues to
fine tune survey
questions
• Can be done via phone
(get broad geographic
sample)
• Can be done quickly
• Small numbers – client
wants large numbers
Y – would be good as a
qualitative method to pair
with a larger quantitative
method
Focus Groups
• Good for qualitative
information gathering
• Could generate some
interesting conversations
about behavior change
motivations
• Wouldn’t get as much
detail as interviews
• Concern about group
think when evaluating
concepts
• Concern about not
sharing details of
personal goals in front of
others
N – interviews would be
better for qualitative
Survey
• Good for large numbers
• Easily replicated across
different
countries/languages
• Can be done online for
broad geographic
distribution
• Can be done quickly
• Unclear what exact
questions to ask
• Doesn’t provide detailed
insights into qualitative
topics
Y – good paired with
qualitative method
53

Want to learn more?
Attend our Workshop
Choosing The Right Methods For Your Research Project
Susan Mercer & Dan Berlin
Full Day Workshop, IA Summit – San Diego, CA
March 26, 2014
http://www.iasummit.org
Contact Us
Susan Mercer
Senior Experience Researcher
smercer@madpow.com
@susanamercer
Dan Berlin
Experience Researcher Director
dberlin@madpow.com
@banderlin
54

DISCUSSION
55