This half-day workshop was designed to give Product Managers, Designers and Engineers about what kinds of research are possible, when to do research, what goes into it, and ultimately how to conduct research and avoid making mistakes when moderating. You can also listen to Example #1 (http://www.designmap.com/audio/modCloth/bad.mp3), Example #2 (http://www.designmap.com/audio/modCloth/subtly_bad.mp3), and Example #3 (http://www.designmap.com/audio/modCloth/good.mp3).
This is a workshop we conducted for a consumer-facing web-based product company. They had a very talented internal design team, but found that they (happily!) had more demand for research than they could fill.
1. Research Workshop
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/
or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA.
2. Agenda
1. Research Landscape
2. Focus: Interviews, Usability Research
3. Best Practices
4. Examples (good, bad, and so-so)
5.Exercises
6. Wrap-Up
Confidential – do not copy or distribute
3. Workshop Goals
This half-day workshop will cover:
•key types of research
•best practices
•actual practice
By the end of this workshop, you’ll have an
awareness of and experience with research and
best practices that will improve the quality and
dependability of your findings.
Confidential – do not copy or distribute
5. JJG’s Elements
text text
text
text
text
text
Confidential – do not copy or distribute
6. Research Methods by Data Source vs. Approach vs. Context of Product Use
text text
text
text
text
text
* updated from Christian Rohrer
Confidential – do not copy or distribute
7. Research Methods by Data Source vs. Approach vs. Context of Product Use
text text
text
text
text
text
Confidential – do not copy or distribute
9. Research Methods by Data Source vs. Approach vs. Context of Product Use
text text
Usability Studies
text
text
Interviews text
text
Confidential – do not copy or distribute
10. st
ra
t eg
y
Interviews
Confidential – do not copy or distribute
11. Interviews
strategy
Description
Goal …to understand underlying motivations, attitudes, and
behaviors (attitudinal).
Location ... often in person, may also be via WebEx or phone.
Context ... does not test a specific product.
Protocol ... describes areas of inquiries and sample questions,
but the session is often wide-ranging.
Confidential – do not copy or distribute
12. Interviews
strategy
Description
Need ...insight into customers or potential customers to
discover new ideas or find inspiration for new ways to
help or delight them.
Have ...a hypothesis about a customer or product opportunity,
time to conduct and synthesize research, access to 10 or
so customers, with ~60-90 minutes, preferably in their
environment, and a budget for stipends.
Can ...comfortably conduct a relaxed open-ended research
session, and can video record the session.
Confidential – do not copy or distribute
13. Interviews
strategy
Limitations
1. Predictions often don’t work.
2. Details aren’t always insights.
3. Biases and assumptions show and color
our insights.
4. Insights aren’t guaranteed.
5. Synthesizing is critical!
6. …
Confidential – do not copy or distribute
14. Interviews
strategy
After the Session
Post-session synthesis focuses on identifying
opportunities based on attitudes / problems.
These activities include things like:
1.design thinking
2.language analysis
3.experience maps
4.affinity diagrams
Confidential – do not copy or distribute
15. sk s
el tr
et uc
on tu
/ s re
ur /
fa
ce
Usability Research
Confidential – do not copy or distribute
16. sk s
Usability Research el tr
et uc
Description on tu
/ s re
ur /
fa
ce
Goal …to understand how they might use a product,
and to uncover problems and opportunities
(behavioral).
Location …may be in person, via WebEx or phone.
Context …tests a specific product (yours or a competitors).
Protocol ...may start with a few attitudinal questions, but
usually is longer and more closely followed.
Confidential – do not copy or distribute
17. skeleton / surface
Usability Research
structure /
Description
Need ...to get a sense of how potential customers will react
to a specific product offering or feature, or fine-tune an
interface.
Have ...a prototype, competitor, or working code, a protocol,
time do conduct research, access to 5 or so customers.
Can ...comfortably conduct a relaxed usability research
session.
Confidential – do not copy or distribute
18. skeleton / surface
Usability Research
structure /
Limitations
1. Predictions often don’t work.
2. Observer effect.
3. Limitations of the artifact.
4. False tasks give false results.
5. …
Confidential – do not copy or distribute
19. Usability Research
After the Session
Post-research synthesis focuses on specific
changes to a UI based on observed behavior,
including:
1.affinity diagramming
2.visual queues with screen shots and photos
and stickies
3.analysis of detailed notes
Confidential – do not copy or distribute
21. Best Practices
Moderating (first steps)
1. listen, be quiet!
2. be neutral and avoid value words
– use incomplete sentences
– parrot to probe and avoid answering ?s
– avoid asking for predictions
– use their language
– don’t assume their process
Confidential – do not copy or distribute
22. Best Practices
Moderating (mastering that…)
4. be calm
5. observations ≠ recommendations
6. note hypotheses up front — is success
proving them right or wrong?
Confidential – do not copy or distribute
23. Best Practices
Moderating (bonus points)
7. behavior first, then attitudes and motivation,
not the other way around
– BUT if you’ve asked lots of questions about
behavior and still don’t have a good sense
of attitude, it’s ok to ask a general question
Confidential – do not copy or distribute
24. sk s
Best Practices el tr
et uc
Note Taking on tu
/ s re
ur /
fa
ce
1. In person, sit behind and slightly to one side
of the moderator. Remotely, sit off screen or
in another room.
2. Be polite, attentive, and moderately
interested. (Do not react too much.)
3. Remember, observations before ideas.
4. Save your questions for the end.
5. Note your top ~5 takeaways immediately.
6. Don’t take your own notes.
Confidential – do not copy or distribute
25. Examples
Confidential – do not copy or distribute
26. sk s
el tr
et uc
Recording 1 on tu
/ s re
ur /
fa
ce
Confidential – do not copy or distribute
27. sk s
el tr
et uc
Recording 1 Findings on tu
/ s re
ur /
fa
ce
1. He thinks my idea is great!
Confidential – do not copy or distribute
28. sk s
el tr
et uc
Recording 2 on tu
/ s re
ur /
fa
ce
Confidential – do not copy or distribute
29. sk s
el tr
et uc
Recording 2 Findings on tu
/ s re
ur /
fa
ce
• Looking at Toyotas
• Doing research online using OEM sites and
Google
• Uses web daily, not into Facebook
Confidential – do not copy or distribute
30. sk s
el tr
et uc
Recording 3 on tu
/ s re
ur /
fa
ce
Confidential – do not copy or distribute
31. sk s
el tr
et uc
Recording 3 Findings on tu
/ s re
ur /
fa
ce
• Looking at Toyotas
• Shows concerns about safety, privacy
• Doing test drives, renting cars, and research
online with OEM sites and Google
• Uses web three times a week, uses Facebook
(mainly Farmville) every time he logs in
Confidential – do not copy or distribute
32. Exercise
Confidential – do not copy or distribute
33. sk s
Usability Research el tr
et uc
Exercise on tu
/ s re
ur /
fa
ce
1. Groups of three
2. One computer recording
3. Using our protocol, rotate through:
• participant
• moderator
• note-taker
1. For each moderator:
• 10 minutes conducting session
• 5 minutes for feedback
Confidential – do not copy or distribute
34. sk s
Best Practices el tr
et uc
Giving Feedback on tu
/ s re
ur /
fa
ce
1. Try, “I like…” / “I wish…” / “What if…”
(IL/IW/WI) or, “How might we…?”
2. State the specific observation and explain
the impact.
3. Be direct, respectful, and sincere (don’t say,
“You need to…”.)
4. If you have something nice to say, please
say it.
5. Pause and wait for reactions.
Confidential – do not copy or distribute
35. sk s
Best Practices el tr
et uc
Receiving Feedback on tu
/ s re
ur /
fa
ce
1. Be introspective — what do you think? Note
your own positive and constructive
feedback. (Hint: if you can’t identify any, this
might not be your thing.)
2. It is personal, but even if you start doing
this, it will be a small part of your craft.
Don’t take it too heavily.
3. This isn’t for everyone — being this neutral
is hard.
Confidential – do not copy or distribute
37. Nota Bene
• For all of these, garbage in,
garbage out.
• Agreements must be
documented somewhere.
• These are some tools, not THE
tools.
Confidential – do not copy or distribute
38. More Reading
1. Don’t Make Me Think, Steve Krug
2. The User is Always Right, Steve Mulder and Ziv Yaar
3. The Elements of User Experience, Jesse James Garrett
4. “From Use to User Interface”, Jeff Patton http://www.slideshare.net/abcd82/from-use-to-
user-interface-presentation
5. “Personas for Product Management”, Marty Cagan http://www.svpg.com/personas-for-product-management/
6. “Summary of Usability Inspection Methods”, Jakob Nielsen http://www.useit.com/papers/heuristic/inspection_summary.html
7. “When to Use Which User Experience Research Methods”, Christian Rohrer
http://www.useit.com/alertbox/user-research-methods.html
8. “Usability Demo Test” by Steve Krug
http://www.peachpit.com/promotions/promotion.aspx?promo=137602
9. User Interface Engineering - Articles about User Behavior
http://www.uie.com/browse/user_behavior/
10. “Three Questions You Shouldn't Ask During User Research” by Jared Spool
http://www.uie.com/articles/three_questions_not_to_ask/
11. Observing the User Experience by Mike Kuniavsky, pages 117-127
12. “Meet the Respondents: Understanding User Personalities” (Part 1 & 2) by Mike Fisher - UX Consultant
http://www.uxbooth.com/blog/meet-the-respondents-understanding-user-personalities-part-1/
http://www.uxbooth.com/blog/meet-the-respondents-understanding-user-personalities-part-2/
13. “My Best Advice for Conducting User Interviews” by Whitney Hess - UX Consultant
http://whitneyhess.com/blog/2010/07/07/my-best-advice-for-conducting-user-interviews/
14. Understanding Your Users by Catherine Courage & Kathy Baxter, pages 275-293
15. From Self-Prediction to Self-Defeat: Behavioral Forecasting, Self-Fulfilling Prophecies, and the Effect of Competitive
Expectations, by Kristina A. Diekmann, Ann E. Tenbrunsel and Adam D. Galinsky
Confidential – do not copy or distribute
39. Good Luck!
Please feel free to contact me if
you have questions or need help:
Audrey
audrey@designmap.com
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/ not copy or distribute
Confidential – do
or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA.
Notes de l'éditeur
For each of the two: description, limitations, what to do afterwards
stop and ask – is this what you hoped/want to cover?
Updated from Christian Rohrer
… or contextual inquiry, or field studies…
People are not good predictors of their own behavior – introspection illusion. "There is a very long history within psychology of people not being very good judges of what they will actually do in a future situation," said the study's senior author, Matthew Lieberman, a UCLA professor of psychology and of psychiatry and biobehavioral sciences. From Self-Prediction to Self-Defeat: Behavioral Forecasting, Self-Fulfilling Prophecies, and the Effect of Competitive Expectations: “ A multiplicity of studies have suggested that people are not very good forecasters at all. For example, work on affective forecasting suggests that judgments of predicted and actual reactions to positive and negative hedonic events often diverge (Gilbert & Wilson, 2000). When predicting emotional states in the future, individuals fail to consider that other events will intercede, thereby lessening the impact of any one event (Wilson, Wheatley, Meyers, Gilbert, & Axsom, 2000). This problem of focalism leads people to predict that their future reactions will be more intense than they actually are (Buehler & McFarland, 2001). Furthermore, people seem to anchor on their current state and feelings when making predictions about their future state and feelings (Loewenstein, 1996; Loewenstein, O’Donoghue, & Rabin, 2000).Not only can individuals not successfully predict how they will feel in the future, but research on behavioral forecasting suggests that people are not very good at accurately predicting their own future behavior. People are only moderately accurate in predicting whether they would engage in certain behaviors and make particular choices in the future (Osberg & Shrauger, 1986; Vallone, Griffin, Lin, & Ross, 1990) and exhibit overconfidence in their self-predictions (Griffin, Dunning, & Ross, 1990; Vallone et al.,1990).Kristina A. DiekmannAnn E. TenbrunselAdam D. Galinsky” These interviews can veer into details that don’t help, either excavating behaviors (vs. motivations) or doing collaborative design. It’s easy for moderators to inadvertently let their bias or assumptions slip, and not gain any real insight. Or let them color their perception – you assume you understand but you’re wrong! There are no guarantees – you’re following your nose to some degree and may change direction or dead end. It takes some work to synthesize! At least a few days.
affinity diagramming (welcome points of contention) who weighs in? and how? post analysis participation -- Look for observation contentions. visual queues with screen shots and photos and stickies
listen! (count to 5) - don’t finish their sentences - don’t interrupt let awkward pauses happen, see if they have any more thoughts (count to 5)if you feel too awkward, parrot back what they just said be neutral, avoid value words if you can (good, bad, easy, interesting, helpful, etc) - ask questions as incomplete sentences (“So would you say that experience was…?” instead of, “Was that a good experience?”) - play their words back to them as a probing tactic (if they say “I don’t use Facebook”, instead of, “Why don’t you like Facebook?” say, “You don’t use Facebook?”) and don’t answer questions, parrot them back and if you need to explain that you’ll answer later avoid asking them to predict their behavior (“Would you use this?” will not work—they want to please you and intent isn’t a good indicator for behavior anyway) - use their language (Tivo story) don’t assume their process ( for example, if you want to know what happened next, ask “what happened next” instead of asking what they did in what you assume the next step was, “So you decided to buy a hammock—then how did you do your research?”) To check if you’re truly neutral, you should really like the person after the session.
be calm, make them comfortable, make yourself comfortable with a little chatting if it helps – [they pick up on your behavior]
behavior first (specific examples), then attitudes and motivation, not the other way around (instead of, “Do you write reviews?”, “Can you tell me about the last time you wrote a review?” and then why, etc. or both but not only the first question. People aren’t necessarily reliable reporters of their own intentions and even if they were, intent isn’t a good predictor for behavior, like Kristen who said she wrote reviews but then couldn’t give one example of doing it) BUT if you’ve asked lots of questions about behavior and still don’t have a good sense of attitude, it’s ok to ask a general question (“So you’ve said you like things on Facebook all the time, that you’re trying to spend less time on it, and that you hate Farmville. How do you feel about Facebook generally?”) (Attitude is not a good predictor of behavior.)
talk about how to ask questions (this isn’t the case now)
bad
bad
subtly bad
bad
good
bad
If you’re really confident, and not at all self-conscious this might not be your thing. If you’re really passionate about your project, offer to switch with another person moderating for one another.