Choosing the Right Research Methods for Your Project (webinar)

Experience Research Director à Mad*Pow
18 Nov 2013

Contenu connexe

Similaire à Choosing the Right Research Methods for Your Project (webinar)(20)


Choosing the Right Research Methods for Your Project (webinar)

  1. Choosing The Right Research Methods For Your Project Susan Mercer – Senior Experience Researcher @susanamercer October 29, 2013
  2. Hello, I’m Susan Mercer  BA and MSc in Geophysics  19 years in software and web UI and UX design  Developer  Designer  Web Producer  Product Manager  Researcher  MS Human Factors, Bentley University  Twitter: @susanamercer 2
  3. Introduction “It is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.” – Abraham Harold Maslow Agenda:  Market Research & User Research  Study Goals  User Research Methods  Choosing the Right Methods 3
  4. Marketing Research & User Research 4
  6. Defining Goals “The most serious mistakes are not being made as a result of wrong answers. The truly dangerous thing is asking the wrong questions.” – Peter Drucker 6
  7. Defining Goals “The most serious mistakes are not being made as a result of wrong answers. The truly dangerous thing is asking the wrong questions.” – Peter Drucker The first key step of a good study is to determine your goals Good Study Goals:  Provide focus to the study  Become more specific as the project progresses  Focus on user behavior, not on interface specifics  Help drive actionable results to impact business or design decisions 7
  8. Defining Goals Consider:  Where are you in the project?  Identifying problems with current behaviors to identify opportunities?  Evaluating the appetite for a product feature?  Gathering feedback on an early design?  Testing an existing product?  What are you trying to learn?  Is there an existing artifact to test?  How much time do you have?  How much of a budget do you have? 8
  10. User Research Methods Generative Evaluative  Interviews  In-Person Usability  Ethnography  Remote Usability  Collaging  Unmoderated Usability  Surveys  Desirability Testing  Card Sorting  Usefulness Testing  Diary Studies  Surveys  Focus Groups  Information Architecture Testing  Eye-Tracking  Biometrics 10
  12. User Interviews What is a User Interview:  A focused one-on-one conversation  In-person or via telephone  Familiar format for both participant and interviewer 12
  13. User Interviews How to ask good interview questions:  Open-Ended Questions   Who, What, When, Where, Why, How Start with broad questions, then use prompts to fill in gaps  Let the participant guide the discussion at first to what they think is important Example:  Tell me about sharing the printer and copier with others.  What works well?  What doesn’t work?  Do you need an ID, card, or code to use the shared printer/copier?  Does your printer keep track of how many copies/dollars you have left on your “budget”?  Do you know how much it costs per page to print?  Is your company actively trying to manage printing costs? If so, how? 13
  14. User Interviews Pros Cons Moderator can ask follow on questions to focus on topics of interest Qualitative data takes longer to analyze than quantitative Gather rich, detailed data Questions may not completely consistent from participant to participant Good for exploratory, formative research Need to manage talkative and quiet participants Small sample size yields effective data Not as efficient when there are many different demographics with truly different needs Can be done remotely via telephone Relatively inexpensive to conduct 14
  15. User Interviews Additional Resources: Understanding Your Users: A practical guide to user requirements. Catherine Courage & Kathy Baxter, Morgan Kaufmann, 2005. Chapter 7 Interviewing Users: How to uncover compelling insights. Steve Portigal, Rosenfeld Media, 2013. Mental Models: Aligning Design Strategy with Human Behavior. Indi Young, Rosenfeld Media, 2008. Chapter 7 15
  17. Ethnography What is ethnography?  Observing users in their natural environment while they perform the tasks of interest  Big “E” Ethnography  “A research strategy that allows researchers to explore and examine the cultures and societies that are a fundamental part of the human experience” (Murchison)  Generally involves immersion of the researcher within the culture to be studied  Longitudinal and can take many years  Little “e” ethnography  UX Researchers perform little “e” ethnography  Typically involves a period of observation followed by an interview  Focuses on how users perform the tasks of interest and their overall environment 17
  18. Ethnography Good to examine:  What information users look for and where they find it  Who and what they interact with to get the task done  The order in which things happen, and whether that is important  Distractions, barriers, and interruptions that the participants encounter 18
  19. Ethnography Pros Cons See interactions with other people and other products/systems Logistics of setting up observations/interviews in someone’s home/office Users will interact with their own items, including cheat sheets or other “coping mechanisms” May be difficult to gain access to individuals or certain environments, or for extended periods of time Good to understand how things are really done vs. how they are supposed to be done Observing/interviewing many people requires more time Can yield insightful “aha” moments and innovative ideas On-site recording logistics can be challenging Observe natural interruptions and other events which participants forget to report in interviews 19
  20. Ethnography Additional Resources: Ethnography in UX, Nathanael Boehm,, Practical Ethnography: A guide to doing ethnography in the private sector, Sam Ladner, Expected soon. Ethnography Essentials: Designing, conducting, and presenting your research, Julian M. Murchison. Josey-Bass, 2010. 20
  21. COLLAGING 21
  22. Collaging What is Collaging: A creative activity to help uncover thoughts and attitudes. Participants are given photographs, large paper and art supplies and asked to create a collage about a central theme. They then explain their collage.  Can be done one-on-one or as a focus group activity  Can make some participants conscience of their lack of creative skills  Their storytelling about the college is the key  Followed by a short interview 22
  23. Collaging How to do it: 1. Prepare materials 2. Define focus 3. Create the collage 4. Have them tell their story What does “Saving Energy” mean to you? 23
  24. Collaging Pros Cons Gather rich, detailed data Qualitative data takes longer to analyze Small sample size Questions are not consistent from participant to participant Good for exploratory research – when you don’t know the questions to ask Some participants are not as open to creative exercises as others Good for uncovering values, motivations, attitudes that are not topof-mind – participants may not even be consciously aware of them Preparation can be time-consuming Good seeding activity for focus groups Can help some participants verbalize thoughts 24
  25. Collaging Additional Resources: Collaging: Getting answers to the questions you don’t know to ask. Kyle Soucy, Smashing Magazine, February 6, 2012. McKay, D., Cunningham, S. J., Thomson, K. Exploring the user experience through collage. CHINZ ’06 Proceedings of the 7th ACM SIGCHI New Zealand’s chapter’s international conference on Computer-human interaction: design centered HCI, p. 109-115. 25
  27. Diary Studies What is a Diary Study:  A semi-longitudinal study used gather user behavior over time  Users are asked to keep a diary for a certain topic  When they interact with a device/website/etc.  Health, food, study habits, banking habits, etc. When to use a diary study:  When a single research session will not truly capture users’ interactions  When the users’ environment plays a role in how they use the interface  If there is plenty of time in the project timeline 27
  28. Diary Studies Pros Cons Captures users’ data in their natural setting Reliant on participants keeping up with their diaries Users will be interacting with their own items Participants self-select content Can capture data over time Management of the data collection mechanism Must have time in the project timeline 28
  29. Diary Studies Additional Resources: Observing the User Experience: A Practitioner’s Guide to User Research, Elizabeth Goodman, Mike Kuniavsky, & Andrea Moed, Morgan Kaufman, 2012. Diary Studies in HCI & Psychology, Demetrios Karis, UPA Boston 2011 Conference. 29
  30. User Research Methods Generative Evaluative  Interviews  In-Person Usability  Ethnography  Remote Usability  Collaging  Unmoderated Usability  Surveys  Desirability Testing  Card Sorting  Usefulness Testing  Diary Studies  Surveys  Focus Groups  Information Architecture Testing  Eye-Tracking  Biometrics 30
  32. Evaluative Testing Usable Engaging Useful 32
  34. Desirability Testing What is Desirability Testing? A set of tools for exploring participants’ emotional reactions to different visual designs. Use this method to:  Select a visual design which most closely aligns with desired traits  Evaluate a product’s visual design vs. competition to uncover differences in brand perceptions or user experience  Evaluate a single design for emotional response 34
  35. Desirability Testing Open-Ended Questions:  Ask for initial reactions  Show a design to a participant and ask for their immediate reactions  Let them tell you whatever they want to tell you Sentence Completions:  Start 5-10 sentences and ask them to finish them based on their reactions to the design Product Reaction Words:  Show them ~ 60 words – both positive and negative connotations  Ask them to select a small number which best represent the design 35
  36. Sentence Completion  Have participants complete sentences  Gather focused feedback without being leading Examples:  This page makes me feel…  Reading information on this page is…  If I used this website all day, I would…  I want to use this website because…  I do not want to use this website because… 36
  37. Product Reaction Words Accessible Dated Hard to Use Simplistic Appealing Dull Helpful Sophisticated Approachable Easy to Use Impersonal Sterile Boring Effective Innovative Stimulating Busy Efficient Inspiring Straight Forward Clean Energetic Intimidating Stressful Clear Engaging Intuitive Time-consuming Comfortable Enthusiastic Inviting Time-saving Compelling Exciting Motivating Too Technical Complex Familiar Organized Trustworthy Confident Fast Overwhelming Understandable Confusing Flexible Patronizing Usable Convenient Fresh Predictable Useful Creative Friendly Professional Valuable Cutting Edge Frustrating Reliable 37
  38. Product Reaction Words Word Cloud: 38
  39. Product Reaction Words Existing Design Customers Customers & Non-Customers Non-Customers Reliable Accessible Appealing Confusing Comfortable Organized Professional Busy Confident Straight Forward Approachable Complex Effective Understandable Efficient Dull Friendly Easy to Use Flexible Hard to Use Inviting Clear Helpful Impersonal Time-saving Convenient Overwhelming Trustworthy Clean Boring Useful Familiar Frustrating Simplistic Intimidating Valuable Stressful Usable Time-consuming 39
  40. Product Reaction Words Number of positive vs. negative words Number of Words 70 60 50 40 Positive 30 Negative 20 10 0 Existing Design A Design B Design C 40
  41. Product Reaction Words Words that the client desires the look and feel to portray Existing Design A Design B Design C Appealing Appealing Appealing Appealing Clean Clean Clean Clean Efficient Efficient Efficient Efficient Engaging Engaging Engaging Intuitive Professional Professional Useful Total Professional Useful Useful 9 20 Useful 16 9 41
  42. Desirability Testing Pros Cons Gather qualitative feedback on emotional response to designs Choice of adjectives could introduce subtle bias Gather focused feedback in a neutral, open-ended fashion Generally, relatively small sample size Product reaction words provides some level of quantitative measurement, but should be used carefully Qualitative feedback can take longer to analyze 42
  43. Desirability Testing Additional Resources: Benedeck, J. & Miner, T. (2002) Measuring Desirability: New methods for evaluating desirability in a usability lab setting. Proceedings of Usability Professionals Association, 2003, 8-12. Rapid Desirability Testing: A Case Study, Michael Hawley,, Users play cards We keep score Magic Results, presentation by Carol Barnum and Laura Palmer Kuhala, S. & Nurkka, P. (2012) Sentence Completion for Evaluating Symbolic Meaning. International Journal of Design, 6(3), 1525. 43
  45. Usefulness Testing What is Usefulness Testing? Usability testing identifies issues that prevent users from performing tasks Usefulness testing evaluates the match between user needs and system features 1. Investigate how users do tasks today 2. Have them use the system or prototype 3. Ask questions about usefulness 45
  46. Usefulness Testing 1. Background Questions:  Investigate how users do things today – what works well and what doesn’t Sample Questions:  Tell me about your process for ordering equipment and supplies from XYZ Co.  What do you typically need to order?  How do you determine whom to contact?  How often do you order supplies?  What happened last time you felt your resolve to eat well sagged?  What sources of support did you rely on?  How did they help you?  How did you get “back on track”? 46
  47. Usefulness Testing 2. Show them the product / prototype Watch for:  Comments about usefulness  “Wow. This would really make it easier to reorder the same things.”  “I don’t like this. It would take me too long to do this part.”  Neutral reactions  “Eh. This is OK.”  Probe: “Might this help you get your work done? Or not?”  Be wary of “This isn’t how we do it today” comments  Business processes may need to change in the future, so try to find out why something would not work. 47
  48. Usefulness Testing 3. Follow up Questions:  After they have seen the system/prototype, how do they think it might work for them? Sample Questions:  What are your overall impressions of the website?  Based on what you saw today, how helpful would the system be to you? Why?  What could we do to make this more helpful to you? 48
  49. Usefulness Testing Additional Resources: Hawley, M. & Berlin, D. Beyond Usability Testing: Assessing the Usefulness of Your Design. UPA Boston Conference, 2011. 49
  51. How to Choose the Right Method(s) 1. Make sure your research goals are clear  Know what you want to learn  Know what you will do with the information you gather  Know what decisions the business needs to make 2. Know your constraints  Timeline?  Resources?  Budget?  Access to Users? 3. Create a Methods Chart  The answers will become clear 51
  52. Methods Chart Method Method 1 Pros • List the advantages for this method for this specific project Cons • List the disadvantages for this method for this specific project Candidate Yes / No – Would this method be a good candidate for this specific project Method 2 Method 3 Etc… 52
  53. Methods Chart Example Method Pros Cons Candidate Interviews • Good for capturing motivations for behavior change • Can get good qualitative details on why users prefer each concept or not • Good to explore issues to fine tune survey questions • Can be done via phone (get broad geographic sample) • Can be done quickly • Small numbers – client wants large numbers Y – would be good as a qualitative method to pair with a larger quantitative method Focus Groups • Good for qualitative information gathering • Could generate some interesting conversations about behavior change motivations • Wouldn’t get as much detail as interviews • Concern about group think when evaluating concepts • Concern about not sharing details of personal goals in front of others N – interviews would be better for qualitative Survey • Good for large numbers • Easily replicated across different countries/languages • Can be done online for broad geographic distribution • Can be done quickly • Unclear what exact questions to ask • Doesn’t provide detailed insights into qualitative topics Y – good paired with qualitative method 53
  54. Want to learn more? Attend our Workshop Choosing The Right Methods For Your Research Project Susan Mercer & Dan Berlin Full Day Workshop, IA Summit – San Diego, CA March 26, 2014 Contact Us Susan Mercer Senior Experience Researcher @susanamercer Dan Berlin Experience Researcher Director @banderlin 54

Notes de l'éditeur

  1. Usability testing & Interviews are very popular, but are not always the best method.Goal: Show you some lesser-known methods and when they are appropriate.
  2. People unfamiliar with the field sometimes ask what is the difference between Marketing Research and User Research?Sometimes the difference is a bit fuzzy, particularly because we sometimes use similar methodsThey like on a continuumMarketing Research primarily focuses on whether people will BUY a productUser Research focuses on how people USE a product
  3. Before we can talk about methods, you first have to be clear about what you want to learnFocus on answering questions that will help your business make decisions
  4. Before we can talk about methods, you first have to be clear about what you want to learnFocus on answering questions that will help your business make decisions
  5. Generative = Background = Before you have an artifact to testEvaluative = Get feedback on some stage of prototype or productRecognize overlap with Marketing Research – Interviews, Surveys, Focus Groups
  6. Not good when:Need consistency in questionsNeed quantitative, statistically significant dataHave a lot of disparate user groups (required sample size increases)
  7. ethnography:
  8. Good for:Identifying key pain points to address with new product or new feature setDetermining how a new product might fit into a complex ecosystem (e.g. operating room)Understanding how people do things today – and what product needs to compete with (e.g. iPad vs. hand-writing notes)
  9. Other creative activities work too: Lego Serious Play
  10. Key part is hearing their storyThat is where the symbolic meaning comes outThat is where you get them to articulate their thoughts that they just surfaced with the creative activityWhat is most important to them?
  11. Generative = Background = Before you have an artifact to testEvaluative = Get feedback on some stage of prototype or productRecognize overlap with Marketing Research – Interviews, Surveys, Focus Groups
  12. We think Usability Testing first – but, that is just one piece of the puzzleUsefulness & Desirability Testing use similar methods, but complement by focusing questions differently