Will You Please Be Quiet Please? How not to skew User Interviews in your Products Favor.
•Download as PPTX, PDF•
1 like•1,031 views
This talk discusses the common mistakes that teams make when executing user research, a common pattern is not allowing a enough space and silence around the user to allow their feedback to come through loud and clear. This talk calls out these anti-patterns and offer tips that will help new researchers avoid them.
Recommended
More Related Content
Viewers also liked
Viewers also liked (16)
Recently uploaded
Recently uploaded (20)
Will You Please Be Quiet Please? How not to skew User Interviews in your Products Favor.
- 2. Pivotal Labs Confidential @rozemaryking Hi. I’m a Product Manager @pivotallabs I’m @rozemaryking
- 8. Will You Please Be Quiet, Please? (Also a great book)
- 9. User Research Anti-Patterns 1. Not doing any 2. Not having clear goals 3. Talking to the wrong people 4. Tainting the interview 5. Not taking stock
- 10. Mistake 1. Not Doing Any
- 11. Mistake 1. Not Doing Any
- 12. Solution 1. Do them and don’t stop Ask some why questions Make a list of your assumptions Prove the ROI We learn something everyday and lots of times it’s what we learned the day before that was wrong. – Bill Vaughan
- 13. Solution 1. Do them and don’t stop
- 14. Mistake 2. Talking to the wrong people
- 15. Solution 2. Find your reps. Reach deep into networks. Utilize the tools that are at your disposal. Go where your customers are, not where you want them to be. –Sean Southey
- 16. Mistake 3. No clear goals
- 17. Solution 3. Define a learning objective Understand your assumptions Use appropriate fidelity. Keep your list focused.
- 18. Solution 3. Define a learning objective Guide to Design Mockup Fidelity
- 19. Mistake 4. Tainting the Interview
- 20. Solution 4. You’re not a friend Build trust Don’t tell them you care Let them know their impact “The worst thing that we can do is to make a product that people don’t want to use.” –Joanna Beltowski
- 21. Solution 4. Questions. Less is more. Start vague, get more specific Act as if leading is death. Get comfortable with silence
- 22. Solution 4. Questions. Less is more.
- 23. Mistake 5. Not taking stock
- 24. Solution 5. Debrief Don’t do this alone Find the patterns Prioritize
- 26. Point Mapping
- 27. User Interview Best Practices 1. Do them and don’t stop 2. Find your reps 3. Define your learning objective 4. You’re not a friend + Less is More 5. Debrief (preferably with your team)
- 28. @rozemaryking Don’t take it personally.
Editor's Notes
- Hi Guys, I’m Rosemary King, I’m a product manager at pivotal labs I want to walk you through my journey as a user researcher, and what I’ve learned since getting started five years ago. I want to focus on the mistakes that are commonly made as we interact with user research participants in interviews that skew the results in favor of our product, and help us avoid more harsh realities
- So Five years ago case commons (my company at the time) was building a case management program for the state of Indiana’s department of child services,
- We had access to some of the best macro research on the challenges that social workers face as they document their cases, which gave us a great view of the what, but we just weren’t talking personally to enough people in Indiana to understand the Why. there were big obstacles in starting. Department stakeholders were really hesitant to give us access. They were worried about a loss of control and meandering scope. We didn’t have a great well of experience on implementing user research and bandwidth was low. There was fear on both sides. Eventually I convinced Indiana to give me a list of super users. At the time I celebrated by high fiving a million angels and got started.
- Super users group was 45 or so people. All in marion county, in the state capital, central office hand-picked, all young, new to the department, not biased to old practices, They gave us great feedback, sometimes tough, but generally they were really excited about roll-out. So we kept on building towards our MVP and believed that we were going to introduce a ground-breaking new tool.
- Me Now really really feels for Me Then.
- Fast forward to roll-out. We went live with the product across the state and we finally could speak to other users We went to Lake county, which is urban, and is economically depressed. Clark County. Very rural area, with tiny offices and a lot of land to cover. The workers in these counties had very different work lives to their central office counter-parts. Very used to the legacy system. Long story long. We had expletives, we had a lot confusion, we had a few people in tears. We were blind-sided. We hadn’t heard any of this type of frustration from our super-user group. How did all of those interviews yield so little of what bubbled in other places after going live. I want to remind you that This was the product that we had invested two years building which a good portion of our user group were saying they HATED. The revelation was that all my mistakes boiled down into one simple concept. I had inserted myself, in my user interview, between the user and the product.
- From that point on, I had to constantly remind myself, Will you please be quiet, please? Don’t make noise that will influence the user. And you can do this in a million different ways. Let’s go back and examine the ways that I got in the way of CaseCommons initial tries at user interviews.
- the problems occurred at each stage of the user interview process. When building out the research plan that you’re going to use to gather information, there are several things you need to do in order to make sure that you’re not setting yourself up for the perfect outcome. You have to check yourself. Are you focusing on the aspects that will help your product succeed? Are you allowing enough space for honest feedback? How are you feeling about what your interviewees are telling you? Resistant to hearing the feedback. Willing to engage with subjects? Eager to make excuses for your product? In other words are your ears open to what’s truly going on? For each one of these anti-patterns there is a positive pattern that can be implemented.
- Of all of the anti-patterns this one is the most egregious and confusing, because you can’t start improving on any of the other inevitable hiccups, until you START. Fail fast, fail often, and learn from it. Case Commons wasn’t close to our users, but really, our stakeholders at the agency held their workers in a deathgrip because they were scared about what felt like a loss of control. This happens a lot. I call it the “Iknowbest” syndrome. It can be hard to overcome, but it’s possible.
- CaseCommons didn’t push so hard against this at first because we didn’t have much bandwidth to learn how to research or implement it. We were super busy building the product. Hmmm. That like seeing someone working furiously to cut down a tree with a penknife, and someone comes up and offers a chainsaw, and the person with the penknife says, “can’t you see I’m busy cutting down this tree?!” Unless you’re the most interesting man in the world, this is not the best approach.
- Just saying DO THEM is a bit glib. A lot of companies invest millions into a product, being told that it’s wrong can be terrifying. Or they see themselves as the best voice for the product. When talking to your boss or stakeholder, work to understand where their reservations are. or Make a list of the assumptions that they might be making about the product. or Put it in terms of economics. If you understand what’s MOST important to your customers, then your priorities come into laser focus. This is never more true than when it comes to enhancements. There are always a million possible enhancements or expansion opportunities.
- A lot of folks feel overwhelmed by the prospect of scheduling interviews, thinking you need to speak to dozens and dozens of people and that stops them from starting. BUT I’ve never heard an experienced researcher say or write, that more than 6 in person user interviews of people, are needed to establish themes and patterns. Augmented with macro research and analytics, 5-10 user interviews are enough to validate each round of assumptions.
- This is actually a big thing for “quite please theme” because we can insert ourselves in the user seat by talking to people who look and act just like us. In my case, I was speaking to a lot of folks in Indiana who looked a lot like me. Hand-picked agency super stars, eager to help. Most were around my age, so were a lot more comfortable with newer technologies. Agency Product Owners were happy because they were within earshot We might have all been one big happy family. Long-time agency veterans, folks who weren’t in the central office, workers from different types of counties that were more rural, more economically diverse, had fewer resources, different kinds of communication. These guys were wild cards.
- In general, a good rule of thumb is try to talk to people that make you a bit uncomfortable, in a variety of ways. When you find yourself having a reaction like discomfort, or eye-rolling, or inward giggling because they said something so funny, this is when you have to put on your big-kid pants, let down your resistance and really LISTEN. Because they are giving you information that you could ever get at yourself. This requires reaching deep into your networks and getting creative to find people that you can’t normally get at. Use a recruitment agency to help you find people of a specific profile who don’t know you from Adam. Strangers don’t care about offending you. .
- Parkinson’s Law of Triviality states: The time spent on any item of the agenda will be in inverse proportion to the sum [of money] involved. This is known as bikeshedding to most of us. Bikeshedding is when it’s easier to argue about whether or not the blue is too blue that really focus on whether or not your tool affectively onboards your customers Letting your users focus on small issues like the font, your colors, animations, and not asking them the real questions that you need to know is like waving a lot of red flags to distract the bull from the matador.
- Define the goal of the exercise. Are you trying to see how people accomplish a specific task in their daily lives, or trying to see how they react to something new or Seeing if they can accomplish a specific thing.
- Assumptions are the mistake that I see most product people make when it comes to thinking about their users. You take it for granted that someone who you hope will use your tool, behaves in a certain. Until these assumptions are truly validated through user affirmation, keep your product developments as light as you can. A paper drawing is enough for this. The messier the better. Because then no one focuses on the design details. Now when you are focusing on usability and have something build and design is being layered on, that doesn’t mean fonts and colors become your focus. You’re still looking to keep in goal oriented. What metric would you like to see move by changing that radio button to a dropdown? What do you believe that change will accomplish in terms of user value?
- This is THE big one. It’s the one you have to worry about in terms of getting quality feedback that you can both trust and apply. I’m originally Canadian so I’m going to use a curling metaphor. I would literally curl people through the interview. Whenever they struggled, or had a hard time. Gently brushing them towards the completion of the exercise, because I thought the most important thing was to get them through it, so they could see the whole thing. Ugh. You can do this so subtly. In the way you set up the interview. How you introduce yourself. Simply by making a noise as they do or say something. Wording a question bady. Asking for agreement.
- You build trust in a number of ways. Make sure your set-up works well. You want a seamless, calm experience. Do get them to sign a consent form, so that they know they are secure, and know their responsibility. Don’t leave things ambiguous. Don’t tell them that you’re involved with building the tool. Why? They don’t want to make you mad, let you down. That makes them uncomfortable. and they will clam up. Don’t have more than one other person in the room, because they will feel like they are on display. Set up a listening room for people besides you and the note-taker. Let them know that their insights will shape the tool and help others. Everyone likes being helpful. This loosens lips. Reinforce that their experience and opinions are valuable. WE ARE TESTING THE PRODUCT NOT YOU Have a partner in the room to take notes so that your connection with the subject is total.
- There was a game you all play, “Carpet is lava” Don’t touch the carpet, you’ll die! In the game of user research, leading is lava. Ways to lead: Assumptive questions: how would you sign-in here? How do you know that it’s obvious it’s a sign-in page. Linked statements, when you link an idea to your question. Some people find a dropdown helpful there. How would you enter it. Asking for agreement – Do you like this color scheme? Implication questions: will that button take you to X page? As opposed to you where do you want to go next? Don’t make noise because you’re uncomfortable with watching someone struggle. How long someone struggles can be really instructive. But it can be so easy, things that seem so innocent can be loaded. Let’s take an example.
- Say your testing spaghetti. You put a big plate of noodles in front of them. And you ask, “How would you eat these noodles?” What’s wrong with asking this question right off the bat? You introduce the idea that they are noodles. You introduce the idea that you would eat them. In fact, you placed them…cooked, on a plate…on a table! What if someone likes their noodles uncooked in a bowl, with milk, outside! When you’re getting started, you want to start really vague and let them lead the conversation. If they go somewhere that you want to go, ask more questions around that topic. So instead of how would eat these noodles. Ask: What are your favorite foods? List the top five. Oh lasagne, tell me more about what kind of lasagna? What do you like about lasagnga? Don’t’ doggedly follow your script. Try to think of it as letting them steer the conversation.
- I did this a fair amount. I listened to 5-10 people. Took copious notes. Got back to the office. And DID NOTHING WITH IT. It’s in my head, I don’t have time to get it all down. I know what I heard. But the brain can’t hold that much. Nor remember that much. So make sure that in every user research round, you carve out time to brain dump the notes, find the themes and figure out how it maps to your goals.
- This is why it’s so important that you have at least one other person listening to these user interviews. You want to filter the data through a lot of different lens. People will perk up at different points of the conversation, PM, Dev, Designer, and business are all listening for different points. It helps you give a 360 view of the conversation. Your memory is not reliable. You are going to get attached to certain sentences. Surfacing each interesting piece and putting it somewhere visibly keeps this stuff at the forefront of your teams brain. Making sure that the insights attached to the higher priority business goals are given a bit more weights is incredibly helpful for keeping your backlog tight and focused. Helps give you your first step. There is no ambiguity about what you could start with next.
- Insert picture of my user research boards.
- Goals are orange Challenges are pink Opportunities are green Themes are blue
- Just a little recap slide for everyone on the patterns to our anti-patterns.
- If I was going to send myself a message, back in time, maybe in a fortune cookie that I would open at the end of one of those disasterous days of user researcher, when I first got beat up by user feedback it would read: DON’T TAKE IT PERSONALLY. I could have really used to have heard that at that moment. The reason why we hesitate to talk to users, or want to hear that they love the product is because we invest ourselves in the things we’re building. But it’s not about us, it’s about building products. Hopefully great ones. Don’t believe that when you hear something negative it’s a putdown on you. It’s not. It’s your customer, trying to get value from the thing that you’re building for them. Don’t resist that, let them help you find out how to make your product better. . Learn to love it when you hear bad things Because bad things lead can lead to clear action, if you let them.